Code Reading Techniques 2019
We have discussed three scenario-based Code Reading techniques, defect-based Code Reading, perspective-based Code Reading, and function-point-based Code Reading. The first and last Code Reading techniques have been applied to requirements specification documents. Perspective-based Code Reading has been customized to requirements, design, code, and usability Code Reading and inspection. Due to experimental variabilities and human nature, both positive and negative findings were reported.
We discussed a cognitive analysis that shed light on what is going on during perspective-based Code Reading and analyzed a simple model to suggest when perspective-based Code Reading can outperform other Code Reading techniques. Perspective-based Code Reading started to see some industrial adoption.
Scenario-Based Code Reading Techniques
Scenario-based Code Reading was developed by Porter, and their paper triggered an active research on software Code Reading techniques. Their method was later renamed “defect-based reading” and the term “scenario-based reading” was reserved for a group of Code Reading techniques that use different ways to decompose Code Reading scenarios.
The family of scenario-based Code Reading techniques includes defect-based Code Reading, perspective-based Code Reading, etc. Defect-based Code Reading concentrates on specific defect classes and perspective-based Code Reading focuses on viewpoints of consumers of a document. Defect-based Code Reading was originally employed for requirements inspection but could be applied to other software artifacts as well.
Scenario-based Code Reading was motivated by the ideas behind the active design review. In an active design review, reviewers are selected based on their expertise. Each reviewer is given a specific area to focus on and required to answer specifically designed questions that can only be answered after a critical Code Reading and analysis of the design, which solves the issue of passive participation. Scenario-based Code Reading is based on operational scenarios that give specific guidance to readers.
The guidance can be a set of questions, an assignment, or explicit instructions on how to conduct reviews and look for defects. To improve effectiveness, the overlap of scenario assignments shall be minimized. Because of detailed instructions on Code Reading and defect detection, this group of Code Reading techniques is considered systematic.
The scenario used here refers to a process-driven or operational scenario, not to be confused with scenarios in the use-case context. It is expressed in the form of algorithms that readers can apply to traverse a document with a particular emphasis. In practice several scenarios must be combined to provide an adequate coverage of a document, since each scenario is focused, detailed, and specific to a particular viewpoint.
Before analyzing and detecting defects in a document, one must understand the document first. In cognitive science, comprehension is often characterized as the construction of a mental model that represents the objects and relationships in the text document. An operational scenario has a set of activities to help a reader build a model of the software artifact and a set of review questions related to the model.
The model is simply an abstraction of the software artifact under review. In the defect-based Code Reading, the model can be different types of defects the document may have. In the perspective-based Code Reading, if the reader reads the code from a tester point of view, the model can be related to how testing can be performed. While the reader builds a model and answers the questions based on the analysis of the model with a particular emphasis, he or she can note down any anomaly and log defects.
To minimize the potential Code Reading overlap, scenarios shall be orthogonal to each other. One team member is responsible for one scenario and multiple members provide complete coverage. The findings of individual team members are aggregated via meeting or non-meeting, which depends on the inspection process bestowed by the organization. The choice of model or abstraction and the types of questions depend on artifacts, the organization’s problem history, or the objectives of Code Reading.
Compared with other Code Reading techniques such as ad hoc or checklist-based Code Reading, the scenario-based Code Reading techniques have the following benefits:
Systematic: Specific steps for individual Code Reading are well-defined.
Focused: Different readers focus on distinct aspects of the document.
Allow controlled improvement: Based on experience and feedback, one can identify and improve specific aspects of Code Reading techniques.
Customizable: Organizations can customize Code Reading techniques to a specific project or organization setting.
Allow training: One can use Code Reading techniques to train oneself in applying the techniques.
Defect-Based Code Reading
Defect-based Code Reading is the first proposed Code Reading technique in the family of scenario-based Code Reading and focused on detecting specific types of defects. The main idea is that if each reader uses different but systematic techniques to search for different, specific classes of defects, he or she and the whole team will have a better chance to detect defects effectively than readers applying ad hoc Code Reading or checklist-based Code Reading.
In this defect-based software Code Reading, each reader is given specific steps to discover a particular class of defects. Each reader’s role or responsibility is specific and narrowly defined, such as ensuring appropriate use of hardware interfaces, identifying untestable requirements, or checking conformance to coding standards and guidelines.
The defect-based Code Reading technique has been applied to detecting defects in software requirements specifications (SRS) and we discuss it in the context of SRS Code Reading.
Taxonomy of Defects in Requirements Specifications
Defects in SRS can be divided into two broad types: omission and commission. An omission defect means important information is missing from the SRS, while a commission type defect means incorrect, redundant, ambiguous, or conflicting information is presented in the SRS. Depending on what is missing, the omission type can be further divided into four groups:
Missing functionality: The specification of the internal operational behavior of the system is not included in the SRS.
Missing performance: The performance specification is not present in the SRS or is not acceptable by end users in an acceptance testing.
Missing environment: The environment in which the system will be operated is not specified or is specified incompletely in the SRS, including infrastructure, middleware, hardware, software, database, personnel, and so on.
Missing interface: The interfaces through which the system interacts and interoperates with outside or communication mechanisms or protocols through which the system exchanges data with outside is not included in the SRS.
The commission type of defects also has four subcategories: ambiguous information, inconsistent information, incorrect or extra functionality, and wrong section:
Ambiguous information: Technical terms, phrases, or anything essential for readers to correctly understand and interpret the SRS is undefined or defined in such a way to cause misunderstanding or misinterpretation.
Inconsistent information: Information in different parts of the SRS contradicts each other directly or indirectly, or actions or behaviors cannot both be correct or carried out by the system.
Incorrect fact: Specifications in the SRS cannot be true in the given context.
Wrong section: The SRS is poorly organized or the specification is put in a wrong section of the SRS.
There are other defect taxonomies. IEEE Std. 1028-2008 categorizes anomaly (defects) as missing, extra (superfluous), ambiguous, inconsistent, not conforming to standards, risk-prone, incorrect, unachievable, and editorial. In the recent survey, Walia and Carver attributed the defects to their sources, i.e., people, process, and documentation.
Defect-Based Code Reading Techniques
When readers are assigned to review and detect defects in a software artifact, they can have the same general responsibilities, e.g., detect as many defects as they can, and their Code Reading activities are not coordinated. Alternatively, in a coordinated team, readers can have separate and different responsibilities.
As the active design review suggests, individual readers can be more effective if they are assigned specific responsibilities and provided systematic techniques to meet those responsibilities. The defect-based Code Reading technique is motivated by the ideas behind the active design review.
In this Code Reading technique, a collection of procedures (operational scenarios) for detecting particular classes of defects is developed. Each reader executes one single scenario, and multiple readers are coordinated to improve the coverage while minimizing the overlap.
One way to come up with operational scenarios is to make use of the defect taxonomy, as discussed by Porter and his colleagues. The general defect taxonomy discussed earlier can be given to ad hoc readers and used to define their responsibilities. To support checklist-based Code Reading, detailed and concrete questions as checklist items can be developed under each defect classes. The checklist is thus a refinement of the taxonomy.
Those questions can be motivated by industrial checklists. For example, to detect defects related to inconsistent information, one can have following checklist items:
Are the requirements mutually consistent?
Are the functional requirements consistent with the system overview?
Are the functional requirements consistent with the operating environment?
The actual defect types, groups, and classes shall depend on the software artifact under concern and be based on the organizational experience. Checklist items shall be customized to detect those defects accordingly. Since the checklist is derived from the defect taxonomy, checklist responsibilities are a subset of the ad hoc responsibilities.
Scenarios for defect-based Code Reading are further derived from the checklist by substituting the checklist items with procedures designed to implement them. Hence, the responsibility of defect-based Code Reading is a subset of checklist-based Code Reading. If defect-based Code Reading scenarios cover all checklist items, defect-based Code Reading and checklist-based Code Reading can detect the same kinds of defects.
The horizontal axis labels the Code Reading technique and denotes the increased degrees of details in Code Reading procedures. As the Code Reading technique becomes more sophisticated (moving to the right), the defect coverage decreases. The Code Reading responsibilities and the classes of defects targeted for the latter Code Reading technique are the subsets of the former Code Reading techniques (moving to the left).
Each scenario was designed for a specific subset of checklist items. Those scenarios were intended to detect defects in areas of data type inconsistency, incorrect functionality, and ambiguous or missing functionality. According to their estimate, those scenarios cover about 50% of the defects present in the SRS. Their Incorrect Functionality Scenario is reproduced in Panel 1: Defect-Based Code Reading (Incorrect Functionality Scenario), and interested readers can refer to their original paper for the other two Code Reading scenarios.
PANEL 1: DEFECT-BASED Code Reading (INCORRECT FUNCTIONALITY SCENARIO)
2.\ For each functional requirement, identify all specified system events.
a.\ Is the specification of these events consistent with their intended interpretation?
3.\ Develop an invariant for each system mode or state, i.e., under what conditions must the system exit or remain in a given mode.
a.\ Can the system’s initial conditions fail to satisfy the initial mode’s invariant?
b.\ Identify a sequence of events that allows the system to enter a mode without satisfying the mode’s invariant.
c.\ Identify a sequence of events that allows the system to enter a mode but never leave.
The operational scenario leads a reader through the SRS and gives the reader instructions where and how to find defects. For example, in (1), “where” would be the input and output data object related to a functional requirement, and “how” is to check if all values generated for the output data object is consistent with its intended function and check whether the output data object is used by some functions.
From this scenario, one can clearly see the influence of active design review. It turns a passive reader into an active reader (e.g., step 3) and therefore improves readers’ effectiveness.
Defect-based Code Reading is a systematic technique that focuses the readers on detecting the specific class of defects that the scenario targets, providing a mechanism to focus the Code Reading according to project or organizational objectives. As scenarios are typically derived from checklist items, the technique suffers the similar drawbacks as checklist-based Code Reading.
For maximal coverage of defects, multiple scenarios have to be defined, which could be an elaborated process. In practice, defect-based Code Reading has to be applied in combination with other Code Reading techniques so that defects not covered by the defect scenarios can still be detected.
Defect-based Code Reading has been applied to requirements Code Reading. In theory, it can be applied to other artifacts. There are no other applications reported yet.
Porter et al.used graduate students in computer science in their experiments to test the effectiveness of defect-based Code Reading by comparing it with ad hoc Code Reading and checklist-based Code Reading. To some degree, they replicated and expanded their experiment and came to the same conclusion. The taxonomy was provided to all readers, but ad hoc readers were not given further instructions. The checklist and the scenarios were developed in the way discussed earlier.
They concluded that defect-based Code Reading had the highest defect detection rate (about 35% improvement over the other two Code Reading techniques), followed by ad hoc Code Reading, and checklist-based Code Reading had the lowest effectiveness. Further analysis confirmed that defect-based scenario readers were more effective than ad hoc and checklist readers at finding defects the scenario was designed to detect, and all readers were equally effective at detecting other defects not targeted by scenarios.
Scenarios helped readers focus on specific defect types, and they didn’t compromise readers’ ability to detect other defect types not covered by scenarios. Porter and Votta replicated the same experiment using industrial professionals and reported the similar results, although the performances of students and professionals were different.
Porter et al. studies were replicated by other researchers. However, the results were not consistent with the original findings, mainly defect-based Code Reading was not significantly better than other ad hoc and checklist-based readings. Hayes used meta-analytic (analysis of analyses) methods to synthesize the series of replicated experiments with seemingly contradictory results.
He suggested that the differences might be caused by subjects’ familiarity with the formal notation used in the SRS and with the software domains used in the experiment (e.g., cars with cruise control were popular in the United States but not in Europe).
Perspective-Based Code Reading
Perspective-based Code Reading was developed by Basili. It provides a set of operational scenarios. Each reader reads a software artifact under review using one of the scenarios from a particular perspective, and the combination of different perspectives yields a better coverage of the document. It was originally applied to requirements documents but subsequently applied to high-level designs, source code, and user interface.
Perspective-based Code Reading, in fact, includes a family of Code Reading techniques. In this section, we present perspective-based Code Reading in its generic form first, then discuss specifics when the technique is applied to different types of software artifacts. Finally, we shed light on why the Code Reading techniques are effective in practice.
A Generic Perspective-Based Code Reading
Software artifacts are created for various purposes. It is important to ask their stakeholders to read and assess if the documents meet their needs and quality expectation, from a particular point of view, which is called perspective. Perspective-based Code Reading focuses on a point of view or needs of the main customers of a document.
Each reader reads from their respective perspective with a narrowly defined focused view, which leads to an in-depth analysis of potential defects or anomalies in the document. The union of multiple perspectives provides an extensive coverage of the document so that no defects will be missed.
Defect reports from individual readings are aggregated at the end of Code Reading. It is not necessarily true that we select one reader for each perspective. How many readers are assigned to a given perspective depends on the organizational and project context.
For example, if more omission defects are expected in a requirements specification, one may assign more readers to the user role, since a user perspective offers a better opportunity for exposing omission defects. If there are not enough resources available, one reader may be assigned to multiple roles and he or she can simulate each perspective in turn.
When read with a particular perspective, defects that a perspective is targeting at would have a higher probability to be detected, while other defects not targeted by the perspective might have a lower probability to be detected. Thus an individual using perspective-based Code Reading may not be necessarily more effective and efficient than one using other Code Reading techniques.
Collectively on the team level, perspective-based Code Reading could be more effective and efficient as each perspective detects different kinds of defects.
The idea of perspective-based Code Reading is not completely new, however. It synthesizes ideas that have already been applied in software inspection but have never been worked out in great detail. The influence of the active design review was already mentioned.
Fagan stated that the code should be inspected by its tester. When reporting AT&T experience on in-process inspection, Fowler stated, “[E]ach meeting participant takes a particular point of view when examining the work product.” Graden also reported that when collecting the software inspection data, each participant must denote the perspective (customer, requirements, design, test, maintenance) by which they have evaluated the deliverable.
Code Reading Scenarios and Template
Perspective-based Code Reading has a set of operational scenarios that provide Code Reading guidance to readers. In an ideal case, those scenarios shall have a minimal overlap in defects detection, which is very hard to achieve in practice.
What scenarios to provide to readers depends on many factors such as artifact types, organization’s history on defects, as well as organizational objectives of software reviews.
The scenarios are developed using a scenario template, which has three sections:
Introduction: The introduction section explains the stakeholder’s interests in the artifact.
Instruction: The instruction section provides guidance regarding how to read and extract relevant information from artifacts or from the descriptions of artifacts and to optionally create other artifacts such as test cases from a tester’s perspective or a high-level design from a designer’s perspective. It helps a reader focus on the most relevant information and actively work with the document to gain a deep understanding of it.
Question: The question section is for reviewers to answer by following the instructions above. Questions are derived from past defect classes.
It shall be noted that to review software artifacts, one can ask real stakeholders and experts to review. One can also ask other readers to review, who will mimic real stakeholders’ perspective. For the latter case, the introduction section of a scenario is important to set up the scope and expectation.
Even when real stakeholders and experts are invited to review a document, they will appreciate the concrete guidance provided in the latter two sections of the scenario. However, people with less experience seem to follow the prescribed perspectives while people with more experience are more likely not to conform to their prescribed perspectives.
Developing Code Reading Scenarios
Scenarios play a pivotal role in perspective-based Code Reading. How are the scenarios created in the first place? Here in this section, we provide some guidance on scenario creation based on Laitenberger and Atkinson.
\1.\ Document identification.
Identify documents subject to Code Reading. This might be a software requirements specification document, software architectural or design description including diagrams and graphical models, source code files, a test plan including detailed test cases, or a graphical user interface mockup, etc.
\2.\ Stakeholder identification.
For the identified artifacts and their descriptions, identify key stakeholders who have a vested interest in the artifacts to be reviewed. Those stakeholders can be subject matter experts, users, or maintainers. They can be the consumer of the artifacts or the producer of upstream documents from which the current documents are based. They can also be peers of the author who created the artifacts. Prioritize those stakeholders according to the potential gain from the Code Reading activities and resource constraint.
\3.\ Perspective understanding.
For the identified key stakeholders, understand their perspectives by deciding and collecting what information and descriptions are important to them. You can interview and survey stakeholders to collect their inputs. After having understood the stakeholders, the scenario writer identifies which documents contain that information and how to identify and extract the information.
\4.\ Introduction development.
The scenario writer writes the introduction to a particular perspective and highlights stakeholders’ interests in the artifacts.
\5.\ Instruction development.
The scenario writer develops the detailed instruction on how to identify and extract, step by step, the required information from the artifact or the descriptions of the artifact.
\6.\ Questions development.
The scenario writer compiles a list of questions for a reader to answer based on the extracted information and the reader’s understanding of the artifact. The checklist, if available, as well as the historical defects can be a source for those questions. The writer makes sure only pertinent questions are included, i.e., those the reader could answer by Code Reading and analyzing the documents from the given perspective.
\7.\ Scenario review, testing, and updating.
After the scenarios are written, the scenario writer should invite stakeholders to review and pilot its usage to get rid of kinks in the scenarios. The above steps shall be iterated as necessary. The scenarios shall also be kept up to date. When new defects are uncovered, or the contents and structures of the documents are changed, the scenario instruction and questions shall be updated as well.
Characteristics of Perspective-Based Code Reading
Perspective-based Code Reading has been developed to satisfy a variety of goals, which are discussed below
Perspective-based Code Reading is adaptable to particular software artifacts, being it requirements specification, design description, source code, and user interface. It is also adaptable to whatever notation used in the document. For example, the requirements might be specified using natural languages or a more formal specification language.
Perspective-based Code Reading is tailorable to organizational and project setting: The number of perspectives, the number of readers using a scenario, and the Code Reading instruction and questions in a scenario are all customizable.
A Code Reading scenario gives a reader a well-defined instruction how to traverse a document. Hence the Code Reading process is repeatable and the effectiveness of defect detection becomes less dependent on the individual’s experience and expertise.
A particular perspective provides particular coverage of part of the document and multiple perspectives combined provide coverage of the whole document. This gives the individual reader a focus, which enables an in-depth analysis. At the same time, multiple perspectives ensure no areas are left unread.
Each reader has a specific, narrowly defined responsibility, with a specific tool (scenario) to meet this responsibility. The reader knows what perspective to simulate, which scenario to use, and where and how to detect defects.
Those benefits will become more evident as we go through the Code Reading techniques for specific types of documents in the remainder of this blog. Perspective-based Code Reading is not without drawbacks. The most noticeable one is the demanded effort. The improved rate of defect detection comes at the price of a higher effort on the reviewer.
Basili reported about 30% more effort, depending on the document under review. Much of the extra effort was spent on creating additional model and documentation, which might be required in a later stage of the development life cycle, e.g., test cases created during the requirement Code Reading can be reused later in the testing phase
Perspective-Based Requirements Code Reading
Many researchers have applied perspective-based Code Reading to requirements documents. In fact, it was first applied to the requirements document when the Code Reading technique was developed. This should not be a surprise, given the importance of a requirements document in the software development process.
Any problems and issues in requirements specification detected and corrected early can save a lot of time and money of the development organization since it will reduce potential rework of design, implementation, and testing, or avoid embarrassment if the product is shipped to customers with residual defects.
Code Reading Scenarios for Requirements Specifications
There are many possible stakeholders of a requirements document. Basili identified three key consumers of a requirements document, i.e., end user, tester, and designer. The requirements describe functionalities and performance expectations that the finished software product must meet; thus the end user is an important stakeholder.
The requirements are the basis for system design and the designer must create a design that provides the functionalities with the performance constraints documented in the requirements; therefore the designer is another important stakeholder. The finished software system must be verified to be conformant to functional and performance specifications, which is typically conducted by a tester; hence the tester is yet another important stakeholder of the requirements document.
From the user’s perspective, requirements must be complete and correct and provide the necessary system functionality. From the tester’s perspective, requirements must be testable with unambiguity so that the tester can construct test cases. From the designer’s perspective, requirements must have enough details with accuracy so
that the designer can design major system components. Failing to satisfy any of the above needs amounts to a requirement defect. Operational scenarios must be developed for the perspectives of the end user, tester, and developer.
Given the overarching impact of a requirements specification, it often involves more inspectors or readers than other types of software artifacts do. Other stakeholders and their perspectives might be important or even critical, depending on a particular environment.
For example, for the software requirements for a nuclear power plant-related system, designer, tester, user, maintainer, verifier, regulator, and the contractor might be identified as a key stakeholder. As a nuclear power plant has a long lifetime (60 years), the maintainer is a critical stakeholder as well.
One of the key ideas behind the active design review is to engage reviewers for active participation. With the same spirit, when a reader reads from a perspective, he or she is asked to create a physical model and answer questions from the same perspective based on the analysis of the model. When Code Reading from the tester’s perspective, the reader is asked to design a set of test cases and answer questions related to the activities being performed.
Similarly, when Code Reading from the developer’s perspective, the reader is asked to create a high-level design, and when Code Reading from the user’s perspective, the reader can develop use cases or draft a user’s manual. The exact document to generate can vary according to the reader’s experience or the organization’s needs.
Basili et al. tested their perspective-based Code Reading techniques on requirements documents using professional software developers as subjects. The techniques were compared to the existing techniques at NASA Goddard Space Flight Center Software Engineering Laboratory. Readers were given unique roles (user, tester, and designer) when Code Reading four requirement documents, two from NASA domain and the other two generics. They reported that perspective-based Code Reading provided significantly better coverage of both domain-specific and generic documents.
The experiment of Basili. was replicated and original findings were confirmed in an academic environment. In the replicated experiment, perspective-based Code Reading was compared to ad hoc Code Reading. Perspective-based Code Reading of requirements was replicated by many other groups.
However, the positive effects were not observed all the time. Regnell et al. questioned whether all the perspectives were really orthogonal and people using them found different kinds of defects. To address that, Maldonado et al. replicated the experiment with students as subjects.
They confirmed that, in general, different perspectives detect different kinds of defects. They also reported that perspective-based Code Reading didn’t outperform checklist-based Code Reading and perspective-based Code Reading and checklist-based Code Reading appeared to be complementary on defect detection.
PANEL 2: PBR – REQUIREMENT Code Reading SCENARIO FOR TESTERS
Perspective-based Code Reading is built around the concept that customers of a document are at the best position to judge if the document meets their needs and quality expectation. A different customer reads the document and detects defects from their respective perspective. This Code Reading scenario assumes a tester’s perspective. From the tester’s perspective, requirements must be testable and unambiguous.
For each requirement or functional specification, create a test case to verify that the implementation satisfies the requirement. Compose test cases into a test suite. While composing the test suite, answer the questions below.
1.\ Do you have all the information to identify the item being tested and to identify your test criteria? Can you create reasonable test cases for each item based on the criteria?
2.\ Is there another requirement for which you would create a similar test case but would get a contradictory result?
3.\ Can you be sure the test generated will output the correct value in the correct units?
4.\ Are there other interpretations of the requirement that the implementer might make due to the particular description of the requirement? Will this impact the test case and its oracle?
5.\ Does the requirement make sense based on your domain knowledge or from what is specified in the general description?
Perspective-Based Design Code Reading
Most software Code Reading techniques target textual documents such as requirements specifications and source code files. Design documents are typically written as textual descriptions annotated with graphical models and diagrams, which is particularly true for object-oriented designs.
An object-oriented approach is more popular nowadays than other paradigms such as a procedure or structured programming, but it is much more complicated and error-prone in both design and implementation. This section discusses how the perspective-based Code Reading technique can be customized to read the object-oriented designs.
Code Reading Scenarios for Design Documents
Unified modeling language (UML) is often used in object-oriented modeling at analysis and design levels. Use-case diagrams are used to capture the interaction with the system and the requirements of the system. Class diagrams are used to document the static structure. The dynamic behaviors can be captured using state diagrams, activity diagrams, sequence diagrams, or collaboration diagrams.
The last two kinds of diagrams capture identical information from different perspectives. What UML diagrams and textual documents to generate depending on the project and team needs.
Laitenberger and Atkinson suggested operation schema, class diagrams, collaboration diagrams, operation pseudocode, and data dictionary, where the operation schema is simply a method description, the operation pseudocode is the pseudocode for the method, and the data dictionary is a catalog of each modeled object with description.
We can identify the following stakeholders of a design: requirements engineer, designer, implementer, tester, and maintainer. A requirements engineer has a high interest to make sure requirements specifications have been fully and correctly considered in the design. A designer is primarily interested in the correctness of the design models and the analysis documents. An implementer wants to make sure all necessary information is provided in the design models so that he or she can realize the design.
A tester wants to know what functionalities are implemented and how to develop test cases to verify the correctness of those operations. A maintainer will extend the design and implement new functionalities in the future and he or she has a vested interest if the design has an acceptable maintainability. The number of Code Reading scenarios to develop for stakeholders depends on the organizational context, history, and objectives.
Stakeholders have vested interests in different documents assembled for a design. A possible association is shown in Table1. The concern of a requirements engineer is the correctness and completeness of the functional specification at the end of the analysis.
Particularly, various analysis models shall be consistent. Thus the requirements engineer has a high interest in the data dictionary, method descriptions (operation schema), as well as the responsibility assigned to each class.
The designer concerns the correctness and completeness of the design diagrams with respect to the analysis. In terms of correctness, design diagrams shall be consistent with analysis diagrams. In terms of completeness, all elements in analysis diagrams shall be reflected in design diagrams with sufficient details.
Thus he or she is interested in the operation schema, collaboration or sequence diagrams, and class diagrams. A sample design Code Reading scenario for a designer is given in Panel 3: PBR – Design Code Reading Scenario for Designers. One can customize it when experimenting or adopting perspective-based Code Reading.
PANEL 3: PBR – DESIGN Code Reading SCENARIO FOR DESIGNERS
Assume a designer's perspective when Code Reading the design artifacts. A designer concerns the correctness and completeness of design diagrams with respect to analysis diagrams.
Get familiar with the purpose of the system and the requirements (use cases). Locate the class diagrams, compare them with analysis diagrams, and make sure classes, attributes, methods, associations, constraints, and other abstract concepts are consistent. As design diagrams are a refinement of analysis diagrams, design diagrams sometimes contain additional elements.
Locate collaboration diagrams and compare them to the corresponding operation schema, and make sure the messages, parameters and their types, and the resulting behaviors are consistent.
Questions: While following the instruction, answer questions:
1.\ Is there anything in analysis documents but not in design documents?
2.\ Are the messages appropriate and consistent from the collaboration diagrams and operation schema?
3.\ Are the system start-up conditions clear and correct?
4.\ Are there corresponding sent messages in operation schema and collaboration diagrams?
5.\ Is the order of messages correct in the collaboration diagram?
6.\ For every change of attributes, objects, states, or links in the operation schema, is there a corresponding sent a message in the collaboration diagram?
7.\ Are there any discrepancies of the functionality described in the operation schema and in the collaboration diagram?
The main responsibility of an implementer is to translate the design into code. As such, the implementer is concerned with the consistency and completeness of the design, as well as whether the design is implementable.
To check for consistency, the implementer reads and detects any contradictions among design elements scattered in multiple diagrams. To check for completeness, the implementer reads and detects any missing elements from diagrams based on the analysis and design models.
The implementer makes a professional judgment regarding whether the design is implementable based on project constraints, technologies and development environment, as well as other considerations. The implementer has high vested interests in the class diagram, collaboration diagram, and pseudocode.
He or she will verify the messages received by an object as documented in the sequence or collaboration diagram are implemented by the class, and the collaboration diagram is consistent with the operation pseudocode.
The tester is concerned if the operations are testable. A productive approach is to walk through test cases and ensure test cases are correct with respect to the associated functions. The tester is thus interested in the operation schema and their collaboration diagram as well as pseudocode that implements the collaboration. A design Code Reading scenario for testers is illustrated in Panel 4: PBR – Design Code Reading Scenario for Testers.
We contrast this tester-based Code Reading scenario with that for the requirements specification. Here since the internal implementation is available as revealed in the pseudocode and collaboration diagrams, white-box testing becomes feasible, i.e., one can develop test cases to cover the various branches and loops, which is not feasible when only the requirements are available.
The maintainer is concerned with the maintainability of the system. To improve the maintainability or modifiability of a system, the designer shall follow well-established design principles such as low coupling and high cohesion as well as the simplicity of the design. Bass et al. have written about the tactics of improved modifiability. The proper places to check whether best practices are followed are collaboration diagrams and pseudocode.
PANEL 4: PBR – DESIGN Code Reading SCENARIO FOR TESTERS
Assume a tester's perspective when Code Reading design artifacts. Ensure the correctness and robustness of all operations and their testability. For this perspective-based Code Reading, test cases are used. If they don’t exist yet, have them created.
Get familiar with the requirements (use cases). Locate the operation schema and identify test cases derived from the schema. Locate the collaboration diagram associated with the operation schema. Walk through the collaboration diagram and simulate the execution of the test cases; check if the correct results are generated, all branches and paths are covered, and the resulting behavior is consistent with the schema.
Questions: While following the instruction, answer questions:
1.\ Do the branches in the collaboration diagram match the condition outcomes in the operation schema?
2.\ Are all possible inputs and their combinations properly addressed in the operation schema and the pseudocode?
3.\ Are the effects of each collaboration diagram consistent with the corresponding operation schema?
Laitenberger compared perspective-based design Code Reading to checklist-based Code Reading, with practitioners (experienced students) as subjects. The software artifacts under Code Reading were UML design documents. Their checklist was developed from scratch. Three Code Reading perspectives were employed (designer, implementer, and tester). For both Code Reading techniques, defects were pooled through the defect collection meeting attended by three team members.
They reported that teams found more defects using perspective-based Code Reading than using checklist-based Code Reading and, on average, the improvement of detection effectiveness is 41%. When considering the defect detection effort, perspective-based Code Reading was also cost-effective for the defect detection phase, the meeting phase, and the overall inspection.
The readers also believed that applying perspective-based Code Reading helped them improve their understanding of the systems.
They used three Code Reading perspectives (user, designer, and implementer). The authors concluded that defect detection effectiveness using both Code Reading techniques was similar, but checklist-based Code Reading was more effective for three-person simulated virtual teams.
Perspective-Based Code Code Reading
In industry practice, code documents are still the most frequently inspected software artifacts. It is thus natural to apply perspective-based Code Reading to code. This section discusses the work in that area.
Code Reading Scenarios for Code Modules
Code modules subject to inspection include implementation source code files as well as their operational descriptions as a specification. We can identify two main stakeholders of those artifacts: code analyst and tester. While a code analyst is concerned if the code implements the right function, a tester is concerned if the code implements the function right.
One could certainly identify more stakeholders. Laitenberger split the tester into the module tester and the integration tester. A maintainer has a high interest in the source code and he or she is concerned if the code is easily understandable to allow future modification with minimal effort. The code can also be checked if it is compliant to coding standards and guidelines. This latter checking can be accomplished with modern tools.
A code analyst locates different functions in the code file, reads and understands them, and forms high-level abstractions. When Code Reading source code, he or she can use any code Code Reading techniques such as stepwise abstraction for structured code or abstraction-driven Code Reading for object-oriented code, which are discussed in a later blog.
The reader then compares the high-level abstraction to the code specification and notes any deviation as a potential defect. The reader has a set of limited questions to answer, specifically designed for his or her Code Reading perspective, based on the code documents and the understanding the reader achieved at this point. A sample Code Reading scenario is shown in Panel 5: PBR – Code Code Reading Scenario for Code Analysts.
PANEL 5: PBR – CODE Code Reading SCENARIO FOR CODE ANALYSTS
Assume the code analyst Code Reading perspective, who has to ensure the right functionalities are implemented in code.
Locate the functions implemented in the code. Use the stepwise abstraction Code Reading technique if it is a structured code; use the abstraction-driven and use-case-driven Code Reading techniques if it is object-oriented code.
Start with the least-dependent code and read bottom-up to form a high-level abstraction of the coded function. For each function, check if the abstraction you derived matches the specification; if there is a deviation, decide if it is a defect and log your findings.
Questions: While following the instruction above, answer questions:
1.\ Does the operation implemented in code match the one described in the specification?
2.\ Is there any operation described in the specification, but not implemented in the code?
3.\ Are data (constants, variables, etc) and their type used correctly?
4.\ Is the calculation performed correctly?
5.\ Is usage of the interface between different modules semantically correct?
A code tester identifies different functions implemented in the code and develops and runs test cases on them to ensure their correctness. The reader then mentally simulates the execution of each function guided by test cases and compares the outcome with specifications. Any deviation shall be noted as a potential defect.
If test cases have been developed during design Code Reading, those test cases can be reused here. The reader also has a set of limited questions to answer, specifically designed for this tester Code Reading perspective.
Those questions are intended to focus the reader’s attention. A sample tester Code Reading scenario is shown in Panel 6: PBR – Code Code Reading Scenario for Code Testers, which is adapted from.
PANEL 6: PBR – CODE Code Reading SCENARIO FOR CODE TESTERS
Assume the tester Code Reading perspective, which has to ensure the functionalities implemented in code are correct.
Locate the functions implemented in code. Determine the dependencies among these functions and build a call graph from them. Start with the nodes with the least dependencies and traverse the call graph backward, and develop or reuse test cases that allow you to mentally simulate the execution of the function implemented by the node to cover all branches and loops.
For each simulated execution, compare the outcome with that in the specification. If there is a discrepancy, decide if it is a defect and log your findings.
Questions: While following the instruction above, answer questions:
1.\ Do you have all required information to identify a test case?
2.\ Are branch conditions implemented correctly?
3.\ Can you generate test cases for all branches and loops? Can you traverse all branches by some use cases? Does the outcome of each test case match the specification?
4.\ Is memory allocated and deallocated properly?
5.\ Can you traverse the call graph?
As we move down from design to implementation, the details are revealed gradually. In this code Code Reading scenario, we are able to ask concrete questions, e.g., on the memory usage and branch implementations and coverage. This Code Reading scenario is meant for module or unit testers as well as module integration testers.
Laitenberger et al. ran a quasi-experiment and two replicates on professional software developers to compare the performances of perspective-based Code Reading and checklist-based Code Reading of C programming files. It is a quasi-experiment since they piggybacked experiments on their industrial training opportunities and they didn’t have control over all experimental conditions.
They used two Code Reading perspectives, analyst and tester. Defects were pooled through defect collection meetings and defects gain or loss effects in meetings were almost non-existing, thus ignored. Meta-analysis was conducted on the three experiments whenever possible. The researchers reported that perspective-based code Code Reading had a slightly higher team defect detection effectiveness than checklist-based Code Reading.
The difference was statistically significant. Although individual using perspectives seems to require more effort (time) to detect a defect, the extra effort seems to be justified on the team level, since the team has a better cost per defect. The difference was significant on the team level using meta-analysis.
The meeting cost of perspective-based Code Reading was lower. Conjectured reasons include an increased understanding enabled by perspectives. When all costs (individual Code Reading and defect collection meeting) were considered, perspective-based Code Reading had a lower cost per defect and less variability than checklist-based Code Reading.
Perspective-Based Usability Code Reading
Usability is concerned with how easy it is for a user to perform the desired task and what supports the system provides to the user to accomplish the task. It comprises the following aspects: learning system features, using the system efficiently, minimizing the impact of errors, adapting the system to user needs, and increasing user confidence and satisfaction.
Experiences indicate that focusing on usability is the easiest and cheapest way to improve the user’s perceived system quality. In this section we discuss how the ideas of perspective-based Code Reading can be applied to inspect usability of software applications.
Code Reading Scenarios for Usability
Usability problems can be detected with usability inspection or usability testing. In usability testing, usability problems are found through the observation of and interaction with users while they use or comment on the user interface. In usability inspection, problems are found through the expertise of the inspectors and the inspection techniques they employ.
The latter costs much less than the former, and heuristic evaluation, cognitive walkthrough, and guidelines and checklists are common inspection techniques in practice. However, their effectiveness (percent of problems detected) is rather low, particularly for non-experts.
To improve the problem of detection effectiveness, Zhang proposed a perspective-based usability inspection method known as use-based Code Reading. They divided the large variety of usability issues along with different perspectives (novice use, expert use, and error handling) and held multiple inspection sessions, each focusing on one particular perspective.
Each perspective provides the inspector a point of view, a list of questions to check, which represents typical usability issues related to the perspective, and a specific procedure to perform the inspection. Just as perspective-based Code Reading of other software artifacts, the focused attention, and well-defined procedure improve the inspector’s problem detection performance related to the given perspective, and the combination of multiple orthogonal perspectives provides a better coverage of the usability spectrum and uncovers more usability issues.
People use computers to accomplish a task, and the usability or lack of it is exhibited during the human-computer interaction (HCI), which can be abstracted as the repetition of the following sequence of steps: form the goal, form the intention, identify the action, execute the action, perceive the system response, interpret the results, understand the outcome, and deal with errors if any.
This sequence of steps can be summarized as execution and evaluation, with possible error correction. Therefore, usability issues are categorized as the gulf of execution (the mismatch between the user’s intention and the computer action) and the Gulf of evaluation (the mismatch between the user’s expectation and system’s representation. This simplistic HCI model guides the development of Code Reading perspectives.
Usability perspectives are scenarios of HCI, and different perspectives emphasize different stages in the HCI model or different aspects of the same stage. Based on whether the user knows how to achieve a certain goal and whether the user executes the action correctly, Zhang et al. identified three perspectives: novice use, expert use, and error handling.
For the novice use perspective, the user does not have a deep knowledge regarding how to use the system to achieve his or her goal. Thus the usability goal is for the novice users with the minimum knowledge to accomplish basic tasks. For the expert use perspective, the user knows how to use the system but wants to accomplish the task efficiently or easily or to achieve other higher goals.
Thus the usability goal is for the expert users to complete tasks efficiently and easily, customize the system to their desire, and use advanced functions or features for improved productivity. For the error handling perspective, the user has a problem with the effect of the previous action and needs to correct the problem.
Thus the usability goal is to minimize the chance to commit an error (error prevention), help the user to understand the situation once errors happen, allow the user recover from errors (error recovery), and deal with system failure appropriately. Questions in Code Reading scenarios help the reader decide if the usability goals can be met or not and report usability gaps if not.
Panel 7: PBR – Expert Use Code Reading Scenarios shows the expert use Code Reading scenarios adapted. The scenarios are designed specifically for a web-based data collection form application.
Although the exact questions depend on the application being assessed, the intention is similar. The above expert use Code Reading scenario aims at checking the user interface for its efficiency, flexibility, and consistency in supporting the user tasks, as well as the appropriateness of its visual appearance and organization, which supports the usability goal of the expert use discussed above.
While the perspective-based usability inspection of Zhang et al. partitions Code Reading scenarios from the use point of view (novice, expert, and error handling), there are alternative ways to partition Code Reading scenarios.
They identified conceptual, presentation, and navigation design perspectives for web applications that are used to guide the interpretation of commonly used heuristics. Combinations of the heuristics and design perspectives have yielded a set of specific Code Reading guidance.
PANEL 7: PBR – EXPERT USE Code Reading SCENARIOS
Assume you are an expert user. Your goal is to fill out the web form and submit it, which can be decomposed into multiple sub-goals in order. For each sub-goal, go through the following stages and check the questions. If your answer to a question is no, raise a usability issue.
Code Reading Instruction 1: Scan through instruction, objects, and actions in the user interface.
Question set 1:
1.\ Is the text easy to read?
2.\ Is the information organized such that the most important information can be read first?
3.\ Are the more frequently selected items arranged on top in the list control?
4.\ Is redundant information avoided or minimized?
Code Reading Instruction 2: Execute the actions for achieving the sub-goal, using shortcuts whenever available. For each action, check the following questions.
Question set 2:
1.\ Are common shortcuts available, e.g., to allow the user to step to the next text field via keyboard?
2.\ Are possible default values provided?
3.\ Does the system compute and remember information for the user whenever possible?
4.\ Can the user make a selection by clicking on a larger area associated with the object to be selected?
5.\ Are unproductive activities avoided or minimized, e.g., navigation, mouse movement, hand movement, eye movement, etc.?
6.\ Are stressful actions avoided or minimized, e.g., clicking the mouse
multiple times in a short period of time, clicking on a small object to make a selection, etc.?
7.\ Is the text easy to read?
Code Reading Instruction 3: Wait for system to respond, if necessary.
Question set 3:
1.\ Does each user action immediately generate perceivable results in the user interface?
General question: In addition to the specific questions in Question set 3, consider this high-level question: Can the web form be redesigned to reduce the user’s unproductive activities?
Zhang et al. used professionals (programmers, domain experts, technique researchers, and cognitive researchers) as subjects and compared the use-based Code Reading and the heuristic evaluation in inspecting the usability of a web-based data collection form. Heuristic evaluation is a popular inspection technique in practice. It gives the same general responsibility and provides a list of usability issues to participants, which is similar to ad hoc Code Reading discussed earlier.
The heuristics used include: speak the users’ language, remain consistency, minimize the users’ memory load and fatigue, flexibility and efficiency of use, use visually functional design, design for easy navigation, conduct validation checks, facilitate data entry, and provide sufficient guidance.
They reported that perspective-based usability inspection found not only more issues related to their assigned perspectives but also more overall problems. When issues from three perspective readings were aggregated, the use-based Code Reading uncovered about 30% more problems. Assigning inspectors more specific, focused responsibilities leads to improved performance, and combining multiple perspectives focused inspections is a good strategy for creating high usability user interfaces.
Why Does Perspective-Based Code Reading Work?
As discussed earlier, perspective-based Code Reading has been adapted and applied to software artifacts such as requirements specification, design description and graphical models and diagrams, source code and specifications, and user interface usability. Some experiments have been replicated internally and externally.
Researchers reported both positive and negative experiences, and sometimes contradicting findings. To navigate through the collection of replicated experiments and associated findings, Ciolkowski performed a meta-analysis, which proved to be challenging, given the fact that the data were reported differently and important information was sometimes missing.
Nevertheless, the author concluded that for requirements documents, perspective-based Code Reading was significantly more effective than ad hoc Code Reading; for design and code documents, perspective-based Code Reading was significantly more effective than checklist-based Code Reading, but for requirements documents, it was less effective. This section discusses a cognitive analysis that sheds light on what is going on during perspective-based Code Reading and how people’s experiences play a role during Code Reading.
We also discuss a simple analytical model that provides indicative information when perspective-based Code Reading can outperform other Code Reading techniques.
Cognitive Analysis of Perspective-Based Code Reading
The cognitive analysis from cognitive science has been applied to the understanding of software inspection meetings. Letovsky video taped a representative code inspection meeting at IBM and analyzed it in detail. Their observational results suggested that participants were attempting to achieve three goals (clarity, correctness, and consistency) by executing three corresponding behaviors (design reconstruction, mental simulation, and document cross-checking).
These categories accounted for 89% of the duration of the session. Robbins and Carver have applied a similar protocol analysis to perspective-based Code Reading of requirements. Unlike the defect collection meeting, individual’s Code Reading of the software document typically happens quietly. To work around that, readers were requested to think aloud, i.e., they just spoke whatever in their mind at the moment during their Code Reading. To facilitate the analysis, verbal reports during Code Reading were coded and categorized.
Perspective-based Code Reading is a complex problem-solving task that is focused on the problem of detecting defects in documents, where the sensory input of the documents, including the requirements document, Code Reading scenario description, and worksheet to log defects, goes through the sensory memory that feeds to the short-term memory.
An individual’s knowledge on the application domain, software and programming experience, the general knowledge on Code Reading techniques, as well as the knowledge on the Code Reading document newly acquired during Code Reading are stored in the long-term memory, which is retrieved and sent to the short-term memory. All problem-solving tasks are performed in the short-term memory and the outcome is formed as a response.
Verbal reports during Code Reading were audio-recorded. Words or rephrases spoken by readers were captured and sometimes cross-checked with the defect worksheet, which was coded according to the model above:
Any mentioning of Code Reading of the requirements document, scenario description, and worksheet was coded as stimulus inputs with their respective codes.
Any mentioning or recalling of previous knowledge or facts on the requirements document, training, perspective Code Reading technique, domain knowledge, software, or programming knowledge was coded as retrievals from long-term memory using their respective codes.
Any mentioning of combining existing knowledge into a single factor making an assumption on the document was coded as manipulation in short-term memory with their respective codes.
Any mentioning or action of writing on the worksheet or logging defect was coded as response output with their respective codes.
The researchers ran the experiment using graduate and upper-level undergraduate students as subjects. The requirements document was about a web-based conference paper submission and review system. Only user and tester perspectives were used. The codes were analyzed using a frequency-based method. The authors were able to confirm:
Most defect identifications were preceded by combining knowledge actions, which corresponds to the “defect trigger.”
The type of defects found by readers was related to the assigned Code Reading perspectives, user, or tester. That is, the perspectives actually lead to detection of perspective-related defects.
Perspective specific experience significantly affected the perspective-based Code Reading process.
The protocol analysis provides a robust approach to understanding the cognitive mechanisms supporting perspective-based Code Reading. It has a potential to reconcile all seemingly contradictory observations on the Code Reading technique.
Alternative Partitioning of Code Reading Responsibilities
We have discussed two scenario-based Code Reading techniques, defect-based Code Reading and perspective-based Code Reading. While defect-based Code Reading partitions scenarios based on defect classes, perspective-based Code Reading decomposes scenarios based on stakeholders’ perspectives to the documents under review.
When a defect or perspective scenario is assigned to a reader, the portion or aspect of the document is assigned to the reader implicitly. There exist other alternatives that partition the Code Reading responsibilities and coordinate Code Reading activities, however.
Ad hoc Partition
A document under review usually has multiple parts. If a template is used to write the document, it might have different blogs and sections. For example, a requirements specification of an IT application may have blogs on web interface, billing, database, etc. Among all those parts, a group of reviewers can decide how to assign different portions to different people. Cheng and Jeffery called this self-set strategy.
The document author or review coordinator can make the assignment as well, based on readers’ interests and expertise. Once the document review responsibilities are divided among readers, each reader can use whatever Code Reading techniques they may possess, e.g., ad hoc Code Reading or checklist-based Code Reading.
This ad hoc assignment reduces the work overload to individual readers and provides a specific area of focus for them, compared to the case where the whole document is given to reviewers and the reviewers are simply asked to detect as many defects as they could.
Function Point-Based Partition
Cheng and Jeffery developed a set of scenarios, function point scenarios, for requirements Code Reading based on function point analysis. Functional user requirements can be categorized into five types: inputs, outputs, inquiries, internal files, and external interfaces. Each of those types can be easily mapped to end-user business function, e.g., an input mapped to a data entry, an inquiry mapped to a user query.
Function point analysis is frequently used to estimate the complexity and effort of the software. The authors, however, used the five functional types to dissect the software system into orthogonal areas, which provides the basis for Code Reading scenario construction. Five function point scenarios are:
Overview: This is an overall description of the entire system. The purpose of this Code Reading scenario is to check the overall relevance, completeness, and correctness of a requirements specification.
File: This combines the internal files and external interfaces functional types, given that they have similar characteristics with respect to the requirements review. This contains the attributes for the purpose of the application.
Input: This is a process that requests the inputs from a user or another system. The input may come as screen input or batch input.
Output: This is the opposite as the input and generates information for a user or another system, which can be in a form of report, or data transferred to other processes.
Inquiry: A process that retrieves data as a response to an input from a user or a process, and outputs the data to a user or another system. Data update is not involved in this process.
Key issues covered by each function point analysis are constructed into domain-specific questions, which are augmented by available checklist items. Those questions are used directly in a scenario description, which also provides instruction to focus the reader on specific areas of the requirements document related to the scenario/functional type.
Cheng and Jeffery used senior undergraduate and graduate students with industrial experience as subjects. They compared the defect detection performances of self-set strategy and function point scenarios and reported that the self-set strategy group detected more defects. The difference was not statistically significant though.
They further reported that there was no significant difference in defect types either. While the Code Reading using the self-set strategy showed a dependency on the individual experience, the same dependence was not observed in the subjects using the function point scenarios. The researchers could not tell if function point scenario-based Code Reading helped the inexperienced readers or inhibited the experienced readers.