- 1. TDT 4242
Tor Stålhane
- 2. Course contents
The course consists of two parts:
• Requirements – a description of what we
shall develop: 11 lectures
• Tests – a description of how we will
check that we have developed what the
customer required: 19 lectures
The curriculum consists of
• A set of slides
• Copies of journal articles
- 3. Course grade
The course grade will be decided based on
• One group exercise on writing
requirements – 10%
• One group exercise on writing a test
strategy for a set of requirements – 10%
• One group exercise on testing a software
package – 30%
• Exam – 50%
- 4. Lecture plan – 1
Tuesdays 14:15 – 16:00 in F3
Fridays 10:15 – 11:00 in F3
• First lecture – Tuesday, January 11, 2011.
• Last lecture – Friday, April 15, 2011.
• Exercise 1 – week 4.
• Exercise 2 – week 7.
• Exercise 3 – weeks 13 and 14.
There will be no lectures during the exercise
periods.
- 5. Lecture plan – 2
Week Course content
What you will learn
Presentation of lecture plan and exercises
2 Goal oriented requirements specification
Quality issues in requirements
Requirements elicitation with boilerplates (templates)
On requirement – testability and traceability – 1
3 On requirement – testability and traceability – 2
Intro. to exercise 1
4 Exercise 1 – Write a requirements specification
Testing vs. inspection – strong and weak sides
5 Testing vs. inspection – strong and weak sides
Testing and cost / benefit analysis
Testing strategies – a general introduction
Black box vs. White box
6
Grey box testing
Introduction to exercise 2
7 Exercise 2 – Write a testing strategy for the requirements specification from exercise 1
- 6. How to prioritize requirements
8 Requirements handling in COTS, Outsourcing, sub-contracting etc.
Requirements in embedded and information systems
Aspects, cross-cutting concerns and non-functional requirements
9 Testing COTS, Outsourcing, sub-contracting etc.
Presentation of the FITNESS tool
Domain testing
10 Coverage measures
Use of test coverage measures for code and requirements
Requirements through user stories and scenarios
11 How to write good user stories and scenarios
Advanced use cases
Test driven development – TDD 1
12 Test driven development – TDD 2
Introduction to exercise 3
13 Exercise 3
14 Exercise 3
Regression testing
15 Non-functional requirements – safety, user friendliness etc.
Testing non-functional requirements
- 7. Main theme
The main message of this course is that
requirements and tests are two sides of
the same coin.
• Requirements without tests will be
ignored.
• Tests without requirements are
meaningless.
- 8. The need for traceability
It must be possible to trace
– From test to requirement. Why do we need
this test?
– From requirement to test.
• Where is this requirement tested?
• Staus: How many of the tests for this requirement
has been executed successfully?
- 9. Challenges in Requirements Engineering
What is a requirement?
•What a system must do (functional):
System requirements
•How well the system will perform its functions (non-functional):
Institutt for datateknikk og System quality attributes
informasjonsvitenskap The RE process:
Inah Omoronyia and Tor Stålhane
Requirements Specification and Testing
An introduction defined satisfy
Ultimately: operational
business
needs
capabilities
TDT 4242 TDT 4242 TDT 4242
Challenges in Requirements Engineering Challenges in Requirements Engineering
Importance of getting requirements right:
1/3 budget to correct errors originate from requirements
Source: Benoy R Nair (IBS software services)
TDT 4242 TDT 4242
- 10. Challenges in Requirements Engineering Challenges in Requirements Engineering
Factors that make a software project challenging: Why projects are cancelled:
Source: Benoy R Nair (IBS software services) Source: Benoy R Nair (IBS software services)
TDT 4242 TDT 4242
Requirements Development - 1 Requirements Development – 2
Requirements Elicitation: Effects of Inadequate Requirements development – Ariane 5:
The process of discovering the requirements for a system by (An expendable launch system used to deliver payloads into
communication with customers, system users and others who have a geostationary transfer orbit or low Earth orbit)
stake in the system development.
Ariane 5 succeeded Ariane 4. Wrong implicit assumptions about the
Requirements gathering techniques parameters, in particular the horizontal velocity that were safe for
Ariane 4 but not Ariane 5.
• Methodical extraction of concrete requirements from high • horizontal velocity exceeded the maximum value for a 16 bit unsigned integer
level goals when it was converted from it's signed 64 bit representation.
• Requirements quality metrics • Ariane 5: component (requirements) should have been designed for reuse –
but the context of reuse was not specified.
Cost of poor requirements in Ariane 5
• Data overflow on launch
• Self-destruction of the complete system
• Loss > 500 Million EUR
TDT 4242 TDT 4242
- 11. Requirements Development – 3 Problem world and machine solution
Effects of Inadequate Requirements development – Airbus: The problem to be solved is rooted in a complex organizational, technical or physical
world.
Requirement: Reverse thrust may only be used, when the airplane is • The aim of a software project is to improve the world by building some machine
landed. expected to solve the problem.
• Problem world and machine solution each have their own phenomena while sharing
Translation: Reverse thrust may only be used while the wheels are others.
rotating. • The shared phenomena defines the interface through which the machine interacts
with the world.
Implementation: Reverse thrust may only be used while the wheels
are rotating fast enough.
Situation: Rainstorm – aquaplaning
Result: Crash due to overshooting the runway!
Problem: erroneous modeling in the requirement phase E-commerce world
Requirements engineering is concerned with the machine’s effect on the
surrounding world and the assumption we make about that world.
TDT 4242 TDT 4242
Formulation of requirements statements Two types of requirements statements
• Descriptive statements: state properties about the system
that holds regardless of how the system behaves. E.g. If train
doors are open, they are not closed.
• Prescriptive statements: States desirable properties
about the system that may hold or not depending on how the
system behaves
• Need to be enforced by system components
Statement scope: • E.g. Train doors shall always remain closed when the
• Phenomenon of train physically moving is owned by environment. It train is moving
cannot be directly observed by software phenomenon
• The phenomenon of train measured speed being non-null is shared by
software and environment. It is measured by a speedometer in the
environment and observed by the software.
TDT 4242 TDT 4242
- 12. Formulation of system requirement Formulation of software requirement
A prescriptive statement enforced by the software-to-be.
A prescriptive statement enforced solely by the software-to-
• Possibly in cooperation with other system components be. Formulated in terms of phenomena shared between the
• Formulated in terms of environment phenomena software and environment.
Example: The software “understand” or “sense” the environment
All train doors shall always remain closed while the train is through input data
moving
Example:
In addition to the software-to-be we also requires the The doorState output variable shall always have the value
cooperation of other components: ‘closed’ when the measuredSpeed input variable has a non-
• Train controller being responsible for the safe control null value
of doors.
• The passenger refraining from opening doors
unsafely
• Door actuators working properly
TDT 4242 TDT 4242
Domain properties Goal orientation in requirements engineering – 1
A domain property:
• Is a descriptive statement about the problem world A goal is an objective that the system under
• Should hold invariably regardless of how the system consideration shall achieve.
behaves
– Ranges from high-level strategic to low-level
• Usually corresponds to some physical laws
technical concerns over a system
– System consist of both the software and its
Example:
environment. Interaction between active
A train is moving if and only if its physical speed is non-null.
components, i.e. devices, humans, software etc
also called Agents
TDT 4242 TDT 4242
- 13. Goal orientation in requirements engineering – 2 Goal statement typology
Goals can be stated at different levels of granularity:
– High-level goal: A goal that requires the cooperation of
many agents. They are normally stating strategic
objective related to the business, e.g.
The system’s transportation capacity shall be increased
by 50%
– Requirement: A goal under the responsibility of a
single agent in the software-to-be.
– Assumption (expectation): A goal under the
responsibility of a single agent in the environment of
the software-to-be. Assumptions cannot be enforced by
the software-to-be
TDT 4242 TDT 4242
Goal types Behavioral goal specialization
TDT 4242
TDT 4242
- 14. Goal categorization – 1 Goal categorization – 2
Goal categories are similar to requirements categories: Functional goal: States the intent underpinning a system
service
• Satisfaction: Functional goals concerned with
satisfying agent request
• Information: Functional goals concerned with keeping
agents informed about important system states
• Stimulus-response: Functional goals concerned with
providing appropriate response to specific event
Example: The on-board controller shall update the train’s
acceleration to the commanded one immediately on
receipt of an acceleration command from the station
computer
TDT 4242 TDT 4242
Goal categorization – 3 Goal refinement
A mechanism for structuring complex specifications at different levels of
Non-functional goal: States a quality or constraint on concern.
service provision or development. A goal can be refined in a set of sub-goals that jointly contribute to it.
Accuracy goal: Non-functional goals requiring the state of Each sub-goal is refined into finer-grained goals until we reach a
requirement on the software and expectation (assumption) on the
variables controlled by the software to reflect the state of
environment.
corresponding quantities controlled by environment agent
NB: Requirements on software are associated with a single agent and
E.g: The train’s physical speed and commanded speed they are testable
may never differ by more than X miles per hour
Soft goals are different from non-functional goals. Soft goals
are goals with no clear-cut criteria to determine their
satisfaction.
E.g: The ATM interface should be more user friendly
TDT 4242 TDT 4242
- 15. Goal refinement tree – 1
Goal refinement: Example
Refinement links are two way links: One showing goal decomposition, the
other showing goal contribution
TDT 4242
TDT 4242
Goal refinement tree – 2
Goal feature annotation
Requirements quality metrics – 1
Qualitative Goal‐Requirements tracing:
An approach to requirements
refinement/abstraction that makes it
less likely to generate trace links that
are ambiguous, inconsistent, opaque,
noisy, incomplete or with forward
referencing items
TDT 4242 TDT 4242
- 16. Requirements quality metrics – 2 Requirements quality metrics – 3
Forward Referencing: Requirement items that make
use of problem world domain features that are not yet
defined.
E, C and D need to be mapped to a requirement item
TDT 4242 TDT 4242
Requirements quality metrics – 4 Requirements quality metrics – 5
Opacity: Requirement items for which rational or
dependencies are invisible.
Noise: Requirement items that yield no information on
Multiple unrelated concept mapping. A is not related to B problem world features. X refers to a concept
undefined in the domain
TDT 4242 TDT 4242
- 17. Requirements quality metrics – 6 Requirements quality metrics
Completeness: The needs of a prescribed Quality metrics on a requirements set provides useful
system are fully covered by requirement understanding, tracking and control of requirements
items without any undesirable outcome. improvement process.
No requirement item mentions the goal
concept Z
TDT 4242
Where do the goals come from? Summary
We get goals from:
Goals can be defined at different levels of
• Preliminary analysis of the current system. abstraction
• Systematically by searching intentional keywords
There are two types of goals: Behavioral or soft goal
in documents provided, interview transcripts etc.
E.g. ‘objective’, ‘purpose’, ‘in order to’.
• Iterative refinement and abstraction of high‐level There are several categories of goals, e.g.
goals: By asking the how and why question. • Functional and non-functional
Results in a goal refinement tree • Goal refinement provides a natural mechanism
• Approaches: KAOS – Goal driven requirements for structuring complex specifications at
acquisition. different levels of concern:
• Goal refinement graph
TDT 4242 TDT 4242
- 18. Requirements
There are three levels of requirements:
• Informal – e.g. Natural language (NL): free text,
no rules apply
Institutt for datateknikk og
informasjonsvitenskap • Semiformal
• Guided Natural Language (GNL): free text but
Inah Omoronyia and Tor Stålhane allowable terms are defined by a vocabulare
• Boilerplates (BP): structured text and an
ontology – vocabulary plus relationships
Guided Natural Language between terms
and
• Formal: e.g. state diagrams or predicate logic
Requirement Boilerplates
TDT 4242 TDT 4242 TDT 4242
Requirements elicitation Humans and machines – 1
Step 1: Step 2: Step 3: Step 4: Given the amount and complexity of RE, we
Capture Transfer Requirements and Refine the requirements model Create a
Requirements in functions into a semi-formal and derive detailed requirements preliminary need to automate as much as possible.
Natural Language requirements model design model
based on the
requirement
model (to be
Humans and machines have different strong
Req.012: The system
shall enable cabin Function 1 Function 2 Function 1 used and and weak points.
temperature regulation refined in
between 15°C and 30°C
…
Req 001
Req 002
Req 011
Req 028 Function 1a Req 001.01
SP3) We want to elicit and analyze requirements in a
Req 012 Req 050
…
Req.124: Cabin
… … Function 1b
Req 001.02
….
way that allows both parties to build on their
Req 124 …
temperature shall not
exceed 35° Function 1c strong sides.
Parallel Steps:
Apply dictionary with common vocabulary; validate and check Requirements consistency and completeness
- 19. Humans and machines - 2 Why BPs and GNL – 1
Machines are GNL and BPs will reduce variation and thus giving
good at observing quantitative data and being
the machines the opportunity to do what they are
deductive, fast and precise. In addition, they are best at: to be fast, precise and consistent.
good at doing consistent repetition of several
By combining humans and machines and let both do
actions.
what they are best at, we get a better result than we
bad at handling variations in written material and
would get if we left the job of handling requirements
pattern recognition. to just one of them.
Humans are good at handling variations in written
material, being inductive. In addition, they are good
at doing error correction.
Why BPs and GNL - 2 GNL and BPs
Template based textual = Syntax + Semantics + Meta Model
The final goal is to allow the machine to assist the Keywords: RMM
- Refinement
developers in analysing requirements for: Guided RSL Boilerplates Reflects requirement,
system and domain Analysis - Specialization
concepts -Correctness
Consistency -Completeness
Requirements expressed on templates -Consistency
Completeness Uses predefined templates based on concepts,
relations and axioms to guide requirements elicitation
-Safety analysis
Safety implications Example:
The <system function> shall provide <system capability> to achieve <goal>
Requirements expressed using a vocabulary guide
Uses predefined concepts, relations and axioms to
guide requirements elicitation
Example:
The ACC system shall be able to determine the speed of the ego-vehicle.
Ontology: General and SP specific
- Requirements classification
- System attributes
- Domain concepts
- 20. What is GNL - 1 What is GNL - 2
Aim:
Free text requirement elicitation with the
• Bridge the gap between unconstrained
assistance of prescribed words from a expression and quality checking when
dictionary. This will give us requirements representing requirements as free text. Quality
which use all terms in a uniform way, this measures:
reducing misunderstandings Correctness, consistency, completeness and un‐
No formal constraints ambiguity (reduced variability)
Requires minimal expertise. • Provide the basis for semantic processing and
checking of requirements.
• Dictionary – Simple taxonomy or more formal
ontology
Approach for GNL – 1 Approach for GNL – 2
Ontology = Thesaurus + Inference Rules Required Activity
• Thesaurus – Domain concepts: entities, Knowledge capture: Information embedded in
domain events from domain experts and ontologist
terms and events Implementation: Formal representation of captured
• Inference Rules – Relations, attributes and knowledge. Language: OWL, Support environment:
axioms Protégé.
Verification: Checking that represented ontology is
• Causality, similarity, reflexivity, correct using
transitiveness, symmetric, disjoint • Classifiers/reasoners
• Domain experts (semantic accuracy)
(contradiction) …
• Mapping of requirement segments to ontology concepts
- 21. Motivation for use of templates - 1 Motivation for use of templates - 1
Text has the advantage of unconstrained Template based textual requirements
expression. There is, however, a need for specification (boilerplates) will introduce some
common limitations when representing requirements
but will also reduce the opportunity to
• Understanding of concepts used to express
introduce ambiguities and inconsistencies.
the requirements and relations between them.
Boilerplates
• Format of presentation
• Provides an initial basis for requirements
Lack of common understanding makes
checking
requirement specifications expressed as free text
prone to ambiguous representations and • Are easy to understand for stakeholders
inconsistencies. compared to more formal representations
TDT 4242 TDT 4242
What is a boilerplate – 1 What is a boilerplate – 2
Boilerplates is a set of structures that can be used to The RE process is as follows:
write requirements. They use high-level concept
classification and attributes
1. Select a boilerplate or a sequence of
boilerplates. The selection is based on the
attributes that need to be included and how
they are organized – fixed terms.
2. If needed, identify and include mode boilerplates
3. Instantiate all attributes
A boilerplate consists of fixed terms and attributes.
It may, or may not, contain one or more modes.
TDT 4242
TDT 4242
- 22. Fixed Terms
Boilerplate examples - 1
Attributes
BP32
The <user> shall be able to <capability>
Attributes:
• <user> = driver
• <capability> = start the ACC system
Requirement
The driver shall be able to start the ACC system
TDT 4242
- 23. Boilerplate examples - 2 Boilerplate examples - 3
BP43 While <operational condition>
BP2
BP32 The <user> shall be able to <capability>
The <system> shall be able to <action> <entity>
BP43 is a mode
Attributes:
•<system> = ACC system Attributes
•<action> = determine • <operational condition> = activated
•<entity> = the speed of the ego-vehicle • <user> = driver
• <capability> = override engine power control of
Requirement the ACC system
The ACC system shall be able to determine the Requirement
speed of the ego-vehicle
While activated the driver shall be able to override
engine power control of the ACC-system
TDT 4242 TDT 4242
Functional requirements example Non functional requirement example – 1
Functional requirements from the SafeLoc system
The robot control system shall stop the robot within 10 Non‐functional requirements and soft goals fits into
milliseconds if a gate is opened to the zone where the robot is the same BPs as functional requirements
operating
The robot shall only be allowed to start when all gates are BP61
closed and the reset button is pushed The <system> shall be able to <action> to <entity>
The robot shall stop if it tries to move into a zone already
occupied by an operator Suitability:
The <system > shall be able to <provide an
appropriate set of functions> to <the user>
TDT 4242 TDT 4242
- 24. Non functional requirement example – 2 Non functional requirement example – 3
Non‐functional requirements and soft goals fits into
the same BPs as functional requirements BP43
While <operational condition>
BP2-1 The <system> shall be able to <capability> BP2
BP12 …for a sustained period of at least The <system> shall be able to <action> <entity>
<number> < unit>
While <normal operational condition> the
Maturity: <system> shall be able to <tolerate>
The <system > shall be able to <operate without <90% of software faults of category...>
failure> for a sustained period of at least <quantity>
<time unit>
TDT 4242
Summing up
The use of boiler plates and ontologies will
• Enforce a uniform use of terms
• Reduce the variability of presentations –
requirements that are similar will look similar
Reduced variation in form and contents simplifies
the use of automatic and semi-automatic tools
for
• Checking requirement quality – e.g
completeness and consistency
• Creating test cases
TDT 4242
- 25. What is requirements traceability
“Requirements traceability refers to the ability
to describe and follow the life of a
Institutt for datateknikk og requirement, in both a forwards and
informasjonsvitenskap backwards direction, i.e. from its origins,
Inah Omoronyia and Tor Stålhane through its development and specification, to
its subsequent deployment and use, and
Requirements Traceability through periods of on‐going refinement and
iteration in any of these phases.”
Gotel and Finkelstein
TDT 4242 TDT 4242 TDT 4242
Traceability Goals - 1 Traceability Goals – 2
• Project Management • Validation
– Status: “When will we finish?” and “what will it cost?” – finding and removing conflicts between requirements
– Quality: “How close are we to our requirements?” – completeness of requirements
• derived requirements cover higher level requirements
• QA manager • each requirement is covered by part of the product
– Improve Quality: “What can we do better?” • Verification
• Change Management – assure that all requirements are fulfilled
– Versioning, documentation of changes (Why? What? • System Inspection
When?)
– identify alternatives and compromises
– Change Impact analysis
• Certification/Audits
• Reuse
– proof of being compliant to standards
– Variants and product families
– Requirements can be targeted for reuse
TDT 4242 TDT 4242
- 26. Habitat of Traceability Links – 1 Habitat of Traceability Links – 2
Pre- vs. Post-Requirements Specification
TDT 4242 TDT 4242
Challenges of traceability – 1 Challenges of traceability – 2
– Traces have to be identified and recorded among
numerous, heterogeneous entity instances – A variety of tool support
(document, models, code, . . . ). It is challenging • based on traceability matrix, hyperlink, tags
to create meaningful relationships in such a and identifiers.
complex context. • still manual with little automation
– Traces are in a constant state of flux since they – Incomplete trace information is a reality due to
may change whenever requirements or other complex trace acquisition and maintenance.
development artefacts change.
– Trust is a big issue: lack of quality attribute
• There is no use of the information that 70% of
trace links are accurate without knowing which
of the links forms the 70%
TDT 4242 TDT 4242
- 27. Challenges of traceability – 3 Challenges of traceability – 4
Different stakeholders usage viewpoint (different questions Different stakeholders usage viewpoint (different questions
asked by different stakeholders): asked by different stakeholders):
• Validation:
• QA management: – Tracability can be used as a pointer to the quality of
– “how close are we to our requirements” and “what can we requirements:
do better” to improve quality. • Completeness, ambiguity, correctness/noise,
• Change management inconsistency, forward referencing, opacity
– Tracking down the effect of each change to each involved – Ensures that every requirement has been targeted by at
component that might require adaptations to the change, least one part of the product
recertification or just retesting to proof functionality. • Verification
• Reuse: – Checking that constraints are not violated (in most cases
– Pointing out those aspects of a reused component that this is an extension of validation context)
need to be adapted to the new system requirements.
• Certification/Audit
– Even the requirements themselves can be targeted for
reuse. • Testing, maintenance (reverse engineering)
TDT 4242 TDT 4242
Traceability meta-models – 1 Traceability meta-models – 2
• A model is an abstraction of phenomena in the real world; a
meta model is yet another abstraction, highlighting properties
of the model itself.
• Meta‐models for traceability are often used as the basis for
the traceability methodologies and frameworks:
– Define what type of artefacts should be traced.
– Define what type of relations could be established between these
artefacts.
Traceability Meta Model
Low-end traceability
TDT 4242
TDT 4242
- 28. Traceability meta-models – 3
European EMPRESS project: Meta model for
requirements traceability
High-end traceability
TDT 4242
Traceability meta-models – 4 Approaches to traceability
Creating trace links:
– Critical tasks in requirements traceability:
establish links between requirements and
between requirements and other artefacts.
– Manual linking and maintaining of such
links is time consuming and error prone
– Focus is on requirements traceability
through (semi‐)automatic link generation.
PRECISE Meta‐model (SINTEF)
- 29. Manual trace links – 1 Manual trace links – 2
This is the classical traceability methods and
the simplest form of traceability. In this approach,
we create a requirements traceability matrices
using a hypertext or table cross referencing
scheme, often using Excel
Two problems
• Long-term difficulty of maintaining a large
numbers of links.
• The static nature of the links (lack of
attributes) limit the scope of potential
automation.
Scenario driven traceability – 1 Scenario driven traceability – 2
• Test‐based approach to uncover relations amongst The method to achieve traceablity uses the idea of
requirements, design and code artifacts (Alexander “footprint”.
Egyed ) When we are dealing with traceability, a footprint
• Accomplished by observing the runtime behavior of contains two types of information:
test scenarios. • The set of classes that were executed when we were
• IBM Rational PureCoverage, open source tool testing a specific scenario.
(org.jmonde.debug.Trace) • The number of methods that were executed in each
• Translate this behavior into a graph structure to class.
indicate commonalities among entities
associated with the behavior
- 30. Footprints – 1 Footprints – 2
E.g. scenario A uses 10 methods in class CAboutDlg Only classes are registered – e.g scenario [s3] uses
and 3 methods in Csettings Dlg classes C, J, R and U
Footprints – 3 Footprints – 4
Some problems: Based on the footprint table, we can make a
requirements-to-class trace table
• There might be scenarios that do not cover any
requirement – e.g. [s3]
• There are scenarios that belong to several
requirements, e.g. [s9]
Such scenarios will get separate rows in the trace
matrix and will be marked with an F (Fixed) or a P
(Probable), depending on how sure we are that a
certain class belongs to this scenario.
- 31. Footprints – 5 Development footprints - 1
Each test scenario will leave a footprint. If we make A solution that enables the project to construct
one test scenario per requirement, then we get one traceability information during development has
footprint per requirement. been suggested by I. Omoronyia et al.
The method requires that each developer
We can make the footprints more fine grained and • Always identifies which requirement – e.g. use case
thus get more information by using methods or – he is currently working on
code chunks instead for classes. • Only works at one use case at a time
This will require more work but also more – better –
traceability information.
Development footprints - 2 Development footprints - 3
The result will be similar to the scenario testing We can extract more info from the development
footprint table. process in order to understand better what has
The resulting table will show which documents, been going on in the project. The next slides shows
classes etc. have been accessed during work on • Types of access: C – Create, U – Update and V –
this particular requirement – e.g. use case. View
Main problem: “false” accesses – e.g. a developer • Timeline – e.g. date or time
looks at some of the code of another requirement • Person – who did what and, more important, who
for info. will have expertise on what?
Each line in the table will show
- 32. Development footprints - 4 Scenario driven traceability – 3
Problems:
– Semi‐automated but requires a large amount of time
from system engineers to iteratively identify a subset
of test scenarios and how they related to
requirement artifacts.
– Requirements that are not related due to non
matching execution paths might be related in some
other form (e.g calling, data dependency,
implementation pattern similarity, etc).
Trace by tagging – 1 Trace by tagging – 2
This method is easy to understand and simple to
implement. The problem is that it depends on
heavy human intervention.
The principle is as follows:
• Each requirement is given a tag, either manually or
by the tool.
• Each document, code chunk, etc. are marked with
a tag which tells which requirement it belongs to
- 33. Trace by tagging – 3 Trace by tagging – 4
There are several ways to create tags, e.g.: The quality of traceability through tagging will depend
• Single level tags – e.g. R4. This gives a standard on that we remember to tag all relevant documents.
trace matrix It is possible to check automatically that all
• Multilevel tags – e.g. R4, R4.1 and R4.2 where R4 documents in the project database is tagged.
is the top level requirement and R4.1 and R4.2 are It is, however, not possible to check that this tagging
sub-requirements. This gives us more detailed is correct.
trace information
Conclusion
• Requirements traceability is an important aspect of
requirements management
• Stakeholders have different traceability information needs
• Traceability can be complex for not trivial projects
• Traceability meta-models provide insight on the type of
traceability information required for a project
• There exist several automated approaches for
requirements traceability. The strength is in a synergy of
different automated approaches
- 34. Tor Stålhane
Requirements Specification and Testing
Requirements testability
- 35. Testability definition
According to ISO 9126, testability is defined
as:
Testability: The capability of the software
product to enable modified software to be
validated.
NOTE - Values of this sub-characteristic
may be altered by the modifications under
consideration.
- 36. Testability concerns
Testability touches upon two areas of concern:
• How easy is it to test the implementation?
• How test‐friendly is the requirement?
These two concerns are not independent and
need to be considered together. We will first
look at it from the requirements side.
- 37. Testability
Three basic ways to check that we have achieved our
goals:
• Executing a test. Give input, observe and check
output. A test can be a
– Black box test
– White box test
– Grey box test
• Run experiments
• Inspect the code and other artifacts
Usually, we will include all of these activities in the
term testing
- 38. When to use what
The diagram on the next slide is a high level
overview of when to use
• T – Tests. Input / output. Involves the
computer system and peripherals.
• E – Experiments. Input / output but
involves also the users.
• I – Inspections. Evaluation based on
documents.
- 39. Concrete requirements from high level goals
T E
E T
I I
T E
E E
TDT 4242
- 40. Testability
In order to be testable, a requirement needs
to be stated in a precise way. For some
requirements this is in place right from the
start:
When the ACC system is turned on, the
“Active” light on the dashboard shall be
turned on.
In other cases we need to change a
requirement to get a testable version.
The system shall be easy to use.
- 41. Testability challenges - 1
Some requirements are more difficult to test
than others. Problems might rise due to:
• Volume of tests needed, e.g. response
time or storage capacity.
• Type of event to be tested, e.g. error
handling or safety mechanisms.
• The required state of the system before
testing, e.g. a rare failure state or a certain
transaction history.
- 42. Testability challenges – 2
We can test the requirements at any level. The
formulation of the test will depend on the level
- 43. Making a requirement testable – 1
One way to make requirements testable is
the “Design by Objective” method
introduced by Tom Gilb.
The method is simple in principle but is in
some cases difficult to use. There are two
problems
• The resulting tests can in some be rather
extensive and thus quite costly.
• The method requires access to the
system’s end users.
- 44. Making a requirement testable – 2
1. What do you mean by <requirement>?
This will give us either (a) a testable
requirement or (b) a set of testable and
non-testable sub-requirements.
2. In case (a) we are finished. In case (b) we
will repeat question 1 for each non-
testable sub-requirement
- 45. Making a requirement testable – 3
Requirement: Reverse thrust may only be
used, when the airplane is landed.
The important questions are
• “How do you define landed?”
• Who should you ask – e.g. pilots, airplane
construction engineers, or airplane
designers?
- 46. Requirements for testability – 1
First and foremost:
The customer needs to know what he wants
and why he wants it. In some cases it is
easier to test if the user actually has
achieved his goal than to test that the
system implements the requirement.
Unfortunately, the “why”-part is usually not
stated as part of a requirement.
- 47. Requirements for testability – 2
Each requirement needs to be
• Correct, i.e. without errors
• Complete, i.e. has all possible situations
been covered?
• Consistent, i.e. not in disagreement with
other requirements.
• Clear, i.e. stated in a way that is easy to
read and understand – e.g. using a
commonly known notation.
- 48. Requirements for testability – 3
Each requirement needs to be
• Relevant, i.e. pertinent to the system’s
purpose and at the right level of
restrictiveness.
• Feasible, i.e. possible to realize. If it is
difficult to implement, is might also be
difficult to test.
• Traceable, i.e. it must be possible to relate
it to one or more
– Software components
– Process steps
- 49. Completeness
All possible situations must be covered.
“If X then….”, “If Y then….” Must also
consider what will happen “If neither X nor
Y…”
Automatic door opener – what is missing?
If the door is closed and a person is
detected then send signal Open_Door. If
no person is detected after 10 sec., send
signal Close_Door.
- 50. Consistency
Consistency is a challenge since we, at least
in the general case, need a complete
overview of all requirements.
In most cases, we can make do with
checking all requirements that are related
to the same event, function or parameter.
- 51. Clear – 1
This is mainly a question of representation
such as choice of
• Diagram notation
• Description language
• Level of details
Who shall understand the requirement?
• Customers
• Developers, including hired-in consultants
• Testers
- 52. Clear – 2
Simple example:
Print the “accounts ledger” for all accounts
This requirement is perfectly clear for
developers who are working in the banking
business.
Other developers might experience some
problems.
- 53. Relevant
Two questions are important:
• Do we really need this requirement?
• Is it at the right level of strictness – i.e. not
too strict and not too lax.
Only the second question is important for
the tester.
• Too strict means more work for the
developers
• Too lax means more work for the tester.
- 54. Feasible
This question is really related to the contract but
we should also consider it here – can we really
do this?
Testers can contribute to the feasibility question by
asking how it should be tested. This will help /
force everybody involved to make the
requirement more clear and thus improve on the
requirement.
Requirements that are difficult to tests are also
usually difficult to implement – mainly because
they are badly defined.
- 55. Some sound advice
The following set of advices on requirements and
testability are quoted from Ludwig Consulting
Services, LLC.
They are not a definition and not “the final words”
on requirements testability. Instead, they should
be used as a checklist.
That one of the following rules are not obeyed does
not mean that the requirement is wrong. It
should, however, be reviewed for potential
problems.
- 56. Modifying Phrases
Words and phrases that include:
• as appropriate
• if practical
• as required
• to the extent necessary / practical.
Their meaning
• is subject to interpretation
• make the requirement optional
Phrases like "at a minimum" only ensure the minimum,
while "shall be considered" only requires the
contractor to think about it.
- 57. Vague Words
Vague words inject confusion. Examples of frequently
used vague verbs are:
• manage
• track
• handle
• flag
Information systems receive, store, calculate, report, and
transmit data. Use words that express what the system
must do.
Requirement: The system shall process ABC data to the
extent necessary to store it in an appropriate form for
future access.
Correction: The system shall edit ABC data.
- 58. Pronouns With No Reference
Example: It shall be displayed.
When this occurs, the writer is usually relying on
a nearby requirement in the requirements
document for the meaning of "it."
As requirements are assigned for
implementation, they are often reordered and
regrouped, and the defining requirement is no
longer nearby.
- 59. Passive Voice
Requirements should be written in active voice,
which clearly shows X does or provides Y.
Passive voice: Z shall be calculated.
Active voice: the system shall calculate Z.
- 60. Negative Requirements
Everything outside the system is what the system
does not do.
Testing would have to continue forever to prove
that the system does not do something.
State what the system does. Substitute an active
verb that expresses what the system must do.
• Change "the system shall not allow X," to "the
system shall prevent Y."
• Use the prefix "un," such as: The system shall
reject unauthorized users.
- 61. Assumptions and Comparisons – 1
The requirement "the system shall increase
throughput by 15%" sounds testable, but isn't.
The assumption is "over current system
throughput." By comparing to another
system, the meaning of the requirement
changes when the other system changes
- 62. Assumptions and Comparisons – 2
An example, sometimes found in requests for
proposals, is:
"The system shall address the future needs of
users."
The writer is probably thinking ahead to after the
contract is awarded. The requirement is
meaningless because whenever it is read, it will
point to the future.
A requirement on change management included in
the project management processes, would make
more sense than making it a requirement for the
system.
- 63. Indefinite Pronouns
Indefinite pronouns are “stand in” for unnamed people or
things, which makes their meaning subject to interpretation.
Some of these may find their way into requirements:
• All • Everybody
• Another • Everyone
• Any • Everything
• Anybody • Few
• Anything • Many
• Each • Most
• Either • Much
• Every
- 64. The implementation
We will shortly look at four concerns related to
implementation and testability:
• Autonomy of the system under test
• Observability of the testing progress
• Re‐test efficiency
• Test restartability
- 65. Autonomy – 1
How many other systems are needed to test this
requirement?
It is best if it can be tested using only the SUT
and the autonomy and testability is
successively reduced as the number of other,
necessary systems increase.
If the other systems needed are difficult to
include at the present we will have to write
more or less complex stubs.
- 66. Example – 1
Requirement: “If the door is closed and a
person is detected then send signal
Open_Door”
• Sensors and actuators can be tested in
the lab.
• The system with a simulated actuator:
simulate a “person detected” signal on the
sensor and check if a Open_Door signal is
sent to the actuator.
- 67. Example – 2
We can build a complete system – door, sensor,
door motor and software system and test by
• Letting persons approach the sensor
• Check if the door opens
– Early enough
– Fast enough
- 68. Observability
How easy is it to observe the
• Progress of the test execution?
This is important for tests that do not produce
output – e.g. the requirement is only
concerned with an internal state change or
update of a database.
• Results of the test?
Important for tests where the output is
dependent on an internal state or database
content.
- 69. Re‐test effiencey
Retest efficiency is concerned with how easy it is
to perform the “test – check – change – re‐
test” cycle.
This includes
• Observability – observe the test result
• Traceability – identify all tests related to the
change
- 70. Test restartability
This is mostly a question of check points in the
code. How easy is it to
• Stop the test temporarily
• Study current state and output
• Start the test again from
– The point where it was stopped
– Start
- 71. Final comments
That a requirement is testable does not
necessarily mean that it is easy to test.
In order to have testable requirements it is
important that
• The testers are involved right from the
start of the project. It is difficult to add
testability later.
• The tests are an integrated part of the
requirement
- 72. Introduction to exercise 1
Tor Stålhane
- 73. The goals ‐ 1
Consider the following goals for an adaptive
cruse controller – ACC:
When the ACC is active, vehicle speed shall be
controlled automatically to maintain one of
the following:
• time gap to the forward vehicle
• the set speed
- 74. The goals ‐ 2
In case of several preceding vehicles, the
preceding vehicle will be selected
automatically
The ACC shall support a range of types
• Type 1a: manual clutch operation
• Type 1b: no manual clutch operation, no
active brake control
• Type 2a: manual clutch operation, active
brake control
• Type 2b: active brake control
- 75. Exercise requirements
• Identify the combinations of sub‐goals that
will jointly contribute to each of the goals
• Build a refinement graph showing the low‐
level goals, requirements , assumptions,
domain properties and associated agents
- 76. ACC state model
- 77. Test vs. inspection
Part 1
Tor Stålhane
- 78. What we will cover
• Part 1
– Introduction
– Inspection processes
– Testing processes
• Part 2
– Tests and inspections – some data
– Inspection as a social process – two experiments
and some conclusions
- 79. Introduction
- 80. Adam’s data ‐ 1
Mean Time to Problem Occurrence – years
Product 1.6 5 16 50 160 500 1600 5000
1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.2
2 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.3
3 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.7
4 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.2
5 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.2
6 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.0
7 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.0
8 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.9
9 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2
- 81. Adams’ data – 2
The main information that you get from the
table on the previous slide is that
• Some defects are important because they will
happen quite often.
• Most defects are not important since they will
happen seldom.
How can we tell the difference?
- 82. Testing and inspection – the V model
- 83. Testing and inspection – 1
The important message here is that testing
cannot always be done.
In the first, important phases, we have nothing
to execute and will thus always have to do
some type of inspection.
This might be considered one of the weaknesses
of traditional software engineering over Agile
development.
- 84. Testing and inspection – 2
In order to understand the main differences
between testing and inspection, we should
consider Fit’s list.
Based on this, we will give a short discussion of
the relative merits of testing and inspection.
- 85. Area of Man Machine
competence
Understanding Good at handling variations in Bad at handling variations in
written material written material
Observe General observations, Specialized, good at observing
multifunctional quantitative data, bad at
pattern recognition
Reasoning Inductive, slow, imprecise but Deductive, fast, precise but
good at error correction bad error correction
Memory Innovative, several access Copying, formal access
mechanisms
Information Single channel, less than 10 Multi channel, several
handling bits per second Megabits per second
Consistency Unreliable, get tired, depends Consistent repetition of several
on learning actions
Power Low level, maximum ca. 150 High level over long periods
watt of time
Speed Slow – seconds Fast
- 86. Man vs. machine – 1
Good when we need the ability to
• Handle variation
• Be innovative and inductive
• Recognize and handle patterns
Not so good when we need the ability to
• Do the same things over and over again in a
consistent manner
• Handle large amount of data
- 87. Man vs. machine – 2
In order to do the best job possible we need
processes where we let each part
• Do what they are best at:
– Man is innovative
– Machine handles large amounts of data
• Support the other with their specialties.
– Machine supports man by making large amounts
of information available
– Man support machine by providing it with
innovative input
- 88. General considerations ‐ documents
Architecture, system, sub‐system and
component design plus pseudo code. Here we
can only use inspections.
Man will use experience and knowledge to
identify possible problems
Machine can support by identifying information
– e.g. find all occurrences of a string.
- 89. General considerations – code (1)
For executable code, we can use inspection,
testing or a combination of both.
The size and complexity – degree of dynamism –
of the code will, to a large degree, decide our
choice.
Other important factors are the degree of
experience with
• The programming language
• The algorithms used
- 90. General considerations – code (2)
Simple code
• Start with inspection – all code
• Design and run tests
Complex code
• Start with inspection – focus on algorithm and
logic
• Decide test completeness criteria – we cannot
test everything
• Design and run tests
- 91. Inspection processes
- 92. Inspections – 1
The term “inspection” is often used in a rather
imprecise manner. We will look at three types
of inspection:
• Walkthrough
• Informal inspection – also called informal
review
• Formal inspection – also called formal review
or just inspection
The first two types are usually project internal
while the last one is used as a final acceptance
activity for a document.
- 93. Inspections – 2
For all types of inspections:
• The quality of the results depends on the
experience and knowledge of the participants.
“Garbage in – Garbage out”
• It might be a good idea to involve customer
representatives.
- 94. The walkthrough process
Walkthrough is a simple process – mostly used
for early decisions for an activity. The
document owner:
1. Makes a rough sketch of the solution –
architecture, algorithm etc.
2. Presents – explain – the sketch to whoever
shows up.
3. Registers feedback – improvements.
- 95. Walkthrough – pros and cons
Pros:
• Easy and inexpensive. Needs no extra
preparation.
• Collect ideas at an early stage of
development.
Cons:
• No commitment from the participants
• May collect many loose or irrelevant ideas
- 96. The informal inspection process
Planning
Rules,
checklists,
procedures
Product Change
Individual Logging
document requests
checking meeting
- 97. Informal inspections – pros and cons
Pros:
• Is simple and inexpensive to perform.
• Can be used at all stages of development
• Usually has a good cost / benefit ratio
• Needs a minimum of planning
Cons:
• No participant commitment
• No process improvement
- 98. The formal inspection process
The formal inspection process described below
is – with small variations – the most
commonly used. The version shown on the
following slides stem from T. Gilb and D.
Graham.
We recommend this process as the final
acceptance process for all important
documents
- 99. Formal inspection process overview
Change
requests
Product
document
Planning
Process
improvements
Rules,
checklists,
procedures
Edit
Individual and
Walk‐ Kick‐off Logging
follow‐
through checking meeting
up
- 100. Distribution of resources
Typical
Activity Range %
value %
Planning 3–5 4
Kick-off 4–7 6
Individual checking 20 – 30 25
Logging 20 – 30 25
Editing 15 – 30 20
Process brainstorming 15 – 30 16
Leader overhead, follow up, entry, 3–5 4
exit
- 101. Initiating the inspection process
• The inspection process starts with a “request
for inspection” from the author to the QA
responsible.
• The QA responsible appoints an inspection
leader.
• First step is always to check that the
document is fit for inspection.
- 102. Planning
Important planning points are:
• Who should participate in the inspections
– Who is interested?
– Who have time available for preparation and
meetings?
– Who has the necessary knowledge concerning
application, language, tools, methods?
- 103. Kick‐off
Important activities here are:
• Distribution of necessary documents:
– Documents that shall be inspected
– Requirements
– Applicable standards and checklists
• Assignment of roles and jobs
• Setting targets for resources, deadlines etc.
- 104. Individual checking
This is the main activity of the inspection. Each
participant read the document to look for
• Potential errors ‐ inconsistencies with
requirements or common application
experience
• Lack of adherence to company standards or
good workmanship
- 105. Logging meeting
The logging meeting has three purposes:
• Log issues already discovered by inspection
participants
• Discover new issues based on discussions
and new information that arises during the
logging meeting.
• Identify possible improvement to the
inspection or development process.
- 106. Improve the product ‐ 1
The author receives the log from the inspection
meeting. All items ‐ issues ‐ in the log are
categorised as one of the following:
• Errors in the author’s document.
• Errors in someone else’s document.
• Misunderstandings in the inspection team.
- 107. Improve the product ‐ 2
• Errors in own document:
Make appropriate corrections
• Errors in someone else’s documents:
Inform the owner of this document.
• Misunderstandings in the inspection team:
Improve document to avoid further
misunderstandings.
- 108. Checking the changes
This is the responsibility of the inspection
leader. He must assure that all issues raised
in the log are disposed of in a satisfactory
manner:
• The documents that have been inspected
• Related documents ‐ including standards
and checklists
• Suggested process improvements
- 109. Formal inspection – pros and cons
Pros:
• Can be used to formally accept documents
• Includes process improvement
Cons:
• Is time consuming and expensive
• Needs extensive planning in order to succeed
- 110. Testing processes
- 111. Testing
We will look at three types of testing:
• Unit testing – does the code behave as
intended. Usually done by the developer
• Function verification testing – also called
systems test. Does the component or system
provide the required functionality?
• System verification testing – also called
acceptance test. Does the hardware and
software work together to give the user the
intended functionality?
- 112. The unit testing process
Unit testing is done by the developer one or
more times during development. It is a rather
informal process which mostly run as follows:
1.Implement (part of) a component.
2.Define one or more tests to activate the code
3.Check the results against expectations and
current understanding of the component
- 113. Unit testing – pros and cons
Pros:
• Simple way to check that the code works.
• Can be used together with coding in an
iterative manner.
Cons:
• Will only test the developer’s understanding
of the spec.
• May need stubs or drivers in order to test
- 114. The system test process
A systems test has the following steps:
1. Based on the requirements, identify
– Test for each requirement, including error handling
– Initial state, expected result and final state
2. Identify dependencies between tests
3. Identify acceptance criteria for test suite
4. Run tests and check results against
– Acceptance criteria for each test
– Acceptance criteria for the test suite
- 115. Systems test – pros and cons
Pros:
• Tests system’s behavior against customer
requirements.
Cons:
• It is a black box test. If we find an error, the
systems test must be followed by extensive
debugging
- 116. The acceptance test process
The acceptance test usually has three actvities –
both involving the customer or his
representatives:
• Rerun the systems test at the customer’s site.
• Use the system to solve a set of real‐world
tasks.
• Try to break the system – by stressing it or by
feeding it large amounts of illegal input
- 117. Acceptance test – pros and cons
Pros:
• Creates confidence that the system will be
useful for the customer
• Shows the system’s ability to operate in the
customer’s environment
Cons:
• Might force the system to handle input that it
was not designed for, thus creating an
unfavorable impression.
- 118. Test vs. inspection
Part 2
Tor Stålhane
- 119. Testing and inspection
A short data analysis
- 120. Test and inspections – some terms
First we need to understand two important
terms – defect types and triggers.
After this we will look at inspection data and
test data from three activity types, organized
according to type of defect and trigger.
We need the defect categories to compare test
and inspections – where is what best?
- 121. Defect categories
This presentation uses eight defect categories:
• Wrong or missing assignment
• Wrong or missing data validation
• Error in algorithm – no design change is necessary
• Wrong timing or sequencing
• Interface problems
• Functional error – design change is needed
• Build, package or merge problem
• Documentation problem
- 122. Triggers
We will use different triggers for test and
inspections. In addition – white box and black
box tests will use different triggers.
We will get back to triggers and black box /
white box testing later in the course.
- 123. Inspection triggers
• Design conformance
• Understanding details
– Operation and semantics
– Side effects
– Concurrency
• Backward compatibility – earlier versions of this
system
• Lateral compatibility – other, similar systems
• Rare situations
• Document consistency and completeness
• Language dependencies
- 124. Test triggers – black box
• Test coverage
• Sequencing – two code chunks in sequence
• Interaction – two code chunks in parallel
• Data variation – variations over a simple test
case
• Side effects – unanticipated effects of a simple
test case
- 125. Test triggers – white box
• Simple path coverage
• Combinational path coverage – same path
covered several times but with different
inputs
• Side effect ‐ unanticipated effects of a simple
path coverage
- 126. Testing and inspection – the V model
- 127. Inspection data
We will look at inspection data from three
development activities:
• High level design: architectural design
• Low level design: design of subsystems,
components – modules – and data models
• Implementation: realization, writing code
This is the left hand side of the V‐model
- 128. Test data
We will look at test data from three
development activities:
• Unit testing: testing a small unit like a method
or a class
• Function verification testing: functional
testing of a component, a system or a
subsystem
• System verification testing: testing the total
system, including hardware and users.
This is the right hand side of the V‐model
- 129. What did we find
The next tables will, for each of the assigned
development activities, show the following
information:
• Development activity
• The three most efficient triggers
First for inspection and then for testing
- 130. Inspection – defect types
Activity Defect type Percentage
Documentation 45.10
High level design Function 24.71
Interface 14.12
Algorithm 20.72
Low level design Function 21.17
Documentation 20.27
Algorithm 21.62
Code inspection Documentation 17.42
Function 15.92
- 131. Inspection – triggers
Activity Trigger Percentage
Understand details 34.51
High level design Document consistency 20.78
Backward compatible 19.61
Side effects 29.73
Low level design Operation semantics 28.38
Backward compatible 12.16
Operation semantics 55.86
Code inspection Document consistency 12.01
Design conformance 11.41
- 132. Testing – triggers and defects
Activity Trigger Percentage
Test sequencing 41.90
Implementation
Test coverage 33.20
testing
Side effects 11.07
Activity Defect type Percentage
Interface 39.13
Implementation Assignments 17.79
testing Build / Package /
14.62
Merge
- 133. Some observations – 1
• Pareto’s rule will apply in most cases – both
for defect types and triggers
• Defects related to documentation and
functions taken together are the most
commonly found defect types in inspection
– HLD: 69.81%
– LLD: 41.44%
– Code: 33.34%
- 134. Some observations – 2
• The only defect type that is among the top
three both for testing and inspection is
“Interface”
– Inspection ‐ HLD: 14.12%
– Testing: 39.13%
• The only trigger that is among the top three
both for testing and inspection is “Side effects”
– Inspection – LLD: 29.73
– Testing: 11.07
- 135. Summary
Testing and inspection are different activities.
By and large, they
• Need different triggers
• Use different mind sets
• Find different types of defects
Thus, we need both activities in order to get a
high quality product
- 136. Inspection as a social process
- 137. Inspection as a social process
Inspections is a people‐intensive process. Thus,
we cannot consider only technical details – we
also need to consider how people
• Interact
• Cooperate
- 138. Data sources
We will base our discuss on data from two
experiments:
• UNSW – three experiments with 200 students.
Focus was on process gain versus process loss.
• NTNU – two experiments
– NTNU 1 with 20 students. Group size and the use
of checklists.
– NTNU 2 with 40 students. Detection probabilities
for different defect types.
- 139. The UNSW data
The programs inspected were
• 150 lines long with 19 seeded defects
• 350 lines long with seeded 38 defects
1. Each student inspected the code individually and
turned in an inspection report.
2. The students were randomly assigned to one out
of 40 groups – three persons per group.
3. Each group inspected the code together and
turned in a group inspection report.
- 140. Gain and loss ‐ 1
In order to discuss process gain and process
loss, we need two terms:
• Nominal group (NG) – a group of persons that
will later participate in a real group but are
currently working alone.
• Real group (RG) – a group of people in direct
communication, working together.
- 141. Gain and loss ‐2
The next diagram show the distribution of the
difference NG – RG. Note that the
• Process loss can be as large as 7 defects
• Process gain can be as large as 5 defects
Thus, there are large opportunities and large
dangers.
- 142. Gain and loss ‐ 3
12
10
8
Exp 1
6 Exp 2
Exp 3
4
2
0
7 6 5 4 3 2 1 0 -1 -2 -3 -4 -5 -6