Glossary
Glossary for Software Reliability
This glossary is intended for use by anyone involved with or interested in software reliability, risk assessment and defect prediction. It contains many common words and phrases frequently used in the context of this specialized discipline of software engineering.
Reliametrics makes no claim to the ownership of these definitions. In fact, we have borrowed freely and extensively from many sources in order to compile and consolidate these definitions. You are encouraged to use the glossary in any way that helps you in the course of your work.
We also request you to help us in our efforts to continually improve the value of this information. Please email to us any recommended additions, corrections or modifications to definitions, as well as information regarding other glossaries you feel are especially relevant to the subject.
Availability (for software) The
probability that a system will be operative at any given instant
during a specified time period. The measure of the availability
combines reliability and time to restore service. The relevant
time domain must therefore be real time, and reliability must
be transformed to that domain if expressed relative to some other
such as execution time. Note that many software failures are
intermittent and recovery may be automatic, without significant
interruption of service. Therefore, time to recover is not the
same as time to repair, since repair may be done off-line.
Availability is the capability of the software
product to be in a state to perform a required function at a
given point in time, under stated conditions of use. Externally,
availability can be assessed by the proportion of total time
during which the software product is in an up state. Availability
is therefore a combination of maturity (which governs the frequency
of failure), fault tolerance and recoverability (which governs
the length of down time following each failure).
Bugs Problems that find their way into
programs and systems. The term "bugs" is generally
synonymous with other terms such as errors, defects, faults,
and failures. It originated in the 1950's when the late Admiral
Grace Hopper actually found an insect jamming the contacts of
an electromechanical switch. A bug can be any segment of software
or hardware that produces an undesired result.
Change Control An umbrella process that
enables a project team to accept, evaluate, and act on changes
in a systematic manner.
Cleanroom Software Engineering An engineering
and managerial process for the development of high-quality software
with certified reliability. Cleanroom was originally developed
by Dr. Harlan Mills. The name "Cleanroom" was taken
from the electronics industry, where a physical clean room exists
to prevent introduction of defects during hardware fabrication.
Cleanroom software engineering reflects the same emphasis on
defect prevention rather than defect removal, as well as certification
of reliability for the intended environment of use.
Common Cause Failure A failure which is
the result of an event(s) which because of dependencies, causes
a coincidence of failure states of components in two or more
separate channels of a redundancy system, leading to the defined
system failing to perform its intended function. (IEC 1508, 3.6)
Common Mode Failure A failure of apparently
independent components or communication links due to an initiating
event that affects them all.
Complexity A measure of how complicated
a section or unit (typically of code or design) is. It represents
how complex it is to understand (although this also involves
cognitive features of the person doing the understanding) and/or
how complex it is to execute the code (for instance, the computational
complexity). The complexity evaluation can be performed by considering
the computational complexity of the functional part of the system
-- i.e., the dominant instructions in the most iterative parts
of the system. The complexity may be also a measure of the amount
of memory used or the time spent in execution an algorithm.
Component Physical part of a subsystem
that can be used to compose larger systems. A reusable encapsulation
that is typically larger scale than an individual class. Components
can be bought and sold and are therefore real, not conceptual.
Any self-contained item or equipment that is part of a larger
item and contributes to the function performed by the larger
item.
Component Failure The failure of a component
of a larger item. This leads to a component fault and an internal
error, which may propagate and lead to a system failure unless
error recovery is performed.
Component Fault A fault in a component
of a larger item. This may lead to an internal error that may
propagate and lead to system failure unless error recovery is
performed.
Configuration Management A discipline
applying technical and administrative controls throughout the
life-cycle to: (i) identify and document physical and functional
characteristics of configuration items; (ii) limit changes to
characteristics of those configuration items; (iii) record and
report change processing and implementation of the system.
Cost of Measurement The cost (e.g., the
person month spent or the total cost) of the project part related
to system assessment performed by measuring software characteristics
with metrics.
Cost per Defect The average cost (i.e.,
effort in person/month) spent for correcting a defect in the
system. It is obtained by dividing the total cost for correcting
the defect by the number of defects corrected.
Cost of Failure A measure of the severity
of the consequences of failure. Depending on the type of system,
different scales may be used, such as duration of down time,
consequential cash loss, number of lives lost, etc. Cost of failure
to user must be distinguished from cost of maintenance to vendor.
Cost/Benefits Analysis The analysis of
benefits and costs related to the implementation of a project
or product.
Coverage A measure of achieving a complete
assessment. In the situation of test coverage, an assessment
of 100% coverage is achieved when every one of the types of tests
specified in the software test plan has been observed, e.g. 100%
decision coverage occurs when every decision has been executed
within the software.
Coverage Analysis An analysis of the degree(s)
of coverage assessed and the sum of the coverage's used, i.e.
analysis of whether 50% decision coverage and 10% path coverage
is rigorous enough for the integrity level.
Critical Failure A failure that is assessed
as likely to result in injury to persons, significant material
damage or other unacceptable consequences. Precisely which failure
modes are critical will obviously depend on the type of system.
Criticality Classification of the consequences,
or likely consequences, of a failure mode, or classification
of the importance of a component for the required service of
an item. Synonymous with severity.
Cumulative Defect Removal Efficiency Sum
of all the defects found by all available methods (review, testing,
etc) before delivery to the customer.
Cyclomatic Complexity A measure of the
complexity of a module's decision structure. Cyclomatic complexity
is a widely used member of a class of static software metrics.
Cyclomatic complexity may be considered a broad measure of soundness
and confidence for a program. Introduced by Thomas McCabe in
1976, it measures the number of linearly-independent paths through
a program module. This measure provides a single ordinal number
that can be compared to the complexity of other programs. Cyclomatic
complexity is often referred to simply as program complexity,
or as McCabe's complexity.
Dead Code Dead code is non-functional
code (1) unintentionally included in the baseline, (2) left in
the system from an original software configuration, not erased
or overwritten, and left functional after a new program is loaded
in, over, or around it, or (3) deactivated code not tested for
the current configuration and environment.
Debug With software, to detect, locate,
and correct logical or syntactical errors in a computer program.
To attempt to determine the cause of the symptoms of malfunctions
detected by testing or by user complaints. The process of locating
and eliminating errors that have been shown, directly or by inference,
to exist in software.
Debugger A program designed to help in
debugging another program by allowing the programmer to step
through the program, examine data, and check conditions. There
are two basic types of debuggers: machine-level and source-level.
Machine-level debuggers display the actual machine instructions
(disassembled into Assembly language) and allow the programmer
to look at registers and memory locations. Source-level debuggers
let the programmer look at the original source code (C for example),
examine variables and data structures by name, and so on.
Defect Non-fulfillment of an intended
usage requirement or reasonable expectation, including one concerned
with safety. A non-conformance between the input products and
the output products of a system development phase. The main purpose
of verification activities (e.g., inspection) is to detect defects
so that they can be corrected before subsequent development phases.
If not detected and corrected during development, defects may
give rise to one or more faults in the delivered system, and
hence to failure in operation. Some defects (e.g., inappropriate
comments in source code) cannot give rise to faults, but may
adversely affect maintainability or other quality characteristics.
A failure observed by a customer, client, or user.
Defect Potential The potential number
of defects that can be encountered during the entire life cycle.
Defect Prevention The set of technologies
that minimize the risk of human error by software staffs.
Defect Removal The action performed to
remove defects in software. This process is a part of the software
maintenance process.
Defect Severity The relative impact of
a defect on the software use.
Defects Count The number of defects identified
in a system or sub-system in a given period.
Dependability Collective term used to
describe the availability performance and its influencing factors:
reliability performance, maintainability performance and maintenance
support performance. Dependability is described in terms of several
external attributes of a system: safety, security, usability,
maintainability, recoverability, availability, and extendibility.
Dependability is used only for general descriptions in non-quantitative
terms. No single measure can be defined for dependability. The
importance of the different attributes, and hence the target
values for the corresponding measures, depend on the type of
system. Synonyms with reliability.
Design Fault A fault caused by inadequate
design of an item. A design fault is caused by human error during
system development. Design faults in software are usually latent,
transient, recurrent, and systematic. Design faults give rise
to failure during operation when activated by a certain combination
of conditions referred to as the trigger. Since these conditions
are encountered at random during operation, design failures can
appear to be random events.
Design Review The process in which the
design is presented to project personnel: managers, developers,
users, etc. for discussion and approval.
Development Process The process by means
of which the software is developed from its requirement analysis
to the final product. It typically includes requirement definition,
analysis, design, coding, assessing, risk evaluating, testing
software, etc. The process of development is structured on the
basis of a specific life-cycle (spiral, waterfall, rapid prototyping)
that states the evolution of the phases of the development process
and the rules that have to be followed for the development process.
Dynamic Analysis Defect removal method
where defects are found during execution of the software. Opposed
with static analysis.
Emulator A combination of computer program
and hardware that mimic the instruction and execution of another
computer or system. (MCCR).
Error In software, an error is usually
a programmer action or omission that results in a fault.
Mistake in engineering, requirement specification, or design.
Mistake in design, implementation or operation which could cause
a failure. The cause of an error is a fault.
Mistake in engineering, requirement specification, or design. Mistake in design, implementation or operation which could cause a failure. The cause of an error is a fault.
An incorrect internal state of a system,
due to a component failure or an external failure. Unless redundancy
or fault tolerant design enables a successful error recovery
to be performed, a system failure may result. An error can be
caused by a faulty item, e.g., a computing error made by faulty
computer equipment.
Error-Prone Module A module in a software
system containing a lot of errors. At first glance, one might
think that errors are normally distributed of a software system.
Empirical research has pointed out that on the average 50% of
the errors are located in about 3% of the modules.
Execution Time A measure of the execution
time spent by a software item. The measure chosen will depend
on the type of system. Generally, it will not be equivalent to
real time. Possible measures are processor time consumed, number
of instructions executed, etc.
Failure The termination of the ability
of a system element to perform a required function. (Note: hardware
can fail but software can only have faults.) (ISO 2382, IEEE
729). The lack of correct performance. The cause of a failure
is an error. A failure that is observed by a customer, client,
or user is a defect.
The inability of a computer system to perform
its functional requirements, or the departure of software from
its intended behavior as specified in the requirements. Failure
can also be considered to be the event when either of these occurs,
as distinguished from "fault" which is a state. A failure
is an event in time. A failure may be due to a physical failure
of a hardware component, to activation of a latent design fault,
or to an external failure. Following a failure, an item may recover
and resume its required service after a break, partially recover
and continue to provide some of its required functions (fail
degraded) or it may remain down (complete failure) until repaired.
Failure Intensity Also known as Failure
Rate. The rate of failure of a system. The instantaneous failure
intensity is the approximate failure rate at a given point in
time (failure intensity is assumed to be a continuous function)
while the overall failure intensity typically reflects the number
of failures over some elapsed interval divided by the length
of the interval. If the failure intensity varies (non-constant),
then the failures are said to come from a non-homogenous process
(often assumed to be Poisson).
Failure Rate The number of failures of
a specified category in a given period in a given number of computer
runs, or in some other given rate of measure. The hardware and
software failure rate of a system or component varies during
its lifetime. For hardware systems, the usual scenario is that
the rate decreases at first as problems are detected and repaired
and finally increases at the end-of-life due to deterioration.
Between these two periods the rate usually remains steady. For
software systems, the rate may initially go up or down depending
on exposure rates, patching/repair rates and fault avoidance
but then decreasing steadily over time as patching/repair and
fault avoidance continue past the exposure saturation point.
Failure Severity The seriousness of the
effect of a failure. This may be measured on an ordinal scale
(e.g., classification as major, minor or negligible) or on a
ratio scale (e.g., cost of recovery, length of down-time). This
value may depend also on the frequency of the error.
Fault (also Software Fault) A fault is
a software defect that causes a failure. The state of an item
characterized by inability to perform a required function, excluding
the inability during preventive maintenance or other planned
actions, or due to lack of external resources. A fault is often
the result of a failure of the item itself, but may exist without
prior failure.
(1) A condition that causes a device or
system component to fail to perform in a required manner. (2)
A defect which may cause a failure under some set of conditions.
(3) Hardware: An accidental condition that causes a functional
unit to fail to perform its required function. (4) Software:
A manifestation of a mistake in acquisition, specification, design
or implementation. A cause of an error in software. Can be synonymous
with "bug". A fault, if encountered or detected, results
in an error. NOTE: (IEC 1508) (1)
(1) From the above definitions it can be
seen that TfaultIerrorIfailureU are part of a chain; (faultIerrorIfailure/faultIerrorIfailure/faultIerrorI).
(2) The definitions of fault, error, failure have to be seen
in the context of the system. The definitions above refer to
one level of the hierarchical system decomposition. An error
is thus the manifestation of a fault in the system, and a failure
is the effect of an error in service. The failure of a system
component can be considered in two different ways depending on
the hierarchical level from which it is seen. If the component
itself is seen as a system (lower level view) this failure is
the result of an internal erroneous state caused by a fault within
the component. On the next higher level the failure of the component
is a fault possibly resulting in erroneous states and a failure
of the next system level. (3) At first sight the definitions
of fault. Error, failure in this international standard seem
to deviate from the definitions given I IEC 50 (IEV).
The reason for using these different definitions
lies in the fact that they refer to a clear hierarchical concept
of system structure. The relationship to the definitions given
in IEC 50 (IEV) may be established by recognizing that the term
RitemS, used in IEC 50 (IEV), refers to a component of the system
without regard to the level of system decomposition from which
it is seen. This means that the failure of an item corresponds
to the view of where this component is seen as a failing (sub)system.
Afterwardsm if viewed from the next higher system level, this
component, or item, is faulty.
Fault Avoidance The use of techniques
and procedures which aim to avoid the introduction of faults
during any phase of the lifecycle and therefore avoiding faults
in the safety-related system. (Adapted from IEC 1508, 3.22)
Fault Density Defined as the number of
faults compared to the volume of the software work product. This
volume for software is often expressed in terms of Lines of Code,
Number of Statements, or Function points, but other measures
or volume may be appropriate given the stage of development.
For example, Faults per number of pages may be appropriate when
discussing fault density of a software design document.
Fault Exposure Ratio The probability that
a fault will cause a failure. This may (and often does) vary
from fault to fault. Alternatively, it may be viewed as an overall
probability regarding the number of faults that would be found
versus the number of faults that would still exist after a sequence
of fault removal or software exposure steps.
Fault Injection The process of purposely
or accidentally inserting or injecting a fault into a software
artifact. Faults may be injected on purpose to test fault tolerance
capabilities.
Fault Masking A condition in which the
occurrence of a fault is masked, often by another fault. Only
when the first fault is discovered and removed does the second
fault become apparent.
Fault-prone Component A software component
in which faults are often found, usually after a series of changes
or updates have been made to the component.
Fault Removal The act or process of removing
the fault. As faults are removed, reliability should generally
increase.
Fault Seeding A process in which faults
are purposely injected into the software so that an organization
can:
a) determine how efficient its fault removal processes are (efficiency
equals the number of seeded faults found versus the number of
seeded faults injected)
b) determine failure modes and effects, including the visibility
of failures in alarms and logs
c) test fault tolerance features
a) determine how efficient its fault removal processes are (efficiency equals the number of seeded faults found versus the number of seeded faults injected)
b) determine failure modes and effects, including the visibility of failures in alarms and logs
c) test fault tolerance features
Fault Tolerance The designed and built-in
capability of a system, or system component, to provide continued
though perhaps degraded, functionality in the presence of generally
expected failures and special-case failures (such as providing
for graceful degradation or continued minimum safety)."
Fault-tree Analysis Analysis (often quantitative)
of a fault tree model in which minimal sets of component failures
which cause system failures (termed minimal cutsets) are enumerated
and studied. A fault tree model is a graphical representation
of the logical relationships between events, including failure
events. Quantitative analysis is used to determine the probability
of the "top" events given the probability of occurrence
for the basic events.
Formal Methods The use of mathematical
techniques in design and analysis. (from Formal Methods and the
Certification of Critical Systems by John Rushby, Technical Report
CSL-93-7 December 1993) System - The use of mathematical techniques
in the design and analysis of the system). Hardware - The use
of mathematical techniques in the design and analysis of a component.
Software - The use of discrete mathematical techniques in the
design and analysis of software components and systems.
A software specification and production
method, based on a precise mathematical syntax and semantics,
that comprises: a collection of mathematical notations addressing
the specification, design and development phases of software
production; a well-founded logical inference system in which
formal verification proofs and proofs of other properties can
be formulated; and a methodological framework within which software
may be developed from the specification in a formally verifiable
manner. Formal methods can be operational denotational or dual
(hybrid).
Formal Verification(for software) The
process of evaluating the products of a given phase using formal
mathematical proofs to ensure correctness and consistency with
respect to the products and standards provided as input to that
phase (ISO 9000-3, 3.6 applied)
FMECA (Failure Mode Effects and Criticality Analysis)
A design review process that examines the ways and likelihood
that a system can fail, the severity and effects on the mission,
and the ability to detect, isolate and recover from the fault.
While the roots of FMECA are in hardware design and defect analysis,
it can be effectively applied to software and processes, as well.
Various industries have their own standards.
Aerospace and defense companies generally use either the MIL-STD-1629
FMECA standard (the C in FMECA represents the Criticality calculation)
or the SAE ARP5580 FMEA standard. Automotive suppliers use SAE
J1739, or the Automotive Industry Action Group (AIAG), Daimler-Chrysler,
Ford, and GM FMEA methodologies. Other industries generally adopt
one of these standards, sometimes customizing them to meet their
own requirements.
Goel-Okumoto Model A particular software
reliability growth model that assumes that the failure/fault
discovery events behave according to a non-homogeneous Poisson
process where the rate is a decaying exponential function with
respect to execution time. The cumulative expected number is
exponential (with asymptote) with respect to time
Growth Models (for software reliability)
Any of a class of models (most assume a non-homogeneous Poisson
process for the distribution of interfailure times) where the
failure intensity will tend to decrease asymptotically with respect
to time (that is, the successive interfailure times will tend
to become larger), thus reflecting an eventual monotonic increase
or growth in reliability.
Halstead Complexity Measures Halstead
complexity measurement was developed to measure a program module's
complexity directly from source code, with emphasis on computational
complexity. The measures were developed by the late Maurice Halstead
as a means of determining a quantitative measure of complexity
directly from the operators and operands in the module. Among
the earliest software metrics, they are strong indicators of
code complexity.
Hazard A condition that is prerequisite
to a mishap/accident (882C)
A state of an item which, in conjunction with certain environmental
conditions, could lead to an accident. A physical situation,
often following from some initiating event, that can lead to
an accident.
A state of an item which, in conjunction with certain environmental conditions, could lead to an accident. A physical situation, often following from some initiating event, that can lead to an accident.
Hazard Probability The aggregate probability
of occurrence of the individual events that create a specific
hazard (882C).
Probability of a system getting into a hazardous state within
a given time interval under given conditions. It is assumed that
the system is not in a hazardous state at the start of the interval.
Probability of a system getting into a hazardous state within a given time interval under given conditions. It is assumed that the system is not in a hazardous state at the start of the interval.
Hazard Severity An assessment of the consequences
of the worst credible mishap that could be caused by a specific
hazard (882C).
Impact Analysis Impact Analysis is the
activity aimed at determining the impact of a change before its
actual implementation, in order to anticipate the extent of its
effects, and thus the associated risk and cost.
Incident An event during operation of
an item that may indicate that a failure has occurred. The true
nature of an incident may not become apparent until after diagnosis.
Some incidents will turn out to be due to mistaken observation.
Others will turn out to be "genuine" and relevant to
the measurement of one or more dependability attributes.
Independent Verification & Validation (IV&V)
Confirmation by independent examination and provision of objective
evidence that specified requirements have been fulfilled (ISO
8402, 2.17), and that the particular requirements for a specific
intended use are fulfilled (ISO 8402, 2.18).
Instantaneous Failure Rate -(See Failure Intensity)
Interfailure Times Times between failures
of a system, often assumed to be exponentially distributed.
Invalid Defect A defect that looks like
a defect in some software product but in reality it is not a
defect. Either it is not a defect, or it is a defect in some
other software product. It is common to receive about 15% of
invalid defects for software vendors. Possible causes are user
errors, hardware errors, and errors in other products.
Life-Cycle All the steps or phases an
item passes through during its useful life. The life may include:
problem identification, requirements engineering, analysis, design,
implementation, testing, assessment, risk analysis, maintenance,
etc.
Log Poisson Model A particular software
reliability growth model that assumes that the failure/fault
discovery events behave according to a non-homogeneous Poisson
process where the rate is inversely proportional with respect
to execution time. The cumulative expected number is logarithmic
with respect to time, thus the name. This model is also known
as the Musa-Okumoto execution time model.
Latent Fault An existing fault that has
not yet been recognized. A fault that is present in a part of
a system but has not yet contributed to a system failure.
Maintainability The maintainability of
software is the ease with which it can be understood, corrected,
adapted, and/or enhanced.
Modification may include corrections, improvements or adaptation
of software to changes in the environment, and in requirements
and functional specifications. Maintainability is defined as
the effort to perform maintenance tasks, the impact domain of
the maintenance actions, and the error rate caused by those actions.
Modification may include corrections, improvements or adaptation of software to changes in the environment, and in requirements and functional specifications. Maintainability is defined as the effort to perform maintenance tasks, the impact domain of the maintenance actions, and the error rate caused by those actions.
The probability that an item can be retained
in or restored to specified condition when maintenance is performed
by personnel having specified skill levels, using prescribed
procedures, resources and equipment at each prescribed level
of maintenance and repair (MCCR). Maintainability can be stated
as a performance or a measure. (BS4778)
Maintenance An important part of the software
life-cycle. Maintenance is expensive in manpower and resources,
and software engineering aims to reduce its cost. Maintenance
activities include: Perfective maintenance - Changes which improve
the system in some way without changing its functionality; Adaptive
maintenance - Maintenance which is required because of changes
in the environment of a program; Corrective maintenance - The
correction of previously undiscovered system errors.
Actions performed by personnel having specified
skill levels, using prescribed procedures, resources and equipment
at a prescribed level of repair, scheduled or reactive, to return
the system to within specified limits of operation.
Major Defect The relevant defect resulted
in the failure of a software system or component that leads it
to non operative conditions.
Markov Model for Software A Markov Model
consists of system states and transitions from one state to another.
A system state represents a combination of both operational and
failed components in the system. The system stays in a state
for a random time, defined by an exponential distribution, and
then goes into another state. A transition from one state to
another state is characterized by a transition rate. A Markov
model can be solved mathematically to obtain reliability and
availability measures. For example, the expected proportion of
time that the system spends in the failure states, which is called
the system unavailability, can be calculated.
Maturity The capability of the software
product to avoid failure as a result of faults in the software.
Attributes of software that bear on the frequency of failure
by faults in the software.
Maturity Level Refers to the 5 Levels
of maturity in the Software Engineering Institute - Capability
Maturity Model.
Mean-Time-Between-Failures (MTBF) MTBF
is the average time (usually expressed in hours) that a component
works without failure. It is calculated by dividing the total
number of failures by the total number of operating hours observed.
Thus, the term is related to the time a user may reasonably expect
a device or system to work before an incapacitating fault occurs.
The total cumulative functioning time of
a population divided by the number of failures. As with failure
rate the same applies to observed, assessed and extrapolated
MTBF. MTBF is used for items which involve repair. Mean Time
to Failure (MTTF) is used for items with no repair. (TNO)
Mean-Time-To-Failure (MTTF) The mean time
elapsed until a failure occurs.
Mean-Time-To-Repair (MTTR) The mean time
to carry out a defined maintenance action. Usually refers to
corrective maintenance. (TNO)
The mean time that a device will take to
recover from a non-terminal failure. The MTTR is usually part
of a maintenance contract, where the user would pay more for
a system whose MTTR was 24 hours, than for one of, say, 7 days.
This means the supplier is guaranteeing to have the system up
and running again within 24 hours (or 7 days). Fault tolerant
devices have a MTTR of zero.
Mean-Value Function The expected value
function of a Poisson process, that is a process where the interarrival
times for the events are exponentially distributed. Each event
increment the process by 1, thus the mean-value function is the
expected number of events by time t.
Metric The defined measurement method
with its measurement scale. Metrics can be internal or external,
and direct or indirect. Metrics include methods for categorizing
qualitative data. Synonymous with measure. A quantitative measure
of the degree to which a system, component, or process possesses
a given attribute.
Minor Defect The defect does not cause
a failure, does not impair usability, and the desired processing
results are easily obtained by working around the defect.
Mishap An undesirable consequence of system
failure, whose cost, whether economic, ecological, or in terms
of injury or loss of life, is unacceptable. In some cases, it
is a synonymous malign failure or accident.
Module A self-contained software item
with a specified function and a defined interface to the rest
of the system. The concept is hierarchical: big modules may consist
of composition of little modules, and a complete program may
be thought of as a module of a larger system. The term is also
applied to a self-contained hardware component or to one consisting
of hardware and software. In programming languages, a module
is a piece of code that can be separately compilable.
Musa-Okumoto model (See Log Poisson model)
Mutation Testing A technique for which
two or more program mutations are executed and tested in order
to detect if the used test cases are capable of detecting mutations.
N-Version Software Software developed
and tested to fulfill a set of requirements where multiple versions
of software are intentionally made independent and different.
Differences can be in some or all of: specifications, design,
use of language, algorithms, data structures, etc.
Operational Profile A characterization
of the conditions of use of a system. The term is generally applied
to software. The profile can sometimes be defined by a partitioning
of the input space and estimate of the probability of encountering
an input from each feature. A system trial must therefore be
performed using a realistic operational profile. Definition of
conditions of use is essential for measurement of reliability
(and some other dependability attributes), since a system will
generally exhibit different reliabilities under different conditions.
Usage is synonymous.
Orthogonal Defect Classification A scheme
for classifying software defects detected throughout the development
cycle that provides feedback about a process stage based on the
defects associated with that stage. Each defect is classified
across multiple dimensions or attributes, such as defect type,
defect trigger, impact, and source code type, that contain orthogonal
or independent values.
Outage An event which causes the state
of a system to be unavailable to some or all of its potential
users. A failure that denies availability for a period of time
to a class of customers.
Patch A change to machine code (object
code) representation of a computer program that bypasses the
compiler (MCCR).
Physical Fault A fault which is the result
of a physical failure.
Predicted Quality Software quality that
is estimated and predicted for the end software product quality
at each stage of development, and which is based on fundamental
design quality and intermediate product quality.
Process Assessment A disciplined examination
of the processes used by an organization against a set of criteria
to determine the capability of those processes to perform within
quality, cost and schedule goals by characterizing current practice
and identifying strengths and weaknesses.
Product A package of software or hardware,
providing functionality designed for use or incorporation within
a multiplicity of systems. (ITSEC 91)
A product is broader than a system. It incorporates all components
that the producer uses, and all the items delivered to customers.
Documentation and confidential source code, for instance, is
part of the product but not part of the system. Result of activities
or processes. Product may include service, hardware, processed
materials, software, or combination thereof.
A product is broader than a system. It incorporates all components that the producer uses, and all the items delivered to customers. Documentation and confidential source code, for instance, is part of the product but not part of the system. Result of activities or processes. Product may include service, hardware, processed materials, software, or combination thereof.
A product can be tangible (e.g., assemblies
or processed materials) or intangible (e.g., information or concepts),
or a combination thereof. Product can be either intended (e.g.,
offering to customers) or unintended (e.g., pollutant or unwanted
effects). Examples of software products are also: specifications,
test reports, executable object code, user manuals, and support
documentation. Something that has value and is generated by a
manufacturing or development process.
Product Liability Generic term used to
describe the onus on a producer or others to make restitution
for loss related to personal injury, property damage or other
harm caused by a product (ISO 8402).
Project Manager The leader of the project.
He/she has the responsibility to: (i) control but facilitating
the information flow among teams; (ii) to ensure that the project
program, milestones and time scales are maintained and deviations
resolved and recorded with change control; (iii) managing related
documentation; (iv) maintaining relationships between partners;
(v) control the behavior of project against the planned activities;
(vi) communicate progress; (vii) establishing external relationships
according to the dissemination activity; (viii) controlling quality
of all activities and deliverables via specific tools; (ix) administrations,
e.g., collection of cost statements on a timely basis, collection
of information for EC management and progress reports on a timely
basis; (x) act as a link between this project and other related
projects; (xi) arrange for the preparation of necessary project
review documentation, etc.
Quality The totality of features and characteristics
of a product or service that bear on its ability to satisfy stated
or implied needs. Not to be exchanged with the "degree of
excellence" or "fitness for use" that meet only
partially the definition. Software quality is defined in the
ISO 9126 norm series. Software quality includes: functionality,
reliability, usability, efficiency, maintainability, portability.
The quality of a system is the evaluation of the extent to which
the system meets the above mentioned features. The response of
the system to these features is called the estimated quality
profile. Quality should not be used as a single term to express
a degree of excellence in a comparative sense nor should it be
used in a quantitative sense for technical evaluations. To express
these meanings, a qualifying adjective shall be used. For example,
use can be made of the following terms: "relative quality"
where entities are ranked on a relative basis in the "degree
of excellence" or " comparative sense" (not to
be confused with grade); "quality level" in a "quantitative
sense" (as used in acceptance sampling) and "quality
measure" where precise technical evaluations are carried
out.
Quality Evaluation Systematic examination
of the extent to which an entity is capable of fulfilling specified
requirements (ISO 8402, 4.6).
A quality evaluation may be used to determine
supplier quality capability. In this case, depending on specific
circumstances, the result of quality evaluation may be used for
qualification, approval, registration, certification or accreditation
purposes.
An additional qualifier may be used with
the term quality evaluation depending on the scope (e.g., process,
personnel, system) and timing (e.g. pre-contract) of the quality
evaluation such as "pre-contract process quality evaluation".
An overall supplier quality evaluation also may include an appraisal
of financial and technical resources.
Quality evaluation is sometimes called
"quality assessment", "quality appraisal",
or "quality survey" in specific circumstances.
The process of measurement, rating and
assessment of quality attributes for an evaluation item to determine
if the specified software quality characteristics are fulfilled.
Random Failures Failures that result from
a variety of degradation mechanisms in the hardware. Unlike failures
arising from systematic failures, system failure rates arising
from random hardware failures can be quantified with reasonable
accuracy.
A sub-class of systematic failures, while
occurring at random times, which result from a variety of degradation
mechanisms at a predictable rate. (Derived from IEC 1508, 3.45)
NOTE: (1) There are many degradation mechanisms occurring at
different rates at different times in operation, failures of
a total equipment comprising many components occur at predictable
rates but at unpredictable times. (2) A major distinguishing
feature between random hardware failures and systematic failures
(See also Systematic Failures) is that systematic failure rates
(or other appropriate measure), arising from random hardware
failures, can be predicted with reasonable accuracy but systematic
failures, by their very nature, cannot be quantified with reasonable
accuracy but those arising from systematic failures cannot be
accurately quantified.
Regression Testing The test that is performed
on a system when it has been produced in a new version/release.
The regression testing has to verify: (i) all the already tested
functionalists that were correct and available in the previous
version are also present in the new version without any problems,
(ii) that the problems solved in the new version have been effectively
solved without causing second level problems. The regression
testing is very important in order to verify if the new release
has solved the problems without adding at the same time new problems.
Relevant Failure A failure that should
be included in interpreting test or operational results or in
calculating the value of a reliability performance measure.
Reliability The probability of a given
system or component to perform its required functions under stated
conditions for a specified period of time (or cycles). The reliability
of a system is generally measured using operating time as the
time domain, i.e., as the probability of non-failure in a given
period of time during which the item is actually operating. For
calculating the probability of failure over real time, operating
time or execution time must be transformed into real time taking
into account the frequency of use of the item, simultaneous operation
of multiple copies, etc.
Limitations in reliability are caused by
faults in requirements, design, and implementation. Failures
due to these faults depend on the way the software product is
used and the program options selected rather than on elapsed
time.
Risk A measure of the severity and likelihood
of an accident or mishap. The probability that a specific threat
will exploit a particular vulnerability of the system. A measure
of the cost of operating a system, derived by combining hazard
probability, danger, and severity of mishap.
Risk is defined as the possibility of loss
or injury. Risk exposure is defined by the relationship RE =
P(UO) * L(UO) Where RE is the risk exposure, P(UO) is the probability
of an unsatisfactory outcome, and P(LO) is the loss to the parties
affected by if the outcome is unsatisfactory. Examples of unsatisfactory
outcome include schedule slips, budget overruns, wrong functionality,
compromised non-functional requirements, user-interface shortfalls
and poor quality.
Safety State in which the risk of harm
(to persons) or damage is limited to an acceptable level (ISO
8402). Freedom from those conditions that can cause death, injury,
or occupational illness, or damage to or loss of equipment or
property, or damage to the environment (882C).
The ability of a system to operate without
unacceptable risk in accordance with its requirements in a consistent
and predictable manner for a given time in a given environment
without mishap. For system safety, all causes of failures which
lead to an unsafe state shall be included; hardware failures,
software failures, failures due to electrical interference, due
to human interaction and failures in the controlled object. The
system safety also depends on many factors which cannot be quantified
but can only be considered qualitatively.
Safety-Critical A term applied to a condition,
event, operation, process, object, function or system whose proper
recognition, control, performance or tolerance is essential to
safe system operation or use; e.g., safety critical function,
safety critical path, safety critical component (882C). A system
whose failure may cause injury or death to human beings (e.g.
an aircraft or nuclear power station control system). Common
tools used in the design of safety-critical systems are redundancy
and formal methods.
Security Attributes of software that bear
on its ability to prevent unauthorized access, whether accidental
or deliberate, to programs and data. The ability of a system
to deliver its required service and data under given conditions
without unauthorized access.
The in-built resistance of a system to
deliberate attempts to influence its operation in an undesired
way, including release of information. (CORD)
Sensitivity A measurement of responsiveness
to or recording slight changes to externally or internally sensed
parameters.
Software Defect A software defect is a
perceived departure in a software product from its intended properties,
which if not rectified, would under certain conditions contribute
to a software system failure (departure from required system
behavior during operational use).
Software Failure System failure due to
the activation of a design fault in a software component. All
software failures are design failures, since software consists
solely of design, and does not wear out or suffer from physical
failure.
Software Fault A design fault located
in a software component. (see Fault)
Software Quality The totality of features
and characteristics of a software product that bear on its ability
to satisfy stated or implied needs, including internal and external
aspects and contributing on functionality, reliability, usability,
efficiency, maintainability, portability of the system under
development.
Software Reliability The measurement of,
prediction of, or improvement to the duration or probability
of failure-free performance under stated conditions.
Software Reliability Measurement (Estimation and Prediction)
Estimation - To determine the previous and current software reliability
by applying statistical inference techniques to failure data
obtained during system test or operation.
Prediction - To determine (or estimate) the probable future software
reliability based on available software metrics and measures.
When failure data are available, estimation techniques that use
models based on this and related data are called reliability
predictions. When failure data are unavailable or not yet available,
and the estimation techniques are based on measurement of the
software process of characteristics of the product, the estimation
technique is called early prediction. (Handbook of Software Reliability
Engineering, McGraw-Hill)
Prediction - To determine (or estimate) the probable future software reliability based on available software metrics and measures. When failure data are available, estimation techniques that use models based on this and related data are called reliability predictions. When failure data are unavailable or not yet available, and the estimation techniques are based on measurement of the software process of characteristics of the product, the estimation technique is called early prediction. (Handbook of Software Reliability Engineering, McGraw-Hill)
Software Testing Software testing is the
process of examining the behavior of a program or part of a program
by executing it on selected or exhaustive sets of input data.
Testing is performed to discover defects in a program. Testing
is not the process of confirming that a program is correct, nor
is it a demonstration that defects are not present.
Stability Attributes of software that
bear on the risk of unexpected effects of modifications. The
capability of the software product to avoid unexpected effects
from modifications of the software.
Static Analysis The analysis of a system
or component directly performed on the code and documentation
without compiling it.
Statistical Usage Testing Software testing
that is viewed as a statistical experiment. A subset of all possible
uses of the software is generated,
and performance on the subset is used as a basis for conclusions
about general operational performance. In standard experimental
parlance, a "sample" is used to draw conclusions about
a "population."
In statistical testing:
(1) expected operational use is represented in a usage model
of the software,
(2) test cases are randomly generated from the usage model and
executed in an environment that simulates the operational
environment, and
(3) failure data are interpreted according to mathematical and
statistical models.
and performance on the subset is used as a basis for conclusions about general operational performance. In standard experimental
parlance, a "sample" is used to draw conclusions about a "population."
In statistical testing:
(1) expected operational use is represented in a usage model of the software,
(2) test cases are randomly generated from the usage model and executed in an environment that simulates the operational
environment, and
(3) failure data are interpreted according to mathematical and statistical models.
Stress Testing Test by applying a high
load.
A performance testing in which the test cases are oriented in
stressing the functionalities of the system.
A performance testing in which the test cases are oriented in stressing the functionalities of the system.
System Test and Evaluation Process A process
that plans, develops, and documents the quantitative demonstration
of the fulfillment of all baseline functional performance, operational,
and interface requirements.
Test Case A set of test inputs, conditions
for execution, and expected outputs with the aim of testing a
particular part of the system or particular functionalities or
particular requirements.
Testing The process of executing a series
of test cases and evaluating the results. Operating a system
(possibly under unrealistic conditions of use) in order to detect
faults. See debug, and software testing.
Test Procedure (1) Specified way to perform
a test (ISO 8402, 1.3 procedure). (2) Detailed instructions for
the set-up and execution of a given set of test cases, and instructions
for the evaluation of results of executing the test cases. (RTCA/DO-178B)
Time May denote either real time (also
referred to as elapsed, calendar or wall-clock time) or execution
time.
Traceability Ability to trace the history,
application or location of an entity by means of recorded identifications
(ISO 8402, 3.16). One of three main meanings: (a) In a product
sense it may relate to (1) the origin of materials and parts,
(2) the product processing history and (3) the distribution and
location of the product after delivery. (b) In a calibration
sense, it relates to measuring equipment to national or international
standards, primary standards, basic physical constants or properties,
or reference materials. (c) in a data-collection sense, it relates
calculations and data generated throughout the quality loop sometimes
back to the requirements for quality for an entity (ISO 8402,
3.16 Notes). All aspects of traceability requirements, if any,
should be clearly specified, for example, in terms of periods
of time, point of origin or identification.
The property that allows to trace a features
or and error along a system or process. For example, it can be
useful to trace, the requirements along the development life-cycle,
the errors along the system structure, the messages along the
communication channel, the email along the network.
Undetectable failures A failure which
cannot be detected using a stated device or function in a stated
environment. For example, a failure may not be detectable using
on-line systems, devices, or functions yet has been demonstrated
to be detectable using special external test equipment. In the
context of normal operations the failure is undetectable. (Derived
from -882 and IEC 1508 3.74 and other standards) NOTE: Other
names used in the past for undetectable failure include: covert
failure, unrevealed failure, latent failure, hidden failure,
or dormant failure.
Unit Test Part of the testing strategy
that focuses on tests to individual program components.
Validation Steps taken to ensure that
the output products of any development phase correctly implement
the input products. Confirmation by examination and provision
of objective evidence that specified requirements have been fulfilled.
In design and development, verification concerns the process
of examining the result of a given activity to determine conformity
with the stated requirement for that activity. It answers the
question: "Are we building the system right?"
The verification includes activities such
as inspection, proof of correctness, static analysis, etc. It
may includes the act of reviewing, inspecting, testing, checking,
auditing, comparing or otherwise establishing and documenting
whether items, processes, services, or documents conform to specified
requirements. It also includes the formal proof of program correctness.
(1) Confirmation by examination and provision
of objective evidence that the particular requirements for a
specific use are fulfilled (ISO 8402, 2.18). (for software) The
process of evaluating software to ensure compliance with specified
requirements (ISO 9000-3, 3.7). (2) The process of determining
whether the system operates correctly and executes the correct
functions.
Verification and Validation (V&V)
The process of verification and validation. In some cases, with
some techniques the distinction of the boundary between these
formal activities is blurred.
(1) Confirmation by examination and provision
of objective evidence that specified requirements have been fulfilled
(ISO 8402, 2.17). (for software) The process of evaluating the
products of a given phase to ensure correctness and consistency
with respect to the products and standards provided as input
to that phase (ISO 9000-3, 3.6) (2) The process of evaluating
the products of a given phase to determine the correctness and
consistency of those products with respect to the products and
standards provided as input to that phase.