Abstract
Embedded systems are often used in safety-critical environments. Thus, thorough testing of them is mandatory. To achieve a required structural code-coverage criteria it is beneficial to derive the test data at a higher program-representation level than machine code. Higher program-representation levels include, beside the source-code level, languages of domain-specific modeling environments with automatic code generation. For a testing framework with automatic generation of test data this will enable high retargetability of the framework. In this article we address the challenge of ensuring that the structural code coverage achieved at a higher program representation level is preserved during the code generations and code transformations down to machine code. We define the formal properties that have to be fullfilled by a code transformation to guarantee preservation of structural code coverage. Based on these properties we discuss how to preserve code coverage achieved at source-code level. Additionally, we discuss how structural code coverage at model level could be preserved. The results presented in this article are aimed toward the integration of support for preserving structural code coverage into compilers and code generators.
. Denotes the program before (
) and after (
) the transformations for which we want to preserve structural code coverage.
consists of a set of nodes 
representing basic blocks (see below), a set of edges 
representing the control flow (also called control-flow edges), a unique entry node 
, and a unique end node 
.
. Is a fragment of 
with well-defined interfaces for entry and exit. We denote the set of program scopes in a program 
as 
. The concrete partitioning of a program into scopes is application-specific. For example, in [
is a sequence of control-flow edges from an entry point of the scope to an exit point of the scope. In case of nested program scopes, the whole inner program scope is a single block in the paths of the outer program scope. A scoped path of a program scope 
is uniquely represented by its starting basic block and the necessary TRUE/FALSE evaluation result of all conditions along the scoped path. We denote the set of scoped paths in a program scope 
as 
. The paths within a program 
, that is, the scoped paths where the program scope subsumes the whole program, is denoted as 
.
is a code sequence of maximal length with a single entry point at the beginning and with the only allowed occurrence of a control-flow statement at its end. We denote the set of basic blocks in a program 
as 
. The set of all basic blocks along a scoped path 
is denoted as 
. Note that in cases of program paths with cycles, 
will contain multiple instances of the basic blocks in the program code. If a scoped path goes through a nested program scope, all the basic blocks from the nested program scope are hidden for this path. The starting basic block of a scoped path 
is denoted as 
.
as 
.
is not evaluated if the result of the operator is already determined by the first argument. See Section 5.4 for further details. The correct identification of hidden control flow is important, for example, to analyze decision coverage.
as 
. The set of all conditions within a program 
is denoted as 
.
is denoted as 
. Note that in cases of program paths with cycles, 
will contain multiple instances of the conditions in the program code. If a scoped path goes through a nested program scope, all the conditions from the nested program scope are hidden for this path. To follow a certain path, it is also important whether a condition evaluates to TRUE/FALSE. Whether a condition has to be evaluated as TRUE or as FALSE is given by the syntactical structure of a program. For a given scoped path 
we denote by 
all the conditions that have to be evaluated as TRUE and by 
all the conditions that have to be evaluated as FALSE to follow 
. It holds that 
and 
.
. Defines the set of all possible valuations of the input variables of a program. (Valuation of a variable means the assignment of concrete values to it. The valuation of an expression means the assignment of concrete values to all variables within the expression.)
. Defines the set of valuations of the input variables that have been generated with structural code coverage analysis done at source-code level. Since exhaustive testing is intractable in practice, 
is assumed as a true subset of the program's input data space 
: 
. If we would consider exhaustive testing (
) there would be no challenge of structural code-coverage preservation.
Defines the set of valuations of the input variables that trigger the execution of expression 
, where 
can be a condition, decision, or a basic block.
,
Defines the sets of valuations of the input variables that trigger the execution of the condition/decision 
with a certain result of 
: 
is the input-dataset, where 
refers to TRUE and 
is the set, where 
refers to FALSE. The following properties always hold for 
, 
:
.
due to the hidden control flow caused by the short-circuit evaluation of ANSI C; see Section 5.4). It follows that
(
the conditions 
, 
, and 
depends on whether the programming language has hidden control flow (see Section 5.4). Above definitions allow to formally describe structural code coverage criteria. We will also use them to describe requirements to preserve structural code coverage.
is executed at least once. Statement coverage alone is quite weak for functional testing [
can be tricky due to hidden control-flow. A statement in a high level language like ANSI C can consist of more than one basi cblock. For example, the ANSI C statement 
has been tested at least once with each possible outcome. Decision coverage is also known as branch coverage or edge coverage. Decision coverage implies statement coverage:
. However, often definitions are used that do not explicitly consider short-circuit operators (e.g., [
conditions in a decision, 
test cases are required to achieve MCDC. Note that MCDC implies DC and CC. A formal definition of MCDC is given in (9) based on the set of input test data 
. It requires that for each condition 
of a decision 
there exists two test vectors such that the predicate symbol 
holds, which ensures that the two test vectors show different outcomes for 
as well as 
but the same outcomes for all other conditions within 
. This is exactly how MCDC is described above
tests whether one of the test data 
, 
is a member of the input dataset 
and the other one a member of the input data set 
. If this predicate symbol is TRUE it is guaranteed that the expression 
evaluates to both, TRUE and FALSE:
tests whether the input-data set 
provides a constant outcome for the evaluation of 
. Actually, the predicate symbol 
is used to test whether there exists a test-data subset 
for a given condition, such that the results of all other conditions remain unchanged. Thus, this predicate symbol is used to ensure that each condition can independently control the output of the decision:
are strongly coupled, iff they have the same input data partitioning for their satisfiability valuation, i.e., 
.) As described in [
by another predicate that formalizes the semantics of the alternative MCDC criterion.
conditions 
tests are necessary. MCC is desired in theory, but MCC tends to be infeasible for industrial code, because there are too many conditions per decision [
has been tested at least once. Since the number of paths within a program typically grows exponentially with the program size (PC is even stronger than MCC), we do not address PC.
into program scopes and cover all possible paths within each program scope. Actually, PC is just the special case of using SPC in combination with only one program scope covering the whole program 
.
" of (13) ensures that in the pathological case of having a program scope that is completely free of conditions, coverage of the only single path in the program scope is guaranteed.
and 
that this code coverage is preserved while the program 
is transformed into another program 
. This scenario is shown in Figure 
, the set of program decisions 
, or program scopes 
may get changed. As shown in Figure 
is preserved to the transformed program. Based on these properties one can adjust a source-to-source transformer or a compiler to use only those optimizations that preserve the intended structural code coverage.
then this coverage is also fulfilled at the transformed program with the same test data:
that achieve the structural code coverage at the original program.
of the transformed program there exists a basic block 
of the original program such that reaching 
with a given test vector implies that also 
is reached with the same test vector.
achieves statement coverage on a given program 
, then (15) provides a sufficient—and without further knowledge about the program and the test data (there is now knowledge about the test data or the program assumed), also necessary—criterion for guaranteeing preservation of statement coverage on a transformed program 
.
is assumed to achieve SC on 
, it holds for each 
that 
. Since (15) states that 
it follows that for each 
we also have 
. Thus, SC is preserved at 
.
of 
such that for all basic blocks 
of 
it holds that 
, then each 
contains at least one input that is not in 
. If 
consists of exactly those inputs, then 
is never reached although SC holds in 
, which implies that SC is not preserved.
given in (16).
is only TRUE if the set of input data 
includes at least the true-satisfiability valuation 
or the false-satisfiability valuation 
of expression 
, where 
is either a condition or a decision. The predicate 
is used for the coverage preservation criterion of CC (and also DC) to test whether the evaluation of any expression 
of the original program to both, TRUE and FALSE, implies that the test data include at least one element of 
, needed for the coverage of an expression in the transformed program
of the transformed program there exists at least one condition of the original program whose coverage implies that 
evaluates to TRUE and there exists at least one condition of the original program whose coverage implies that 
evaluates to FALSE.
achieves condition coverage on a given program 
, then (17) provides a sufficient—and without further knowledge about the program and the test data, also necessary—criterion for guaranteeing preservation of condition coverage on a transformed program 
:
is assumed to achieve CC on 
, it holds for each 
that 
and 
. Since (17) states that for each 
it holds that
we also have
.
of program 
such that for all conditions 
of program 
it either holds that
: 
: 
is kept equal or simply is inverted. This would result in the simpler criterion given in (20)
given in (16), which is also used for preserving CC.
of the transformed program there exists at least one decision of the original program whose coverage implies that 
evaluates to TRUE and there exists at least one decision of the original program whose coverage implies that 
evaluates to FALSE.
achieves decision coverage on a given program 
, then (21) provides a sufficient—and without further knowledge about the program and the test data, also necessary—criterion for guaranteeing preservation of decision coverage on a transformed program 
is assumed to achieve DC on 
, it holds for each 
that 
and 
. Since (21) states that for each 
we also have 
and 
. Thus, DC is preserved at 
.
such that for all conditions 
it either holds that
or
there is an adequate decision 
of the original program such that decision coverage is preserved. For example, consider the following code transformation:
inlined style
noninlined style
is used in the same way as the real criterion: it is used to express that only input data that fulfill MCDC at the original program have to be considered for coverage preservation
by a single condition from the original program 
.
that are used to achieve MCDC at the original program. To ensure coverage preservation in general, it would be necessary to ensure that the criterion holds for all possible sets of test data 
that achieve MCDC at the original program, which tends to be intractable in practice.
given in (24). The predicate 
is similar to the predicate symbol 
, with the difference that it performs the control check on all members of two sets of input data. The predicate 
is used for the coverage preservation of MCDC to test whether the condition 
of the original program refers to TRUE for one input data set 
or 
and refers to FALSE for the other. Besides 
, also the predicate 
(10) is used to describe the preservation criterion for MCDC coverage
of a decision 
of the transformed program there exist two sets of input data 
and 
whose members achieve the 
criterion needed for MCDC coverage. Further, there has to be a condition of the original program such that the 
is a subset of either the true-satisfiability valuation or the false-satisfiability valuation (tested with the predicate 
). 
the same requirement as 
.
achieves MCDC coverage on a given program 
, then (25) provides a sufficient—and without further knowledge about the program and the test data, also necessary—criterion for guaranteeing preservation of MCDC coverage on a transformed program 
is assumed to achieve MCDC on 
, it holds for each 
and for each 
that there exist at least two test vectors 
such that 
. Since 
as defined in (10) for each condition is the formal definition of MCDC it directly follows that
.
with a condition 
such that for all input-data subsets 
it either holds that
condition coverage is not fulfilled; this case is already excluded by assumption of having MCDC coverage at 
)
; this case is already excluded by assumption of having MCDC coverage at 
)
do not provide MCDC coverage at the transformed program 
) which in each case violates the preservation of MCDC: Case (a) and (b) violate the preservation of MCDC since they are in contradiction with the requirement that MCDC is achieved at the original program. Case (c) states that there exists a condition in the transformed program for which there are no test data to achieve unique cause coverage, which is required for MCDC.
given in (31).
is only TRUE if there is at least one condition from the set of conditions 
whose true-satisfiability valuation is a subset of the input data 
or there is at least one condition from the set of conditions 
whose false-satisfiability valuation is a subset of the input data 
. The predicate 
is used for the coverage preservation criterion of SPC to test whether for a condition in the transformed program with true/false-satisfiability valuation 
there exist two conditions in the original program whose true/false coverage are a subset of 
of the transformed program there exists a scoped path 
such that the reachability of the first basic block of 
implies the reachability of the first basic block of 
. Further, Equation (
of 
that has to be evaluated to TRUE, there exists a condition 
of a scoped path in the original program that will imply the True evaluation of 
(by predicate 
). Finally, Equation (
of 
that has to be evaluated to FALSE, there exists a condition 
of a scoped path in the original program that will imply the FALSE evaluation of 
(by predicate 
).
achieves scoped path coverage on a given program 
, then (32) provides a sufficient—and without further knowledge about the program and the test data, also necessary—criterion for guaranteeing preservation of scoped path coverage on a transformed program 
is assumed to achieve SPC on 
, it holds for each 
and each 
with 
that there exists test data 
with
.
such that for all scoped program paths 
it either holds that
is not executed with the given test data 
)
that has to be evaluated to True evaluates to False for all test vectors of 
)
that has to be evaluated to False evaluates to True for all test vectors of 
) which in each case violates the preservation of SPC.
denotes the statement at source line 1. If there are multiple statements at the source line, we identify them by an additional index letter, for example, 
, 
, and so forth. Similarly, 
and 
address the decisions and conditions at source line 
. Based on this notation we use our formalism to describe by a graph the coverage relations between different program elements. 
denotes that the coverage of type 
at element 
implies the coverage of type 
at element 
. The possible coverage types are "R" (reaching), "T" (true-evaluation) and "F" (false-evaluation). A dotted line between elements from the original and the transformed code denotes that the two elements have the same coverage. A dotted arrow from an element of the original code to an element of the transformed code denotes that a coverage of the original node by the concrete test data implies also the coverage of the element in the transformed code. Structural code coverage is achieved, if all elements of the transformed code have a connection to an element of the original code. Elements of the transformed graph, for which coverage is not preserved, are marked by a surrounding box.
is surrounded by a box, which means that this coverage is not preserved during optimization. This means that the second condition at line 1 of the transformed program (
but not of 
implies that SC is still preserved. Also DC is preserved. However, coverage criteria that are based on the coverage of conditions are not guaranteed to be preserved. For example, preservation of CC or MCDC is not guaranteed. For simplicity, we do not discuss preservation of SPC, because analyzing SPC preservation would make it necessary to unroll the program to make the different scoped program paths explicit.
of ANSI C/C++ have a short-circuit evaluation semantics, which in fact already is conditional control flow. The short-circuit evaluation of ANSI C/C++ states that the second argument of the operators 
is not evaluated if the final result of the operator is already determined by the first argument.
if (
if (
if (
if (
/
/
(
= (
(where the condition 
is reached with an evaluation to True). The problem also arises for statement coverage (SC) because inlining reduces the reachability valuation of (
