more on Testing...
* test PLAN focuses on how you are going to handle
testing for a project;
* not the details of testing a particular SUT <- software under test
* note, then, that test CASE design has to be done
separately for each SUT
* based on the approach(es), features, etc.,
test cases are determined for a SUT
* and remember: each test case MUST include/MUST specify
the EXPECTED OUTCOME for that test case;
* we'd LIKE/HOPE that, together, our set of test cases for
SUT should detect all or most of any defects lurking
within that SUT;
* we'd also like the set of test cases to be "small" --
each test case DOES consume effort;
* IDEALLY, we'd like a set of test cases that is likely
to catch most of any defects within a SUT,
SUCH THAT a "SMALLER" set could not catch these defects
[BY THE WAY - the IDEAL mindset for testing is to WANT
to "BREAK" the SUT -- to go with the goal of how can I "break"
this software?]
* often test case specifications may be given in a table:
seq # cond to be tested test data exp'd result actl result if successful
----- ----------------- --------- ------------ ----------- -------------
* a project might also have a formal way of logging defects as they
are found, and tracking whether they have been corrected yet, etc.
* defect logging and tracking is considered to be good practice
in industry;
* ONE possible example of a "defect life cycle":
* a fault occurs, someone logs that that indicates a defect,
(include any info they can) - submitted
* job of FIXING this is assigned to someone --
when they (think) it is fixed, they note that - fixed
* someone -- manager? submitter? someone -- verifies
that the defect is (seems to be) fixed - closed
* (and all this provides data that can later be used
to assess quality metrics for a project...)
* SEVERITY of defects (in terms of IMPACT on the software)
is also recorded;
* an example of such a categorization:
* critical - show stopper!
* major - has a large impact
* minor - an isolated defect
* cosmetic - no impact on functionality
* SO: "good" sets of test cases ARE critical to a project;
and there are NUMEROUS approaches to trying to achieve
such test cases;
* some EXAMPLES: (keeping in mind you are not limited to
using JUST one of these approaches...!)
* black box, or functional, testing
* design test cases based on the SUT being a "black box" --
like you do not know what is inside;
* "data-driven" test case design?
* based on the function of the SUT
* white box, or clear box, or structural, testing
* design test cases based on knowledge of the
code within the SUT --
you want to exercise all of that code;
* (there's so-called grey box testing, too --
for example, based on state diagrams)
* black box testing
* design test cases based on the SUT being a "black box" --
like you do not know what is inside;
* test cases can be - and are - determined solely from
the specification
* EXHAUSTIVE testing -- every possible input! --
would be most thorough, BUT typically is infeasible!
(too costly, too time consuming, etc.)
* SO: how do you determine the right "smattering"
of inputs to try?
* one approach: equivalence class partitioning
* divide the input space into equivalent classes
* do this such that, IF the software works for
ONE test case from an equivalence class,
it is likely to also work for all
of the other values in that equivalence class;
* NOTE that programs often fail on "special" values;
one example of such "special" values are those lying
on BOUNDARIES between equivalence classes (IF applicable)
* test cases that include boundary values have high yield
* a boundary value test case is a set of input
data that lies on (or near) the EDGE of an
equivalence class;
* ONE approach to this:
* choose a value on the EDGES of the class
* choose values JUST OUTSIDE the edges <-- BOTH ways!
that is,
0 <= x <= 1.0,
0.0, 1.0 are boundaries,
-0.1, 0.1 are AROUND the edge 0,
0.9, 1.1 are AROUND the edge 1
* white box testing --
devising the test cases based on knowledge of what
IS in the SUT
* control-flow based criteria
* data-flow based testing
* mutation testing
...control-flow is most commonly-used;
* statement-coverage - criterion: each statement is executed
at least once during testing
* branch-coverage - criterion: each edge (of the control-flow graph)
should be traversed at least once during testing
* NOTE that these are not necessarily equivalent!
(consider SUT with an IF that does not have an ELSE --
you can have a set of test cases that exercise every
statement, but not necessarily every edge/branch (if don't
include one test case where the IF's condition is true,
AND one test case where the IF's condition is false...)