If you can't verify the device exhausively, you have to specify what you did verify and to what extent. You also have to specify how the testing was accomplished. An analysis of the impact of variation of the test parameters will reduce the breadth and depth of testing, but such reductions must be carefully considered, and those must be incorporated into MTBF calculations, most of which are so poorly done that they are little more than fabrications.

It means that in order to get the necessary work done within a reasonable amount of time, you plan the testing before the first bit of hardware or firmware is designed and design to the test spec. Then you put the necessary manpower to work in order to test the system against the design specification.

The key to a reliable system, however, is that requirements analysis must precede design. Most designs are performed by rote, without real consideration of the requirements. More often than not, effort devoted to things that "would be nice" in a later version exceeds that devoted to meeting specified requirements, either because no specification exists or because the designer doesn't know how to work from a specification.

The result of a proper analysis and design is a specification against which a system can be tested under normal and extra-normal conditions. System behavior must be entirely predictable in accordance with the system specifications. It must perform exactly as specified in order to pass acceptance testing. Under no circumstances should the designer/coder team be paid for any or all of their effort unless it does so.

Now, if the software tools aren't thoroughly specified and tested, how can the work product they generate be reliable?

RE