Should confidence levels be included in high level mission requirements, e.g., KPPs? If so, please give examples of their proper and improper use.
There is nothing "improper" about using confidence intervals for KPPs. But KPPs with "absolute thresholds" are actually LESS restrictive than KPPs with explicit confidence intervals.
Open full Question Details
Let's consider a typical "absolute threshold" parameter:
"The system shall have the capability to lethally engage threats with a hit probability of no less than 70 percent at 500 yards."
You could decide to test this parameter by firing at 10 targets and declaring success with 7 hits. But if you add a (typical) 80% confidence requirement, your test program would need at least 9 hits in 10 trials to declare success. (The 7 hits in 10 trials actually produces only 35% confidence that the system meets or exceeds the 70% parameter.)
Confidence intervals in system specifications (ref MIL-STD-961E) can provide a clear understanding between the government and contractor of what level of proof is needed to accept a system design. (Perhaps with a test program of 1,000 trials and less than 280 failures to demonstrate 90% confidence of 70% hit probability)
But confidence intervals in capabilities documents (between the sponsor and the acquirer within the government) may prove problematic during operational test. Rather than firing a few rounds under operational conditions to validate the results from DT, the operational tester could be obliged to fire hundreds of rounds to reach the same statistical confidence which was verified during development. (Due to a smaller sample size, 100 trials with 28 failures only provides a 60% confidence of 70% hit probability.)
It is very important that the CDD sponsor collaborate with the MDA-designated developer when preparing a CDD. Good communication will help the sponsor build KPPs which are achievable in terms of cost, schedule and technological maturity.