Headquarters U.S. Air Force Integrity - Service -

Headquarters U.S. Air Force Integrity - Service -

Headquarters U.S. Air Force Integrity - Service - Excellen ce RI3:Risk Identification, Integration, & illities Synergy with Cost Estimating, Earned Value Management and Auditing John Cargill Air Force Cost Analysis Agency/FMAE ASMC RPDI 4 March 2010 Version 1.0 1 No One Likes to Recognize Risk System Engineers -- Life is a lot easier (initially) when you are not forced to deal with risk and can seek the comfort of group think Contractors -- Win source selections by minimizing statements of risk. Nothing can go wrong until it does. Program Managers -- Behave as if nothing will go wrong with their programs (until it does). Good managers have low risk User Community -- Force changes to the system that invariably result in higher risks and costs (Requirements creep because of revised statements of I want)

Budgeteers/POMers -- Request budgets less than what they will realistically need to cover uncertain future events. Management reserve and contingency funding are verboten/taboo Cost Estimators Never quite independent, and often not skilled enough in communicating their operations research methods to decision-makers. Dont enjoy bucking the low risk engineers. EVMers Almost never crank assessments of risk into their EACs. The answer is wInsight, what was the question? Integrity - Service - Excellence 2 For Your Consideration The trouble with forecasting is that it's right too often for us to ignore it, and wrong too often for us to completely rely on it Dont rule out risk, even if it is difficult to characterize or quantify It is far better to foresee even without certainty than not to foresee at all. --Henri Poincare in The Foundations of Science, page 129. A pragmatic remark from one of the foundation builders of chaos theory. Integrity - Service - Excellence

3 Precision It is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits and not to seek exactness where only an approximation is possible.- Aristotle What our boy, Aristotle, really meant to say was: Do not pretend to know more than you do. Do not carry extra decimal places past the noise or uncertainty around your inputs You got risk and uncertainty let your COV reflect it Close enough for Government work And above all, dont verticate the S-Curve (CDF)! (attributed to LtGen Hamel, USAF) Integrity - Service - Excellence 4 Sorta Begs the Question With all this reluctance to recognize risk, what has been the outcome? Integrity - Service - Excellence 5 The Score Keeper March, 2006

GAO-06-391 DEFENSE ACQUISITIONS: Assessments of Selected Major Weapon Programs GAO assessed 52 systems that represent an investment of over $850 billion (comment: scope of the report) programs that began with immature technologies have experienced average research and development cost growth of 34.9 percent (comment: started with low TRLs which translate to technical problems and cost growth) Programs consistently move forward with unrealistic cost and schedule estimates, use immature technologies in launching product development, and fail to solidify design and manufacturing processes at appropriate points in development. (comment: overall summary of the problem, i.e., what Congress, OSD and SAF look for when evaluating a program) Integrity - Service - Excellence 6 The Score Keeper March, 2007 GAO-07-406SP DEFENSE ACQUISITIONS Assessments of Selected Weapon Programs GAO assessed 62 weapon systems with a total investment of over $950 billion, some two-thirds of the $1.5 trillion DOD plans for weapons acquisition Fully mature technologies were present in 16 percent of the systems at development start------the point at which best practices indicate mature levels should be present. (This means 84% of programs

had immature technology) Programs that began development with immature technologies experienced a 32.3 percent cost increase, whereas (This means 4/5ths of the programs grew by a 1/3rd) Those that began with mature technologies increased 2.6 percent. Integrity - Service - Excellence 7 The Score Keeper March, 2008 GAO-08-467SP DEFENSE ACQUISITIONS Assessments of Selected Weapon Programs Of the 72 programs GAO assessed this year, none of them had proceeded through system development meeting the best practices standards for mature technologies, stable design, or mature production processes by critical junctures of the program Integrity - Service - Excellence 8 HASC Authorization Language: As Dr. Phil says, Hows that working for you? In addition, the committee believes that over the past decade the acquisition of space systems has been plagued by cost overruns and schedule delays. The lack of enforcement of internal DoD procurement rules results in systemic problems leading to multiple space acquisition failures. These problems include reliance on immature

technology, overdependence on contractors for program management, and a lack of government systems engineering and cost analysis expertise. p.13. The committee is alarmed by the number of space acquisition programs experiencing unexpected cost growth over the past decade. Virtually every major space acquisition program has experienced or sits dangerously close to a Nunn-McCurdy breach. p.13 Integrity - Service - Excellence Historic Problem Full disclosure of system risk does not reach decision makers that they might make informed decisions Obvious risks frequently do not make the risk cubes Risk is downplayed at every level Filters applied at every level in the acquisition chain Decision Makers such as Service Acquisition Executives, Milestone Decision Authorities and OSD seldom can accept acquisition chain statement of risk and technical maturity

GAO reports suggest ~4/5ths programs certified as mature go forward with immature technology Integrity - Service - Excellence 10 Fearlessly Going Where No Man Has Gone Before Risk and Uncertainty Known + Historical database of somewhat analogous programs = Outcome Decision quality actionable risk assessments Integrity - Service - Excellence 11 AFSO21/D&SWS is Part of the Answer Funding Our Priorities We will fund transformation through organizational efficiencies, process efficiencies, reduction of legacy systems and manpower while sustaining GWOT and ongoing operations in support of the Joint Fight.

- Michael W. Wynne, SECAF I n t e g r i t y - S er v i c e - E x c e l l en c e 1 The Status Quo is Out AFSO21 -- The USAF will do less with less -- Do what is valued by our customers -- Employ tools and techniques smartly to reduce waste and nonnon-valuevalue-added work, to maximize value to the warriors I n t eg r i t y - S er v i c e - E x c el l en c e 4 4 Develop and Sustain Warfighting Systems (D&SWS) Integrity - Service - Excellence 12 Project Description and Background 3 Initiatives with the goal of institutionalizing one AF level process to manage investments in technologies to ensure they are mature for AF systems TD 1-12 Tech Maturity Yardstick Comprehensive Qualitative Criteria Tech Performance

Manufacturability Integrability Other ilities Better Assessment of Risk Improved Tech Forecasting TD 1-13 High Confidence Tech Transitions Early & complete lifecycle transition planning IPT Approach maximize coordination Stage/gated transition of technology Clearly defined entrance/exit Criteria TD 1-14 Identify & Prioritize Tech Needs Enterprise process to gather & prioritize tech needs Focus S&T on highest priority needs Game-changing Tech Pushinfluencing capability planning Integrity - Service - Excellence 13

What is Effective Risk Management? Level of Mitigation Excessive Risk Management Excessive Risk Management i sk R t ve en Effective Risk i t ec gem f f Management E na Ma Negligent Risk Management Significance of Risk Essentially Pareto economics approach to risk Asks the question where should I focus my limited resources during risk mitigation what risk(s) can I not tolerate? need decision support for an informed

in order to make informed decision Seldom possible to mitigate to zero Integrity - Service - Excellence 14 Characteristics of Decision Support Systems Large number of decision alternatives Outcomes or consequences of the decision alternatives are variable Alternatives evaluated on the basis of multiple criteria Criteria may be qualitative or quantitative Typically more than one decision maker (or interest group) involved in the decision-making process Decision makers have different preferences with respect to the relative importance of evaluation criteria and decision consequences

Decisions made are often surrounded by uncertainty Integrity - Service - Excellence 15 GAO-093SP Cost Estimating and Assessment Guide of March 2009 Integrity - Service - Excellence 16 Risk vs Notional Technology Readiness Levels Range of Program Cost Outcomes Over Time Risk Reduction AOA SDD Components and prototypes in relevant environment LRIP FRP Systems in operational environment Range of outcomes is reduced as we eliminate risk and some of the previously probable cost outcomes Cost risk levels

Very High TRL 1 2 3 4 High Med-High 5 6 Med Med-Low 7 8 Low 9 Risk is retired as a program progresses and technology matures Integrity - Service - Excellence Return +1 Backup 17 COMPONENTS OF RISK ANALYSIS

Risk assessment Identifying the risk at hand that could materialize as an issue Bound or quantify level of potential harm Risk communication Inform others in a way that helps them to make optimal decisions Requires risk analysis be viewed from the beginning as a decision support tool Risk assessment must be communicated in such a way that it is actionable Risk management (SE an any other name) Action to take once risk is identified Risks can be managed on many different levels Integrity - Service - Excellence 18

Need for RI3 Process What is Missing? 1 Technology Readiness Assessments (TRAs) are necessary but not sufficient to capture risk Natural tendency for advocates to downplay risk RI3 sheds additional light on areas that seem to have traditionally been underrated in terms of risk interestingly, literature from the field of cognitive psychology generally suggests that people often have difficulty in characterizing the relative risks of various activities appropriately (possibly due to group- think) thus resulting in underestimation of their effects.1 1 Risk Identification, Integration & Illities Guidebook ver 1.2, 15 December 2008, p.5 Integrity - Service - Excellence 19 Need for RI3 Process What is Missing? 2 Common approach to identifying program risk

Consistent Validated against historical programs Visible to all levels in the acquisition chain Automated hand-offs (export) to systems engineering and program information reporting systems Active Risk Manager (ARM) Probability of Program Success (PoPS) Of use to program management, cost estimating, earned value management, logistics and auditing communities Integrity - Service - Excellence 20 Surveyed Globe for Good Ideas Efforts surveyed across DoD, other agencies, internationally, universities, corporate world NASA-originated AD2 methodology viewed favorably by

members in OSD AT&L and SAF AQR British Ministry of Defence provided good input British System Readiness Levels (SRLs) are used in conjunction with TRLs Also in conjunction with a full-blown risk analysis assessment Integrity - Service - Excellence 21 Relationship to OSD Checklists Various OSD checklists are available on the DAU website TRA deals primarily with setting up for a TRA, not how to conduct a TRA PRR in discussion with MRA team PDR, CDR Only Navy NAVAIR appears to use the checklists Observations on checklists Checklists are excellent sets of questions Checklists are much broader in scope than RI3 questions, but lack the depth Integrity - Service - Excellence 22

Bout Those Dreaded Checklists Many checklists are too long for day-to-day usage by a Program Office NAVAIR checklist 800+ questions: if everything is important, then nothing is important Traditional checklists are 1-dimensional Many checklists are like IG inspection clipboard checklists Do you have a Budget? End of subject!! Example of multi-dimensional question (RI3-style approach) Do you have a Budget? Is it adequate? Is the total amount sufficient? What is the confidence level of your budget? Is the total amount phased correctly? Is the total amount sufficient by Appropriation? Are there pending changes that would impact the budget? Are there risks not captured in your budget? Integrity - Service - Excellence 23 ilities Threads RI3 Development Team down-selected to the following list* Design Maturity and Stability (stability of requirements)

Scalability & Complexity Software Integrability Testability Reliability Maintainability Human factors People, organization, & skills List was driven by observations of past program problems and is consistent with International Counsel on Systems Engineering (INCOSE) standards Integrity - Service - Excellence 24

Development Philosophy Standard AF Risk Cube Leverages off of standard AF risk processes Risk Cube 5 Likelihood Definitions 4 Consequence Definitions 3 DoD Guide 2 PROBABILITY OF OCCURRENCE 1 1%-20% LEVEL LIKELIHOOD

1 Not Likely 2 Low Likelihood 21%-40% Likely 1 2 41%-60% Highly Likely 61%-80% Near Certainty 81%-99 % 3 Proposed AF Definition 4 1 Minimal or no consequence to technical performance Minimal consequence to technical performance but no overall impact to the program success. A successful outcome is not dependent on this issue; the technical performance goals will still be met. 2

Minor reduction in technical performance or supportability, can be tolerated with little or no impact on program Minor reduction in technical performance or supportability, can be tolerated with little impact on program success. Technical performance will be below the goal but within acceptable limits. 3 Moderate reduction in technical performance or supportability with limited impact on program objectives Moderate shortfall in technical performance or supportability with limited impact on program success. Technical performance will be below the goal, but approaching unacceptable limits. 4 Significant degradation in technical performance or major shortfall in supportability; may jeopardize program success Significant degradation in technical performance or major shortfall in supportability with a moderate impact on program success. Technical performance is unacceptably below the goal. 5 Severe degradation in technical performance; Cannot meet KPP or key technical/supportability threshold; will jeopardize program success Severe degradation in technical/supportability threshold performance; will jeopardize program success; or will cause one of

the triggers listed below 5 3 4 5 Likelihood Definitions Consequence Definitions Integrity - Service - Excellence 25 Why the RI3 New Colors and Ratings Make Sense Current Risk Cube 5 4 3 2 1 1 2 3 4 Consequence 5 I am a good program manager running

a good program I am not red Bad things happen to program managers who show red Bad things happen to programs that dont quickly move out of the red Tendencies Move all red except for the (5,5) outside of program control down to yellow When you are green you do not have to worry about risk There is often a tendency for persons outside a program to believe that if the CTEs of a system are all at TRL 6, then the technical risks related to those technologies have been paid down. Unfortunately, this is not the case. RI3 Guidebook, p.2 Integrity - Service - Excellence 26 Ratings versus Colors: Proposed But Not Yet Accepted These ratings could be thought of as creating 2 intermediate colors Red/Yellow (4 ratings) Reduces tendency to try to avoid high numbers because theyre red Green/Yellow (2 ratings) Reduce items that get ignored because theyre green

RI3 Ratings Original Colors 5 5 2 3 4 4 5 4 4 2 3 3 4 4 3 3 2 2

3 3 4 2 2 1 2 2 3 3 1 1 1 1 2 2 3 1 2 3

4 5 1 2 3 4 5 Integrity - Service - Excellence 27 Some Sample Questions: Integrability Are there interactions / integration issues that could be affected by proprietary or trust issues between/ among suppliers? Have key sub-systems, at whatever level of readiness (breadboard, brassboard, prototype), been tested together in an integrated test environment and have they met test objectives? Software Are personnel with development-level knowledge of the existing, reused software part of the new software development team? Maintainability Is modeling and simulation used to simulate and validate maintenance

procedures for the unit under test and higher levels of integration? Explanatory discussion with potential best practices on each question are included in RI3 guidebook and Excel-like worksheet/tool Questions are technical and shy away from programmatic Total of 101 question covering the 9 illities -- Answer only those that are appropriate for the Unit Under Evaluation (UUE) Integrity - Service - Excellence 28 illity Preamble 2.0 Scalability & Complexity The issues of scalability and complexity are subtle issues that are often overlooked (or aspects are overlooked) with disastrous consequences. Both are complicated issues that have many dimensions. In the case of scalability, the issues are often associated with size, and weight in both directions, e.g. going from meter-scale to 10s of meters-scale; going from micron-scale features to nano-scale features; extrapolating to super light-weight systems or to super high-density systems. In the case of software/communications, the scale may require going from regional to global. All of these scale changes have tremendous impacts on manufacturing, integration, testing and operation. In the case of complexity itself, the primary problem is the lack of clarity in the definition of complexity itself. What makes a system complex? For instance with respect to the program itself: Are multiple contractors involved? Are multiple Uniformed Services involved? Are multiple government agencies involved? Are multiple non-US government agencies involved? Are multiple funding sources involved? Are there critical dependencies with other programs (e.g. cryptographic equipment)?

Integrity - Service - Excellence 29 Question Amplifying Instructions 1.0 Design Maturity and Stability 1.01 Are hardware and software design requirements stable (have they been finalized)? Changing requirements are a significant factor in cost and schedule overruns. Continually changing requirements indicate that the design is in a state of flux. This in turn leads to a high probability that critical aspects will be overlooked, or that designs will become obsolete before they are implemented. Requirements creep is also a cause for scrap and rework, further increasing costs. It is critical to freeze requirements (or put them under strong configuration control) and to obtain the necessary buy- in from the customers before continuing to advanced stages of design. New requirements can be included as part of a future, pre-planned improvement. Integrity - Service - Excellence 30 RI3 Questions Perception by Various Communities Engineers feel RI3 questions are systems engineering questions Program Managers feel RI3 questions are questions they want to ask their staff Cost Estimators feel RI3 questions are risk questions for their cost models in widespread use at Eglin AFB

Auditors have responded that RI3 questions are good questions to ask to ascertain whether risk had been addressed directly Note: AFAA auditors at Eglin AFB have already incorporated some of the RI3 questions into their audit questions list EVM practitioners in general have not yet responded with any enthusiasm to the concept of risk, but NDIA has included risk in their ANSI standards for EVM Reality RI3 questions are appropriate for anyone who deals with risk Integrity - Service - Excellence 31 Usage of RI3 to Feed Risk Management Processes Risks RI3 Guidebook Questions: Integration ilities 5 Tool 4 3

2 1 1 2 3 4 Risk Management Active Risk Manager (ARM) compatible file Step 2. Risk Identification 5 Consequence Tool Similar output for cost estimation being investigated Additional Summary Displays Integrability Testability Tool

Reliability Maintainability PoPS Human Factors Scalability & Complexity People, Organization, Skills Design Maturity & Stability 0 1 2 3 4 5 Integrity - Service - Excellence 32 How to Begin RI3 Methodology Project XYZ System A System B System C CTE Non-CTE Subsystem a

Subsystem b Component Subsystem c Component Component Start with system level gross evaluation (top-down) Break down into subsystems, note Critical Technology Elements (CTEs), and evaluate TRL at appropriate level To assess integration and ilities, must evaluate CTEs + units that interface with CTEs, even if they are not CTEs themselves Then, proceed back up tree as appropriate Integrity - Service - Excellence 33 RI3 Signature Evolves over Time RI3 can be used to support risk identification both in support of milestones as well as pre-MS A activity Input to PoPS Risks could actually increase as more knowledge is obtained; however one expects risk to decrease as a program matures

Integrability Integrability Testability Testability Reliability Reliability Maintainability Maintainability Human Factors Human Factors Scalability & Complexity Scalability & Complexity People, Organization, Skills People, Organization, Skills Design Maturity & Stability Design Maturity & Stability 0 TRL 1 Basic Principles Observed

TRL 2 Concept Formulati on 1 TRL 3 Proof of Concept 2 3 TRL 4 Breadboard in Lab 4 5 TRL 5 Breadboard in Rep Environment 0 TRL 6 Subsystem Prototype in Rep Environment

1 2 TRL 7 System Prototype in Ops Environment Integrity - Service - Excellence 3 TRL 8 System Qual 4 5 TRL 9 Mission Proven 34 A Glimpse at a Potential RI3 Tool Instantiation Software Development S1 Will engineering hardware models or prototypes be available for software testing in the appropriate time frame? S2

Have mechanisms or forums been established to ensure appropriate interactions between simultaneously working software development teams? S3 Have mechanisms or forums been established to ensure appropriate interactions between the simultaneously working hardware and software development teams? S4 Has the hardware/ software interaction been simplified to the maximum extent? S5 Has the interoperability of reuse/OTS software with both internal and external system elements been considered? S6 Has the ability of reuse/OTS software to provide required safety, security, and privacy been confirmed? S7 Has the ability of reuse/OTS software to isolate faults in the integrated reuse/OTS been confirmed? Integrity - Service - Excellence 35 Description of RI3 Historical Test Stayback Team Historical Program The Mushrooms Documentation up to PDR Only

RI3 Guidebook, Tool Review Materials Full Historical Documentation Predict Risks al ar t i P t r ac f o t x E In Review Full History Omniscient Team The Know-It-Alls Full Team Interview Program Office, Compare Predictions to Truth Integrity - Service - Excellence Results Test Metrics RI3 Revisions 36

Results: RI3 Historical Test Completed Nov 21, 2008 Realized Risks / Issues Number Correctly Predicted by Team Could be Predicted by Program Office Escaped Prediction (Type 1 Error) 22 13 6 3 RI3 v1.0 Tool could have predicted 86% of Issues Modified tool to be more perceptive Correctly Predicted by Team Team members predicted a risk, which in fact became an issue Could be Predicted by Program Office If the program personnel had the RI3 tool available, issue that arose

would likely have been predicted as a risk by RI3 Team did not predict risk in exercise due to lack of information Escaped Prediction Questions did not yet capture an issue that arose Integrity - Service - Excellence SECAF Pathfinder Programs SECAF established High Confidence Criteria Pathfinder programs Initiative led by Col Kevin Keck and Col (s) Fred Gregory Eglin AFB Pathfinders Small diameter Bomb II Hard Target Void Sensing Fuze Usage of RI3 methodology is

part of the effort Risk identification Recommend usage of assistance from TEAS and/or AAC/EN personnel Integrity - Service - Excellence 38 Notional Example Application Problem determine the engineering risks and cost impacts of two competing approaches Using RI3 Calculator provide risk metrics for input to cost and EVM models Develop estimate/EAC for most risky approach Integrity - Service - Excellence 39 Notional RI3 Summary of Two Alternatives

Notional Micro Munition Approach 1: likely to make changes Driven by dispersion patterns May drive guidance design Possible change seeker geometry Approach 2: considering cost changes Switching to explosive for warhead Simplifying heritage Ad hoc net Primary Risk/Integration drivers Changes to CDD Predator weapons Thermalactual beyond tested values MIL SPEC exceded Vibrationalnot characterized Prime developed models (no IV&V) Limited testing over threat set Payload 1 Design Maturity & Stability 2 Scalability & Complexity 3 Integrability 4 Testability

5 Software 6 Reliability 7 Maintainability 8 Human Factors 9 People, Organization & Skills 2 3 2 2 0 0 0 0 2 4 4 4 4 0 0 0 0 0 Guidance & Control 1 Design Maturity & Stability 2 Scalability & Complexity 3 Integrability 4 Testability 5 Software 6 Reliability 7 Maintainability 8 Human Factors 9 People, Organization & Skills Check 4 3 2

4 2 2 0 2 3 3 2 3 2 2 0 0 2 Structure 0 0 3 2 0 0 0 0 2 0 0 2 2 0 3 0 0 0 1 Design Maturity & Stability

2 Scalability & Complexity 3 Integrability 4 Testability 5 Software 6 Reliability 7 Maintainability 8 Human Factors 9 People, Organization & Skills Integrity - Service - Excellence 40 RI3 Results Feed and Document Cost Risk Model Payload 1 2 3 4 5 6 7 8 9 Design Maturity & Stability Scalability & Complexity Integrability Testability Software Reliability Maintainability Human Factors People, Organizatio n & Skills 2

3 2 2 0 0 0 0 2 4 4 4 4 0 0 0 0 0 Guidance & Control 1 2 3 4 5 6 7 8 9 Design Maturity & Stability Scalability & Complexity Integrability Testability Software Reliability Maintainability Human Factors

People, Organizatio n & Skills Check 4 3 2 4 2 2 0 2 3 3 2 3 2 2 0 0 2 Structure 1 2 3 4 5 6 7 8 9 Design Maturity & Stability Scalability & Complexity Integrability Testability Software Reliability

Maintainability Human Factors People, Organizatio n & Skills 0 0 3 2 0 0 0 0 2 0 0 2 2 0 3 0 0 0 9 scores 3 scores 1 overall risk rating Integrity - Service - Excellence 41 Risk Ratings Around Point Estimate Of Work Remaining Boundary Interpretation for expert input Focus on Uncertainty Captured

Experts never provide total range of outcomes Default is 70% of the range Reasonable to us 90% for catalog or commodity items Integrity - Service - Excellence 42 Subjective Uncertainty Bound Interpretation continued Adjusted for skewness AF CRUH p.vii 43 Integrity - Service - Excellence 43 Risk Adjusted Estimate and Determination of Skewness R i s k LM MH M Split between the tails of the triangular distribution Integrity - Service - Excellence 44 Risk, Boundary and Skewness Adjusted Expected Value

Resulting distribution Best Case Most Likely Case Worst Case $474 $600 $1,295 Expected Value $790 Integrity - Service - Excellence 45 RI3 Risk Adjusted Cost Confidence Curve Integrity - Service - Excellence 46 So, Where Does EVM come in? If you have actuals, why estimate what you already know? Focus on what you know and estimate what you dont Necessary tenets of EVM

TENET 1: Acquisition Program Baseline (APB) must reflect program/ product reality Doesnt always happen TENET 2: Categorization of costs must be accurate -- Dont remember the last time when NR and REC and SEIT/PM were accurate and consistent across programs TENET 3: Program schedules must be executable Not until late in program will this be known as many schedules are not stable until after CDR Integrity - Service - Excellence 47 Traditional EVMS Performance Factors: Invalid For Predicting The Future Reality check We do know something about the past We do know something about the future Real knowledge, not assumption We do know something about the program Risk is retired as work packages are completed

The future will not be like the past, except by coincidence We can focus on the future with some degree of certainty You can bet on it! The new reality The best available information concerning the future lies in the collective judgments of those closest to the actual program 48 Integrity - Service - Excellence Change of Focus and Leap of Faith Contract Performance vs Future Risk Factor Risk and Uncertainty How much have I gotten for my dollar? (CPI) Sunk Cost How many times the budgeted cost for my remaining work will I have to pay? Future Cost Integrity - Service - Excellence

49 Performance Factor Traditionally we use CPI or SCI based performance factors that traditionally get no where close to the eventual cost of the program In the EVM literature, D.S. Christensen took EAC sci from Worst Case, to Most Likely, to Best Case -- where is the EAC reality in that? Real interest is in the risk associated with work remaining Performance EAC CumACWP ( BAC BCWP ) x CGPF Factor Actuals Remaining Work Sunk Cost Future Cost PMI Project Management Body of Knowledge (PMBOK) uses Planned Value and Earned Value for BCWS and BCWP Integrity - Service - Excellence 50 Performance Factor Continued

Not difficult to go back into CPRs or DCARC 1921s of historical programs and compute risk factors Air Force Cost Risk and Uncertainty Analysis Handbook lists default Most Likely estimate multipliers by risk rating Big Time Caveat In example following we will assume high schedule/Technical Risk with a multiplier of 2.0 1.Always check any multiplier and risk rating system against growth of historical programs in the commodity area of interest. 2.Un-calibrated multipliers can be dangerous. 3.Kids, Dont try this at home! Integrity - Service - Excellence 51 Performance Factor Continued Example: High Risk assessed on the budgeted $200M work remaining Basic Triangle

1.0 x $200M = $200M Most Likely Case 2.0 x $200M = $400M Worst Case (High Risk) ($180M+$200M+$400M)/3 = Expected Value EAC = $260M Bound Interpretation with 70% Uncertainty Captured and corrected for Skewness (3% to left tail, 27% to right tail) .785 x $200M = $157M Best Case 1.0 3.145 x $200M = $629M Worst Case .9 x $200M = $180M Best Case x $200M = $200M Most Likely Case ($157M+$200M+$629M)/3 = Expected Value EAC = $329M $329M/$200M = 1.645 = Performance Factor Integrity - Service - Excellence 52 Acquisition End Game

Low Risk For Full Rate Production We are not at Low Risk Once we are through our pre-EMD risk reduction phase Once we enter EMD Once we start testing prototypes Once we start LRIP We are at Low Risk only after we have a proven design supported by OT&E findings We have a proven production line ready to enter FRP Until then, there are a lot of bad things that can happen, probably will happen, and will have cost and schedule

impacts Integrity - Service - Excellence 53 Now that you have been drug through the mud Time for some initial conclusions Integrity - Service - Excellence 54 54 RI3 Conclusions Air Force Risk Identification RI3 Process is now an Air Force Best Practice Covered in AF Pamphlets and guides SecAF Directed for all Air Force High Confidence programs Being required for all Air Armament Center programs

RI3 is being reported by OSD to Congress as the Best of the best risk identification practices RI3 is being looked at by GAO Implies You may see this again Integrity - Service - Excellence 55 TD-1-14 Stagegating September 2006 GAO report: GAO recommends that DOD strengthen its technology transition processes by developing a gated process with criteria to support funding decisions; expanding the use of transition agreements, GAO Report to Congressional Committees, BEST PRACTICES Stronger Practices Needed to Improve DOD Technology Transition Processes, September 2006. GAO-06-883 Initiative focuses on Technology Transition process Ensure early and complete life-cycle transition planning Create a common understanding of the technology transition processes to be applied at all life cycle stages Initiative goal is improved transition success

Improved planning using exit criteria enhances probability and speed of the transition, increasing confidence of acquisition programs REDUCE PROGRAMMING RISK! Key aspect is ensuring the right people are involved earlier for increased collaboration between researcher, acquisition organization, and stakeholders Integrity - Service - Excellence 56 Headbone Connected to the Neckbone Moving from the Ruler to the Gate Guard RI3 Stagegating Historical situation (only slightly tongue-in-cheek) Lab Matures technology to a TRL 4 Throws it over the fence as a TRL 6 Program Manager Matures the TRL 4 to TRL 5 Sells it to MDA as a TRL 6 MDA Concurs on Milestone passage Program Manager runs into technology problems finding it difficult to invent on schedule Declares Nunn-McCurdy breech GAO Finds one more time we went ahead with immature technology and excessive risk Integrity - Service - Excellence 57 Scope the Initiative September 2006 GAO report: GAO recommends that DOD strengthen its technology transition processes by developing a gated process with criteria

to support funding decisions; expanding the use of transition agreements, GAO Report to Congressional Committees, BEST PRACTICES Stronger Practices Needed to Improve DOD Technology Transition Processes, September 2006. GAO-06-883 Initiative focuses on Technology Transition process Ensure early and complete life-cycle transition planning Create a common understanding of the technology transition processes to be applied at all life cycle stages Initiative goal is improved transition success Improved planning using exit criteria enhances probability and speed of the transition, increasing confidence of acquisition programs REDUCE PROGRAMMING RISK! Key aspect is ensuring the right people are involved earlier for increased collaboration between researcher, acquisition organization, and stakeholders Integrity - Service - Excellence What is New? Formalized process to develop the strategy to mature and transition a new technology

List of detailed activities needed for technology maturation Mechanism to ensure a robust execution of the strategy: Stage-Gate process A Stage is where the activities occurs the team completes key activities (technology and programmatic) to advance the project to the next gate and focuses on the changing roles and responsibilities A Gate is a decision point on whether a project is a go, nogo, re-directed or put on hold (TRL based / driven) The decision is based on EXIT CRITERIA for each gate Integrity - Service - Excellence 59 What is New? A formalized process, the mechanism (stage-gate criteria) and detailed activities and milestones necessary to transition from phase to phase Stage-Gate Process Phases Stage-Gates Separate Feasibility / Formulation Complete

Team Ready > Mgmt Approve > Next Phase N Proof of Concept Readiness Level 3 N Mgmt Approaves Refinement Breadboard Readiness Level 4 N N Transition Integration N Development Brassboard Readiness Level 5 N

Mgmt Approaves N Mgmt Approaves Prototype Relevant Environ Readiness Level 6 Mgmt Approaves N Integrity - Service - Excellence 60 To Be Process Tech Development & Transition Strategy (TDTS) Replaces the TTP TDS is subset of TDTS required at Milestone A As program progresses TDTS Morphs to LCMP Owner: Acquisition PM Tech Development & Transition Strategy (TDTS) Replaces TTP, but a gated approach defining depth required at each phase. Integrated Strategy (Technology Development and

Acquisition) Example: As team approaches: MS-A (TRL-4) Gates/checklist ensures TDS is complete MS-B (TRL-6) Gates/checklist ensures LCMP is complete Subset Becomes Tech Development Strategy (TDS) LCMP (Public Law 107-314, Sec 803) Acquisition Approach Supporting Rationale R&D Strategy Performance Goals CSP and Spirals Describe Tech Demo CSP and Exit Criteria Develop Test Plan Goal / Exit Criteria Ensure Maturity Level Exec Summary Mission/Rqmts Program Summary Program Mgmt Business Strategy

Risk Mgmt Cost and Performance Mgmt Test Approach Product Support Concept Integrity - Service - Excellence Tool Description - TurboTPMM Facilitates development of the Transition Strategy for Tech Maturation and Transition USAF added graphic user interface to model The TurboTPMM S/W tool features: Automates the stage-gate process Easy to use, walks user through the process Turbo-tax like graphic user interface

Questions aligned with acquisition framework Ensures application of Systems Engineering Follows Project Management fundamentals DAU also Collaborating with TurboTPMM Designed DesignedtotoAsk Askthe theRight RightQuestion Questionatatthe theRight RightTime Time Integrity - Service - Excellence TurboTPMM Baseline Planning 63 Integrity - Service - Excellence Where Do You Get these New Fangled Deals? RI3 Calculator and Guide are available online at Air Force Institute of Technology http://www.afit.edu/cse/page.cfm?page=164&sub=110

TDTS Guidebook: Available on DAU Acquisition Community Connection (ACC) https://acc.dau.mil/CommunityBrowser.aspx?id=314696&lang=en-US TurboTPMM: Gunter AFB is hosting alpha version of TurboTPMM URL: https://www.tdr.gunter.af.mil/GCSS-SBX031 UserName: TPMM.testuser Password: [email protected][email protected]#$ Integrity - Service - Excellence 64 Challenge to the Various Communities No more group think Recognize enthusiasm and hope can become blinders Anticipate risk -- it is real, it is there Communicate the risk to the Program Managers, PEOs and MDAs Work with the Engineers, Cost Estimators, Auditors

They will help you communicate risk status Better yet, they can help document your risk management plans are working Work with the EVMers Dont gotta like em, but they have a direct line to PM Communicate the risk posture They, too can help document your risk management plans are working Integrity - Service - Excellence 65 Challenge to the EVM Community Fundamental challenge to the EVM community is whether it will embrace risk assessments as a general EVM practice Will the EVM community join with the Systems Engineering and Cost Estimating communities in utilizing risk assessments to generate performance factors?

NDIA introduced concept of risk to EVM in 2007, but the EVM community as a whole has not embraced it PREDICTION: RISK IS COMING TO EVM Integrity - Service - Excellence 66 Challenge to Cost Estimating Community Work with the Systems Engineers they need your help Take an engineer to lunch Have an engineer take you to lunch Translation work jointly look for synergy do risk together Attend engineering sufficiency reviews Have engineers attend cost sufficiency review Participate in each others fact finding trips

Use the RI3 questions as cost questions Ideally, SEs, PMs, Costers, EVMers and Auditors will embrace the RI3 questions as their own Did I say you might see RI3 again? Karen Ritchy, GAO, is having a team look at the RI3 questions to see which would make good audit questions Integrity - Service - Excellence 67 EVM Risk Metric I propose a new risk metric based on the concept that Estimates from program office are less than from Component (service)/agency headquarters Estimates from service/agency headquarters are less than OSD CAPE Estimates from CAPE are less than actual costs found by audit agencies upon completion of program

Essentially a comparison of traditional EVM with Independent Cost Estimate May give too much credence to independent estimates (Program Office Estimate Technical Baseline Estimate) (Independent Cost Estimate Independent Cost Estimate TBE) Integrity - Service - Excellence 68 Challenge to the Acquisition Community Change Follow a formal risk identification process Dont just assume risk away Stop sanitizing risk Try your culture telling the truth to the decision makers Follow a formal technology development stagegating process Dont go forward to Milestone B with immature technology Integrity - Service - Excellence

69 The End 70 Integrity - Service - Excellence Questions and Point of Contact John Cargill YD-1515-03 Air Force Cost Analysis Agency Operating Location Eglin AFB, FL 850-883-3460(w) 703-371-0891(c) [email protected] Integrity - Service - Excellence 71 BACKUP SLIDES Integrity - Service - Excellence 72 Final Note About NON-LINEAR TRLs Going from System/subsystem model or prototype demonstration in a relevant environment TRL6 System prototype demonstration in an operational

environment TRL7 Is much harder than Component and/or breadboard validation in relevant environment TRL5 AFCAA System/subsystem model or prototype demonstration in a relevant environment TRL6 Integrity - Service - Excellence 73

Recently Viewed Presentations

  • The Dimensions of Meaning - Universitas Negeri Yogyakarta

    The Dimensions of Meaning - Universitas Negeri Yogyakarta

    Syntagmatic…the relation of the lexeme with other lexemes with which it occurs in the same phrases or sencences. Syntagmatic---the mutual association of two or more words in a sequence so that the meanings contribute to the meaning of the larger...
  • 12.215 Modern Navigation Thomas Herring (tah@mit.edu), MW 11:00-12:30

    12.215 Modern Navigation Thomas Herring ([email protected]), MW 11:00-12:30

    Title: PowerPoint Presentation - 12.215 Modern Navigation Author: Thomas Herring Last modified by: Thomas Herring Created Date: 9/6/2002 1:51:31 AM
  • Versologie UDSLB 1. Ver  Ver  zkladn rytmick a

    Versologie UDSLB 1. Ver Ver zkladn rytmick a

    Kdo vypoví, jak hoří, málo hoří! 8.1.2 Příklad normovaných typů strof W. Shakespeare: Sonet 66 Znaven tím vším, já chci jen smrt a klid, jen nevidět, jak žebrá poctivec, jak pýchou dme se pouhý parazit, jak pokřiví se každá čistá...
  • The Importance of Critical Thinking

    The Importance of Critical Thinking

    The Legal Environment of Business A Critical Thinking Approach 5th Edition Nancy K. Kubasek Bartley A. Brennan M. Neil Browne * * * * * * THE LEGAL ENVIRONMENT OF BUSINESS CHAPTER 1 Critical Thinking and Legal Reasoning * Legal...
  • U.S. General Services Administration FEDERAL ACQUISITION SERVICE Customer

    U.S. General Services Administration FEDERAL ACQUISITION SERVICE Customer

    gsaglobalsupply.gsa.gov. Source for all your products needs. Requisition based ordering. Guaranteed full compliance. GSA is accountable for everything from order placement through delivery and billing. GSA Global Supply is the trusted and reliable supply system providing a vast selection of...
  • ELT materials development - GEORGE E.K. WHITEHEAD

    ELT materials development - GEORGE E.K. WHITEHEAD

    Grammar Translation method. It originated from the practice of teaching Latin; in the early 1500s. It was one of the most commonly used methods until the 1950's. It is still common in classrooms today (especially in Korea) Students learn grammatical...
  • Definition of your CRM and online strategy

    Definition of your CRM and online strategy

    ----Summary---- Total price is #19 for 4 ticket(s) to Shrek At Wimbledon Odeon On 9 March at 19.38 Tickets can be collected at (venue) on day of show. By selecting 'Buy Now' you agree to the …
  • Document #: TX005846 The Prophets, Jesus Christ, and

    Document #: TX005846 The Prophets, Jesus Christ, and

    The Book of Isaiah contains not only the words of the original Isaiah, but also the words of some of his later followers. The Book of Isaiah has many prophecies that were fulfilled in the life, Death, and Resurrection of...