Evaluation Procedures Brian Lines, Ph.D Assistant Professor University
Evaluation Procedures Brian Lines, Ph.D Assistant Professor University of Kansas Dept. of Civil, Environ. & Arch. Engineering [email protected] 785-864-6503 Expertise-Driven Project Delivery (XPD) CONTRACT AWARD 0 SCOPE PROCUREMEN DEVELOPMEN T & T SELECTION PRE-AWARD CLARIFICATION PHASE 1 - SELECTION
1 PERFORMANCE METRICS Evaluation Procedures 1. 2. 3. 4. 5. Blind Evaluations Keep it Fair Streamlined Scoring Keep it Simple Individual Evaluations Avoid Groupthink Offer Detailed Debriefings Plan Before You Sign Take-Away Tools 1-5-10 Evaluation Scale Evaluation Scoresheets Evaluation Matrix Debriefing Template PHASE 1 - SELECTION 1
Blind Evaluations: Keep it Fair PHASE 1 - SELECTION 1 Evaluation Concerns How are you deciding who wins? Do you have a favorite? Is it Fair? Is it confusing? Will you let me optimize my profit? PHASE 1 - SELECTION 1 Blind Evaluation Make it Fair The evaluated proposal documents MUST NOT contain any names that can be used to identify who the Proposer is. (company names, personnel names, project names, or product names) PHASE 1 - SELECTION
1 1 Improving Fairness Evaluated Proposals Non-blind evaluation process allows for bias (whether intentional or unintentional bias) ractors t n o C h Smit Inc. Fair evaluation process is blind/anonymous (evaluators do not know firm names) 103 m
r i F : e Cod PHASE 1 - SELECTION 1 Variation in Evaluation Scores Scope Definition vs. Proposal Variation 27% 25% 25% 24% A/EScope DB/CMAR Definition Poor DBB Construction
Moderat e High 18% 17% 15% 14% 13% 8% Cost Technical Proposal Interviews N = 347 projects (1,850 PHASE 1 individual - SELECTION 9%
7% Past Performance Evaluation Weights 1 Evaluation Criteria Weight Evaluation Risk Assessment 15-20% Committee Score* Value Assessment 10-15% Committee Score* Past Performance Surveys
05-10% Numerical* Interviews 30-40% Committee Score Cost / Financial Proposal 15-30% Numerical* TOTAL 100% *Red Text indicates blind evaluation PHASE 1 - SELECTION 1 Case Study
GMP Management Cost Control Approach Similar Experience Contract Exceptions Goals KPI Insurance Safety Plan Average Size of Safety Plan Average Size of Safety Plan 356Pages Pages 356 Time Spent Evaluating Safety Plan Time Spent Evaluating Safety Plan 2.5Weeks Weeks 2.5 Safety Plan Score (Standard Deviation) Safety Plan Score (Standard Deviation)
1 Streamlined Scoring: Keep it Simple PHASE 1 - SELECTION QUESTION Evaluation Scales What types do you normally use? PHASE 1 - SELECTION 1 1 Typically 30-45 minutes once all details are included Sample Evaluation Team Training Presentation RFP for Financial Software System PHASE 1 - SELECTION 1 Agenda Big picture
What am I evaluating first? (blind proposals) What comes next? (interviews + software demos) Evaluation procedures and logistics Evaluation schedule PHASE 1 - SELECTION Big Picture Goal of evaluations = select the best 1 Company Project Team Software system Role of evaluators = dont make a decision. Vendors must differentiate themselves form their competitors Focus on info / level of detail needed to make a SELECTION There will be additional clarifications after the selection. Selected vendor will be brought into Pre-Award Clarification Evaluators can ask any additional questions SMEs can be brought in to confirm any details Additional software verification / demo can be requested
PHASE 1 - SELECTION What am I evaluating first? step1 = Anonymous Proposals Evaluation Team Weights Execution Methodology (2pg) 10% Risk Assessment Controllable (2pg) + Non-Contr 20% (2pg) 1 10% Value Assessment (2pg) Evaluation team provided with Functional Response & Technical = 40% total
Response? Weights Procurement 15% Cost Proposal 2.5% Past Performance Surveys for Project Teams 2.5% Past Performance Surveys for Software = 20% total PHASE 1 - SELECTION What comes next? step2 = interviews & software verifications
1 Shortlisting procedures If necessary, a shortlist will be determined Shortlisted firms will be notified and invited to the interviews + verifications Evaluation team provided with Functional Response, Technical Response, Vendor Project Team, List of Subs (if any), and all References (surveys) Weights 20% Evaluation Team Interviews with each Project Team Software Verification with each Vendors system PHASE 1 - SELECTION 20% = 40% total 1
Evaluation Summary Response Item Proposal Anonymous Evaluation Form Response Weight Signature Page 1 -- -- Project Team 2 -- -- List of Subcontractors 3
10% Past Performance Surveys for the Project Team 7 -- 2.5% Past Performance Surveys for the Software Product 8 -- 2.5% Proposal Assumptions 9 -- -- Cost Proposal
10 -- 15% Comp RFP Requirements 11 -- -- Software Verification (potentially short-listed) -- -- 20% Interviews of the Project Team (potentially short-listed) --
-- 20% PHASE 1 - SELECTION 1-5-10 Rating Scale High performance range Average level of expertise Low Performance Range 10 9 8 7 6 5 4 3 2 1 ABOVE AVERAGE
(clear justification, differentiation above the average, demonstrates high performance capability, definitely want to interview/demo) 1 AVERAGE (difficult to differentiate, no significant difference, acceptable to meet requirements, could interview/demo) BELOW AVERAGE (clear differential below the average, does not demonstrate high performance capability, dont need to interview/demo) PHASE 1 - SELECTION Evaluation Scoresheets PHASE 1 - SELECTION 1 Procurement Schedule Internal Activity
DEADLINE TO SUBMIT PROPOSALS Distribute Anonymous Proposals to Evaluation Team Send individual scores to Procurement by 12pm noon Consensus Scoring Meeting Short-List Notifications INTERVIEWS & SOFTWARE VERIFICATIONS Scoring of Interviews/Software Verifications Initiate Steering Committee Meeting / EPMO Process NOTIFICATION TO SELECTED VENDOR PHASE 1 - SELECTION Date 1 Individual Evaluations Avoid Groupthink PHASE 1 - SELECTION 1
QUESTION 1 Evaluation Procedures How should the evaluation process be conducted? Group vs. Consensus vs. Individual Evaluation Scores? PHASE 1 - SELECTION Evaluation Committees 3-7 individuals Knowledge of the project and scope Represent various perspectives Beware of potential bias Preferences for certain vendors Previous work history Overly technical PHASE 1 - SELECTION 1 Evaluation procedures & logistics Evaluation procedures:
1 Anonymous proposals Focused on Clear Differential Independent Prohibit individuals from lobbying others Treat each evaluation category separately (rather than discussing an overall ranking) Use an Evaluation Matrix to compile the total points PHASE 1 - SELECTION 1 Simple Scoring Methodolgy NO 1 2 3 4 5 6
CRITERIA Proposal Cost Interview Rating NTR Rating TC Rating VA Rating PPI Rating WEIGHTS 250 350 150 100 100 50 RAW DATA FINAL POINTS Vendor A Vendor B Vendor C $ 57,000 $ 65,000 $ 55,000 8.5 5.1 5.1 9.5 6.5
Interview - Site Superintendent 15 2.3 3.7 8.3 - 6.7 - PPI Average Rating (all entities) 5 9.86 9.30 9.29 9.22
9.40 9.80 PHASE 1 - SELECTION What to Look For Actionable plans = step-by-step solutions 1 Is there an actual solution or do they say they will figure it out later (to be determined)? Risks should be listed by priority Prioritization is important Budget, Schedule, Quality are often critical Are the risks project-specific? Or generic? Generic risks include: communication, teamwork, landscape, weather, safety Is it clear and easy to read? Does it make sense from an Owners perspective? Do they use marketing information? How often is their solution just to do what youd expect them to do? PHASE 1 - SELECTION 37
1 Differentiators Indicators of Expertise Risks prioritized in order of importance Project-specific risks Actionable steps to mitigate issues Planning for project delivery & execution Red Flags Risk priority is arbitrary or misaligned with Client concerns General / Generic Risks No Plan (we will figure it out later) Will survey & verify upon award
Will provide solutions Will communicate with Long, detailed, technical, confusing Marketing information Clear & easy to read Identifies what support is needed from the client PHASE 1 - SELECTION 38 What to Look For 1 Its about what they say (the type of risks identified and how theyre prioritized)
- AND - how they say it ! (solutions with actionable steps and potential impacts to budget, schedule, or quality) PHASE 1 - SELECTION 39 1 Offer Detailed Debriefings PHASE 1 - SELECTION Debrief Process Offer to ALL bidders successful & unsuccessful Include language in the RFP to offer this. Notify bidders at each step that they can request a debrief (i.e. if cut at the short-list or upon final selection notification, etc.) Conduct debriefs in a discussion format it is important to walk through it with the vendor. PHASE 1 - SELECTION 1
Debrief Document PHASE 1 - SELECTION 1 Advantages of Debriefings 1 Get better proposals next time!!! PHASE 1 - SELECTION What are Evaluators Seeing? Evaluators are looking for justification. If vendors do not differentiate themselves, evaluators are trained to give vendors equivalent scores. It is NOT about convincing the Owner to hire you with marketing information. It IS about telling the Owner how youll execute the project (this is how to show your expertise and your understanding of the project requirements) PHASE 1 - SELECTION 1
Evaluation Comments - Examples LOW Proposal Scores No risks related to the building itself. Many risks were related to RFI processes (risk 4) and transferring risk solution back to the Owner, particularly risks 1, 3, and 5 which stated a solution of meeting with the owner without offering options or recommending best approaches $32M Construction A great deal of delineation from our stated program requirements. For example, item 2 indicated they are not an expert in the reactor approach (did not know the long term desire for decommissioning?) $240M building re-design No statements that related to the specific requirements Inaccuracy of information vs. the RFP documents in risks 3-7. Identified issues but solutions were vaguefairly boiler plate Overly selling us, we want to deliver your visiondid not provide any actionable steps in risks 1-3 PHASE 1 - SELECTION 1 Evaluation Comments - Examples HIGH Proposal Scores identified structural elements and specific solutions for building code concerns good information on floor plates in Controllable Risk item 3 Demonstrated phasing and potential impacts $30M Construction
explained connectivity between Light Rail link, building entrance, and building circulation considerations. $240M building design Noted the costs for the laboratory structure and have included it within their base bid good! Others have either excluded this or not commented. $30M DB Construction Risks 2-4 demonstrate good recognition of existing building potential issues & solutions. That will benefit the construction schedule as well as building life $25M construction renovation PHASE 1 - SELECTION 1 Evaluation Comments High Interview Scores Lead Arch: Was able to draw out their project plan & presented timeline with dates as follows: 1) info gathering for Feb 17 visioning session 2) June to SD for building use & tie to fit scenarios/alternatives 3) DD January 2013 tie to code, integrate Light Rail link to circulation plan, atrium usage Mechanical: Provided their project plan and schedule and it and aligns with Arch team!!! Obviously worked closely together in review of their proposal. All members of Proponent team have commented on getting mechanical earlier (cores, window areas, etc.)
PHASE 1 - SELECTION 1 1 Plan Before You Sign PHASE 1 - SELECTION Plan before you Sign 1. Immediately address any major cost discrepancies (1-2 days) [if applicable] 2. Kickoff Meeting: Contractor presents their plan identifies anything they need from owner and WHEN set clarification phase schedule discuss any red flags and/or design concerns 3. Finalize project deliverables 4. Summary Meeting PHASE 1 - SELECTION
1 Greatest Sources of Project Change 1 Delays (schedule) Change Orders ($) PHASE 1 - SELECTION Non-Controllable Risks Common Types: 1 Coordination with the Client or other Third Parties Clients Schedule or Budget Constraints Unclear information in Scope & Requirements Unknown Conditions (i.e. known unknowns) Potential Trigger Points for cost, schedule, scope change Anything that can cause the Consultant/Contractor to be inefficient PHASE 1 - SELECTION
Examples of Real World Partnering (addressing common causes of Project Change) 1. 2. 3. 4. 5. 6. 7. 8. 9. Owner-provided items/tasks/deliverables Owner timely decision-making Owner resource commitments Owner has an overly prescriptive scope Owner has an issue/error Owner has a budget/schedule constraint Owner scope changes Concealed or unknown conditions Coordination with 3rd Parties and AHJs PHASE 1 - SELECTION 1 Leveraging the High Performers
Expertise 1. Cost Verification 1 Provide a detailed cost breakdown Identify why the cost proposal may be significantly different from competitors Review big-ticket items Review value added options Identify how payments will be made and all expectations regarding finances 2. Preplan in Detail Coordinate the project/service with all critical parties Revisit the sites to do any additional investigating Prepare a high level project schedule Prepare a schedule of client activities Prepare a detailed project work plan (transitioning, training, safety, security, staffing, etc)
PHASE 1 - SELECTION Leveraging the High Performers Expertise 3. Align expectations Review and address all assumptions Clearly identify the clients roles and responsibilities Technical review of product/system & demonstrations if necessary Potential deal breakers What is included and excluded in the proposal Review any unique requirements with the client Review and approve all contract terms and conditions 4. Identify and mitigate all uncontrollable risks Identify all risks or activities not controlled by the Offeror Identify the impact of the risks Identify what the client can do to mitigate the risks Address how unforeseen risks will be managed PHASE 1 - SELECTION
1 Impact of Partnering Traditional Low-Bid Pre-Contract Clarification 11 10 2 Total awarded cost $14M $10M 3 Ave. Schedule 5-6 mo. 4-5 mo. 4.4%
6.0% 0 7 No CRITERIA 1 Number of projects analyzed 4 Percent awarded cost below budget 5 Average BV-PA duration (days) 1 Change Order Rate: Cut in HALF Delay Rate: Cut in HALF PM Satisfaction Rating: 34% INCREASE Lines, B. C., Sullivan, K. T., Hurtado, K. C., & Savicky, J. (2015). Planning in Construction: Longitudinal Study of Pre-Contract Planning Model Demonstrates Reduction in Project Cost and Schedule Growth. The International Journal of Construction Education and Research, 11(1), 21-39. doi:10.1080/15578771.2013.872733 Download from: http://www.tandfonline.com/doi/abs/10.1080/15578771.2013.872733 PHASE 1 - SELECTION
Evaluation Procedures 1. 2. 3. 4. 5. Blind Evaluations Keep it Fair Streamlined Scoring Keep it Simple Individual Evaluations Avoid Groupthink Offer Detailed Debriefings Plan Before You Sign Take-Away Tools 1-5-10 Evaluation Scale Evaluation Scoresheets Evaluation Matrix Debriefing Template PHASE 1 - SELECTION 1
Vivo en mi barco pequeño que se llama "Itsuro." Hemos viajado por Alaska durante los últimos 10 años. ¡Alaska nos encanta! Durante el año escolar, vivo en Santa Cruz con mi esposo, Shawn. Shawn es consejero de carreras en Cabrillo...
irksome (a.) annoying and tedious. The irksome task of cleaning the bathroom always annoyed Irma. malevolent (a.) producing harm or evil. With all the bad things happening to her, Melody was sure there was a malevolent spirit haunting her. ......
The first General Stream Adjudications took place in the 1920s - Sevier, Weber and the Virgin River basins. So, we've talked about adjudication in its broadest sense, but what I really want to talk about is the concept of a...
Building a Bridge. Connecting Higher Education with Alternative Certification Programs. AESA Conference . School Leadership Partnership Program. Master's degree with a Concentration in School Leadership. ... PowerPoint Presentation Last modified by:
Sample Problem 5.3 - Problem and Plan. Applying the Volume-Temperature and Pressure-Temperature Relationships. PROBLEM: A balloon is filled with 1.95 L of air at 25°C and then placed in a car in the sun.
Trees (continued)Theorem: An undirected graph is a tree if and only if there is a unique simple path between any two of its vertices.. Proof: Assume that . T. is a tree. Then . T. is connected with no simple...
Fourteen Arguments for the Existence of God. The Argument from Religious Experience. God as the "Ultimate Being Deserving of Reverence" FR, p. 30; peterkreeft.com # 19. Belief in God—that Being to whom reverence and worship are properly due—is common to...
Portland District overview. Civil Works district ~1300 civilian employees. 8 military officers. Mission: The dedicated people of the Portland District provide vital public engineering services to the Pacific Northwest and the Nation during peace and war to strengthen our security,...
Ready to download the document? Go ahead and hit continue!