Technology Maturity and Risk Reduction (TMRR) Phase
MATERIEL SOLUTION ANALYSIS (MSA) PHASE
TECHNOLOGY MATURITY AND RISK REDUCTION (TMRR) PHASE
ENGINEERING & MANUFACTURING DEVELOPMENT (EMD) PHASE
Request for Proposal
After conducting contract preparation activities, the next step is to develop the documentation and strategy for the Request for Proposal (RFP) to acquire capabilities via Agile software development.
Performance Work Statement (PWS) or Statement of Objectives (SOO)
A Performance Work Statement (PWS) or Statement of Objectives (SOO) is a key document in the Request for Proposal (RFP) package. The RFP describes the purpose of the work to be performed, with the contract requirements set forth in clear, specific, and objective terms with measurable outcomes. A PWS describes the government’s requirements, the tasks that must be accomplished to satisfy the requirements, and when/how the government will know when the outcome has been satisfactorily achieved. A SOO, by contrast, describes the high-level outcomes or objectives the government wants to achieve, giving the vendors an opportunity to propose innovative solutions to meet the objectives defined by the government.
There is no specific format for a PWS/SOO, but they usually contain a brief description of services, background, objectives, scope, period of performance, quality control, and quality assurance.
In the case of Agile development, to the government must first identify if it wants to procure a service or a product. As recommended in the Contact Preparation section, a services contract is often the preferred contract for Agile efforts because it offers the most flexibility to change requirements through the life of the contract. The following questions can be used to help drive the critical thinking behind the development of a PWS/SOO for an Agile development contract.
Questions to help develop the PWS/SOO:
- Has the government determined that Agile is the only acceptable development strategy? Is the PMO open to other development methods proposed by the vendor to best deliver the product or system?
- Will the government or the contractor define the details of the Agile development process?
- What are the government vs. contractor roles and responsibilities?
- What will the government provide to the contractor (e.g., facilities, information)?
- What is the government’s “definition of done”?
- How is the contractor expected to interface with the PMO, testers, engineers, and other contractors?
- What are the government’s expectations of the contractor’s relationship with other contractors in the program?
- What are the government’s expectations of the contractor’s support of systems engineering reviews and test?
- What are the data rights requirements? How will data be transferred by the vendor during transition-out period of the contract?
- What is the minimum set of deliverables required?
- What is the minimum set of metrics that the contractor is required to track and report?
- How will the government measure performance?
- What are the incentives/disincentives associated with the performance measures?
- Does the government have specific job categories in mind for Agile development support, or does the government expect the offerors to propose their own teams to provide an optimal mix?
The PMO must decide who will be responsible for defining the Agile development processes and methodology. In some cases, the government may already have well-established Agile processes within the program, and the government may want to dictate the specific Agile development process that will be utilized in a PWS. Below is sample PWS language that can be used if the government “owns” or directs the Agile methodology or procedures:
Government Directed Agile Methodology:
The program will utilize an Agile development methodology for this program. The program will provide capability deliveries every 6 months in a product release. Each release will be composed of four 6-week sprint cycles. Sprint cycles are composed of a 1-week design period that includes requirements definition and prioritization, a 4-week development and operational testing cycle, and a 1-week end-user demonstration period. The contractor will work with the government in a team-based Agile environment. The program will create and maintain the system architecture, requirements backlog, and roadmaps that will be the basis for the contractor’s work. The contractor shall support the government in the development and estimate of user stories, a release plan, systems engineering products, and acceptance criteria.
Alternatively, the government may want the vendor community to propose the Agile development process and methodology that best suits the outcomes or objectives identified by the government. In this case, it may not even be necessary for the government to dictate the use of Agile, but instead simply to describe its expectations and objectives and utilize a SOO document format. Below is sample SOO/PWS language that can be used if the government wants the contractor to propose the development methodology and processes that will best meet the objectives identified by the government.
Contractor proposed development methodology:
The contractor shall propose a development methodology to best meet the requirements of the contract. At a minimum, the contractor shall develop and deploy usable capabilities to the program in 6-month increments. Each increment should include testing, IA, and deployment to ensure capabilities can be released to the user in an incremental fashion. The proposed solution shall include an explanation of how project and contract management, communication/collaboration with the government, security and privacy requirements, documentation, and reporting will function in conjunction with the proposed development methodology. The contractor shall identify the roles and responsibilities of the contractor versus the government under the proposed development methodology.
Under a services type contract, the government can provide specifications around the type of support that will be provided under the contract. Examples of Agile services requirements include user story collaboration, user centered design, code development, automated testing, usability testing, integration support, data migration, deployment support, DevOps support, configuration management, user training, metrics reporting, transition support, facilitation support, and Scrum Master coaching and training. It is also important to include transition-in and transition-out support as part of the PWS.
Due to the iterative nature of Agile, it is important to have a common “definition of done” agreed upon by the entire team. Therefore, the PWS should include a clear “definition of done.” It will be used to determine whether individual user stories are completed and can be accepted. The government can require the offerors to include a definition of done in their proposed Agile solution that the government will evaluate as part of the technical proposal. As a team matures, the definition can be expanded to include more elements. The contract should include provisions to allow for re-definition by agreement between the government and contractor.
The initial “definition of done” could include any of the following examples:
- All code has been pair programmed or has been code inspected.
- Capability has been delivered to the satisfaction of the user representative
- Coding standards have been met and re-factored where necessary.
- All 508 Compliance, Security requirements, Legal requirements have been factored into released code.
- Decision maker has accepted the final deliverable
- Capabilities are delivered in compliance with the Key Performance Parameters
Request for Proposal (RFP) or Request for Quote (RFQ)
The focus of the RFP/RFQ will depend largely on whether the government is pursuing a services or a product contract.
If pursuing a services contract, the most important aspect of the proposal should focus on the qualifications of the individual team members and the experience of the collective team executing the Agile development process. The government should request that the technical or management proposal include resumes of the proposed Agile team members, and provide specific examples of relevant experience executing the Agile development process on similar and related efforts.
If pursuing a product contract, the government should focus mainly on how well the offerors describe their plan to meet the government requirements. In this respect, if the contractor delivers to the government requirements, the government should not mandate use of Agile processes, but rather allow the contractor to propose a methodology that best meets the requirements identified by the government.
The table below gives examples of language that can be used in an Agile development RFP, including language for Section L (instructions to offerors) and Section M (evaluation factors for award). The government may use a combination of the below sample evaluation elements, but the authors of this site strongly advise against using too many evaluation factors/sub-factors (more than 3), which will greatly increase the time to evaluate and award the contract. The government should only use technical evaluation factors/sub-factors that will generate proposal discriminators during the evaluation process. Furthermore, it is considered a best practice in the contracting community to define Section M (evaluation factors for award) before writing Section L (instructions to offerors), as reflected in the below table.
Please note, it is not advised that the government use the below proposal instructions and evaluation criteria when issuing task orders under existing contract vehicles. The government is advised to take full advantage of the flexibilities offered under FAR 15.505 fair opportunity provisions. In these cases, the government should not use a formal evaluation plan or detailed evaluation criteria (for more information, see the Contract Vehicles section under Contract Preparation).
|Section M (Evaluation Factors for Award)
Section L (Instructions to Offerors)
|Agile Implementation Strategy
|The government will evaluate the completeness, feasibility, and appropriateness of the proposed Agile implementation including techniques for release planning, plans for engaging end users, methods for capturing and applying lessons learned, testing processes, reasons behind the composition of their Agile teams and the rationale behind the proposed development talent and project oversight (tied to Product Vision), how the offeror will make resources available within schedule and budget constraints, and the offeror’s approach to configuration management.
|The offeror shall describe their approach to manage the Agile implementation,
|Roles and Responsibilities
|The government will evaluate the completeness, feasibility, and appropriateness of the offeror’s described roles, responsibilities, and expectations of the contractor, government PMO, and PMO team members (e.g., test) in executing the Agile process, thoroughly identifying and explaining all assumptions made by the offeror.
|The offeror shall describe the roles and responsibilities of the Agile team, to include government personnel.
|The government will evaluate the completeness, feasibility, and reasonableness of the offeror’s plan to appropriately resource the Agile development team within schedule and budget constraints. The government will assess the offeror’s proposed team structure and staffing, and their strategy to keep the team motivated and engaged throughout the project lifecycle.
|The offeror shall describe their strategy to resource and manage team performance.
|The government will evaluate the completeness and comprehensiveness of the technical solution to ensure it addresses project and contract management, communication/collaboration with the government, security and privacy requirements, documentation, and reporting (include additional elements as needed).
|The offeror shall describe their proposed solution to meet all the government requirements.
|Product Development Roadmap
|The government will evaluate if the offeror’s proposed product development roadmap correlates with the offeror’s stated objective and timeframe for implementation of the offeror’s proposed Agile methodology. The government will evaluate if the product development roadmap demonstrates where testing, training, security, privacy, and cut-over planning will be conducted, and the reasonableness of the approach.
|The offeror shall provide a product development roadmap.
|The government will evaluate the completeness, feasibility, and appropriateness of the offeror’s proposed quality control and performance measurement approach, including how proposed performance standards will be monitored, evaluated, and reported. The government will evaluate if the measures and metrics proposed align with the proposed technical solution.
|The offeror shall propose a performance management strategy for the government to use throughout contract performance, to include specific measures and metrics to measure team performance.
|The government will evaluate the completeness and reasonableness of the offeror ‘s proposed strategy to mitigate the risk of building a technical debt during the program and resolve the backlog of technical debt on the program.
|The offeror shall describe its strategy to manage and technical debt on the project.
|Transition from Waterfall to Agile
|The government will evaluate:
|The offeror shall describe the proposed approach to transition from the government’s current waterfall approach to the offerors’ proposed Agile methodology.
|The government will evaluate the offeror’s approach to planning releases to determine if it presents a thorough and comprehensive approach to satisfying government objectives and priorities for delivery of system functionality.
|The offeror shall describe its approach to initial release planning, structuring incremental releases of software capability, and responding to changing government priorities both during release planning and during execution.
|The government will assess the release plan proposed by the offeror for realism and feasibility of the plan and the resources proposed for its implementation. The government will also assess the offeror’s understanding of the objectives of the solicitation, DHS, and for the degree to which the proposed plan satisfies the objectives specified in the solicitation.
|The offeror shall describe the release of the capability sought in the solicitation. The offeror shall describe what it proposes as the number and length of iterations necessary to satisfy the objectives of the solicitation and shall describe the products produced by each iteration contributing to the overall capability.
|The government will evaluate if the offeror provides a complete, comprehensive, and compelling rationale for why the Agile methodology proposed is the most appropriate and will deliver the best results. The government will evaluate:
|The offeror shall describe why the Agile methodology being proposed for this effort is the most appropriate for the requirements in this RFP.
|Government Agile Implementation Support
|The government will evaluate the offeror’s approach to supporting the government as they implement an Agile approach for the first time.
The government will assign higher scores to offeror’s that can provide specific successful examples of how they have supported the government implement Agile for the first time on other programs and organizations. The offerors shall describe how these organizations or programs were successful in adopting Agile practices.
|The offeror shall describe its approach to supporting the government in its implementation of an Agile strategy.
|Agile Requirements Management
|The government will evaluate the completeness, comprehensiveness, and rationale for their proposed strategy to manage the Agile requirements. The offeror shall provide a detailed explanation for how they plan to work with the government and user reps to continuously balance tradeoffs across user needs, business goals, and technical requirements.
|The offeror shall describe their strategy to manage the Agile requirements.
|The government will evaluate:
|Offerors shall provide a description of the Agile development team, to include the number of personnel to support the Agile development process and specific individual names linked to their role on the Agile team. offerors shall describe if the Agile team as proposed has worked on previous projects/programs supporting Agile development and the success of those programs/projects. The offerors shall identify all personnel on the Agile development team and the role(s) they will play in supporting the Agile process. The offeror shall provide resumes of personnel that describe the details of everyone’s overall experience and education. Resumes should include specific references to experience and training with Agile processes. The resumes should also identify the individuals’ experience working on previous Agile development projects and their role(s) on the projects. (recommend setting a specific page limit on resumes, or providing a resume template for consistent review of resumes)
|The government will evaluate:
|The offeror shall describe how the full Agile development methodology proposed has been utilized on other relevant programs and projects. For each customer reference, provide the following:
*Evaluating relevant experience in the technical/management proposal is different from evaluating past performance as a separate and distinct factor. Generally, past performance must be evaluated for all major acquisitions; relevant experience (as part of the technical evaluation factor) is not required but it is highly recommended by the authors of this site.
Another popular approach to evaluating Agile solicitations is including challenge-events at which vendors present live demonstrations of their capabilities. For example, the government can require vendors to submit a working prototype based on a common dataset in a 24-hour product development challenge. This event be conducted in conjunction with the technical proposal and/or oral proposal, or can be used in place of a traditional written technical proposal.
Product challenges work well for Agile development solicitations because they require vendors to demonstrate or prove their ability to deliver capability. They also reduce the bid and proposal expenses for the vendors if the challenge replaces the technical proposal. On the other hand, product challenges can create a risk for the government because it has no guarantee that the team that worked on the challenge will be the same team that will work on the resultant contract. In theory, vendors can utilize their “A” team for the product demonstration but subsequently staff the project with less experienced staff. Therefore, it is recommended that the government combine a product-challenge with a “lightweight” technical proposal that presents the staffing plan and proposed development methodology processes. The use of Youtube videos for recorded oral proposals can be used as a “lightweight” technical proposal.
The Department of Homeland Security, Flexible Agile Support for the Homeland Request for Proposal featured a technical challenge. The details of the challenge can be accessed here.
Below is suggested language that can be used for a video-based oral proposal:
The offeror shall submit a video that is no longer than X minutes in length, that addresses the following evaluation areas (insert evaluation criteria, questions, or scenarios). It is recommended the video features Agile project team members proposed for this effort.
NOTE: The quality of the video submissions will not factor into the government’s feedback. The government encourages low-cost video production, such as the cell phone video cameras.
Submission Instructions – Video submissions must be posted to Youtube.com for the government to access. Videos may be marked public or private. By XX date, the vendor must send a Youtube link and password to (email address) in order for the government to access the video submission. The government will not provide access to the vendor’s video outside of the government evaluation team without the permission of the vendor. Do not provide a shortened URL, such as youtu.be.
The contract should include the agreed pricing methodology or model. Some other considerations for pricing an Agile development include:
- Defining the invoicing periods and pricing instructions for both individual iterations and the project as a whole.
- Definitions of price per unit of work (UoW). Examples of a UoW include price per story point, price per function point, and price per feature point.
- Points are related to an estimate of size or effort; it is critical that each “point” is clearly defined.
Two typical ways to determine price per point:
- An average based on several previous projects
- Customized amount: supplier average point value for a few iterations during which time detailed costs are tracked. Then the supplier and customer agree to a custom price per point (can be FP or T&M NTE).
- Fixed price considerations:
- Fixed price per iteration (perhaps calculated by reference to the amount of work required for that iteration or the business value of the relevant development items)
- Fixed price per unit of work (price per user story, or feature point); may be difficult to estimate if the user stories have a different scope or value)
Traditional projects often include a detailed prescriptive list of what the vendor should deliver (usually in the form of lengthy documents) and how the artifacts are accepted. Onerous documentation requirements (e.g., earned value management) take time away from the team that should be focused on delivery. A detailed list of deliverable documents is not required for Agile; when applied appropriately, Agile is a structured and disciplined process and the government will receive “empirical” evidence of performance in the form of working software. The government should request “just enough documentation” to validate that the capabilities are being delivered in compliance with agency governance and IT system lifecycle requirements. Examples of the type of deliverables that could be requested for Agile include:
- Agile metrics reporting (e.g., team velocity)
- System design documentation
- Architecture reference/compliance documentation
- Automated testing results
- Lessons learned
- Periodic progress reports
- Training manual
- Information support plan
Key Questions to Validate Contract Strategies
- Do the contract strategy and timelines support frequent capability releases?
- Is the program pursuing a services-based contract or completion/product delivery contract?
- Is the government the prime systems integrator?
- Does the program have dedicated contracting support?
- Is the contracting officer co-located with the program?
- Does the contracting environment support Agile development processes?
- Do existing contract vehicles support Agile delivery?
- What is the level of engagement between the contracting officer and Agile team?
- How is the government monitoring the contractor’s performance?
- Does the contract include a process for determining how many high-priority items identified by the Product Owner can be developed during the current sprint?
- Does the contract have to include a definition of “Minimal Viable Product” for each feature?
- Does the contract identify a process for developing and reprioritizing the sprint backlog?
- Does the contract describe how improvements agreed at a sprint review meeting are implemented?
- GAO Report 12-681 Effective Practices and Federal Challenges in Applying Agile Methods, July 2012
- Going Agile: The new mind-set for procurement officials, Deloitte, May 2017
- Agile Contracts – PWS Templates, GSA Tech Guide
- Agile Contracts – Task Order Template, GSA Tech Guide
- Agile Contracts – Interview Questions for Agile roles, GSA Tech Guide
- Modular Contracting, 18F, Jan 2017
- DODI 5000.74 Defense Services Acquisition, Jan 2016
- OMB Contracting Guidance to Support Modular Development, Jun 2012
- Contracting for Agile Software Development in the DoD, Eileen Wrubel, Jon Gross, SEI, Aug 2015
- Agile Government Contracting, John Stenbeck, Kevin Jans
- Enabling Acquisition Success for Agile Development, Frank McNally, ASI Government, March 2014
- TechFAR Handbook