Materiel Solutions Analysis (MSA) Phase

A general misconception exists that Agile software development means that no long-term plan. Agile development does involve long-term planning and cost estimation is a critical activity in Agile programs. It requires early, upfront analysis that demonstrates a high-level understanding of the program and its associated costs and benefits. Programs must provide high-level answers to questions about what they plan to deliver, how, and when; how much it will cost; and what benefits will be gained. They use this information to justify the investment, as well as planning and funding decisions.

Programs use cost estimates and benefits assessments in a variety of products and for many different purposes throughout the acquisition phases, including trade-off analyses, such as AoA, business case analysis (BCA), and economic analysis (EA), and life cycle cost estimates (LCCEs).  Cost estimates and benefit assessments generally mature over time as more information becomes known about the program. The estimates and assessments should consider the nuances of the Agile development process Initial program-level estimates will have low fidelity for high-level increments, but will be followed by detailed estimates prior to the development of each release. A comprehensive and competent estimation process and methodology give senior stakeholders the confidence to relax rigorous oversight, and provide the program with valuable cost information to continuously improve performance and management.

Costs and benefits for Agile development can vary widely, as programs differ in complexity, size, and scope. Several areas have risen to the surface as key distinguishers that affect all cost elements and benefit areas across the life cycle. These areas include:

• Quality and experience of the team used by both the government and developer
• Fit of Agile development to program needs and constraints
• Agile development methods used by both the government and developer
• Degree of Agile enablement by external environment (stakeholders and processes)

Cost impact data and benefit realization data for Agile development is still maturing, particularly for the government environment. As with any software methodology, the extent to which best practices and discipline are used affects the costs and benefits and should be considered in any cost estimate or benefit assessment.

The major techniques used to develop cost estimates for acquisition programs include analogy, parametric, engineering build-up, and extrapolation from actual costs. Generally, the cost estimating technique used for an acquisition program matures from the analogy to actual cost method as the program becomes more defined and actual data becomes available. The method chosen depends on where the program is in its life cycle. Early in the program, definition is limited and the analogy method is appropriate. Once a program is in development, the cost estimator can use actual cost and technical data from the development phase to estimate the remainder of the program. Thus the cost estimate has more uncertainty in the early phases and becomes more certain as the program matures and risks materialize.

Cost estimating techniques for an Agile development do not necessarily differ from techniques used in a traditional development program. Initial estimates will be high-level and will be refined as additional program definition and data become available. However, an Agile development program intentionally does not have as much definition available as the development phase nears as that of a traditional program. Near-term releases have detailed definition and engineering build-up techniques can be used. Later planned releases will not have the same level of definition and the cost estimator has to rely on the artifacts such as the product vision and product roadmap and estimation techniques such as analogy (to the near-term releases). Definitions of releases will mature over time and the cost estimator can then apply engineering build-up methods as well as actual data collected from completed releases. Once the costs for the program have been ‘fixed,’ the cost analysis activities focus on assessing how much work can be accomplished within the funds available.

Estimating costs in an Agile environment requires a more iterative, integrated, and collaborative approach than in traditional acquisition programs. Traditional programs often treat cost analysis as a separate activity, rather than as an integrated team endeavor, but cost estimation on an Agile program is a team-based activity. Ideally, the cost estimator is tightly coupled with the systems engineers and development team as each Agile release is scoped, developed, and tested. Ongoing collaboration among the users, development team, systems engineers, cost estimators, and other stakeholders is critical to ensure agreement on requirements prioritization in the product backlog, and to gain a thorough understanding of the amount of effort required for each release. It also enables an integrated assessment of the operational and programmatic risks, technical performance, cost drivers, affordability, and schedules.

As with other software development methods, product size is usually the biggest cost driver when developing a software development cost estimate. Agile developments typically use cost estimating strategies based on relative measures of size, such as story points. No set formula exists for defining the size of a story, so release teams can use various techniques centered on small team collaboration to reach consensus on the number of points for each story. Teams base story-point estimates on the amount of effort involved in developing the feature, its relative complexity, and the inherent risks. For example, a small, simple story could be assigned one point, whereas a moderately complicated story could receive eight points –meaning that the associated product would take eight times as long to develop. This strategy provides a way to compare the size of one story to another, thus ultimately enabling programs to comparatively measure the size of the entire pool of requirements. Some common scoring systems that teams may use to assign story point complexity measures include Fibonacci series, ideal days, or small-medium-large.

After developing user stories, the Agile team prioritizes them into what becomes the product backlog, and then constructs a release from the product backlog. The number of user stories that comprise a release is based on the sprint schedule measured against the team’s estimated velocity (a measure of productivity unique to the Agile development method). Programs can initially measure team velocity using historical values or by making a forecast based on the skill sets of the team and the team’s experience with the specific product or technology. After the first sprint, the team selects the user stories for the next iteration and the team working on each story makes a more fine-tuned estimate of the appropriate story points for that sprint. The team can then estimate whether it can build the proposed set of stories within the sprint timeframe, given the team’s actual velocity.

Teams regularly conduct this activity and reassess the points of the backlog items based on insight obtained from recent developments. This iterative approach increases the fidelity of estimates over time. The release-level estimate is at a high level, while sprint-level estimates are more refined and detailed to help the team use the cost estimates to manage the project effectively.

As the program progresses through the development process, it should examine how closely the initial estimates match actual effort, if stories were added or removed from a release, and if the estimating methodology or value for the story changed. Tracking the planned versus actual progress of completed iterations against the story points that remain to be completed, using a burn-down chart, can help to track the progress of the Agile development.

Cost impacts encompass cost increases and cost decreases seen in Agile development experiences. Cost increases include the initial investment (and in some cases sustained investment) necessary to implement the development. Cost decreases reflect the reduction in costs expected once development of a release has been completed and deployed and the program has reached a sustainment state. These decreases reflect areas with real potential for tangible benefits such as cost savings and cost avoidances. Some areas have both an initial cost increase and a decrease in the recurring costs (e.g., the program must invest money to save money). The cost impacts are relational measures and should be interpreted in comparison with a traditional development environment. The technical baseline and cost impacts for each major cost area are:

• Program Management/Systems Engineering – Agile programs require intensive government participation and involvement to manage the overall development process. The government program office must closely engage with the contractor development team on a daily basis, and must enlist dedicated support from additional government resources (e.g., cost estimators, testers, users, contracting officer). This higher level of engagement and additional Agile roles to be filled (such as product owner) may require an increase in government resources.
• Software Development – Agile development seeks continual user engagement and feedback throughout the development process. This can result in an increase in unplanned work during development as changes are incorporated into the development phase rather than addressed post-deployment (both in-scope or out-of-scope changes), as in the traditional waterfall approach. Agile developers are typically cross-trained due to the smaller team size and fast pace of delivery. Development teams also work on smaller batches of software at a time. The combination of multi-skilled developers and smaller batch development may increase productivity. Agile development may reduce software growth due to uncertainty around requirements, because in Agile requirements are prioritized and traded to meet cost and schedule. However, programs must still account for software size uncertainty due to other reasons.
• Integration and Test – The cost estimate for integration and testing must consider the impact of the short, frequent, integrated real-time integration and testing characteristics of Agile developments, and determine if the costs differ from those for a traditional waterfall approach. Integrating and testing large batches of functionality, as in traditional waterfall approaches, is inherently more complex than performing these tasks on small batches of functionality, and developing testing techniques and detecting errors during tests are more difficult on larger batches of software. Frequent testing also allows for earlier detection of errors, when they are easier and less costly to fix than fixing those same errors found in tests performed at the end of development. The government must also evaluate the impact on regression testing. Automated testing is not unique to Agile development, but the majority of integration and testing in Agile developments is highly automated, allowing delivery of short, frequent releases of software.
• User Participation – User representation on the Agile release team is necessary to help prioritize requirements, assist in creating user stories, conduct user acceptance testing, and report feedback on deployed capabilities. The costs of maintaining continuous long-term user representation on the Agile team should be factored into the cost estimate.
• Fielding/Deployment – An Agile release deploys new capabilities in short, frequent releases (typically every 6–12 months). The degree of change and level of complexity of each release may require further consideration of deployment costs. Automated deployment is not unique to Agile development but serves as an Agile enabler. Agile development depends on a high degree of automated deployment usage to deliver short, frequent releases of software effectively. The amount of onsite deployment activity is a key driver of cost and could yield a significant increase in deployment efforts if each release requires heavy onsite deployment activity.
• Training – Software development releases and training can be asynchronous but Agile development will lead to a larger number of training deliveries than traditional waterfall developments. End user engagement and training may be required with the rollout of each release. The amount of onsite training is a key driver of cost and could yield a significant increase if heavy onsite training is required. Since users are deeply involved throughout the development process, this increases the built-in usability incorporated within the product and may decrease the complexity of training materials and the amount of training time required.
• Sustainment – As new capabilities are deployed, the program must operate and maintain capabilities deployed from prior releases. The program must evaluate the cost impacts of sustaining multiple, frequent, and overlapping releases. The continual and highly automated nature of testing in Agile may yield greater error detection during development and therefore fewer defects cross over to sustainment, where they become costlier to fix. Continual user engagement throughout the development process may reduce efforts associated with enhancing and optimizing deployed software during the sustainment phase.

Overall, comparing Agile development with the traditional waterfall method reveals areas of both cost increases and decreases. Program management/system engineering may experience a slight increase as the program embraces and learns the Agile process and incorporates a higher level of engagement. Software development shows a complex mix of increases and decreases, but overall higher productivity and reduced software growth due to requirements change likely result in a cost decrease. Automated integration and testing and deployment are key enablers of Agile development, and the degree to which a program can automate these areas offsets the increased number of tests and deployments. The amount of onsite fielding/deployment and training required is a key driver of cost increase in these areas. The less onsite presence necessary, the lower the potential cost increase.  Sustainment represents the largest area of cost decrease, due to higher quality code, fewer defects, and fewer software enhancements and optimization post-deployment. Sustainment costs usually comprise the majority of lifecycle costs as systems are sustained for years, so any impact here can prove significant.

The list below shows a range of potential life cycle cost impacts by major cost area. Analysis to-date is too immature to translate any of the cost impact trends into real dollar values. Additionally, as mentioned previously, many variables factor into the specific cost analysis of each of these areas. Therefore, a relational scale is used to summarize the general trends and slope directions associated with each area of cost increase and cost decrease. The scaling technique selected to compare each cost element is a relative technique in which a symbolic measure was assigned to the alternatives as follows:

• ++ denotes that costs are significantly increased;
• + denotes that costs are increased;
• = denotes that costs not impacted;
• – denotes that costs are reduced;
• – – denotes that costs are significantly reduced

The scaling measures reflect a range of cost increases and cost decreases attributable to Agile development relative to a traditional waterfall development approach.

Life Cycle Impact Assessment

In summary, Agile development may lead to a net decrease in total life cycle costs, especially because the largest area of cost decrease is in sustainment.

The cost increases and decreases described here are not necessarily independent. For example, without a high degree of automated integration and test the program many not fully realize increases in quality of code and the resulting decrease in sustainment costs. The general cost impact range presented for each area can serve as a guide, but programs must assess their unique situation to determine more specific impacts.

Agile has many potential benefits. The table below identifies some of the key benefit areas to consider.

Agile Benefit Considerations

Data from several studies has shown that programs can realize benefits through schedule reduction and improvements in productivity, quality, and customer satisfaction. However, findings to-date differ significantly across studies and ranged from marginal improvement to significant improvement.

Benefit analysis should consider the iterative nature of Agile and the resulting ability to deliver functionality earlier than with the traditional waterfall approach, which does not field capability until the end of development. Benefits will accrue as software is fielded. The figure below compares the two approaches.

Benefit Realization Comparison

The work breakdown structure (WBS) constitutes the cornerstone of every program because it defines in detail the work necessary to accomplish a program’s objective. The WBS relates the elements of work to be accomplished to each other and to the end product and ensures that no portions of the work are omitted or double counted. The WBS provides a basis for identifying and displaying resources and tasks for developing a program cost estimate.

Other MITRE research efforts have developed a comprehensive Agile WBS framework that programs performing Agile development can leverage to define activities and effectively communicate the work to be done. These efforts identified and reviewed existing Agile development activity breakdowns and decompositions. Open source and cost community research identified several commercial activity structures but no single comprehensive life cycle model. The activities were analyzed, compiled, modified, and augmented to develop a comprehensive Agile WBS for government programs. The WBS can be mapped to common life cycle cost elements. The Agile WBS has been presented and discussed at various forums and conferences within in the cost community.

The Agile WBS, shown below, provides a high-level, extensible structure for describing the efforts and products applicable to an Agile program. It is intended to serve as a starting point or reference to assist organizations in managing their transition to an Agile development methodology and as a guide for programs to identify activities to consider. The Agile WBS can be tailored to each program and can incorporate greater detail over time as more information becomes known. The activities identified can be mapped to traditional WBS structures for greater understanding and utility. The Agile WBS should be combined with any additional components a program may have that are outside the scope of Agile development.

The Agile WBS spans both overarching program management and systems engineering activities as well as release-level activities and sprint-level activities that are repeated with each release and sprint, respectively. The WBS includes both development and sustainment activities; however, it deviates from traditional WBSs in that development- and sustainment-related activities are not distinguished from one another at the top level. In Agile, all phases of software lifecycle development could take place in a single iteration and the Agile WBS was structured to reflect the essential nature of Agile. It does not explicitly identify sustainment, but the activities would transition from development to sustainment as time and purpose dictate. The appropriate development activity would include efforts related to sustainment when it transitions. This also means that while the activities are presented in a logical order to generate the framework, they are not necessarily in chronological order.

High-Level Agile WBS

Program management and systems engineering (PM/SE) activities include those traditionally performed by a Program Management Office (PMO) team or by systems engineers who are not necessarily members of a development team. They include traditional activities that remain unchanged in an Agile program, such as resource management, contract development, governance and decision making, roadmap development, acquisition strategy formulation, etc. PM/SE activities unique to an Agile effort include creating and grooming a product backlog, and “defining done.”

All other activities fall into the category of release activities: those that involve planning, executing, and reviewing/demonstrating a release. Many of the activities for a given release take place within a sprint; therefore, “sprint activities” encompass the activities related to planning, executing, and reviewing/demonstrating the work performed in a sprint, as well as a sprint retrospective to improve future sprints. Executing a sprint involves potentially all phases of the development lifecycle, to include design, architecture development, implementation, testing, documentation, deployment, and training. Other activities, such as code reviews and engineering analysis or experimentation, happen on an as-needed basis and may vary per program. As noted earlier, sustainment activities are an integral part of each release and as such are not explicitly distinguished within the WBS. Sustainment activities should be included within the appropriate elements as they transition from development to sustainment.

The budget for an Agile development program is based on the government program cost estimate, given the technical baseline requirements for the IT system. The program cost estimate must consider the Agile development costs along with all the other costs to plan, manage, acquire, maintain, and dispose of the program. One benefit of Agile is that once a budget has been established the program can be structured to “build to budget.” The funding that the program receives then drives the number of releases it can manage in each year and the totality of delivered requirements within the entire development period of performance. However, programs must be careful to track burn down metrics to ensure progress against the backlog is sufficient. Within DoD, resistance to allocating budget for a program until all the requirements are fully defined and approved often poses a challenge. DoD executives and Congress often require a clear understanding of the level of requirements maturity for budget authorizations.

Given the iterative, segmented nature of Agile development, Agile programs can scale up or down more easily than traditional programs that deliver capability in 5–10 year increments. Even if the Agile program experiences a major funding cut or cancellation during development, deployed releases will already have provided capability to the user, which is not guaranteed under a traditional development.

As illustrated in the figure below, under traditional waterfall development projects scope and quality are fixed, while schedule and cost vary and generally increase as requirements are further defined during the project. Agile development projects attempt to fix schedule and cost, while adjusting the scope of specific releases to fit these constraints. The scope of each delivered release is defined by the prioritization of requirements that deliver end-user functionality within the constraints of the cost, schedule, and quality parameters. Thus, treating cost as an independent variable drives the prioritization of requirements and facilitates release planning.

Differences In Traditional Waterfall vs. Agile Cost Variables

As noted, Agile programs can absorb budget cuts more easily than traditional acquisition programs. It is easier to cut funding from an IT program that can defer a small release than to “break” a weapon system increment or significantly delay delivery to the user. Thus, IT programs are often the primary target for budget cuts rather than weapon system programs that operate under inflexible requirements documents. IT programs that employ Agile techniques must ensure they have sufficient support at the Service/Agency, OSD, and Joint Staff levels to avoid constant risks to their funding.

Decomposing a software development project into several deployable releases enables projects using Agile methods to adapt to both permanent reductions in the lifecycle budget and temporary reductions in the development budget. Traditional, waterfall-based development projects find it far more difficult, both technically and contractually, to accommodate changing budgetary conditions.

Key Questions to Validate Estimates:

• Does the team have good historical data that it can refer to when making new estimates?
• Is the team’s velocity updated regularly and used to scope sprints?
• Does the development team have strong knowledge/experience in the user domain?
• Does the development team have strong knowledge of/experience with the technologies?
• Do all development teams use a consistent estimation method?