Defects lurk in requirements: Reduce project costs with disciplined requirements engineering

By Shan Bhattacharya, Field Application Engineer for LDRA Ltd. 

Developing and integrating medium- to large-scale military and aerospace systems are daunting tasks. The systems unite various disciplines, subject-matter experts, customers, support personnel and managers over long development lifecycles. Breaking down an initial vision from an operational concept to a polished and robust fielded product takes disciplined execution, technical ingenuity and tenacity.

Over the last decade or so, requirements-driven development, coupled with various proven processes, has been widely accepted as the most effective mechanism of coordinating and executing these engineering feats. Despite great efforts however, minimizing escaped defects remains crucial for long-term mission and organizational success.

As most experienced engineers in this field understand only too well, the cost of detecting and correcting defects grows by orders of magnitude as they progress from one software development phase to the next. Catching defects as early as possible is critical to controlling cost and meeting the customers’ technical requirements. Powerful tool chains, testing at all tiers, peer reviews at every phase, coding standards adherence, modeling and other methodologies have been employed to manage defects at most of the phases of development. Yet, the ripple effects of requirements-related defects continue to plague programs.

Requirements engineers scope and define the capabilities, constraints, and characteristics of a system to carve out the problem so design can then define the solution. They try to communicate and allocate mission, technical and programmatic problem spaces, so engineers may design and implement solutions. Mission-specific requirements may describe operational life cycle, whereas technical requirements may describe expected behavior at interfaces and modes and states, and programmatic requirements describe parameters such as cost and schedule.

Though requirements documents at some scopes can contain diagrams such as use cases and sequence diagrams, most requirements are written in text. Poorly written requirements are often inconsistent, ambiguous, have loopholes, describe too much implementation detail and have verifiability issues. Consider the example below:

The software shall provide a robust user-friendly graphical user interface that, to the extent possible, minimizes operator error.

Though this may be an explicit request from the customer, the phrases such as “user friendly”, “the extent possible”, and “minimizes operate error” make verification difficult. It is imperative that requirements engineers translate such customer wishes into a series of requirements that very clearly describe user-interface needs and error management.

If done correctly, the customer should feel that their vision is captured correctly and engineers should understand how they must develop the user interface and its capacity to handle user errors. These requirements should be further decomposed so testers can construct test-cases and procedures at the right level of detail, while management can better quantify the cost of implementation and test, risks and churn.

Requirements that describe accuracy, error budgets, computing resources, modes and states, and other complex behavior quite often fall victim to unclear wording and entice engineers to make assumptions as shown in the following example:

The vehicle management system software shall utilize no more than 50% of the available computing resources.

How does one test the requirement above? Does it imply average resource consumption when in operation? Does this include boot time and built in tests? Are spikes in usage that exceed the thresholds during rare and off nominal cases subject to this constraint?

To understand the impact of this poorly worded requirement, consider the following scenario. Legacy systems, as they continue to evolve, are often called on to perform increasingly processor-intensive tasks. As new requirements and tests for these systems start to exceed throughput thresholds, processor intense tests may be left out or simplified to keep program managers from repeatedly revisiting negotiations with the customer. The original intent of the requirement may have been to signal hardware obsolescence for planned upgrades, but re-interpretation of throughput threshold measurement may be an
easy way out for stakeholders. Re-factoring software may be required to meet
the newest interpretation of the requirement, and this often results in defects
being inserted without gain of new functionality.

A whole host of problems may be avoided at the start of a program by clearly describing processor throughput constraints in the requirement.

Requirements written at too high a scope, with too much detail, compound details, loopholes, unnecessary implementation specs, inaccurate data, unclear sequences and other such issues easily devolve a well-intentioned requirements document into an artifact that collects dust on a shelf. Developers then wield code to their vision of a product.

Despite all this pointing to what can go wrong, it may still be difficult to swallow that 60-70% defects in a product are requirements related.

If so, consider that requirements are typically derived from one or more system-level specifications to various hardware and software requirements for subsystems. They are then decomposed to lower level requirements and eventually realize a large specifications tree. Further decomposition results in detailed design and architecture documents, which often involve some sort of modeling language to layout architecture and timing.

Each tier of artifacts are ideally trying to describe the “What” at its tier and leaving the “How” to its child tier. As with the children’s game of “telephone”, the imperfections in each requirements document are compounded by the interpretation and assumption in the levels below. A simplified illustration of a specification tree below, clarifies the ripple effect that a poorly constructed requirement may have on latter phases of development.

Figure 1:

In reliable safety- and mission-critical embedded applications with tight constraints and complex behavior, implementation is the realization and refinement of specifications. These systems must often meet intense certifications such as CMMI and DO-178B. The demands on traceability from the highest level of requirements to the lowest tiers of architecture, code, and test, require constant arbitration and analysis. Communication and change control must operate fluidly amongst all stakeholders to meet the challenges of integrated design and development.

The tight coupling among tiers of requirements documents require that changes made at any level, must be reconciled very quickly vertically and horizontally across the specification tree. Defects encountered at lower tiers such as the software design and architecture layers are often very closely tied to requirements related issues in the tiers above.

When one considers the various issues outlined above are a small subset of factors driving requirements related defects, it is easy to imagine that the bulk of defects in embedded software products are tied to requirements in some form or fashion. Though there are no silver bullets to resolving these issues, a conscious effort towards rigorous requirements engineering that ensures traceability and transparency with all stakeholders can go a long way to reducing defects and ensuring project and organizational success.

Shan Bhattacharya is a Field Application Engineer for LDRA Ltd.  He graduated from Cameron University and began his career in factory automation and robotics. He continued his career with various defense contractors including Lockheed Martin where he served as a Lead Engineer and finished his time as a Deputy IPT Lead.  Shan has been with LDRA since 2007 and provides consultation for clients in various industries focusing on requirements management, software certifications, and development best practices.