Issue 84, January 2008

Software reliability modeling and prediction during product development is an area of reliability that is getting more focus from software developers. The use of software reliability growth models plays an important role in measuring improvements, achieving effective and efficient test/debug scheduling during the course of a software development project, determining when to release a product or estimating the number of service releases required after release to reach a reliability goal. Many new trends in software development process standardization, in addition to established ones, emphasize the need for statistical metrics in monitoring reliability and quality improvements. In this article, we investigate the standard Gompertz and modified Gompertz models and show their applications in modeling software reliability growth using RGA 6.

There are essentially two approaches to performing statistical reliability prediction for software. The first approach, based on design parameters, estimates the number of defects in the software using code characteristics such as numbers of lines of code, nesting of loops, external references, input/output calls, etc. The second approach is reliability growth analysis based on statistical correlations of actual defect detection data obtained during testing. Many models are used to describe reliability growth in software such as Crow-AMSAA, standard Gompertz, modified Gompertz and Lloyd-Lipow.*

This article addresses the techniques available for the second approach using the Gompertz reliability growth models. Let us start with an overview of the basics of the Gompertz model and the modified Gompertz models. Then, we will discuss the application of these models for software development.

The Standard Gompertz Model
The standard Gompertz reliability growth model is often used when analyzing success/failure data and reliability data obtained in developmental reliability growth programs (the previous two issues of the Reliability HotWire, Issues 82 and 83, discussed reliability and success/failure data and other reliability growth models that can be used to model them). The standard Gompertz model is most applicable when the reliability data follow a concave shape, as shown in the next figure.

The standard Gompertz model is mathematically given by the following 3-parameter equation:

where:

• 
• 
• 
• T: time, launch number or stage number, T > 0
• R: the system's reliability at T.
• a: the upper limit that the reliability approaches asymptotically as T ® ∞, or the maximum reliability that can be attained.
• ab: initial reliability at T = 0.
• c: the growth pattern indicator (small values of c indicate rapid early reliability growth and large values of c indicate slow reliability growth).

The estimated parameters in RGA 6 are unitless. The solution for the parameters, given T_i and R_i, is accomplished by fitting the best possible line through the data points. Many methods are available, all of which tend to be numerically intensive. For more details about the parameter estimation method used in RGA 6, click here.

The Modified Gompertz Model
Sometimes reliability growth data with an S-shaped trend, such as the one shown in the next figure, cannot be described accurately by the standard Gompertz or logistic curves. Since these two models have fixed values of reliability at the inflection points, only a few reliability growth data sets following an S-shaped reliability growth curve can be fitted to them. A modification of the standard Gompertz curve overcomes this shortcoming by considering a shift in the vertical coordinate:

where:

• T: time, launch number or stage number, T > 0.
• R: system's reliability at T.
• d: shift parameter.
• d + a: upper limit that the reliability approaches asymptotically as T ® ∞.
• d + ab: initial reliability at T = 0.
• c: growth pattern indicator (small values of c indicate rapid early reliability growth and large values of c indicate slow growth.

The Reliability vs. Time plot for this model looks like the figure shown next.

The parameters of the modified Gompertz model can be estimated using linear regression and confidence bounds can also be estimated; for more details about how to estimate confidence bounds, click here.
Many scenarios can be modeled with the S-shape behavior of the modified Gompertz. The S-shape behavior essentially distinguishes between multiple "phases" in the product reliability growth program. In the beginning of testing (first phase), scarcity in discovering problems and delays in identifying fixes can cause the improvement in reliability to be small. During the second phase, fixes become available and, with any additional discovered failures and implemented fixes, significant improvements are observed and the reliability grows at a fast pace. In the third phase, fewer and fewer failure modes are uncovered, as most of the failure modes have already been discovered. In addition, the reliability growth program might start running into limitations of technology and design that slow down the reliability growth.

Application to Software Reliability Growth
The standard and modified Gompertz models are praised for their simplicity and ability to produce valid estimates of the future reliability of the software and to provide estimates sufficiently early in the testing/debugging phase. The standard Gompertz model can be a good model to use in software reliability growth in cases where the bugs are discovered fairly early and improvements are made swiftly. The modified Gompertz model, on the other hand, is more appropriate to describe an S-shaped reliability growth curve trend with a lower rate of debugging and growth at the early stage, a higher rate later on as more fixes are found and successfully implemented, and ending with a slower rate of debugging toward the completion of the program, where the law of diminishing marginal returns causes slower improvements to the software. Time, in the standard and modified Gompertz models, can be represented by accumulated execution time, stages, accumulated CPU time, accumulated clock time or calendar time. Defect data can be represented in the form of a tally of discovered bugs, mean time between new discovered bugs, number of test cases and failed tests, etc.

Example 1
A software development company tracks the number of discovered bugs during the development of a new software application. The following figure shows the record of bugs discovered during 10 stages of the development cycle. Each stage consists of a number of performed tests (about 1100 tests) and the number of tests that failed the test. At the end of each stage, fixes are implemented in the software and a new version is compiled for the next stage of testing. This figure also shows the calculated parameters for the standard Gompertz model.

The next plot shows the estimated reliability plot along with the lower 90% one-sided confidence bound.

The next figure shows the calculation of the demonstrated reliability at the end of the test (10^th stage of testing) with a 90% lower confidence estimate.

Because the "a" parameter in the standard Gompertz model is equal to 1, the testing/debugging program has the potential to grow the reliability of the software further than what is currently demonstrated. To achieve a reliability goal of 99%, for example, about 6.46 more stages of testing would be required.

Example 2
A software development company follows a process called the Capability Maturity Model (CMM), which is a process used in software development for controlling improvements and achieving high software quality. Like many other software development and quality assurance initiatives, CMM emphasizes statistical analysis to quantify and control the improvement process. The CMM process has different levels. To achieve Levels 4 and 5 of CMM, the company must set quantitative quality goals for the software product and measure and control its improvement using statistical techniques. The company is using reliability growth models to monitor and measure the software reliability.

In the current software development project, the company experienced slow growth at the initial testing stages, which relied on random exploratory testing. It was then able to improve the efficiency of the testing cases after better identification of the user profile and risk areas in the software. The last few stages of testing revealed software faults at a slower rate and the company expects that the software is reaching its maturity.

The following data set, shown entered in RGA 6, shows the test case results obtained at each of 11 stages of testing. The standard Gompertz model and the modified Gompertz model were fitted to the data set. The modified Gompertz model was a more appropriate model for this data set.

The reliability growth plot is shown next.

The achieved reliability is 0.9965, as shown next.

The asymptotic limit based on this reliability growth program is a + d = 0.9974. That reliability level is estimated to be achieved after about three stages of testing (at the 14.68^th stage), as shown next.

Final Comments
Readers interested in seeing an additional software reliability growth application may wish to examine this case study.

*Other common models include Musa, Goel-Okumoto, Pareto and Yamada

Copyright © 2008 ReliaSoft Corporation, ALL RIGHTS RESERVED

Comments or Questions: Webmaster@ReliaSoft.com
weibull.com is a service of ReliaSoft Corporation
Copyright © 1992-2008 ReliaSoft Corporation, All Rights Reserved
Last updated: 06/26/2008 06:59:22 PM

SITE [Home] [Site Contents]
LEGAL [Copyrights & Notices]
[Terms of Use] [Linking Guidelines]
[On-Line Privacy Statement]