Issue 71, January 2007

How Good Is Your Assumed Distribution's Fit?

After fitting a distribution model to a data set when performing life data analysis, we are often interested in diagnosing the model's fit or comparing the fit of different distributions. In addition to the engineering knowledge that should always govern the choice of a distribution model, there are many statistical tools that can help in deciding whether or not a distribution model is a good choice from a statistical point of view. These tools can also be used to compare the fit of different distributions. This article presents a survey of various statistical tools available in Weibull++ that can be used to assess the fit of a distribution model and compare it to other distributions.

For the remainder of this article, we will use the following data set to explain the different ways to asses the fit of one or multiple distributions. For comparisons, we will use as an example the Weibull distribution and the exponential distribution. (Note, however, that the concept can be used to compare more than two distributions.)

Table 1: Sample data set

Probability Plots
Probability plotting is a graphical method that allows a visual assessment of the model fit. Once the model parameters have been estimated, the probability plot can be created. The next figure shows a comparison of the probability plots of two choices of distributions using the same data set.

Figure 1: Comparing the probability plots of two distributions using the same data set

The plots show that the Weibull distribution fits the data well and is a better fit than the exponential distribution.

Note:
This method can be used if the Least Square Parameter Estimation (Rank Regression) method has been used to estimate the parameters of the distribution. It should not be used if the MLE (Maximum Likelihood) parameter estimation method has been used to fit the distribution model. Even though it is common practice to plot the MLE solutions along with median ranks (i.e. points are plotted according to median ranks and the line according to the MLE solutions), this is not completely representative. The MLE method is actually independent of any kind of ranks. For this reason, the MLE solution often appears not to track the data on the probability plot. This is perfectly acceptable, as the two methods are independent of each other, and in no way suggests that the solution is wrong.

Correlation Coefficient
The correlation coefficient, usually denoted by ρ (Rho), is a measure of how well the linear regression model (the probability line) fits the data. In the case of life data analysis, it is a measure of the strength of the linear relation (correlation) between the median ranks and the data. The population correlation coefficient is defined as follows:

where σ_xy is the covariance of x (times-to-failure) and y (median ranks), σ_x is the standard deviation of x, and σ_y is the standard deviation of y.

The estimator of ρ is the sample correlation coefficient, , given by:

The range of is -1 ≤ ≤ 1.

The closer the value of is to 1 or -1 (or the closer the absolute value is to 1), the better the linear fit. Note that +1 indicates a perfect fit (i.e. the paired values (x_i, y_i) lie on a straight line) with a positive slope, while -1 indicates a perfect fit with a negative slope. A correlation coefficient value of zero would indicate that the data are randomly scattered and have no pattern or correlation in relation to the regression line model.

Using the data set presented in Table 1 and using the Rank Regression on X method to estimate the parameters, we make the following comparison:

Table 2: Comparing the correlation coefficients of two distributions using the same data set

The above table shows that the Weibull distribution is a very adequate model (i.e. || is close to 1). Also, the absolute value of the correlation coefficient for the Weibull distribution is greater than that for the exponential distribution (i.e. the Weibull distribution is statistically a better fit).

Note:
The correlation coefficient value is displayed by default in Weibull++ below the estimated parameter values on the Main tab of the Standard Folio's Control Panel.

Likelihood Value
When using the MLE (Maximum Likelihood Estimation) method to estimate the parameters of the distribution model, the likelihood value can be used to assess the fit of the distribution to the data set. The likelihood value (or function), L, is the basis of the MLE parameter estimation method. It is mathematically formulated as follows:

where:

• R is the number of units with exact times-to-failure.

• M is the number of suspended units.

• P is the number of units with left censored or interval times-to-failure.

• θ₁, θ₂, ..., θ_k are the parameters of the distribution.

• T_i is the i^th time to failure.

• S_j is the j^th time of suspension.

• is the ending of the time interval of the l^th group.

• is the beginning of the time interval of the l^th group.

Unlike the correlation coefficient, the likelihood value is not constrained by a certain range of possible values. L can have any value and therefore cannot be used by itself to make a judgment about the fit of the distribution model. L can, however, be used to compare the fit of multiple distributions. The distribution with the largest L value is the best fit statistically.

Note that the likelihood values shown in Weibull++ are actually the log-likelihood values, not the likelihood values. The log-likelihood function is used instead because it is much easier to work with than L for parameter estimation. Using the log-likelihood function does not affect the validity of the results.

Using the data set presented in Table 1 and using the MLE method to estimate the parameters, we make the following comparison:

The above table shows that the log-likelihood value for the Weibull distribution is greater than that for the exponential distribution (i.e. the Weibull distribution is statistically a better fit).

Note:
The log-likelihood value is displayed by default in Weibull++ below the estimated parameter values on the Main tab of the Standard Folio's Control Panel.

Modified Kolmogorov-Smirnov (KS) Test
The standard Kolmogorov-Smirnov (KS) test can only be used when determining the fit of a continuous distribution with known parameters. In life data analysis, the parameters are typically unknown and need to be estimated from the sample data. Therefore, another type of KS test is used, called the Modified KS test.

If the data set is made of N failure times (t₁, t₂, ..., t_N), we can define S_N(t) to be the function giving the fraction of data points to the left of a given value t_i (i = 1, 2,, ..., N). S_N(t) is constant between consecutive t_i values, and jumps by the same constant 1/N value at each t_i.

The Modified KS test uses D_max, the maximum of the absolute difference between S_N(t) and the fitted cumulative distribution function, Q(t). [Ref. 1]

What makes the Modified KS test useful is that its distribution in the case of the null hypothesis (i.e. data set drawn from the fitted distribution) can be calculated, at least to a useful approximation, thus giving the significance of any observed non-zero value of D_max.

The Modified KS test returns the probability that D_CRIT < D_max. A high probability value, close to 1, indicates that there is a significant difference between the theoretical distribution and the data set.

Using the data set presented in Table 1 and using the MLE method to estimate the parameters, we make the following comparison:

The above figure shows that the value of P(D_CRIT < D_max) for the Weibull distribution is smaller than that for the exponential distribution (i.e. the Weibull distribution is statistically a better fit).

Note:

• The Modified KS test can be used for small sample sizes.

• The Modified KS test result can be obtained in Weibull++ by selecting Goodness of Fit Results from the Data menu.

Chi-Squared Test
The chi-squared test relies on the idea of binning the data, or grouping the data into a suitable number of intervals (as in histograms). Binning involves a loss of information, and there is also often considerable arbitrariness in how the bins are chosen. The optimal number of class intervals, k, for a data set with sample size N may be estimated from Sturges' Rule [Ref. 1]:

Suppose that N_i is the number of data points in the ith bin and n_i is the number expected according to the fitted distribution. The chi-squared statistic is then [Ref. 1]:

where the sum is over all the bins. Large values of χ² indicate that the null hypothesis is rather unlikely. In other words, it is not likely that the N_i's are drawn from the population represented by the n_i's, (i.e. the fitted model actually fits the data).

The χ² value follows a distribution that can be approximated by the chi-squared probability function, especially when the number of bins is much greater than 1 or the number of data points in each bin is much greater than 1.

The chi-squared test returns the probability that χ²_CRIT < χ². A high probability value, close to 1, indicates that there is a significant difference between the theoretical distribution and the data set.

Using the data set presented in Table 1 and using the MLE method to estimate the parameters, we make the following comparison:

The above figure shows that the value of P(χ² _CRIT < χ²) for the Weibull distribution is smaller than that for the exponential distribution (i.e. the Weibull distribution is statistically a better fit).

Note:

• The chi-squared test is not valid for small sample sizes; a minimum sample size of 25-35 is required for results to be valid. [Ref. 1]

• The chi-squared test is less powerful than the Modified KS test for any sample size. [Ref. 1]

• The chi-squared test result can be obtained in Weibull++ by selecting Goodness of Fit Results from the Data menu.

References
1. Kececioglu, Dimitri, Reliability and Life Testing Handbook, Volume I, Prentice Hall, Inc., New Jersey, 1993.

Copyright 2007 ReliaSoft Corporation, ALL RIGHTS RESERVED