Reliability HotWire

Issue 97, March 2009

Hot Topics
Using Multiple Reliability Tools to Augment Limited Data

In last months Hot Topics article, Joe the reliability engineer was forced to make a decision regarding the B10 life of two products with very limited data. Having to make an important decision based on only a few data points is common when suppliers have expensive hardware and/or limited access to testing equipment. In this case, additional research is often necessary. Other reliability tools, such as comparing test data to field data, relating failure modes seen during the test to a failure modes and effects analysis (FMEA) and even using engineering judgment may be employed to substantiate the conclusions reached with limited test data. This article will discuss some follow-up questions that Joe might ask in order to strengthen his analysis.

To download the complete *.rso7 file for this example, right-click here to save the file to your computer. You must have Weibull++ installed in order to be able to open this file. Free demonstration copies of the software are available for download from

For health and safety reasons, federal regulators had banned a substance that was found in Product A. This ban required that all manufacturers discontinue the use of Product A in three months' time. After two months of frantically searching for a suitable replacement, the engineers at Joes company found a candidate, Product B. After discussions with management, Joe was given 5 samples of Product B, which were made with rapid prototyping (soft) tools. He rushed these samples to an independent laboratory for testing. In the meantime, he found the results from the 10 tests on Product A that had been ordered by his predecessor. When the results from the tests on Product B arrived, Joe quickly ran the analysis that was described in last month's issue of Reliability HotWire. He found that the B10 life of Product A had a median estimate of 311 hours, while the B10 life of Product B had a median estimate of 453 hours. However, because Joe had so few samples, when he added 90% 2-sided confidence bounds to the estimates, there was overlap between the estimates (135 hours to 719 hours for Product A and 218 to 947 hours for Product B). Therefore, Joe was unable to tell if there was a difference between the two products at the 90% confidence level. This left Joe feeling uneasy, but his company needed a decision in less than 2 weeks regarding whether Product B was a suitable substitute for Product A. So Joe decided to try to find more information about the two products to help him make a decision.

Actions Taken
Joe starts by making some phone calls. First, he talks to his contact, Connie, at the laboratory that had tested the products. He asks her about the test procedure and what types of failure modes were observed during the test of Product B. She tells him that four of the failures of Product B were fatigue failures, which were confirmed by the striations on the fracture surfaces of the specimens. The other failure (the one that occurred at 402 hours) did not show any evidence of fatigue; rather, that specimen failed due to a failure of the test apparatus. Joe also asks Connie if she can remember anything about the testing of Product A. Although those tests were run several years ago, she looks up her notes from the tests. She says that six of the tests on Product A were done four months earlier than the other tests. She also says that the failure analysis showed that the six early tests showed both fatigue and impact failures, but she does not know which failure times were the impact failures. She says the later four tests lasted much longer than the original six and exhibited only fatigue failures.

Joe goes to his data file (right-click here to save Joes original file to your computer), duplicates his original analysis Folio and changes the name of the new Folio to Folio2. In Folio2, he clicks the Alter Data Type icon.

In the Data Sheet Setup window that appears, Joe selects the My data set contains suspensions (right censored data) option and clicks OK. This adds a column to the Data Sheet so that Joe can specify that the failure at 402 hours was a suspension, since it failed due to the failure of the test fixture. Joe then calculates the Data Sheet and obtains the parameters shown in Figure 1.

Figure 1: Updating Data Point Classification for Product B

Next, Joe calls his friend Fred, who works for a division of Joe's company that uses Product A in the field. Joe asks Fred how long he thinks Product A typically lasts, and Fred says that a "bad" Product A would last 2000 hours, while a "good" Product A would last 3000 hours. Joe knows from his Product A laboratory test data set that he has seen quite a few failures at times significantly less than 2000 hours, so he asks Fred if he has ever experienced a Product A that lasted less than, say, 1500 hours. Fred cannot remember that happening in all the years he has been using Product A, but says that he will look back at the company's failure reporting and corrective action (FRACAS) system to see what additional data he can find.

Joe is still puzzled by the early failures seen in the Product A data, so he asks his colleague Jen to let him look at the FMEA that has been done on Product A. Fortunately for Joe, his company is very diligent in ensuring that their FMEAs are living documents, and they have updated the Product A FMEA frequently. He finds out the following things from the FMEA:

  • The primary failure mode of Product A in the field is fatigue.
  • The test results that he has for Product A were done on a mixture of specimens. Six specimens that were made with rapid prototyping tools were tested first. Based on these tests, a small design change was implemented. Four additional specimens that were made with production (hard) tools were tested to validate the design change.

Joe knows from his experience on other jobs that products made with hard tools last longer than those made with soft tools. His engineering judgment tells him that he can expect an increase in life in Product B of about two times when the switch to the hard tools is implemented.

Joe goes back to his Product A data and updates the subset ID column to reflect his new knowledge about the data set, as shown in Figure 2.

Figure 2: Specifying Product A Subset ID Data

He clicks the Batch Auto Run icon.

In the Batch Auto Run window, Joe selects both Product A Soft Tools and Product A Hard Tools for consideration in the batch process in order to separate the data into two Data Sheets based on the subset ID column. The calculated parameters for these two data sets are shown in Figures 3 and 4.

Figure 3: Parameters for Soft Tool Data of Product A

Figure 4: Parameters for Hard Tool Data of Product A

Just as he finishes separating the Product A data, Joe receives an e-mail from his friend Fred that contains a spreadsheet with times-to-failure data for Product A from the field. This data set contains 119 suspensions and 10 failures. He copies this data into a new Folio, called Folio3. Figure 5 shows the calculated parameters for this data set.

Figure 5: Field Data for Product A

First, Joe looks at the laboratory test results. He adds MultiPlot2, including the data from the Product A Soft Tools, Product A Hard Tools, and Product B Data Sheets in Folio2. Figure 6 shows the resulting MultiPlot.

Figure 6: Probability Plots of Laboratory Test Data for Products A and B

The line on the left represents the tests done on Product A made using soft tools, the middle line represents the tests done on Product B made using soft tools and the line on the right represents the tests done on Product A made using hard tools. Joe is not surprised to see that the Product A Soft Tools line has a different slope than the others, as it includes failures from both impact and fatigue, while the other two lines have only fatigue failures. Also, he notes that the Product A Soft Tools line is well to the left of the Product A Hard Tools line, which is what he expected given his previous experience with rapid prototype and production tooling.

Joe changes the plot type to Contour Plot and selects the 90% confidence level to obtain the plot shown in Figure 7.

Figure 7: Contour Plots for Laboratory Test Data for Products A and B

The left contour represents Product A made using soft tools, the middle contour represents Product B made using soft tools, and the right contour represents Product A made using hard tools. It is clear from this plot that at a 90% confidence level these populations are different.

Next, Joe adds MultiPlot3, including Product A Hard Tools from Folio2 and the field data for Product A from Folio3 in order to see how well the laboratory test predicted the field data. This plot is shown in Figure 8. Joe is pleased to see how well the laboratory test predicted the life of Product A in the field. He quickly checks the contour plots, shown in Figure 9, to ensure that there is overlap in the contours at the 90% confidence level. (Note that since there are many more data points in the field data set, the resulting contour is smaller than that of the laboratory data set.) This comparison gives Joe additional confidence that he can make a decision based on the results of the laboratory tests.

Figure 8: Probability Plot of Laboratory and Field Data for Product A made with Production Tooling

Figure 9: Contour Plot of Laboratory and Field Data for Product A made with Production Tooling

Given the following findings, Joe is confident that he can recommend Product B as a suitable substitute for Product A:

  • Products made using production tooling last longer than those using soft tooling. Joes experience says that the difference in life is at least a factor of 2.
  • Product B made using soft tooling has a longer life than Product A made using soft tooling, as seen in the probability and contour plots. In spite of the fact that Product A using soft tooling showed both impact and fatigue failures, every data point in the Product B data set is longer than the Product A data set for products made using soft tooling.
  • Product B using soft tooling exhibits the same failure modes as Product A using hard tooling and Product A in the field.
  • The analysis for Product A using hard tooling laboratory data predicts the life of Product A in the field very well.

Copyright 2009 ReliaSoft Corporation, ALL RIGHTS RESERVED