Issue 105, November 2009

Operational Mission Profile Testing in RGA 7

During a development program, it is common practice for systems to be subjected to operational testing in order to evaluate the performance of the system under conditions that represent actual use. It is also common for reliability fixes to be implemented in conjunction with this testing and to perform reliability growth analysis on the data obtained. However, when the system must be tested for a variety of different mission profiles, it can be a challenge to make sure that the testing is applied in a balanced manner that will yield data suitable for reliability growth analysis. In this article, we show how reliability teams can use RGA 7's new Mission Profile Folios to incorporate operational mission profiles as part of their planning to:

• Create and manage an operational test plan that effectively balances all of the mission profiles that need to be tested.
• Track the expected vs. actual testing conducted for all mission profiles and validate that the testing has been conducted in a manner that will yield data sets that are appropriate for reliability growth analysis.

If there are any significant variations from the test plan that could jeopardize the analysis results, RGA can automatically group the data at specified "convergence points" so the growth model can be applied appropriately.

Introduction
Usually, stated mission profile conditions are used for operational testing. These mission profile conditions are typically general statements that guide testing on an average basis. For example, a copier might be required to print 10,000 pages by time T=15 days and 20,000 pages by time T=30 days. In addition, the copier is required to scan 200 documents by time T=15 days, 400 documents by time T=30 days, etc.

Because of practical constraints, these full mission profile conditions are typically not repeated one after the other during testing. Instead, the elements that make up the mission profile conditions are tested under varying schedules with the intent that, on average, the mission profile conditions are met. In practice, reliability corrective actions are generally incorporated into the system as a result of this type of testing.

In order to have valid reliability growth assessments, it is required that the operational mission profile be conducted in a structured manner. Therefore, the testing methodology described in this article involves convergence and stopping points during the testing.

A stopping point is when the testing is halted for the expressed purpose of incorporating delayed corrective actions. While there may be more than one stopping point during a particular testing phase, for simplicity, the methodology with only one stopping point will be described in this example. However, the methodology can be extended to the case of more than one stopping point.

A convergence point is a time during the test when all the operational mission profile tasks either meet their expected averages or fall within an acceptable range. At least three convergence points are required for a well-balanced test and the end of the test must be a convergence point. The test times between the convergence points do not have to be the same.

The objective of having the convergence points is to be able to apply the Crow Extended model directly in such a way that the projection and other key reliability growth parameters can be estimated in a valid fashion. To do this, the grouped data methodology is applied. Note that the methodology also can be used with the Crow-AMSAA (NHPP) model for a simpler analysis without the ability to estimate projected and growth potential reliability.

Example
Let’s explore the concept of operational mission profiles through a practical application.

A company that manufactures construction equipment is planning the reliability test of their new backhoe loader. The loader has four distinct functions that need to be tested:

• Traveling on asphalt roads.
• Loading up to 5,000lb using the front loader.
• Statically rotating 180 degrees in construction zones.
• Digging up to 12ft using the backhoe.

To ensure that the test is balanced, the reliability team decided to use operational mission profiles for all four functions to be tested. They decided to require convergence points at times T=2500, 5000 and 7200 hours.

Based on product specifications, they have specific usage requirements for each of the four functions. For example, in 2,500 hours of field use, it is expected that the average user will have driven the loader for 2,000 miles on asphalt roads, have used the front loader with 5,000lb loads for 18,750 times, used the backhoe to dig 12ft trenches for 4,990 times and performed 180 degree rotations in construction zones for 28,125 times. The reliability team gathers all this information to build an expected usage profile over the 7200 test hours, as shown in Table 1.

In reality, it is almost impossible to stay within the expected profiles for all of the functions at all times during the test. Even the most organized and balanced test will be ruled by special and common causes of variation that will result in divergence from the expected usage profiles at times during the test. The idea behind operational mission profiles is to set convergence points as targets in the test, where there will be a concerted effort to "catch up" or "slow down" in order to meet that expected average at the convergence point. In this example, this methodology involves monitoring the actual usage profiles for all of the four functions and using that input as actionable information in order bring the usage to its expected average at predefined points. It is essentially a test management methodology that assures that the overall test is balanced and representative of how the product will be used in the field.

RGA 7 has a built-in utility that allows the monitoring of operational mission profiles and the automated grouping of reliability growth test data into groups based on the predefined convergence points.

When a new mission profile is added, the first step is to specify the convergence points for the reliability growth test, as shown in Figure 1.

Figure 1: Specifying convergence points in RGA 7

The next step is to create operational mission profiles for each one of the distinct functions, such as the expected traveling distance in miles, as shown in Figure 2.

Figure 2: Setting expected traveling distance profile

After the expected usage has been entered for each one of the four profiles, the test can start. The reliability team will be monitoring actual usage for each of the four profiles, and take appropriate actions in order to meet the expected usage goals at each of the three convergence points. At the same time, failure time data can be logged using a formal reliability growth methodology, in this case the Crow Extended model.

Figure 3 shows the expected and actual usage profile for the front loads over the duration of the reliability growth test. We can see that the loads were lower than expected at the beginning of the test, so the management action was to accelerate front load usage to higher rates in order to meet the first convergence point. During the second period, from the first convergence point to the second convergence point, the actual usage starting exceeding the expected one, so the management action was to completely stop testing front loads from the time when the expected usage was met until the next convergence point of 5,000 hours.

Figure 3: Expected vs. actual usage for front loads throughout the reliability growth test

RGA 7 allows the user to plot expected vs. actual usage for all the profiles simultaneously in order to get a high level overview of the degree of test balance achieved, as shown in Figure 4. From the graph we can see that all profiles met their expected averages at the convergence points of 2,500, 5,000 and 7,200 hours, so the execution of this reliability growth test was successful in terms of managing all the operational mission profiles simultaneously.

Figure 4: Expected vs. actual usage for all profiles throughout the reliability growth test

When failure times data are collected during the test, as shown in Figure 5, the data can be grouped based on the convergence points to assure that the overall results reflect the balanced usage profiles for all four functions.

Figure 5: Failure times data for the reliability growth test

By clicking the mission profile analysis icon, , we can associate an operational mission profile with this data set, as shown in Figure 6.

Figure 6: Associating a mission profile with a data set

Using this utility, the data set is automatically grouped based on intervals specified by the convergence points in the associated mission profile. The grouped data and the final results are shown in Figure 7.

Figure 7: Grouped data and results based on associated mission profile

Conclusions
Mission profiles are a powerful way to simulate the expected field usage of a complicated system with different distinct functions. RGA 7 allows you to create mission profiles and provides an easy way to visually track and manage that the actual usage for each of the profiles meets their expected average at the predefined convergence points. Then the same convergence points can be used to automatically group data to assure that the analysis is based on a balanced test. This ultimately reduces the test management and execution complexity, while bringing the reliability growth analysis results closer to what the actual customer is going to be experiencing in the field.

References
Portions reprinted, with permission, from:
Crow, Larry, H., "A Methodology for Managing Reliability Growth During Operational Mission Profile Testing" Reliability and Maintainability Symposium, 2008, pp: 48-53, 10.1109/RAMS.2008.4925768.© 2008 IEEE.

Copyright © 2009 ReliaSoft Corporation, ALL RIGHTS RESERVED