Using automation to design written procedures for high-risk industries

A Texas A&M University research team’s assessment of how different machine learning algorithms predict procedure performance, based on the characteristics of the procedure and the workers following them, has been awarded the Human Factors Prize, an annual award established to recognize excellence in human factors/ergonomics research.

S. Camille Peres, PhD, of the Texas A&M School of Public Health Department of Environmental and Occupational Health and Anthony McDonald, PhD, of the Wm Michael Barnes ‘64 Department of Industrial and Systems Engineering led the team, which included Nilesh Ade, a doctoral student in the Artie McFerrin Department of Chemical Engineering. Their findings, published in the journal Human Factors, detailed how the researchers collected data from 25 operators performing tasks in a simulated offshore oil production facility at Shell’s Robert training center in Louisiana. The operators were given four procedures that are typically performed as part of normal offshore oil production, two of which are frequent procedures and two of which are less frequent. Each of the procedures consisted of between eight and 23 steps.

The operators were video recorded while following written procedures to perform each of the procedures, and their performance was analyzed by a panel of reviewers later. Step performance was scored as either correct or incorrect. If operators skipped a step, performed it out of order or incorrectly, needed assistance or took too long to complete it, that step was coded as incorrect.

The researchers also collected data on operator experience, performance in signing off on individual steps as they were completed and self-reported familiarity with actual procedures. In addition, Peres and colleagues also analyzed the written procedures, measuring the total number of steps and complexity of each step as well as the readability and language processing factors such as word and character types and use of capital letters in each step.

With the data collected on operator and procedure characteristics and step performance, McDonald and colleagues then tested four algorithms to analyze how well they predicted whether operators would complete the steps correctly. Of interest was whether machine learning algorithms could accurately predict operator performance based on operator and procedure characteristics. All four algorithms performed similarly and had an acceptable level of accurately predicting whether steps would be followed correctly.

The analysis also found that the characteristics of both procedure steps and operators are crucial. For example, steps with fewer than 20 words were more likely to be performed correctly, and greater operator familiarity with procedures increased the odds of correct step performance. These findings indicate the importance of optimized language in written procedures as well as operator experience.

However, as notable as these results are, the researchers caution that further research is needed for such algorithms to be used in designing procedures. The similar performance of the algorithms was surprising, and although it may be due to the small sample size in this study, further research is needed. Additionally, more data from different operators and more detailed measures of step complexity and operator experience would be helpful. The researchers also note that the simulation environment used may not fully capture real procedure performance, which is affected by stress and fatigue. Further studies should also attempt to determine how stress and fatigue affect step performance.

Despite these limitations, the findings of this study serve as a significant first step in the use of machine learning to design written procedures in high-risk industries. With additional data and analysis, procedure design may join many other fields that are being revolutionized by these new technologies. Most importantly, this study is an important example of how multidisciplinary collaborations—safety (Peres), machine learning (McDonald) and chemical engineering (Ade)—can result in important novel findings that can facilitate safety in high-risk industrial settings.