Four studies, two methods, one accident – An examination of the reliability and validity of Accimap and STAMP for accident analysis
Introduction
A variety of systemic accident analysis methods have been developed since the 1990s, partly as a way of responding to the increasing complexity of socio-technical systems across a range of domains including healthcare, nuclear power, rail and marine transportation (Waterson et al., 2015, Waterson et al., 2017). These systemic accident analysis (SAA) methods draw on sociotechnical systems and control theory (e.g., Accimaps - Rasmussen (1997), STAMP - Leveson (2004)) or resilience engineering (e.g., FRAM - Hollnagel (2004)). They commonly illustrate the diversity of causal factors across different levels of the systems, their interactions and the role played by external influences such as political, cultural, financial, and technological circumstances (Branford, 2011). The extent to which methods for systemic accident analysis produce outcomes which are valid (e.g., the degree to which the accident analysis method successfully identifies the causes of an accident) and reliable (e.g., the degree to which accident analysts using the same accident analysis method produce similar causal representations) are often viewed as an important criterion for judging their appropriateness for accident analysis (Waterson et al., 2017). According to Underwood and Waterson (2013), a lack of validation is the key issue which may influence the use of the system approach by practitioners. Likewise, Jacinto and Aspinwal, 2004, Stanton and Young, 1999, Stanton and Young, 2003 argue that the ability to learn the right lessons from accidents and to use them with confidence is dependent upon valid and reliable methods. As such, reliability and validation studies of accident analysis methods are widely viewed as desirable and a prerequisite for their use (Ryan, 2015).
There have been a few attempts to conduct various forms of comparative studies of Accimap and STAMP (Johnson and de Almeida, 2008, Katsakiori et al., 2009). Salmon et al. (2012) conducted a self-reflective comparative study of Accimap, HFACS and STAMP based analysis of the Mangatepopo gorge incident carried out by different experts. Underwood and Waterson (2014) evaluated the ATSB, Accimap and STAMP methods using two criteria: (1) coverage of systems theory concepts and (2) usage characteristics (e.g., reliability and validity). Their findings, in combination with others (e.g., Waterson et al., 2017) point to a need for a more systematic assessment of the validity and reliability of accident analysis methods. The aim of the current study is, therefore, to systematically assess the validity and reliability of two systemic accident methods: Accimap and STAMP. We identified four studies which analysed the same accident using Accimap and STAMP methods. Two studies applied Accimap (Kee et al., 2016, Lee et al., 2017), while of the other two applied STAMP (Kim et al., 2016b, Kwon, 2016). Three studies were published in peer-reviewed journals (Kee et al., 2016, Kim et al., 2016b, Lee et al., 2017) and the final one was an MSc dissertation (Kwon, 2016). In what follows, we first describe how the validity and reliability of accident analysis methods have been defined and evaluated in the literature (Section 2). Section 3 describes the criteria and procedure we used in order to assess reliability and validity across the four studies. The results, both qualitative and quantitative, are presented in Section 4. A final section (Section 5) discusses our findings and considers their implications for systemic accident analysis.
Section snippets
Validity
According to Branford (2007) the validity of accident analysis methods can be considered from two perspectives. One relates to the validity of the method itself that is, the extent to which the method is designed in a way that does what it is intended to do. The focus, in this case, is whether the process underlying the method is appropriate for its intended purposes. Both Accimap and STAMP methods, in theory, are valid from the first perspective, because they are developed to provide insights
A Brief description of the Sewol Ferry accident
On April 16, 2014, Sewol Ferry, the South Korean ship carrying 476 passengers from Inchon to Jeju Island, sank disastrously. The 18-year-old Japanese-built ship was purchased by a company named Chonghaejin, which added two more floors to the ship to hold more passengers, making the ship extremely unstable. During the voyage, when the ship made a sharp turn, lost its balance and started to list. When Captain Jun Seok Lee communicated with the Vessel Traffic Service (VTS) for help, the Captain
Findings
Table 1 summarises the comparison of four studies based on the framework. In subsequent sections of this paper, we refer to the studies with reference to their number in Table 1 (e.g., Study 1 is Lee et al., 2017).
The four studies have the same overall goal which was to apply a systems approach to analyse the South Korea Ferry accident and identify the range of contributing factors. Studies 1 and 2 applied the Accimap to analyse the accident and identify the contributing factors as well as
Summary of findings
The aim of this study was to systematically assess the validity and reliability of two accident methods (Accimap and STAMP). Compared with other studies with the similar aim (Hollnagel and Speziali, 2008, Underwood and Waterson, 2014, Waterson et al., 2017), this study carried out much more systematic comparison using both quantitative and qualitative analyses of outcomes and procedures of four different studies investigating the same accident. The results of the causal factor comparison
Conclusions and future work
This study sets out to systematically assess the validity and reliability of two systemic accident analysis methods (Accimap and STAMP) by comparing both quantitative and qualitative outputs of four studies which analysed the same accident (South Korea Sewol Ferry accident). The study has shown that STAMP (68% overlap of causal factors) tends to be more reliable than Accimap (35% overlap). The study has also shown that STAMP and Accimap identify a very different set of causal factors (8%
Acknowledgment
The work described in this paper was carried out whilst the first author was an academic visitor in the ‘Human Factors and Complex Systems Research Group’ at Loughborough University during the period 2016–2017. We would like to thank Professor Nancy Leveson for sharing one of her MSc student's dissertations which was analysed in this paper.
References (50)
- et al.
Integrating systemic accident analysis into patient safety investigation practices
Appl. Ergonom.
(2018) - et al.
Analysis of soma mine disaster using causal analysis based on systems theory (CAST)
Safety Sci.
(2018) - et al.
Developing a contributing factor classification scheme for Rasmussen's Accimap: Reliability and validity evaluation
Safety Sci.
(2017) - et al.
Designing and evaluating a human factors investigation tool (HFIT) for accident analysis
Safety Sci.
(2005) - et al.
Evaluating safety in the management of maintenance activities in the chemical process industry
Safety Sci.
(1998) - et al.
An investigation into the loss of the Brazilian space programme's launch vehicle VLS-1 V03
Safety Sci.
(2008) - et al.
A systemic analysis of South Korea Sewol Ferry accident - striking a balance between learning and accountability
Appl. Ergonom.
(2017) - et al.
Assessment of accident theories for major accidents focusing on the MV SEWOL disaster: similarities, differences, and discussion for a combined approach
Safety Sci.
(2016) - et al.
A STAMP-based causal analysis of the Korean Sewol ferry accident
Safety Sci.
(2016) Human error identification in human reliability assessment: Part 2: detailed comparison of techniques
Appl. Ergonom.
(1992)
Applying the Accimap methodology to investigate the tragic Sewol Ferry accident in South Korea
Appl. Ergonom.
A new accident model for engineering safer systems
Safety Sci.
What-you-look-for-is-what you-find – the consequences of underlying accident models in eight investigation manuals
Safety Sci.
What you find is not always what you fix – how other aspects of accidents decide recommendations for remedial actions
Acc. Anal. Prevent.
'When Food Kills': a socio-technical systems analysis of the UK Pennington 1996 and 2005 E. coli O157 Outbreak reports
Safety Sci.
Risk management in a dynamic society: a modeling problem
Safety Sci.
Systems based analysis methods: a comparison of Accimap, HFACS and STAMP
Safety Sci.
The System Theoretic Accident Modelling and Process (STAMP) of medical pilot knock-out events: Pilot incapacitation and homicide-suicide
Safety Sci.
Commentary: analysis, investigation and judgement: the post-hoc application of human factors analyses to incidents
Appl. Ergonom.
Giving ergonomics away? The application of ergonomics methods by novices
Appl. Ergonom.
Sand, surf and sideways: a systems analysis of beaches as complex roadway environments
Safety Sci.
Graphic representation of accident scenarios: mapping system structure and the causation of accidents
Safety Sci.
Systemic accident analysis: examining the gap between research and practice
Acc. Anal. Prevent.
Systems thinking, the Swiss Cheese Model and accident analysis: a comparative systemic analysis of the Grayrigg train derailment using the ATSB, AcciMap and STAMP models
Acc. Anal. Prevent.
A STAMP-based approach for designing maritime safety management systems
Safety Sci.
Cited by (66)
Quantification analysis of potential risk in railway accidents: A new random walk based approach
2024, Reliability Engineering and System SafetyLearning not blaming: Investigating ten fatal road traffic collisions using STAMP-CAST
2023, Transportation Research Interdisciplinary Perspectives