Elsevier

Safety Science

Volume 113, March 2019, Pages 310-317
Safety Science

Four studies, two methods, one accident – An examination of the reliability and validity of Accimap and STAMP for accident analysis

https://doi.org/10.1016/j.ssci.2018.12.002Get rights and content

Highlights

  • A choice of accident analysis methods influences analysis outcomes.

  • Data source used and analyst’s background and biases influence analysis outcomes.

  • STAMP tends to be more reliable than Accimap.

  • Accimap captures soft factors, but STAMP captures decision making and its context.

  • The combined use of two methods can be very useful, but practically challenging.

Abstract

The validity and reliability of human factors and safety science methods are some of the important criteria for judging their appropriateness and utility for accident analysis, however these are rarely assessed. The aim of this study is to take a closer look at the validity and reliability of two systemic accident analysis methods (Accimap and STAMP) by comparing the results of four studies which analysed the same accident (the South Korea Sewol Ferry accident) using two methods. Studies 1 and 2 used Accimap whilst Studies 3 and 4 applied STAMP. The four studies were compared in terms of analysis procedure taken, level of detail, causal factors identified, and the recommendations for improvements suggested by the methods. The results of the causal factor comparison indicate that the reliability (degree of overlap of causal factors identified from the same method, i.e. inter-analyst overlap) of STAMP (65%) is higher than Accimap (38%). The validity (degree of overlap of causal factors identified from two different methods) is as low as 8%. The comparison of recommendations indicates that STAMP-based analyses produce a wider range of recommendations across multiple system levels while Accimap-based analyses tend to focus on whole system-related recommendations. These findings suggest that the use of a more structured method like STAMP can help produce a more reliable accident analysis results.

Introduction

A variety of systemic accident analysis methods have been developed since the 1990s, partly as a way of responding to the increasing complexity of socio-technical systems across a range of domains including healthcare, nuclear power, rail and marine transportation (Waterson et al., 2015, Waterson et al., 2017). These systemic accident analysis (SAA) methods draw on sociotechnical systems and control theory (e.g., Accimaps - Rasmussen (1997), STAMP - Leveson (2004)) or resilience engineering (e.g., FRAM - Hollnagel (2004)). They commonly illustrate the diversity of causal factors across different levels of the systems, their interactions and the role played by external influences such as political, cultural, financial, and technological circumstances (Branford, 2011). The extent to which methods for systemic accident analysis produce outcomes which are valid (e.g., the degree to which the accident analysis method successfully identifies the causes of an accident) and reliable (e.g., the degree to which accident analysts using the same accident analysis method produce similar causal representations) are often viewed as an important criterion for judging their appropriateness for accident analysis (Waterson et al., 2017). According to Underwood and Waterson (2013), a lack of validation is the key issue which may influence the use of the system approach by practitioners. Likewise, Jacinto and Aspinwal, 2004, Stanton and Young, 1999, Stanton and Young, 2003 argue that the ability to learn the right lessons from accidents and to use them with confidence is dependent upon valid and reliable methods. As such, reliability and validation studies of accident analysis methods are widely viewed as desirable and a prerequisite for their use (Ryan, 2015).

There have been a few attempts to conduct various forms of comparative studies of Accimap and STAMP (Johnson and de Almeida, 2008, Katsakiori et al., 2009). Salmon et al. (2012) conducted a self-reflective comparative study of Accimap, HFACS and STAMP based analysis of the Mangatepopo gorge incident carried out by different experts. Underwood and Waterson (2014) evaluated the ATSB, Accimap and STAMP methods using two criteria: (1) coverage of systems theory concepts and (2) usage characteristics (e.g., reliability and validity). Their findings, in combination with others (e.g., Waterson et al., 2017) point to a need for a more systematic assessment of the validity and reliability of accident analysis methods. The aim of the current study is, therefore, to systematically assess the validity and reliability of two systemic accident methods: Accimap and STAMP. We identified four studies which analysed the same accident using Accimap and STAMP methods. Two studies applied Accimap (Kee et al., 2016, Lee et al., 2017), while of the other two applied STAMP (Kim et al., 2016b, Kwon, 2016). Three studies were published in peer-reviewed journals (Kee et al., 2016, Kim et al., 2016b, Lee et al., 2017) and the final one was an MSc dissertation (Kwon, 2016). In what follows, we first describe how the validity and reliability of accident analysis methods have been defined and evaluated in the literature (Section 2). Section 3 describes the criteria and procedure we used in order to assess reliability and validity across the four studies. The results, both qualitative and quantitative, are presented in Section 4. A final section (Section 5) discusses our findings and considers their implications for systemic accident analysis.

Section snippets

Validity

According to Branford (2007) the validity of accident analysis methods can be considered from two perspectives. One relates to the validity of the method itself that is, the extent to which the method is designed in a way that does what it is intended to do. The focus, in this case, is whether the process underlying the method is appropriate for its intended purposes. Both Accimap and STAMP methods, in theory, are valid from the first perspective, because they are developed to provide insights

A Brief description of the Sewol Ferry accident

On April 16, 2014, Sewol Ferry, the South Korean ship carrying 476 passengers from Inchon to Jeju Island, sank disastrously. The 18-year-old Japanese-built ship was purchased by a company named Chonghaejin, which added two more floors to the ship to hold more passengers, making the ship extremely unstable. During the voyage, when the ship made a sharp turn, lost its balance and started to list. When Captain Jun Seok Lee communicated with the Vessel Traffic Service (VTS) for help, the Captain

Findings

Table 1 summarises the comparison of four studies based on the framework. In subsequent sections of this paper, we refer to the studies with reference to their number in Table 1 (e.g., Study 1 is Lee et al., 2017).

The four studies have the same overall goal which was to apply a systems approach to analyse the South Korea Ferry accident and identify the range of contributing factors. Studies 1 and 2 applied the Accimap to analyse the accident and identify the contributing factors as well as

Summary of findings

The aim of this study was to systematically assess the validity and reliability of two accident methods (Accimap and STAMP). Compared with other studies with the similar aim (Hollnagel and Speziali, 2008, Underwood and Waterson, 2014, Waterson et al., 2017), this study carried out much more systematic comparison using both quantitative and qualitative analyses of outcomes and procedures of four different studies investigating the same accident. The results of the causal factor comparison

Conclusions and future work

This study sets out to systematically assess the validity and reliability of two systemic accident analysis methods (Accimap and STAMP) by comparing both quantitative and qualitative outputs of four studies which analysed the same accident (South Korea Sewol Ferry accident). The study has shown that STAMP (68% overlap of causal factors) tends to be more reliable than Accimap (35% overlap). The study has also shown that STAMP and Accimap identify a very different set of causal factors (8%

Acknowledgment

The work described in this paper was carried out whilst the first author was an academic visitor in the ‘Human Factors and Complex Systems Research Group’ at Loughborough University during the period 2016–2017. We would like to thank Professor Nancy Leveson for sharing one of her MSc student's dissertations which was analysed in this paper.

References (50)

  • S. Lee et al.

    Applying the Accimap methodology to investigate the tragic Sewol Ferry accident in South Korea

    Appl. Ergonom.

    (2017)
  • N. Leveson

    A new accident model for engineering safer systems

    Safety Sci.

    (2004)
  • J. Lundberg et al.

    What-you-look-for-is-what you-find – the consequences of underlying accident models in eight investigation manuals

    Safety Sci.

    (2009)
  • J. Lundberg et al.

    What you find is not always what you fix – how other aspects of accidents decide recommendations for remedial actions

    Acc. Anal. Prevent.

    (2010)
  • R. Nayak et al.

    'When Food Kills': a socio-technical systems analysis of the UK Pennington 1996 and 2005 E. coli O157 Outbreak reports

    Safety Sci.

    (2016)
  • J. Rasmussen

    Risk management in a dynamic society: a modeling problem

    Safety Sci.

    (1997)
  • P. Salmon et al.

    Systems based analysis methods: a comparison of Accimap, HFACS and STAMP

    Safety Sci.

    (2012)
  • D. Schmid et al.

    The System Theoretic Accident Modelling and Process (STAMP) of medical pilot knock-out events: Pilot incapacitation and homicide-suicide

    Safety Sci.

    (2018)
  • S. Sharples

    Commentary: analysis, investigation and judgement: the post-hoc application of human factors analyses to incidents

    Appl. Ergonom.

    (2017)
  • N.A. Stanton et al.

    Giving ergonomics away? The application of ergonomics methods by novices

    Appl. Ergonom.

    (2003)
  • N.J. Stevens et al.

    Sand, surf and sideways: a systems analysis of beaches as complex roadway environments

    Safety Sci.

    (2016)
  • I. Svedung et al.

    Graphic representation of accident scenarios: mapping system structure and the causation of accidents

    Safety Sci.

    (2002)
  • P. Underwood et al.

    Systemic accident analysis: examining the gap between research and practice

    Acc. Anal. Prevent.

    (2013)
  • P. Underwood et al.

    Systems thinking, the Swiss Cheese Model and accident analysis: a comparative systemic analysis of the Grayrigg train derailment using the ATSB, AcciMap and STAMP models

    Acc. Anal. Prevent.

    (2014)
  • O.A. Valdez Banda et al.

    A STAMP-based approach for designing maritime safety management systems

    Safety Sci.

    (2018)
  • Cited by (66)

    • Learning not blaming: Investigating ten fatal road traffic collisions using STAMP-CAST

      2023, Transportation Research Interdisciplinary Perspectives
    View all citing articles on Scopus
    View full text