Measuring the impact of national guidelines: What methods can be used to uncover time-varying effects for healthcare evaluations?

https://doi.org/10.1016/j.socscimed.2020.113021Get rights and content

Highlights

  • Different methods can be used to measure the time-varying and treatment effects of guidelines.

  • We compare difference-in-differences and semiparametric varying-coefficient methods.

  • The 2015 UK national suspected-cancer referral guidelines are used as an example.

  • Two of the three methods identify time-varying effects.

  • There are no differences over time between men and women or for age.

Abstract

We examine the suitability of three methods using patient-level data to evaluate the time-varying impacts of national healthcare guidelines. Such guidelines often codify progressive change and are implemented gradually; for example, National Institute for Health and Care Excellence (NICE) suspected-cancer referral guidelines. These were revised on June 23, 2015, to include more cancer symptoms and test results (“features”), partly reflecting changing practice. We explore the time-varying impact of guideline revision on time to colorectal cancer diagnosis, which is linked to improved outcomes in decision-analytic models. We included 11,842 patients diagnosed in 01/01/2006–31/12/2017 in the Clinical Practice Research Datalink with England cancer registry data linkage. Patients were classified by whether their first pre-diagnostic cancer feature was in the original guidelines (NICE-2005) or was added during the revision (NICE-2015-only). Outcome was diagnostic interval: time from first cancer feature to diagnosis. All analyses adjusted for age and sex. Two difference-in-differences analyses used either a Pre (01/08/2012–31/12/2014, n = 2243) and Post (01/08/2015–31/12/2017, n = 1017) design, or event-study cohorts (2006–2017 vs 2015) to estimate change in diagnostic interval attributable to official implementation of the revised guidelines. A semiparametric varying-coefficient model analysed the difference in diagnostic interval between the NICE groups over time. After model estimation, primary and broader treatment effects of guideline content and implementation were measured. The event-study difference-in-differences and the semiparametric varying-coefficient methods showed that shorter diagnostic intervals were attributable to official implementation of the revised guidelines. This impact was only detectable by pre-to-post difference-in-differences when the pre/post periods were selected according to the estimation results from the varying-coefficient model. Formal tests of the parametric models, which are special cases of the semiparametric model, suggest that they are misspecified. We conclude that the semiparametric method is well suited to explore the time-varying impacts of guidelines codifying progressive change.

Introduction

Many countries have bodies that develop national healthcare guidelines; for example, The National Institute for Health and Care Excellence (NICE) in England, The American College of Physicians in the USA, and the National Health and Medical Research Council in Australia. A guideline's impact is likely to vary over time, dependent on structural and behavioural change, and may even start before publication if the guideline codifies improving clinical practice. As randomised controlled trials are often impracticable or unethical, there is considerable interest in methods applicable to observational data for quantifying the time-varying impacts of health-policy change (Craig et al., 2012). The current paper explores methods for capturing the time-varying effects of guidelines on health outcomes within repeated cross-sectional data with a control group. We illustrate these methods using the NICE suspected-cancer referral guidelines, first introduced in 2005 (NICE, 2005) and revised in 2015 (NICE, 2015). The guidelines aim to expedite cancer diagnosis for symptomatic patients attending primary care, because late diagnosis is associated with poor survival (Elliss-Brookes et al., 2012). The 2005 guidelines were based mainly on secondary care evidence, and included signs, symptoms and abnormal test results (collectively termed “clinical features”) suggesting a risk of undiagnosed cancer 5% (NICE, 2005). The 2015 revision was based solely on primary-care evidence and added new clinical features. It also lowered the threshold risk to 3%, to liberalise investigation enough to expedite cancer diagnosis without overwhelming clinical services or increasing patient harms from over-investigation (NICE, 2015). The NICE guidelines apply in England, but are also endorsed in Wales and Northern Ireland, and are supported by The Independent Cancer Taskforce (Kumar, 2015).

One notable feature of these guidelines is that they codify the evidence base and best practice that develops over time, a characteristic not commonly encountered in intervention-evaluation studies. The most conservative measure of the effect of the guidelines is to measure the impact from the date of their official implementation, and this is the primary impact we measure. Broader effects can also be measured, such as those linked to guideline content and occurring before official implementation. This requires identifying a key moment in time from which we can explore change, which can be challenging.

We explore three methods that use a control group to assess the impact of revising the suspected-cancer guidelines on time to diagnosis of colorectal cancer – an outcome linked with improved survival in decision-analytic models (Tørring et al., 2013). The three methods are pre-to-post and event-study difference-in-differences, and a semiparametric varying-coefficient model.

The pre-to-post difference-in-differences method is widely applied in health-services research (Streeter et al., 2017), and was used to quantify the impact of the original 2005 NICE suspected-cancer referral guidelines on the timeliness of cancer diagnosis (Neal et al., 2014). The outcome measure was diagnostic interval, i.e. the time between first presentation of a cancer symptom to primary care and diagnosis (Weller et al., 2012). Diagnostic intervals were shorter in 2007–08 compared with 2001–02; however, this was only attributable to the new guidelines in two cancer sites (oesophagus and cervix) (Neal et al., 2014). It is possible that the guidelines had little impact, and/or that Neal et al.‘s chosen control group did not capture the secular trend well. Alternatively, the pre-to-post difference-in-differences method may not be well equipped to measure time-varying changes in impact. This possibility is one focus of our paper.

The event-study difference-in-differences has been used to explore the phased impact of policy, by introducing a series of dummy variables into the difference-in-differences regression model, each indicating time relative to a new policy (Wing et al., 2018). These dummies are typically yearly or monthly, as used to examine the effects of US government school-desegregation policies on racial mix over time (Reber, 2005), and health-insurance mandates on mammography and breast cancer diagnosis (Bitler and Carpenter, 2016). This method can also be used to explore the time trends associated with pre-intervention anticipatory effects; for example, to examine the impact of changes in medical care insurance (Alpert, 2016; Brot-Goldberg et al., 2017) and hospital reform (Kolstad and Kowalski, 2012).

Finally, we consider the semiparametric varying-coefficient model (Li et al., 2002). Here, the pattern of time-varying effects is not restricted by any pre-specified parametric functional form, such as occurs with the common practice of adding interaction terms to a regression model. To our knowledge, the semiparametric varying-coefficient model is rarely applied in healthcare research. It has been used to explore time trends in the control and intervention arms within randomised controlled trials (Gilbert et al., 2019; Wasan et al., 2017), and the time-varying aspects of tumour characteristics on disease-free survival in breast cancer in an observational dataset (Natarajan et al., 2009).

There are three main ways in which the semiparametric model, applied to repeated cross-sectional data, provides additional insights over parametric models in healthcare studies assessing the impact of interventions, particularly when the intervention is a guideline as we consider here. First, the method can account for the time-varying impact of all control variables (for example, age, sex, and intervention). Second, applying the semiparametric varying-coefficient model, it is possible to test the validity of the conventional parametric specifications, such as the pre-to-post and the event-study difference-in-differences methods. Third, once the varying-coefficient model has been run, treatment effects of guideline implementation, as well as the combining effects of both guideline content and subsequent implementation can be estimated. Given the novelty of the method, our aim is to illustrate the method's potential and any assumptions required without being prescriptive about which of these features are ultimately used by researchers.

Our study adds to the existing literature in the following five ways. First, we describe the three methods and summarise their distinguishing characteristics. Second, we show how the semiparametric model can be used in preliminary analysis for data visualisation of the time-varying aspects of covariates. Third, we show that the pre-to-post and event-study difference-in-differences models are special cases of the semiparametric varying-coefficient model, enabling a formal test of the parametric models against the varying-coefficient model. Fourth, we show how the semiparametric method can be used in causal analysis to measure treatment effects. Finally, we report the first comparison of these difference-in-differences methods and the semiparametric varying-coefficient model to assess the impact of national guidelines in patient-level health records, using NICE referral guidelines for suspected colorectal cancer as an illustration.

Section snippets

Study setting and inclusion criteria

The setting was primary care and the data sources were the Clinical Practice Research Datalink (CPRD), a dataset of observational, anonymised, patient records in UK primary care, with partial linkage to National Cancer Registration and Analysis Service (NCRAS) and Office for National Statistics (ONS) data (Herrett et al., 2015). Ethics approval by the Independent Scientific Advisory Committee (protocol 16_037) was obtained on October 30, 2017. The CPRD reduced temporarily in size from 2013 to

Pre-to-post difference-in-differences

In the pre-to-post difference-in-differences method (Eq. (1)), the outcome of interest (diagnostic interval) is measured in four groups, control (NICE-2005) and treated (NICE-2015-only), before and after the treated group is managed according to the new guidelines. The effect attributable solely to official implementation of the new guidelines is estimated by the coefficient for the interaction term (δ3) between the dummy variables dT (indicating before or after revised guideline

Study cohorts

The CPRD provided 25,011 patients with an incident colorectal cancer diagnosis between January 1, 2006 and December 31, 2017. Of these, 13,169 were excluded, mostly because they did not present with features of colorectal cancer (n = 9693) or because they resided in Scotland (n = 3105) where different guidelines apply. Patients diagnosed following screening (n = 52), or with multiple index cancers (n = 52), or who did not attend general practice (n = 4) were also excluded, leaving 11,842

Discussion

This is the first study, to our knowledge, that compares methods to measure the time-varying impacts of national guidelines using repeated, cross-sectional patient-level data.

Our data source was the CPRD – one of the largest database of anonymised patient records in the world, and a widely used resource for primary care research and epidemiology studies (Herrett et al., 2015). Robust methods were used for case identification in the electronic medical record (Watson et al., 2017). Our sample

CRediT authorship contribution statement

Sarah Price: Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Xiaohui Zhang: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Data curation, Writing - original draft, Writing - review & editing, Supervision. Anne Spencer: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing, Supervision, Project administration, Funding

Acknowledgements

This research arises from the Cancer Research UK funded study to explore the impact of NICE guidelines for early cancer diagnosis [C56843/A21550] in which Sarah Price is the Research Fellow and Anne Spencer is the joint principal investigator. This research is linked to the CanTest Collaborative, which is funded by Cancer Research UK [C8640/A23385], of which Sarah Price is an affiliated Research Fellow and Anne Spencer is Senior Faculty.

We would like to thank Professor Willie Hamilton for his

References (36)

  • Z.C. Brot-Goldberg et al.

    What does a deductible do? The impact of cost-sharing on health care princes, quantities, and spending dynamics

    Q. J. Econ.

    (2017)
  • D. Card et al.

    Minimum-wages and employment – a case-study of the fast-food industry in New-Jersey and Pennsylvania

    Am. Econ. Rev.

    (1994)
  • P. Craig et al.

    Using natural experiments to evaluate population health interventions: new MRC guidance

    J. Epidemiol. Community Health

    (2012)
  • L. Elliss-Brookes et al.

    Routes to diagnosis for cancer – determining the patient journey using multiple routine data sets

    Br. J. Canc.

    (2012)
  • W. Hamilton et al.

    The risk of colorectal cancer with symptoms at different ages and between the sexes: a case-control study

    BMC Med.

    (2009)
  • T. Hayfield et al.

    Nonparametric econometrics: the np package

    J. Stat. Software

    (2008)
  • E. Herrett et al.

    Data resource profile: Clinical Practice Research Datalink (CPRD)

    Int. J. Epidemiol.

    (2015)
  • N. Kreif et al.

    Examination of the synthetic control method for evaluating health policies with multiple treated units

    Health Econ.

    (2016)
  • Cited by (2)

    • Trends in time to cancer diagnosis around the period of changing national guidance on referral of symptomatic patients: A serial cross-sectional study using UK electronic healthcare records from 2006–17

      2020, Cancer Epidemiology
      Citation Excerpt :

      Our analytical method allowed us to explore trends in the difference in diagnostic interval between groups aligned by their index feature(s) to the revised (New-NICE) or original (Old-NICE) guidance [20]. The method was derived to explore the time-varying and gradual impact of emerging clinical evidence that is legitimised into clinical practice by official guidance revision and implementation [20]. Our findings build on previous analysis of the original 2005 NICE guideline’s impact on diagnostic interval [13].

    View full text