Measuring the impact of national guidelines: What methods can be used to uncover time-varying effects for healthcare evaluations?
Introduction
Many countries have bodies that develop national healthcare guidelines; for example, The National Institute for Health and Care Excellence (NICE) in England, The American College of Physicians in the USA, and the National Health and Medical Research Council in Australia. A guideline's impact is likely to vary over time, dependent on structural and behavioural change, and may even start before publication if the guideline codifies improving clinical practice. As randomised controlled trials are often impracticable or unethical, there is considerable interest in methods applicable to observational data for quantifying the time-varying impacts of health-policy change (Craig et al., 2012). The current paper explores methods for capturing the time-varying effects of guidelines on health outcomes within repeated cross-sectional data with a control group. We illustrate these methods using the NICE suspected-cancer referral guidelines, first introduced in 2005 (NICE, 2005) and revised in 2015 (NICE, 2015). The guidelines aim to expedite cancer diagnosis for symptomatic patients attending primary care, because late diagnosis is associated with poor survival (Elliss-Brookes et al., 2012). The 2005 guidelines were based mainly on secondary care evidence, and included signs, symptoms and abnormal test results (collectively termed “clinical features”) suggesting a risk of undiagnosed cancer 5% (NICE, 2005). The 2015 revision was based solely on primary-care evidence and added new clinical features. It also lowered the threshold risk to 3%, to liberalise investigation enough to expedite cancer diagnosis without overwhelming clinical services or increasing patient harms from over-investigation (NICE, 2015). The NICE guidelines apply in England, but are also endorsed in Wales and Northern Ireland, and are supported by The Independent Cancer Taskforce (Kumar, 2015).
One notable feature of these guidelines is that they codify the evidence base and best practice that develops over time, a characteristic not commonly encountered in intervention-evaluation studies. The most conservative measure of the effect of the guidelines is to measure the impact from the date of their official implementation, and this is the primary impact we measure. Broader effects can also be measured, such as those linked to guideline content and occurring before official implementation. This requires identifying a key moment in time from which we can explore change, which can be challenging.
We explore three methods that use a control group to assess the impact of revising the suspected-cancer guidelines on time to diagnosis of colorectal cancer – an outcome linked with improved survival in decision-analytic models (Tørring et al., 2013). The three methods are pre-to-post and event-study difference-in-differences, and a semiparametric varying-coefficient model.
The pre-to-post difference-in-differences method is widely applied in health-services research (Streeter et al., 2017), and was used to quantify the impact of the original 2005 NICE suspected-cancer referral guidelines on the timeliness of cancer diagnosis (Neal et al., 2014). The outcome measure was diagnostic interval, i.e. the time between first presentation of a cancer symptom to primary care and diagnosis (Weller et al., 2012). Diagnostic intervals were shorter in 2007–08 compared with 2001–02; however, this was only attributable to the new guidelines in two cancer sites (oesophagus and cervix) (Neal et al., 2014). It is possible that the guidelines had little impact, and/or that Neal et al.‘s chosen control group did not capture the secular trend well. Alternatively, the pre-to-post difference-in-differences method may not be well equipped to measure time-varying changes in impact. This possibility is one focus of our paper.
The event-study difference-in-differences has been used to explore the phased impact of policy, by introducing a series of dummy variables into the difference-in-differences regression model, each indicating time relative to a new policy (Wing et al., 2018). These dummies are typically yearly or monthly, as used to examine the effects of US government school-desegregation policies on racial mix over time (Reber, 2005), and health-insurance mandates on mammography and breast cancer diagnosis (Bitler and Carpenter, 2016). This method can also be used to explore the time trends associated with pre-intervention anticipatory effects; for example, to examine the impact of changes in medical care insurance (Alpert, 2016; Brot-Goldberg et al., 2017) and hospital reform (Kolstad and Kowalski, 2012).
Finally, we consider the semiparametric varying-coefficient model (Li et al., 2002). Here, the pattern of time-varying effects is not restricted by any pre-specified parametric functional form, such as occurs with the common practice of adding interaction terms to a regression model. To our knowledge, the semiparametric varying-coefficient model is rarely applied in healthcare research. It has been used to explore time trends in the control and intervention arms within randomised controlled trials (Gilbert et al., 2019; Wasan et al., 2017), and the time-varying aspects of tumour characteristics on disease-free survival in breast cancer in an observational dataset (Natarajan et al., 2009).
There are three main ways in which the semiparametric model, applied to repeated cross-sectional data, provides additional insights over parametric models in healthcare studies assessing the impact of interventions, particularly when the intervention is a guideline as we consider here. First, the method can account for the time-varying impact of all control variables (for example, age, sex, and intervention). Second, applying the semiparametric varying-coefficient model, it is possible to test the validity of the conventional parametric specifications, such as the pre-to-post and the event-study difference-in-differences methods. Third, once the varying-coefficient model has been run, treatment effects of guideline implementation, as well as the combining effects of both guideline content and subsequent implementation can be estimated. Given the novelty of the method, our aim is to illustrate the method's potential and any assumptions required without being prescriptive about which of these features are ultimately used by researchers.
Our study adds to the existing literature in the following five ways. First, we describe the three methods and summarise their distinguishing characteristics. Second, we show how the semiparametric model can be used in preliminary analysis for data visualisation of the time-varying aspects of covariates. Third, we show that the pre-to-post and event-study difference-in-differences models are special cases of the semiparametric varying-coefficient model, enabling a formal test of the parametric models against the varying-coefficient model. Fourth, we show how the semiparametric method can be used in causal analysis to measure treatment effects. Finally, we report the first comparison of these difference-in-differences methods and the semiparametric varying-coefficient model to assess the impact of national guidelines in patient-level health records, using NICE referral guidelines for suspected colorectal cancer as an illustration.
Section snippets
Study setting and inclusion criteria
The setting was primary care and the data sources were the Clinical Practice Research Datalink (CPRD), a dataset of observational, anonymised, patient records in UK primary care, with partial linkage to National Cancer Registration and Analysis Service (NCRAS) and Office for National Statistics (ONS) data (Herrett et al., 2015). Ethics approval by the Independent Scientific Advisory Committee (protocol 16_037) was obtained on October 30, 2017. The CPRD reduced temporarily in size from 2013 to
Pre-to-post difference-in-differences
In the pre-to-post difference-in-differences method (Eq. (1)), the outcome of interest (diagnostic interval) is measured in four groups, control (NICE-2005) and treated (NICE-2015-only), before and after the treated group is managed according to the new guidelines. The effect attributable solely to official implementation of the new guidelines is estimated by the coefficient for the interaction term () between the dummy variables (indicating before or after revised guideline
Study cohorts
The CPRD provided 25,011 patients with an incident colorectal cancer diagnosis between January 1, 2006 and December 31, 2017. Of these, 13,169 were excluded, mostly because they did not present with features of colorectal cancer (n = 9693) or because they resided in Scotland (n = 3105) where different guidelines apply. Patients diagnosed following screening (n = 52), or with multiple index cancers (n = 52), or who did not attend general practice (n = 4) were also excluded, leaving 11,842
Discussion
This is the first study, to our knowledge, that compares methods to measure the time-varying impacts of national guidelines using repeated, cross-sectional patient-level data.
Our data source was the CPRD – one of the largest database of anonymised patient records in the world, and a widely used resource for primary care research and epidemiology studies (Herrett et al., 2015). Robust methods were used for case identification in the electronic medical record (Watson et al., 2017). Our sample
CRediT authorship contribution statement
Sarah Price: Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Xiaohui Zhang: Conceptualization, Methodology, Software, Validation, Formal analysis, Resources, Data curation, Writing - original draft, Writing - review & editing, Supervision. Anne Spencer: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing, Supervision, Project administration, Funding
Acknowledgements
This research arises from the Cancer Research UK funded study to explore the impact of NICE guidelines for early cancer diagnosis [C56843/A21550] in which Sarah Price is the Research Fellow and Anne Spencer is the joint principal investigator. This research is linked to the CanTest Collaborative, which is funded by Cancer Research UK [C8640/A23385], of which Sarah Price is an affiliated Research Fellow and Anne Spencer is Senior Faculty.
We would like to thank Professor Willie Hamilton for his
References (36)
The anticipatory effects of Medicare Part D on drug utilization
J. Health Econ.
(2016)- et al.
A varying-coefficient panel data model with fixed effects: theory and an application to US commercial banks
J. Econom.
(2017) - et al.
Antimicrobial-impregnated central venous catheters for prevention of neonatal bloodstream infection (PREVAIL): an open-label, parallel-group, pragmatic, randomised controlled trial
Lancet Child Adolesc Health
(2019) - et al.
The impact of health care reform on hospital and preventive care: evidence from Massachusetts
J. Publ. Econ.
(2012) - et al.
Adjusting for unmeasured confounding in nonrandomized longitudinal studies: a methodological review
J. Clin. Epidemiol.
(2017) - et al.
Evidence of increasing mortality with longer diagnostic intervals for five common cancers: a cohort study in primary care
Eur. J. Canc.
(2013) - et al.
First-line selective internal radiotherapy plus chemotherapy versus chemotherapy alone in patients with liver metastases from colorectal cancer (FOXFIRE, SIRFLOX, and FOXFIRE-Global): a combined analysis of three multicentre, randomised, phase 3 trials
Lancet Oncol.
(2017) Semiparametric difference-in-differences estimators
Rev. Econ. Stud.
(2005)- et al.
Health insurance mandates, mammography, and breast cancer diagnoses
Am. Econ. J. Econ. Pol.
(2016) - et al.
Cancer recording and mortality in the General Practice Research Database and linked cancer registries
Pharmacoepidemiol. Drug Saf.
(2013)
What does a deductible do? The impact of cost-sharing on health care princes, quantities, and spending dynamics
Q. J. Econ.
Minimum-wages and employment – a case-study of the fast-food industry in New-Jersey and Pennsylvania
Am. Econ. Rev.
Using natural experiments to evaluate population health interventions: new MRC guidance
J. Epidemiol. Community Health
Routes to diagnosis for cancer – determining the patient journey using multiple routine data sets
Br. J. Canc.
The risk of colorectal cancer with symptoms at different ages and between the sexes: a case-control study
BMC Med.
Nonparametric econometrics: the np package
J. Stat. Software
Data resource profile: Clinical Practice Research Datalink (CPRD)
Int. J. Epidemiol.
Examination of the synthetic control method for evaluating health policies with multiple treated units
Health Econ.
Cited by (2)
Trends in time to cancer diagnosis around the period of changing national guidance on referral of symptomatic patients: A serial cross-sectional study using UK electronic healthcare records from 2006–17
2020, Cancer EpidemiologyCitation Excerpt :Our analytical method allowed us to explore trends in the difference in diagnostic interval between groups aligned by their index feature(s) to the revised (New-NICE) or original (Old-NICE) guidance [20]. The method was derived to explore the time-varying and gradual impact of emerging clinical evidence that is legitimised into clinical practice by official guidance revision and implementation [20]. Our findings build on previous analysis of the original 2005 NICE guideline’s impact on diagnostic interval [13].