By Rob Whiteman, CIPFA CEO
The public sector should make policy decisions based on a rational and rigorous analysis of sound evidence to achieve defined outcomes for service users. Sounds like common sense when you just put it down on paper, doesn’t it? Evidence-based policymaking is not a new idea, and most professionals in government agree that strong evidence will support proper understanding of a problem as well as measurement of success. At CIPFA, we are no different and believe firmly that evidence is also integral to strong public financial management.
Despite it seeming like a common-sense, evidence-based policy making has its barriers. Even in situations where there’s the political will to take an evidence-led rather than partisan approach to policy, data can be challenging to wrangle. The data must be trustworthy and of high quality, as must the people tasked with analysing it. It must be timely and free from bias. And crucially, it needs to be robust enough to provide a full picture of a problem.
COVID-19 has also highlighted the reality that data does not always unambiguously point to a single clear policy decision. These cases require political judgement to weigh available evidence and competing outcomes to drive a policy decision.
Data that only underpins one piece of the policy puzzle, particularly those problems that manifest over time, can distort decision making, sometimes with serious effects. As a result, it’s important that any major policy decision is informed by data that examines the past, present and future together.
Decisions underpinned by only one of these data types can impact drastically on outcomes for service users. Here in the U.K., we have seen in real time, the impact of a decision driven by historic data alone in the recent debacle around A Level exam (university entry exam) results. The algorithm in question took past performance for schools by subject, and students’ GCSE (high school exam) results to create a distribution of grades. This system didn’t allow for outliers at historically underperforming schools, effectively imposing grades on young people that bore no reflection of their academic ability. This in turn led to chaos in university offers and significant public scandal. This has been a perfect case study in how public policy can go wrong when decisions and outcomes are overly weighted on historic data sets.
Data sets rooted exclusively in the present can offer an interesting snapshot of a problem at a particular moment. However, with no indication of trends over time or what an organisation is likely to face in the future, current data provides minimal actionable insight.
Predictive data has for some time been seen as the nut to crack in the public sector, offering opportunities to improve outcomes across a variety of policy areas. Predictive data can be useful in the public sector prevention agenda, projecting forward a current problem and the pros and cons of implementing early interventions. However, predictive data must be used alongside the contexts of both current and historical evidence.
Unforeseen events such as the COVID-19 pandemic can have a material impact on any predictive data model. Such events by their very nature cannot be predicted and as a result, can throw off decisions modelled on future projections. Predictive models can also be thrown off by fast changing environments. Particularly in the last couple of years, the public policy landscape has been moving at a frenetic pace. Let’s take Brexit as an example. Three Prime Ministers have been in office since the referendum result was announced in June 2016, each with different policy approaches, so any predictive model developed over this time with the intention of modelling the impact of Brexit on the U.K. can only be speculative at best.
Put simply, all analytical models are wrong, but some are useful. Predictive models should be used as a guide and a framework for decision making, rather than taken as a firm statement of what will happen in the future.
The most useful data is holistic, underpinned by an understanding of past trends, the current context, and projections into the future. It’s essential that users of analytics understand the strengths and weaknesses of the data they’re using. Examining data in the round allows public sector leaders to take evidence-based decisions on policy that lead to the best outcomes for their citizens and communities. And, as COVID-19 continues to increase demand and further stretch resources, robust data will be key to ensuring support is targeted where it’s needed most.
This article first appeared in Forbes.