Office for Statistics Regulation

TQV: support for Analytical leadership!

In our latest blog, Statistics Regulator Oliver Fox-Tatum explores what we mean by effective analytical leadership and how our TQV (Trustworthiness, Quality and Value) framework supports this. The topic of analytical leadership will be at the forefront of our upcoming virtual event for government analysts on Thursday 3 February 2022. If you’re interested in finding out more please email us regulation@statistics.gov.uk

So… what do we mean by Analytical leadership?

Effective analytical leadership ensures that the right data and analyses are available, and that analysts are skilled and resourced to answer society’s most important questions.

Crucially, it ensures that data and analyses are used at the right time to inform key decisions, and that they are communicated clearly and transparently. When done well, this supports confidence in the analyses themselves, but also in the decisions based on these analyses.

We began exploring this concept in our review of statistical leadership last year and see the review’s findings as relevant for all government analysis produced across the UK.

But I’m not an analyst… is this relevant for me?

Yes! Everyone in government has an important role in championing the use of analytical evidence and being confident in engaging with analytical experts.

Whether as a senior minister communicating analysis publicly; an official drawing on analytical evidence for a policy decision, or for an external communication; or an analyst showing leadership in the provision of new data to answer the most important question of the day, strong analytical leadership needs to be demonstrated at all levels and parts of government.

We all have a stake in ensuring that the data and analyses produced and used across the UK can realise their full potential in supporting:

vital operational and policy decision making
confidence in the analyses and in the decisions based on them
citizens’ and society’s broader information needs – the wider public good.

How does the TQV (Trustworthiness, Quality and Value) framework support this?

The usefulness of TQV is as a simple framework – thinking about Trustworthiness, Quality and Value as a set of prompts is useful in challenging individuals, teams and organisations about how they approach their work and achieve their goals.

Stopping to reflect can be a powerful means to think again, to see what works and what else can be done. It is helpful for everyone using data and analysis in their work – not just for analysts, but non-analysts too – and is a particularly valuable tool as a culture of TQV evolves.

When considered together (and they always should be!) the TQV pillars form a pyramid of strength that ensures that:

the Value (V) of analysis for decision making and the information needs of citizens and wider society is maximised;
the Quality (Q) of the data and methods used is assured;
and the Trustworthiness (T) of both the data and decisions based on them, is supported.

So, when a government minister invests in the analytical capability of a department by providing additional resources for training, or new IT infrastructure to support automation … they are thinking T and Q. And when they choose to publish management information around a key policy area of wider public interest for transparency, that is thinking T and V!

Or when a press officer checks the accuracy of the text alongside a chart to be used in a Tweet with an analytical colleague before posting – that is thinking Q. Or if a policy colleague reaches out to an analytical team when developing a new performance measure for a key policy – that is thinking Q and V!

And not least, when an analyst pushes to attend to a key policy meeting to develop their skills and knowledge in an emerging policy area – they are thinking V. Or their permanent secretary asks them to provide an account of the latest published evidence at a press briefing as they value their objectivity, professionalism, expertise and insight as an organisational asset – that is thinking TQV!

It’s true to say that T, Q and V are equally important and shouldn’t be considered in isolation, as each support and reinforce each other.

But crucially, if we all take time to stop and think TQV when working with data and analysis, we can ensure we are truly supporting confidence in those analyses and the range of important decisions that they inform, as well as ensuring that they serve the public good.

Guest Blog: Challenges and Opportunities for Health and Care Statistics

The COVID-19 pandemic has thrust health and social care statistics into the headlines. Never has there been more scrutiny or spotlight on health statistics – they are widely quoted, in Number 10 briefings, in the news, across social media, on Have I Got News for you… and everyone (it seems) is an expert. Nearly 2 years on from the first news reports of the ‘coronavirus’, the public appetite for data and statistics has continued to grow. This has created new challenges for health and care statistics producers, as well as highlighting existing areas for improvement, as set out in the recent Office for Statistics Regulation’s COVID-19 lessons learned report. The report noted the remarkable work of statistics producers, working quickly and collaboratively to overcome new challenges.

I joined the Department of Health and Social Care early in the pandemic, first leading the Test & Trace analytical function and for the last year as the department’s Head of Profession for Statistics. I have experienced these challenges first-hand and have been impressed throughout by the professionalism and commitment of colleagues across the heath sector to produce high quality and trustworthy statistics and analysis.

One of the recommendations of the OSR report (lesson 7) calls for us to build on the statistical achievements of the last two years and ensure stronger analytical leadership and coordination of health and social care statistics. I reflected at the beginning of the pandemic that it was hard to achieve coherence, given the number of organisations in England working rapidly to publish new statistics. We have made substantial improvements as the pandemic has gone on, the COVID-19 dashboard one of many notable successes, but I want to go further, and apply this to other areas of health and social care.

To address this, I have convened a new Health Statistics Leadership Forum alongside statistical leaders in the Office for Health Improvement and Disparities, NHS England/Improvement, NHS Digital, NHS Business Services Authority, Office for National Statistics, and the newly formed UK Health Security Agency. The forum is chaired by the Department for Health and Social Care in its overarching role and brings together Heads of Profession for statistics and lead statisticians from across the health statistics system in England.

We will use this monthly forum to ensure collaboration across all our statistical work. And we have a broader and more ambitious aim to build a culture (that transcends the complex organisational landscape) which values analytical insights, supports innovation and ensures there is a clear, joined up narrative for health statistics in the public domain.

We have set five immediate priorities

Coherence in delivery of advice and statistics
We will work collaboratively to ensure that our statistical portfolios are aligned, and we provide complimentary statistical products – working in a joined-up way across the system
Shared understanding of priorities
Ensuring health statistics address the highest priority areas, are relevant and useful for public debate and provide clear insight to inform decision making at the highest level.
Consistent approach to transparency
We will ensure alignment of both our internal and external reporting so that the right data is quoted in statements and policy documents – clearly sourced and publicly available in line with the Code of Practice for Statistics.
Shared methodologies and definitions
We will have clear principles for coherence of methodologies and definitions, an expectation of common definitions where it makes sense to do so, and an escalation route via the forum for disagreement.
Build a joined-up statistics community
We will build a joined-up health statistics community through sharing our guidance on good practice, our approaches to induction, a shared seminar programme and annual town hall event, joint recruitment, managed moves, and secondments or loans.

Government statisticians have achieved so much as a community to provide statistics and analysis in really challenging times over the last two years, but there are lessons to learn and things we can do better. I am confident that our Leadership Forum will ensure that we maintain this collaborative approach to delivery, and bring health statistical leaders together to make that happen.

What do we mean by ‘statistics that serve the public good’?

‘The public good’ is a phrase which you might not have come across before. When I first joined the Office for Statistics Regulation (OSR) nearly two years ago, I had no real idea what it meant, but I knew that it was something very important to OSR; something which was mentioned in nearly every meeting I went to. What I know now is that OSR’s vision – that statistics should serve the public good – is fundamental to all that my colleagues and I do.

So what is serving the public good? It means that statistics should be produced in a trustworthy way, be of high quality, and provide value by answering people’s questions: providing accountability, helping people make choices, and informing policy. As statistics are part of the lifeblood of democratic debate, they should serve a very wide range of users. When they meet the needs of these users, they serve the public good.

But needs can change quickly, and statistics can be used in ways that also do not serve the public good – precise numbers can be used to give a misleading picture of what the statistics actually say, too much weight can be put on statistics, or they can be described incorrectly.

At OSR it is our job to support confidence in statistics. Having a really strong understanding of what it means for statistics to serve the public good is crucial to this.

Over the last 22 months, I’ve been leading a research programme aimed at doing just this, as existing research on understanding public good is relatively sparse.

My first step in exploring public good was to publish a literature review which explores the current evidence on public good. We have also analysed how researchers think their research will provide public benefits and we are currently running studies to explore what members of the public and our own team think about the public good.

A key theme coming from this research is the importance of being able to communicate statistics in a way that is understandable to everyone who is interested and needs to be informed by them. This is not an easy thing to do. Statistics potentially have many different audiences – some people may confidently work with statistics, whereas others may not have much experience of statistics, but want to be able to understand them to help make decisions about their lives.

Differences in how people understand statistics are often attributed to an individual’s literacy or numeracy abilities – we often hear the term “statistical literacy” when this type of understanding is being talked about.

We think it is wrong though to think of statistical literacy purely in terms of a deficit in knowledge. Rather, we think that producers of statistics need to understand what people find easy to grasp and what they find counterintuitive and think, “How do we work with that to make sure that the real message of the statistics lands properly?” It is our role in OSR to guide producers to do this.

To help us with this, we will be kicking off some new projects this year aimed at increasing our understanding of how different segments of the public regard statistics.

The public good may seem like a mysterious concept but, by working to build the evidence sitting behind the phrase ‘public good’ and understand how the statistical system needs to respond to meet it, we are hoping to make it much less so.

We hope that our research work, which we are undertaking in collaboration with others, will not only highlight the role that statistics play in the choices that people make and the risks to the public value of statistics in a changing environment, but also that publicly researching this area will stimulate others with the capacity and expertise to work in this area.

It’s beginning to look a lot like Census…

It may be too early to talk about Christmas for some – not for me. I have decided on the design theme, got my advent calendar, am close to finalising the Christmas menu and have started ordering gifts. I am all over it! And getting more excited by the day.

Christmas means different things to different people, but it is certainly my favourite census related celebration. Weren’t Mary and Joseph off to Bethlehem to register as part of a census? Timely then as part of my, possibly a bit too keen, Christmas preparations, I am pleased to say we have published our phase 2 assessment reports for the 2021 Census in England and Wales and the 2021 Census in Northern Ireland.

If you read my previous blog, you’ll know I have been leading OSR’s work on the Censuses in the UK and today is a bit of a milestone moment for me as part of this assessment. The publication of these reports is a culmination of a range of work which kicked off three or four years ago and it has been so interesting and rewarding to speak with users and stakeholders of Census data and statistics throughout – it’s been an absolute gift!

Our reports recognise the efforts of the Office for National Statistics and the Northern Ireland Statistics and Research Agency in delivering Census live operations and their continuing work to produce high quality, extremely valuable Census data and statistics. I wanted to take the opportunity to specifically thank all of the individuals and teams who have taken the time to engage with me and my colleagues as part of this assessment process. You have been open with us, kept us up to date with developments and have taken on board our feedback throughout the process. All the while getting on with the more important job at hand, working on Census itself.

As we get closer to the festive season, I wish you all a well-deserved break and raise a glass of sparkling elderflower pressé to one and all.

Volunteering at COP26 – being part of something big

I felt immensely excited and proud to be one of the 1,000 volunteers helping at COP26 in my home city of Glasgow. Volunteers came from different corners of the globe, to perform a range of tasks. I was a shuttle hub volunteer which means I helped those going to COP26 get from there and back by bus shuttle from the key transport hubs – over 7,000 people on one day in the hours I was there.

What gets me out of bed in the morning is feeling like being part of a greater whole. COP26 makes me realise that climate change and net zero is the bigger picture. As VIPs with their outriders whizzed by me, I was focussed on people keen to get their day going, like the lady who travelled for days from the islands of Palau or the man from Samoa to make sure they get their voices heard alongside the more well-known world leaders. The Volunteers logged the peoples from different parts of the world that they’ve welcomed to Glasgow – now covering almost all the 196 nations at COP26.

Glaswegians had varied responses to COP26. Most saw it as a great opportunity for the world. Some reflected the views of Greta Thunberg, that it’s more about “blah blah blah” than substance. Some bemoaned the inconvenience to their daily routine, although Glaswegians are renowned for their humour and make a joke of any temporary disruption to their day.

Coming from a mining family, I started my career in the coal industry, and was enormously proud to work in a nationalised industry. A phrase I heard a lot back then was that Britain was an island founded on coal surrounded by fish.

I was one of 20 young graduates that joined the National Coal Board (NCB) in 1979 and many, like me, came from coalfield communities. Early in 1980, I was sent on behalf of the NCB’s Economics Intelligence Unit, to monitor the Select Committee on Energy at the House of Commons to feed back on its inquiry into alternative sources of energy. I reported back that I didn’t see renewables taking off or being any competition to coal. I was deeply attached to coal as a nationalised resource. The prevailing economic thinking at the time was the mixed economy – nationalised industries, public corporations and private enterprise.

I so wish now that I could have heard statistics about the threat to the planet (those statistics came much later) and the imperative of reducing our dependency on fossil fuels and ramping up clean alternatives. At that time in the early 1980s the environmental problem of burning 120m tonnes of coal in the UK was making coal a little cleaner. I was astonished to learn that more coal is produced in the world today (~7.6 billion tonnes) than early 1980 (~3.8 billion tonnes), according to statistics from the International Energy Agency.

Climate change statistics are important, but change will rely on leadership

During these weeks at COP26, I saw and heard the leaders and politicians on the TV radio and through the media. As I spoke with those going to COP26, I heard their hopes and fears. I realised that climate change is more than the science; political leadership is crucial. Leaders need to be skilled in the appropriate use of data as part of their role in persuading the public to make the behavioural changes and embrace the consequences. E M Forster said “Only Connect!…Only connect the prose and the passion and both will be exalted”. Leaders connect the prose and the passion and need to skilfully deploy the statistics. Leaders can give statistics a social life.

As we’ve learned from the Covid-19 pandemic, the role of senior leaders in government is also vital in providing support to statisticians. The lessons learned throughout the pandemic should show us how to better help people to understand the choices open to us all to respond to the challenges of net zero.

Statistics play a key role in helping the public understand the bigger picture on climate change. They are also essential for helping governments design and monitor policies that reduce or prevent greenhouse gas emissions (mitigation) and prepare us for the expected impacts of climate change (adaptation). Because it’s such a complex topic, climate change statistics add insight when related sets of statistics are brought together to tell a clear story of what is happening and why. They also add value when they are presented in ways that help different types of users understand what the data is telling us, and when users can access the information they need. Our recent review of the UK’s climate change statistics looked at exactly these two things: their coherence and accessibility.

I’m nostalgic for people in Britain as a society having a concept of being part of a greater whole, that realises we’re dependent on each other. We need to connect with each other locally and globally and we’ll appreciate being part of something much bigger than ourselves.

Will public interest in statistics last?

We recently hosted a lively public event where we engaged with a varied audience answering questions about our work during the pandemic. We are delighted that the public is taking such a keen interest in statistics, data and evidence.

We work hard to ensure that numbers needed and consumed by the public uphold the scrutiny and standards of the Code of Practice for Statistics, offering ‘trustworthiness, quality and value’ to anyone using them.

Before the pandemic, our relationship with the general public (endearingly referred to here as ‘armchair epidimiologists’) was predominantly through third-party intermediaries like the media, government statistical producers, specialist user groups, academia, think-tanks and the third-sector to name but a few.

Understanding of the importance of statistics and public interest in them has changed, with people embracing statistics and realising how others can use or potentially misuse them.

This is shown in the number of cases we have considered; In the period from April 2020 to March 2021, we considered nearly three times the number of cases than in the previous year. 76% of the cases we looked into were related to the pandemic, and 48% of cases related to quality, reliability and trustworthiness of statistics – the first time this has been the most common category. This makes what we do much simpler to explain, as we continue to challenge the use of any official statistics that raise issues or cause concern.

We see statistics as an asset that frame our understanding of the world, help inform our choices and as a starting point for debate. This could be on the size of the economy, the number of people in the country, the rate of crime, the health and well-being of the population, and on more specific topics such as the levelling up agenda, or statistics on climate change.

But this isn’t just for institutional decision makers – like the Bank of England, or a Secretary of State. Statistics also support the choices made every day by a very wide range of people; individuals, businesses, community groups and so on.

We are considering how we should go beyond intermediaries and engage more directly than we do already with the general public on issues we care about. As part of our ongoing work and commitment as statistics regulator, we encourage you follow our twitter, read our newsletter, visit our website and contact us with any thoughts or questions you might have.

Communicating data is more than just presenting the numbers

There has been a lot of talk about a UK Health Security Agency (UKHSA) technical report. It includes information on COVID-19 case rates in England for vaccinated and unvaccinated groups (Table 5). For some the immediate reaction to these data has been outright disbelief, others have used the data to support pre-existing, and incorrect, views that vaccines are not effective. Neither of these reactions is right. Understanding the issues properly is extremely complex, but what we do know with some certainty, is that while the vaccine will not stop the spread of the virus completely, it has been shown to help improve outcomes.

We have seen considerable improvements to the presentation of the data in the latest UKHSA report, which should support better interpretation of these data in future. However, it provides an important lesson about the harm that can be done if data are not clearly presented and well explained. There is more to communicating data than just presenting the numbers. Producers of data should do everything they can to minimise the potential for misinterpretation or misuse.

As well as presenting data clearly, producers need to guide users on how the data they publish can, and perhaps most importantly cannot, be used. They also need to explain the choices that are made in producing the analysis, along with the implications of these choices. Producers of data are the experts in the data they publish, their insight and understanding of these data can help ensure its appropriate interpretation and use.

To return to the example of COVID-19 case rates by vaccination status, the choices made by UKHSA have been a source of much debate because the consequences of the choices are so significant. It is important these choices and their implications are explained, and it is perhaps even more important that the guidance on interpretation of the data is clear. As an organisation of experts, UKHSA is in a position to help the public understand what the data mean, including explaining why the findings may not be as counter intuitive as they first appear while leaving no uncertainty around its views that vaccines are effective. UKHSA can help explain where there is value in using the data (e.g. looking at changes in case rates over time or across age bands, within the vaccinated group) and where there is not (e.g. understanding vaccine effectiveness).

Guidance on interpretation

The report is clear that a direct comparison of rates for vaccinated and unvaccinated is not appropriate. The report opens with the text:

“These raw data should not be used to estimate vaccine effectiveness as the data does not take into account inherent biases present such as differences in risk, behaviour and testing in the vaccinated and unvaccinated populations.”

This is a helpful health warning. By thinking about questions people may have and how they may try to use data, producers can pre-empt potential issues and provide explanation that will support appropriate interpretation. In the context of COVID-19 case rates some of the key questions might be:

What are the data measuring? And is this what people want to know about? Many people want to know the risk of COVID-19 infection for those who are double vaccinated compared with those who have not had a vaccine. The data in UKHSA’s Table 5 do not do this. It does not show infection rates in the population. The table shows case rates i.e. the rate of COVID-19 positive tests in those who come forward for testing, something the government continues to ask people to do[1]. As a result, the data may have inherent biases.
Are the two groups in question comparable? It is easy to see that there may be different behaviours in people who have had two vaccinations compared to those who have had none. One hypothesis is that those with two vaccinations are more likely to get tested, meaning the case rates will look relatively higher for this group compared to the unvaccinated group. There will also be different risks associated with each group, the vaccination programme prioritised vulnerable groups and frontline health and social care workers, so includes those who are more at risk of infection. We haven’t seen evidence to quantify the impact of these risks and behaviours, but it’s likely there will be an impact.
Are there other sources of data which could be considered? There are increasingly other sources of information which demonstrate vaccines are highly effective. The UKHSA has done a significant amount of research and analysis into vaccines. This is outlined in the vaccine surveillance report, which sets out effectiveness against different outcomes (infection, symptomatic disease, hospitalisation and mortality). There is further information via UKHSA’s monitoring of the effectiveness of COVID-19 vaccination In addition, the Office for National Statistics (ONS) has published an article on the impact of vaccination on testing positive as well as an article on deaths involving COVID-19 by vaccination status. All of these examples take into account characteristics of those in the sample and try to adjust for differences. As a result, they offer a better indication of vaccine effectiveness.

Implications of choices

To undertake any analysis choices need to be made. These choices and their implications should be explained.

In the calculation of COVID-19 case rates the most significant choice is the decision on what population estimates to use to calculate the rates (“the denominator”). There are two obvious choices: the National Immunisation Management Service (NIMS) or the ONS mid-year population estimates. Each source has its strengths and limitations and we don’t yet know the true figure for the denominator.

In the context of case rates the choice of denominator becomes even more significant than when it is used for take up rates, because the numbers of people with no vaccine are low. The population of people vaccinated is relatively well known, NIMS includes all those who have been vaccinated and is a good estimate of this population.

The difficulty comes in understanding the total number of the population who have not had a vaccination. There are many advantages to using NIMS, not least because it is consistent with international approaches to considering immunisations and allows for analysis which would not be possible using aggregate population estimates. However, we also know that NIMS overestimates the population. Similarly, there are strengths in using ONS mid-year estimates, but we know these can have particular difficulties for small geographic breakdowns. We also know that the time lag created by using mid-year 2020 estimates has a disproportionate impact in older age groups – for example, it means that in more granular age bands some older age groups show more people having been vaccinated than the ONS population suggests exist. There is more information on the strengths and weaknesses of each source in NHSE&I’s notes on denominators for COVID-19 vaccination statistics. The chief statistician at Welsh Government has published a blog which outlines the impact of the different choices for vaccine uptake in Wales.

Looking just at the adult population, Figure 1 shows the different results which come from using the two different denominator options for the population who have never had a COVID-19 vaccine.

Figure 1: COVID-19 case rates per 100,000, England by age band

*see notes at end for the data and more details of sources used in this figure.

Using the NIMS denominator, the positive case rates for those who are not vaccinated is below the case rate for those who are vaccinated (except in the 18-29 age band). Using the ONS mid-year 2020 estimates as a denominator, the positive case rates for those who are not vaccinated is higher than for those who are vaccinated (in all age groups below 80). While we don’t yet know the true figure for the unvaccinated population, this seemingly simple choice has a huge impact. It is particularly problematic in this circumstance because any error in the total population estimate is applied in its entirety to the unvaccinated population.

As an example, for the 70 to 79 population, the NIMS figure is just 4% higher than the ONS mid-year estimates (5.02 million and 4.82 million respectively). These figures can then be used in combination with the data on total people vaccinated from NIMS to estimate the total number of people not vaccinated. In doing this, the difference of nearly 200,000 in the total population estimates is applied entirely to the relatively small number of 70 to 79 year olds who are not vaccinated. It means the NIMS estimate for the unvaccinated population in the 70 to 79 age band is 363% higher than the estimate of those not vaccinated based on the ONS mid-year estimates. So, an estimate 4% higher at the 70 to 79 age band has led to an estimate 363% higher in the estimate of the unvaccinated population at that age band. This has a huge impact on the case rates for this group, and the conclusions drawn from the data.

An understanding of the impact of choices is essential in supporting appropriate interpretation of the data. In this scenario, we don’t have enough information to know the true figure for the unvaccinated population in each age group. We hope that the work UKHSA is doing to improve the NIMS data (including removing duplicates) along with the work ONS is doing on population estimates and the 2021 Census, will improve our understanding. It is really positive that ONS and UKHSA are working together to try and solve this issue, which is so important across so many statistics. Given this uncertainty, knowledge of the implications of the different choices can help users interpret the presented data with caution.

The message from all this is that data has huge value, but it is also really important that those publishing data consider how the data may be used, what the strengths and limitations of the data are and think about how to present data in a way that minimises the potential for misuse.

In time, there will be changes to what we know about the population, and producers of analysis should not shy away from updating their outputs when new evidence comes to light. In doing this, they should also clearly explain any changes and choices, as this transparency will support trust and understanding. It will help the good data shine out and ensure statistics serve the public good.

[1] https://www.nhs.uk/conditions/coronavirus-covid-19/testing/get-tested-for-coronavirus/

Data, sources and notes for case rate calculations:

Table 1: COVID-19 case rates per 100,000, England by age band

Age band	Rates among people vaccinated (2 doses) (NIMS)	Rates among people not vaccinated - NIMS	Rates among people not vaccinated - ONS
18-29	546.0	671.3	1089.9
30-39	1084.3	816.5	2159.1
40-49	1945.2	834.0	2207.1
50-59	1252.1	585.5	1907.2
60-69	837.6	390.7	1964.7
70-79	636.1	311.8	1443.7
80 or over	434.1	334.1	312.1

The calculations in Figure 1 and Table 1 are based on publicly available data. There are slight variations compared to the UKHSA data in Tables 2 and 5. For example, it is not clear exactly what date UKHSA has used for total vaccinated or unvaccinated populations, and there are regular revisions to the published data on the coronavirus dashboard, which may have caused changes since data were used in the UKHSA analysis. These differences are not big enough to impact on the conclusions in this blog.

NIMS data from coronavirus dashboard download, downloaded 31 October 2021. Metrics downloaded: “vaccinationsAgeDemographics”. Variables used:

Total population = VaccineRegisterPopulationByVaccinationDate (for 24 October)
Total vaccinated population = cumPeopleVaccinatedSecondDoseByVaccinationDate (for 10 October i.e. 14 days before end of period, as only cases with vaccination 14 days or more before positive test are included in UKHSA table of cases)
Total not vaccinated = Population estimate – cumPeopleVaccinatedFirstDoseByVaccinationDate

Total number of cases by age, and, cases by vaccination status by age, taken from COVID-19 Vaccine surveillance report week 43. UKHSA report shows total cases for four weeks to 24 October 930,013. Data downloaded (31 October) from coronavirus dashboard for four weeks up to 24 October (inclusive) gives 935,386. Additional cases likely due to cases reported between UKHSA analysis and 31 October.

ONS population estimates based on ONS 2020 mid-year estimates taken from NHSE&I weekly vaccine statistics, 28 October 2021 (downloaded 31 October).

Six steps to make your model a role model

My favourite number is 6. It was the number Roberto Carlos, my footballing idol, wore when he played for Brazil. As a kid I remember trying to recreate his infamous banana free kick versus France (1997) over, and over again. Practice makes perfect they say. So here, in honour of Senhor Carlos, I am going to provide 6 pieces of advice from our latest publication Guidance for Models so you can all practice turning your models, into a role models. If you just want to know why we made this guidance, skip to the end of this blog.

Every year we throw away 2.5 billion takeaway cups in the UK

Don’t create your model using the same criteria as a takeaway disposable cup: use once and throw away. A model, or components of it, should be reusable. Reusable not just by you, but by others too. It should also be robust (see testing) and adaptable (see modular). Do not make a disposable cup just for coffee, if someone else is also just for tea, when you can work together to make a sturdy cup for both tea and coffee.

Weed your borders

Like beautiful garden borders, models need maintenance. Borders may need mulching in the winter, pruning in the spring and lots of watering in the summer. Likewise, model dependencies may need updating, bugs resolving, and code changes made as a result of changes to data schemas. Teams need to plan their time accordingly to make sure the model still works and remains fit for purpose, especially as others may rely on your model’s outputs for their own model input.

Don’t be like a tragic episode of Grand Designs

We’ve all seen a Grand Designs episode when the keen, wannabe architect wants to design, build and project manage on their own dream home. It often ends in disaster: over budget, late, relationship breakups, and a lot of stress. Likewise, you need the correct people involved at the correct time to make your model a success. Model design plans should be scrutinised and checked. Experts should be consulted early on, instead of after things go wrong. Lastly, like all house builds, the model should be verified and signed off by someone you trust to make sure it is safe and secure.

It took Jean-François Champollion years to decipher the Rosetta Stone

Model documentation should be accessible and understandable to a wide range of audiences. You should not just publish detailed, technical documentation as not everyone will be able to understand the purpose or scope of your model. This may lead to your model being misused. You should explain your model as best as you can. If the nature of the model means it is hard to explain, you should describe how users can interpret your model and its outputs at the very least. Your should open source your code if possible, and provide installation guides, code comments and examples as well.

Be as analytical in choices as you are on Netflix

Aim to be as detailed in your model decisions as you are when choosing your next TV show to binge watch. Like choosing between a horror and a rom-com, you must understand what kind of model you need based on your scope and aims. You should seek advice and guidance from experts. Like the impossibilities of trying to stay up to date with the latest shows using a 90’s copy of the Radio Times, your decisions should be based on relevant and up to date information. Lastly, don’t overcommit and carry on if things don’t seem to be going well. Use regular checkpoints to reassess against your original needs. No one wants to force themselves through another experience like season eight of Game of Thrones.

Be as ethically minded as you are when switching your energy supplier

You read an article last week about the environmental benefits of veganism and want to give it a go? Great. You switched your electricity supplier to a 100% renewable electricity supplier? Way to go. You stopped going to that pub that treated its workers poorly? Power to the people! Now also understand that data, design choices and model selection all can have ethical implications. Power can be given, or taken, from certain groups based on the models we create and use. Ethics should not just be a tick box exercise; it should be the cornerstone of your model design and development.

Sure, nice analogies, but why did you actually create this guidance?

Last year the global pandemic thrust us into the limelight following a series of high profile uses of statistics for decision making. One of the biggest pieces of work we did last year was our review of the approach to developing statistical models to award 2020 exam results. “Algorithms” were blamed, with one headline stating “Dreams ruined by an algorithm” (BBC NI website). As such, we have been concerned about the threat of undermining public confidence in statistical models more broadly.

That exam review work took us into new frontiers by commenting on the use of statistical models that influence public life, not just those that produce official statistics. But statistical models are just one range of tools used by government analysts, data scientists and statisticians. Increasingly, newer techniques such as machine learning (ML) are being tested and deployed in the production of statistics and used to inform decisions. Furthermore, with the creation of the Office for Artificial Intelligence and the Government’s National AI Strategy, we are likely to see an increased use of more advanced Artificial Intelligence (AI) techniques going forward. 

As a result, we identified this as a crucial time to provide guidance for the use of models, regardless of whether they are statistical models, machine learning models or AI models. There have been a number of publications for ethical guidance for models (Ethics, Transparency and Accountability Framework for Automated Decision-Making, Data Ethics framework) as well as the creation of the UK Statistics Authority’s Centre for Applied Data Ethics. There are also a number of technical guides on how to develop models (Aqua Book). However, we saw that there was no current guidance that suitably brought together social, ethical and technical aspects for all elements of model creation: data, design, development, delivery and deployment.

We believe our role as a regulator, and our experience of the exam review from last year, puts us in a prime position to provide this socio-technical guidance for models. As a result, we have published our alpha release version of our model guidance with the aim to obtain feedback and comments from a wide range of users.

If you have any feedback, please get in touch! We aim to release an updated version of the guidance in early 2022.

Transparency: How open communication helps statistics serve the public good

Over the past 18 months we’ve talked a lot about transparency. We’ve made public interventions such as our call for UK governments to provide more transparency around COVID data, and it’s been prominent in our vision for the future of analysis in government, including in our Statistical Leadership and State of Statistical System reports.

But what do we mean when we talk about transparency? Why do we care? And what can be done to support it?

What do we mean by transparency?

Transparency is about working in an open way. For us, transparency means being open about the data being used. Explaining what judgements have been made about data and methods, and why. Being clear about the strengths and limitations of data – including what they can tell us about the world, and what they can’t. It also means making sure data and associated explanations are easy to find and clearly presented. It is at the core of many of the practices outlined in the Code of Practice for Statistics.

Why does it matter?

The pandemic has increased the public appetite for data and drawn attention to the significance of data in decision making. Many of us will have become familiar with the phrase “data, not dates” – a phrase which UK government used as it set out its road map for easing coronavirus restrictions. In a context when so many have been asked to give up so much on the basis of data it is especially important that the data are understood and trusted. Transparency is essential to this.

Transparency supports informed decisions. Appropriate use of data is only possible when data and associated limitations are understood. We all make daily decisions based on our understanding of the world around us. Many of these are informed by data from governments, perhaps trying to understand the risk of visiting a relative or judging when to get fuel.

We also need this understanding to hold government to account. Clearly presented data on key issues can help experts and the public understand government actions. For example, whether the UK is taking appropriate action to tackle climate change? Or how effectively governments are managing supply chains?

Transparency gives us a shared understanding of evidence which supports decisions. It allows us to focus on addressing challenges and improving society, rather than argue about the provenance of data and what it means. It supports trust in governments and the decisions they make. It allows us to make better individual and collective decisions. Ultimately, it ensures that statistics can serve the public good.

What is government doing?

We have seen many impressive examples of governments across the UK publishing increasingly large volumes of near real time data in accessible ways. One of the most prominent being the coronavirus dashboard and equivalents in other parts of the UK, such as the Northern Ireland COVID-19 Dashboard.

It has become routine for data to be published alongside daily Downing Street briefings, and through its additional data and information workbook Scottish Government has put in place an approach which enables it to release data quickly when necessary. We have also seen examples of clear explanations of data and the implications of different choices, such as the Chief Statistician’s update on the share of people vaccinated in Wales.

However, this good practice is not universal. Transparency regularly features in our casework. We have written public letters on a range of topics including Levelling Up, fuel stocks, hospital admissions and travel lists. We want to see a universal commitment to transparency from all governments in the UK. This should apply to data quoted publicly or used to justify important government decisions. Where data are not already published, mechanisms need to be in place to make sure data can be published quickly.

The Ministerial Code supports this ambition by requiring UK Government ministers to be mindful of the Code of Practice for Statistics – a requirement that is also reflected in the Scottish and Welsh Ministerial Codes and the Northern Ireland Guidance for Ministers. In response to a recent Public Administration and Constitutional Affairs Committee report the UK Government itself said:

“The Government is committed to transparency and will endeavour to publish all statistics and underlying data when referenced publicly, in line with the Code of Practice for Official Statistics.”

What is OSR doing?

We want to see statistics serve the public good, with transparency supporting informed decisions and enabling people to hold government to account. Over coming months, we will:

build our evidence base, highlighting good examples and understanding more about barriers to transparency.
continue to intervene on specific cases where we deem it necessary, guided by the UK Statistics Authority’s interventions policy.
work with external organisations and officials in governments to support solutions and make the case for transparency.

What can you do?

We’re under no illusion: OSR can’t resolve this on our own. Whether an organisation or individual we need your help.

You can question the data you see. Does it make sense? Do you know where it comes from? Is it being used appropriately?

You can raise concerns with us via regulation@statistics.gov.uk – our FAQs set out what to expect if you raise a concern with us. We’d also love to hear from other organisations with an interest in transparency.

And you can keep up to date with our work via our newsletter.

Reflections on lessons learned from COVID for health and social care data

You may have noticed that the last 18 months or so have been rather unusual. In fact it’s getting difficult to think about what things were like before masks, distancing and the universal smell of alcohol gel. And there’s another change to which we have become accustomed – the daily parade of statistics, the use of graphs on the news, and the huge presence of scientific and statistical discussion, both in the media and among ordinary people who are not even statisticians!

The scale and ambition of the health data being made available would have been unthinkable just two years ago, as would be the complexity and sophistication of the analyses being conducted. But the Office for Statistics Regulation’s ‘Lessons Learned’ report argues that we should not be complacent: we need to press harder for more trustworthy, better quality, and higher value statistics.

There are a few recommendations that stand out for me. First, Lesson 9 focusses on improved communication. Back in May 2020 I stuck my neck out on the Andrew Marr show and criticised the press briefings as being a form of ‘number theatre’, with lots of big and apparently impressive numbers being thrown around without regard for either accuracy or context. This attracted attention (and 1.7m views on Twitter). But although some dodgy graphs continued to appear, the presentation of statistics improved. Crucial to communication, however, is Lesson 1 on transparency – it is essential that the statistics underlying policy decisions, which affect us all, are available for scrutiny and are not cherrypicked to avoid those that might rock some political boat. This requires both constant vigilance, and appropriate clout for professional analysts.

Lesson 7 deals with collaboration, reflecting the extraordinary progress that has been made both in collaboration across governments and with academic partners, all of whom have shown themselves (against archetype) to be capable of agile and bold innovations. The Covid Infection Survey, in particular, has demonstrated both the need and the power of sophisticated statistical modelling applied to survey data. Although of course I would say that, wouldn’t I, as I happen to be chair of their advisory board, which has enabled me to see first-hand what a proper engagement between the ONS and universities can achieve.

Finally, Lesson 3 addresses the idea that data about policy interventions should not just enable us to know what is happening – essentially ‘process’ measures of activity – but help us to evaluate the impact of that policy. This is challenging; Test and Trace has come in for particular criticism in this regard. For statisticians, it is natural to think that data can help us assess the effect of actions, with the randomised clinical trial as a ‘gold-standard’, but with an increasing range of other techniques available for non-experimental data. Again there is a need to get this up the agenda by empowering professionals.

An over-arching theme is the need for the whole statistical system to be truly independent of political influence from any direction. While this is enshrined in legislation, a continued effort will need to be made to make sure that work with data lives up to the standards expressed in the Code of Practice for Statistics, in terms of trustworthiness, quality and value. The pandemic has shown how much can be achieved with the right will and appropriate resources, and OSR’s ‘Lessons Learned’ point the way forward.

David Spiegelhalter is a Non-Executive Director of the UK Statistics Authority, which oversees the work of the Office for Statistical Regulation.