DEEP-DIVE: UK Government Data Maturity in 2024
James
Produced in partnership with Hewlett Packard Enterprise
In today's rapidly evolving digital landscape, data is a foundation for the efficient functioning of the UK public sector. The unprecedented volumes of data generated and collected daily offer a wealth of opportunities to enhance the operations of government departments and unlock new ways to deliver public services.
This report explores how far public sector organisations have come in their use of data, and where they're headed next. We have spoken to DDaT leaders from the UK Civil Service and industry to capture their views on best practice, identify common challenges, and discuss the technical, operational and cultural strategies they’re deploying now and in the future.
Why data matters in the public sector
This vast potential underlines the critical role of data in fostering an efficient, responsive, and innovative public sector. A study by McKinsey & Company found that public sector organisations using data analytics can increase their operational efficiency by up to 20%, highlighting the transformative impact of data on public sector productivity. Additionally, data-driven insights can help mitigate risks, enhance transparency, and ensure that public resources are used efficiently, contributing to a more resilient and accountable Civil Service.
Beyond operational efficiency, by leveraging data analytics, government departments can tailor services to meet the specific needs of different communities, ensuring that policies and services are more inclusive and equitable.
Integration and collaboration
For many public sector observers, the opportunity to bring together disparate datasets from different agencies and departments presents a powerful opportunity for service transformation. A flagship cross-government programme in that arena is the Integrated Data Service (IDS), which is being delivered by the Office for National Statistics. This initiative, which holds the same high-priority programme status as national projects like HS2, aims to transform the way disparate government data is used by creating a streamlined, efficient, and scalable analytical platform.
According to Fiona James, Chief Data Officer at the ONS, the IDS provides the tools and capability to break down silos, and unlock the potential of data for all of government, enhancing the speed and efficiency in which data can be shared, analysed and employed. 
An example of this collaborative effort is the recent integration of labour market data with health policy data. By working with the Treasury and Number 10, the IDS provided insights into the health conditions that keep people out of work. This information has been used to inform interventions by the Department for Work and Pensions (DWP) and the NHS, ensuring targeted support for those in need.
Another significant advance introduced by the IDS is a new indexing method for data integration. Unlike traditional, resource-intensive manual data linkages, this standardised approach allows for the de-identified combination of personal data across various reference frames. “Because we have already done this indexing, we can use that data again and again, without having to go back to Step A to do that bespoke linkage. So, there we have a model which is highly scalable and can easily expand to other organisations like HMRC for example,” says James.
User-first data projects
While many of the early iterations of data projects were inward-looking and designed to solve internal operational challenges, increasingly, data transformation programmes are outward-looking and focused on improving user experience and services for citizens and partner agencies. The Welsh Government’s celebrated StatsWales project is currently beta testing ahead of a relaunch that will significantly change the way it functions in order to deliver greater value for its diverse users. The free-to-use service allows users to create, view, manipulate and download data tables from a wide variety of Welsh public sector data sources, so user-centric thinking was built into the work from the outset.
The revised version of StatsWales will typify the more accessible approach to data services, suggests Stephanie Howarth, the Welsh Government’s Chief Statistician. “It was very much ahead of its time when it launched, but has been overtaken by other kinds of technological developments in the world of open data.
“We know that the site has a lot of users, but they experience quite a significant amount of frustration as it currently operates, because it hasn't necessarily been designed in the way that you would design a site now, with user research or user engagement, and applying current digital and data standards. So the new project is all about creating an open data service that meets those modern demands for an open, accessible data service,” says Howarth.
The project typifies the wider trend for public sector data projects to move beyond purely technical focus towards user-centric service design. In its discovery phase, the project team spoke to 31 different service users and identified around 70 different user needs to be fulfilled. Ben Marvell, a consultant on the project, explains how the research informs the new iteration of the service: “The key to addressing a lot of those needs is very much around that data accessibility. It ranges from making sure it has programmatic access for data scientists, and then also making a user interface that's intuitive and easy to understand for a member of the public who's just after one little piece of information that's relevant to them.”
Building collaboration in data programmes
The ONS’s Fiona James suggests that moving past the historical siloed mentality in government departments and fostering a spirit of collaboration in inter-agency projects is a priority. “It's tricky, when you're building something new, for people who you're working with to recognise that they have to step out of their departmental need, and focus on solving the bigger cross-government problems. Essentially acting in the broader public interest rather just in the interest of their departments.”
She points out that structural issues around the way departments are set up can contribute to those challenges, with the need to work to the priorities of different individual Ministers. In the IDS project, the solution has been partly to encourage departments to think of themselves as both consumers and providers of data, encouraging a more open approach.
There's a real duty on us and other public bodies to make our data available for other people to use, whether that's other public and private sector organisations, the public, or media, whoever.
The Welsh Government’s Stephanie Howarth sees opportunities not just for collaboration between public sector bodies, but for a wider network of partners. “Coming at this from an open data lens, there's only so much we can do ourselves in public sector organisations. I think there's a real duty on us and other public bodies to make our data available for other people to use, whether that's other public and private sector organisations, the public, or media, whoever. One part of this process is how we unlock that data, so other people can unlock the value from it.”
Building momentum in transformation projects
In order to get organisational support for a transformation programme, it's important to rapidly demonstrate return on investment, believe many DDaT leaders. At the Cabinet Office, Lead Analyst Rosa Alonso heads up the team responsible for the directory of arms’ length bodies, which assesses performance and value for money being delivered by those organisations. She believes that while a data strategy is essential, it’s important that practical implementation continues alongside that strategic development. 
“Sometimes you spend so much time thinking about the strategy on a more theoretical and conceptual basis, that when you actually try to implement it, six months or nine months have passed. So while we develop the strategy, we're trying to develop a simultaneous implementation roadmap, starting with a proof of concept to try things as we go. This conceptual strategy plus implementation is working well for us and it's helped us to get momentum, ” she says.
Driving cultural change and developing an appetite for transformation continues to be a challenge, suggests Paula Lender-Swain, Public Sector Director UK at Hewlett Packard Enterprise. She points to the need to develop successful examples in local government to demonstrate how multi-agency data projects could unlock new opportunities.
"There are already strong green-shoots of multi-agency innovation at a city level, with strong personalities within the city mayors who say 'yes, it's under my jurisdiction, so I'm going to agree that these things happen, and put some money into it.’ Moves from the new government could help accelerate this innovation at a local level, with technology being used to unlock insights across services for more holistic and pre-emptive services aligned to the citizen versus the traditional departmental silos.”
Public sector organisations are still learning how to demonstrate value from longer-term data projects, suggests Lender-Swain. “One school of thought is that you have to just build, you have to bite the bullet and build infrastructure from the bottom up that will support that data narrative. I think those smaller organisations, in local government, may be able to do that because it's easier to achieve at that scale.
“The problem with bigger organisations is the business case and the ROI - how long it will take and what value that would bring back - sometimes aren't tangible or aren't good enough. Data modernisation requires senior buy-in and an understanding about why data should be central to the organisation’s services and broader public sector data landscape, in order to truly reap the societal benefits of data-driven services.”
Progress in Data Standards
A deep focus on data standards remains a critical but sometimes neglected foundation for data-driven transformation, as Fiona James at ONS points out. “We've put a lot of effort into remediating our own data, focusing on areas like the quality of metadata. When it comes to the use of AI, for example, that stuff matters, and the more that we've got control over that up front now, the better the explainability we will have further down the line. Part of our data strategy is getting every analyst to focus on the metadata, and understand that the input is just as important as the output.”
Data standards and governance continue to be a challenge across the public sector, with varying degrees of data literacy across the Civil Service partly responsible. Rosa Alonso at the Cabinet Office points out that while data literacy is strong in DDaT teams, the collection of data can fall to non-specialists, which in turn creates challenges.
“The people who collect and provide the data may not have the best data literacy, which causes us to receive patchy data. While we ask them to follow certain guidelines and templates, they may not always follow them because they do not fully understand why consistency is so important for analysing data. The result is that we spend energy and effort on data validation and data cleaning, which means we have less time to implement new technologies that would add a lot of value.”
Ethics and Privacy
Ethical data use requires respecting individuals' privacy, obtaining informed consent, and ensuring data security to safeguard citizens' rights and prevent misuse.
With rapid growth expected in the use of AI in government, there is an onus on DDaT teams to be vigilant about potential biases in data analysis and algorithms, which can lead to unfair treatment of certain groups and skewed policy decisions.
At ONS, the transparency policies are built on five pillars, also known as the Five Safes: Safe people; Safe projects; Safe settings; Safe outputs; Safe data. “The key theme that we've put front and centre is around trust and transparency in our data. We do a lot around maintaining high levels of trust in both ONS itself and in our data, supported by doing a lot more education around how data is being used,” says Chief Data Officer Fiona James.
Challenges around data ethics and privacy are inevitably underpinned by the issue of public trust, suggests HPE’s Paul Lender-Swain, who believes that transformative projects depend on that support.
“It all comes down to public trust - that’s the driver of all of this. We've gone through a decade of lack of public trust, with some high-profile public sector issues in government and healthcare. When that changes, we'll start to be able to do more things and be a bit more flexible with how we're doing it, because the policies of governance are there. They're good policies, but I'm not sure whether there's enough trust in the system that people in government will be happy to do some of the things that might be perceived to be more risky,” she says.
Cloud first: adoption progress
It has been more than a decade since the UK Government adopted its cloud first policy, but in 2024, it is clear that progress remains inconsistent across and within different public sector organisations. The House of Data research report by Hewlett Packard Enterprise confirms that picture, with 73% of DDaT respondents saying they still store more than half of their data on-premises.
CDDO’s guidance on the policy acknowledges that a 100% Cloud approach isn’t always possible. “By Cloud First, we mean the Public Cloud. However Public Cloud is not always achievable, therefore community, hybrid or private deployment models are acceptable in specific circumstances.”
Organisational size, in-house skills and budgets all impact the relative progress of Cloud adoption, suggests Rosa Alonso at the Cabinet Office. “There are two kinds of analytical teams. The bigger, well-resourced analytical teams have hired data engineers, technical architects and solution architects, and they're very well advanced in the migration process to using cloud-based products - they have developed incredible infrastructures over the last three or four years. The other type of analytical teams may be much smaller and made up of professions like statisticians, economists, operational researchers and social researchers. We manage and collect data and analyse it, but we don't have the same skills.”
A growing chasm between public-cloud and on-premises solutions has been creating an often-unplanned hybrid IT operating model. HPE’s Paula Lender-Swain believes that a more flexible, hybrid-by-design approach can be beneficial for both operational and financial reasons.
“When you've got data in the cloud, every time you interrogate that data, bring it down to do something with it and take it back up again, it costs you money. And so, there are organisations that are talking to us about bringing that data back on prem and how they do that, while still benefiting from the principles of cloud technology,” she says.
Conclusions
The transformation of data use in the UK public sector is driven by organisations’ commitment to enhancing operational efficiency, driving innovation, and improving public services. Key takeaways from this report highlight the significant strides and ongoing challenges in this journey.
- Operational Efficiency and Inclusivity: Leveraging data analytics can significantly enhance performance, while also fostering more inclusive and equitable services by tailoring solutions to diverse community needs.
- Integration and Collaboration: Projects like the Integrated Data Service (IDS) led by the Office for National Statistics, exemplify the transformative potential of inter-agency data integration. Initiatives like these can streamline data use across government departments, allowing for more targeted and effective policy interventions.
- User-Centric Approaches: The evolution of data projects from internal operational focus to user-centric designs is evident in initiatives like the Welsh Government’s Stats Wales. This shift prioritises accessibility and user satisfaction, ensuring data services meet modern digital and data standards.
- Cultural and Structural Changes: Overcoming historical silos and fostering a collaborative spirit among government departments is crucial. Encouraging departments to see themselves as both consumers and providers of data can lead to a more holistic approach to public service delivery.
- Demonstrating Value: Rapidly showing return on investment is essential for gaining organisational support for data transformation projects. Simultaneously developing strategy and implementation can help build momentum and demonstrate the practical benefits of data initiatives.
- Data Standards and Governance: Ensuring high-quality data standards and governance is foundational for effective data use. Focus on metadata quality and data literacy across all levels of the Civil Service can reduce the time spent on data validation and enhance the overall analytical capabilities.
- Ethics and Privacy: Ethical considerations and maintaining public trust are paramount, especially with the growing use of AI in government. Adhering to robust governance principles helps ensure data is used responsibly and transparently.
- Cloud Adoption: Despite a decade-long cloud-first policy, progress in cloud adoption varies significantly across public sector organisations. A hybrid approach to data storage and management may provide operational and financial flexibility, accommodating the diverse capabilities and needs of different departments.
It’s clear that the UK public sector is on a promising path toward a data-driven future, with significant achievements and valuable lessons learned. Continued focus on integration, user-centric design, collaboration, and ethical practices will be key to unlocking the full potential of data in transforming public services.
