Open for business: it’s time to realise the value of putting data at the heart of government

Widespread adoption of open data ecosystems could add up to 1.5% to the UK’s GDP, according to a study by McKinsey Global Institute. Given the vast volumes of data held and produced by public sector organisations, it’s fair to say government has a vested interest in seizing this opportunity.

In fact, further analysis by the Open Data Institute has suggested better access to - and sharing of - data across the public sector could account for 0.5% GDP growth.

But if that is the prize on offer, what's the reality?

In this article, we join forces with data software experts Snowflake to examine the progress the public sector has made in:

• making open data a key part of wider digital transformation
• aspects of data sharing, from efficiency to governance
• improving interoperability of data systems to boost cross-government collaboration

We’ve gathered the views of data and transformation experts at Snowflake, as well as leaders at government organisations that produce and handle huge amounts of data. These include Ordnance Survey, the Office for National Statistics and Met Office.

All of the spokespeople agreed it’s an exciting time to be driving forward an open data agenda across the public sector - but that the journey is really just beginning.

Why change must happen

According to Winston Chang, Global Public Sector CTO at Snowflake, parts of the public sector know data presents a huge opportunity to build better services, and do so cost-effectively. However, he adds: “We’re sitting on so much value but we haven’t yet fully figured out how to access it.”

He frames the current quest for more accessible and better-connected information as “the data-centric epoch”, stating: “The public sector needs to reorganise and design missions around the value of data. The catalyst for that is technology. And at present, tech is usually applied one use case at a time. A few parts of government use it at enterprise level and that’s the goal: data needs to transact across organisational boundaries.”

It’s a sentiment that John Kimmance, Chief Customer Officer at Ordnance Survey (OS), echoes. The government-owned company has around 6,000 organisations - including all major Whitehall departments, and devolved and local administrations - signed up to use its data. They do so in a vast range of ways.

“The speed at which organisations need and want data is accelerating,” says Kimmance. “We’re using new technologies like Earth Observation and AI to identify where change is happening, so that we can get new data into the database faster.”

This is crucial in an era when technology is causing seismic changes in society, and consequently the policies designed to manage rapid evolution. Data can fuel success in emerging policy areas, Kimmance comments - but cautions precision and accuracy are key. “There are aspects of modern life that obviously require highly accurate data. Autonomous driving is one example.

“Another is Net Zero and sustainability. That involves monitoring habitat changes, considerations such as where to build, and how to mitigate development by understanding natural barriers like marshes and reed beds. When we capture and monitor data over time we can get a good view of how climate adaption policy is working on the ground.”

While some organisations are forging ahead with implementing technology that can drive data-led service delivery, many barriers to wider progress remain. Technical debt, resourcing gaps and scalability are commonly cited challenges, states Chang at Snowflake.

“It takes time and energy to overcome these infrastructure issues,” he says. “But there is an understanding that challenges can be overcome through collaboration. It’s a word on the lips of every leader in every geography and sector. You can’t complete a mission without being a team.”

Chang feels passionately that collaboration extends to the tech stack. “We should apply the first principles of leadership and teamwork to technology. Plenty of open source tech still has a hard time becoming interoperable. The answer is a data platform that has open connectors, to connect with anything else that helps with the mission.”

The open data opportunity

Open data is key to building bridges between different parts of the public sector, allowing closer collaboration and the promise of improved, sustainable service delivery.

While those goals look attractive on paper the road to achieving them is long and winding. Knowing what data will be useful, how to format it and keeping it secure are just three of the quandaries data and tech leaders grapple with.

Met Office - a Snowflake customer, part of DSIT, and best known as the home of weather forecasting - ingests around 500 billion observations a day from technology such as satellites and wind farms, in the UK and internationally.

“Open data is an intriguing challenge for us,” says Charles Ewen, Chief Data and Information Officer at Met Office, speaking before July’s General Election. “We produce more data than the speed of light can support distributing. We just can’t make all of that ubiquitously available. It’s about making choices.”

What’s more, the amount of data dealt with by the organisation looks set to exponentially increase. Ewen’s other role is Senior Responsible Owner for SC2020 Plus, a £1.2bn GMPP transformational investment programme that will fuel the use of supercomputing.

Ewen explains that what data Met Office shares, and how it is presented, is led by users; an array of stakeholders in different parts of government and the private sector, with very different requirements. “Our mission is to democratise complex data that everyone can access and understand. There’s no single way in which we produce the data. A scientist, for example, might want a huge ‘cube’ of data to find links and correlations. But an insurer who is worried about property flood risks probably needs geospatial data."

“Our job is to make available accurate derived data - data on demand - that’s useful for a specific case.”

Finding efficient and effective ways to make open data work is one aspect occupying technology experts. Data security and control is another.

Snowflake’s Chang says striking a balance is imperative, and it is possible to find equilibrium: “There are ways to manage data that allow scale but also provide the privacy and security privilege needed. You can code across security, making the technology dynamic to adapt to the environment.”

Chang asserts the need to standardise legal agreements around the way data is shared, from an open layer to other tiers that have increasingly stringent levels of control. At present, he observes, public sector organisations tend to create them from scratch.

Met Office’s Ewen agrees there must be a better way to align data standards. “There is a lot of work going on around standards,” he says. “We understand the challenge in terms of data consistency and exchange, but there’s also a recognition that huge organisations need support to change their practices.”

Ewen speaks of live efforts to establish an “architecture of trust” for security and other aspects of open data governance. He explains: “All of the things you need to put in place to reassure a supplier you will use their data in a professional, appropriate, secure way.”

This is something that leaders at the Office for National Statistics (ONS), another key supplier of vital data across government, are well aware of. “We constantly work with partners like HMRC and DWP to establish boundaries of trust,” comments ONS Chief Data Architect, Charles Baird. “They have some highly sensitive data so are rightly concerned about how we ingest and share it.”

Innovating for interoperability 

While open data challenges can prompt rolled-up sleeves, interoperability issues can create a collective gnashing of teeth.

Baird at ONS concedes: “Government needs a better layer of interoperability, and we’re part of that. How do we give users the tools to work on projects in isolation but also the ability once their task is completed to connect with what other people are doing? It means using data consistently across platforms.”

As part of the solution, ONS has been rolling out a better discovery layer for its Integrated Data Service. It allows better data linkage at scale for both internal and external users.

Kimmance says OS has also honed its systems. He explains: “We have a single access point, the OS Data Hub, which enables our customer base to access the data. It’s available in traditional product format. Now we also have OS Select+Build. This allows the selection of attributes within our data for customised content. In addition, organisations can download the data to store and use it locally, or consume it through APIs.”

These developments are encouraging, says Chang at Snowflake. But there’s always more to be done - not least with legacy technology still prevalent. “One of the biggest things holding back governments around the world is a mentality of ‘it’s how we used to do things’,” he observes. “But technology has completely changed. We need to reflect on the assumptions associated with legacy systems, and reimagine an approach based on new capabilities.”

Chang argues that more widespread collaboration will naturally present solutions. “Building data sharing programmes forces interoperability,” he says. “Data-sets don’t easily work together so require engineering. Once that’s in place, and legacy technology has been replaced, you can turn to master data management and build a holistic ecosystem where data strategy can thrive.”

Departments seem determined to make headway. Baird at ONS cites its engagement with  CDDO/Cabinet Office on standards to make data sharing more consistent, boosting interoperability.

OS’s Kimmance also explains how the organisation has innovated to improve access and make data use more intuitive. OS provides a Unique Property Reference Number (UPRN) associated with each UK address. “It’s a great way to link data-sets,” he says.

A UPRN can be used like a National Insurance number, or car registration plate. A UPRN can be up to 12 digits in length and serves as a unique identifier for an addressable location: a building, a bus stop, a post box, and so on.

The UPRN numeric identifier applies a ‘common standard’ for addressable buildings and objects, which then makes it possible to collate, share, and connect data from various sources. The unique identifier helps reduce ambiguity in a location being considered. Essentially, the UPRN ensures everyone refers to the correct location. It enables greater accuracy of detail between separate organisations, and therefore the unification of data.

An open-hearted approach to data

Continuous innovation will be key to successful use of government data, whoever is using it and whatever it’s need for.

Ewen at Met Office comments: “Change is coming to data ingestion and data sharing. Artificial Intelligence (AI) will disrupt the way we build weather forecasts but also what people ask us for. We’re in the early stages of planning and structuring around how those new requirements will look. Met Office data will in any case be needed to train and initialise AI models. We’ll collaborate with industry and academics to provide value in that area.”

AI is also on OS’s mind, says Kimmance: “It will make a big impact. We just launched our first product entirely generated using Machine Learning. We will increase our use of AI, ethically, to ensure we deliver for customers better and faster.

“That could include investing in machine-to-machine learning, which is of interest across government: structuring our data to support automated interactions through APIs between machines and data. Asking questions and getting answers without always needing human input.”

Snowflake’s Chang concludes that an inherent desire to collaborate can help the public sector march forward into a future of open data and seamless interoperability. “What I like about government is it doesn’t do just things at a proprietary level. We get involved in helping to share learnings. Creating a community means we can reach a technology and data transformation tipping point faster.”