Sharing, optimising and governing data are generally agreed to be the three key challenges facing data leaders and their teams across the public sector. What makes the perfect solution hard to find is how widely approaches to handling data differ from one government department to another.
Data is all too often locked away in silos, making true interoperability the stuff of dreams. Organising data usefully to provide valuable insights and drive certainty in decision-making may be possible, but often proves difficult at best.
Now the advent of open table formats is changing the game. Promising to combine cutting-edge tools for data analytics with much more flexibility around data sharing, the technology is attracting interest across government.
In this article, we partner with data software experts Snowflake to consider aspects of open table format adoption, including:
We’ve gathered the views of Snowflake, the Department for Science, Innovation and Technology (DSIT), Ordnance Survey and NHS National Services Scotland as the quiet revolution of open table formats gathers pace.
Rightly, many organisations are taking time to investigate and understand the power of these new data formats. But it’s fair to say, as one observer tells us, that awareness is growing and the technology maturing, so adoption - and all of the possibilities that brings - will swiftly follow.
While the Covid-19 pandemic might be a period many people would rather forget, it turned the spotlight on how government organisations structure, use and share data.
Undoubtedly, access to vital data about citizens and critical infrastructure aided the response to the crisis, involving an array of organisations from across the public sector which suddenly needed to collaborate on a previously unseen scale.
But could the government’s strategy have been helped if data was held and used more intuitively before the pandemic was even headline news?
He recalls encountering stumbling blocks when the Scottish Government sought to harness data to help mitigate Covid’s outcomes: “There was huge demand because the First Minister and others were reliant on data to make decisions. But, thinking back, we found on-premise platforms inflexible and we were largely unable to scale them. It was very hard to deploy new technology or drive collaboration across organisations at a time when that was really needed.”
Much work has now been completed to overhaul and modernise systems across government, which King believes will leave organisations in a much better place to cope with future demand spikes or requests for rapid insight and decisions.
“From our point of view, moving into the cloud has allowed our team of professionals to deliver fantastic content for 20,000-plus users across health and social care,” he says. “Our output includes hundreds of data products, from vaccinations to finance information, which in turn drives thousands of dashboards for all of those users, as well as analytics and AI.”
Moving data to Snowflake has been a big part of the modernisation programme at NHS National Services Scotland. King is eager to capitalise on the chance to structure data to allow rapid, accurate decision-making; and to meet a wide range of objectives, including using data to support the implementation of Public Health Scotland’s latest operating plan.
The modernisation agenda also aims to take advantage of the emerging capabilities of open table formats - with the opportunity to separate some key data challenges, and the potential to revolutionise the way data is handled and used.
One area King says he is excited about is the chance to “less tightly couple” data storage and data management layers. “This could transform the data storage layer into an ecosystem model; and have the effect of commoditising data management, which would be great for data sharing, interoperability and supporting different technology use cases.”
“We have adopted open table format within our Core Data Services Platform,” Wilson says. “The aim is to improve internal data access, for use in innovation and data analytics.
“Many organisations are still relatively early in the development of internal data platforms but already seeing the benefits of open table formats for large-scale analytics. To do this, they have developed processes to transform data on ingest into open table formats.”
She adds: “That said, open table formats are still to be widely adopted for data sharing. We are still looking to investigate and adopt cloud-optimised data formats, such as Iceberg, for external data sharing. It’s on the OS Data Hub Future Product Roadmap.”
It’s an example of the opportunity on the table to use new data formats to boost interoperability. OS alone shares data with 6,500 public sector users and 2,000 premium data users directly and through hundreds of licensed partners, says Wilson. Cross-departmental collaboration could be about to expand exponentially.
Brunger adds: “Open table formats are making it easier to read and play with data, improving interoperability by accessing and harnessing different tools organisations use, for mutual benefit.”
While interest in open table formats is ramping up, no new technology comes without concerns. For that reason, anticipating and mitigating issues of security and governance is high on the agenda for vendors and users alike.
OS’s Wilson says the organisation is handling growing demands to make its data available within external data marketplaces and platforms, including Snowflake, as well as making it easier for third parties to directly access and use the data.
“We need to better understand the technical implications of integrating data into multiple platforms,” she states. “It means different formats, types of access, and potentially increased complexity and cost.”
For his part, Snowflake’s Brunger expects everyone will soon “play by the same rules” with open table formats. He predicts: “We’ll see the rise of the data custodian and the data catalogue - those are in advanced development. Everyone will be able see and access what data is available; read, write and manage it; and apply the same security to it.”
Wilson agrees with the sentiment: “Data ownership accountabilities must shift closer to operational day-to-day management, in closer partnership with DDaT teams. That includes better enablement by establishing data governance technologies to support structures such as data catalogues, and data observability and testing tools.”
Meanwhile, Brunger also urges organisations to plan for use of open table formats into strategies at the earliest possible stage: “This is about data architecture - design and build, upfront and early. Iceberg is foundational, and should be written into strategy for modernising or integrating platforms right at the beginning.
“Establishing in access controls early removes the need to export large data-sets. You can then have more confidence that the data isn’t being moved around and manipulated.”
This is already being tackled head-on by OS, says Wilson: “Geospatial data is inherently complex and large. Needing to access and use data for broader purposes has required upfront thinking and design to ensure that we manage and make our data accessible once and more cost-effectively.
“It has been easy to design systems that exchange and store the data needed for use locally within the consuming application. However, this approach introduces a range of data governance challenges as each system exchanges data at different times and frequencies; increases the need for more data quality checks/tests; and increases the cost to maintain long-term as data is moved and stored in multiple systems.”
The OS has moved to mitigate these challenges, Wilson adds: “We have implemented the Core Data Services Platform, which has been designed to store and maintain data once, but provide access to different users via a range of optimised access services, notably open table formats, cloud-optimised data format, and APIs.
“We regularly have to adopt new standards and formats for data sharing. We proactively encourage, sponsor and engage in innovation initiatives that demonstrate standards and interoperability in action, prior to formal adoption. This ensures that when we start to publish data in a new format, these are supported by the tools and technologies used by our customers. It also provides us with the knowledge and understanding of what is required for us, as a data publisher, to implement new data access and sharing formats.”
But he cautions open standards have a big role to play: “I believe that open standards and open data formats are fundamental to modernising our IT systems and enabling interoperability. Open standards are key to reducing the risks and costs associated with legacy systems and creating a more flexible environment that meets the evolving needs of our users.
Open standards are not just about technology - they are about fostering innovation and ensuring a level playing field for suppliers. They allow both open-source and proprietary software providers to compete fairly for government contracts, ultimately delivering better value.
“By giving priority to open standards for software interoperability, data sharing and document formats, we can build systems that communicate seamlessly, support innovation and improve public services. This is a critical step in creating a government IT landscape that is open, cost-effective and future-ready.”
Finding a “common language” as part of a wider bid to create open standards around this emerging technology - in areas such as open access and consistency between disparate tools - will also be crucial, Brunger believes.
According to King at NHS National Services Scotland, considering all of the opportunities and challenges of open table formats in the round is key.
“Fundamentally, the insight we need to deliver all of the interacting public services in Scotland effectively relies on our ability to share data efficiently for collaborative analytics,” he comments. “The role of open table formats in that aspect is as important as enabling more effective and trustworthy data sharing overall.
“The wrinkles are being ironed out and it’s clear to see where the technology can add value. Open table formats have an important role to play within government.”
It will be interesting to see which departments continue to take a watching brief and others that are keen to be in the vanguard, investigating the plentiful opportunities open table data promises - and setting the standard for all areas of government to get involved.