Written by Robin Sutara, Field Chief Technology Officer at Databricks and Milos Colic, EMEA Public Sector Tech Lead at Databricks.
Data sharing is one of the critical enablers for each of the five missions of the National Data Strategy (NDS). Without data, we simply cannot make informed decisions, and importantly evaluate the impact they have on society.
Government is an industry that employs federation by design to drive specialisation and efficiency concerning critical topics. Whilst federation serves a pivotal role in designing valuable policies, standards and legislation, the downside of federation is the lack of a complete view of data across the government as a whole. Data sharing, data exchange and data marketplaces are tools that allow departments to exchange data products to form a complete view of the problems they are modelling and designing policies for.
The UK data economy was estimated to be c.£125 billion in 2021, with cumulative trending upwards. Mission one of NDS focuses on enabling the growth of the overall UK economy by leveraging data. The government collected, provided, and shared open data can be crucial for addressing many challenges across all industries.
Whether that data comes from DEFRA, ONS, Ordnance Survey, or another department or agency, it can be used to solve various problems ranging from national security and defence, fisheries, border control, fraud, trade and many more citizen-facing services.
When solving these problems and enabling use cases such as policy design - the key term is “trust”, or more specifically, “data trust”. Mission two of NDS is designed to drive pro-growth and pro-trust data regimes and ecosystems. The very nature of data (as long as we adhere to best practices and principles of FAIR data systems) is that with sharing and circulation, the value of data increases. With every step of the data through the data supply chain, new insights are created, further information is extracted, and new data products are produced.
The underlying purpose of the NDS is to promote data-driven decision making for the government and to drive better decision making through better utilisation of data.
The quality and efficiency of decision-making are directly linked to the quality of the data used to analyse situations and plan the appropriate actions. This is true even when data consumption is in the form of a dashboard, a spreadsheet, or a report - these are just examples of derived data products. One of the main challenges in this domain is the completeness of the data; it becomes imperative that different departments exchange data that provide various aspects of the same entities to extract complete information.
An example that highlights the need for a complete view of the data is that of Child Protection. To best ascertain the need for actions to secure the child’s safety while minimising unnecessary actions, one has to have complete information about the child in question.
Finally, it is vital that the security and resilience of data infrastructure are ensured. Data security breaches can directly affect and erode the data trust and the trust in the government’s ability to protect its citizens. On the one hand, we have a strong need for data sharing to be open, on the other hand, we have the pull to be more closed due to risk adversity.
This adversity can be rooted in many things: proprietary IP produced and accumulated over decades of hard work, lack of data trust due to past incidents - both real situations and situations taken out of context, standards designed without the practitioners in the loop. Whilst most of these situations cause an empathetic response - we need to challenge the status quo.
How to reconcile these two motivations? Through the application of modern technologies designed from their inception to be open, secure, infrastructure-source agnostic, governed and accessible.
Delta sharing is an example of a technology that fulfils all of the requirements above. An open source data sharing protocol that provides a fortified access management and governance suite of capabilities while abstracting the total cost of ownership and many traditional manual steps, allowing for time to value of data to be significantly accelerated. Because of these reasons, it has been adopted across private and public sectors and in the global context.
Through the application of modern technology, we can simplify laborious processes. And, if we follow best practices and standards, we can increase transparency and trust in data.
Reusability is one such best practice design principle fundamental for achieving a circular data economy. It also reduces the data's time to value, leading to a pro-growth data ecosystem and data economy. While doing so, we are driving the missions of the National Data Strategy and ensuring that the UK economy can benefit from data as a comprehensive enabler across sectors.