Publishing data via portals
Whether you're a government department aiming to increase the impact of your open data, a research institution sharing datasets with the community, or an organisation holding valuable private data, publishing that data is worthwhile, not just to meet regulatory requirements, but to enable innovation in line with your mission. However, publishing static datasets alone falls short:
- Each dataset requires redundant infrastructure.
- Every user must repeat the same transformation work.
- High friction means valuable ideas may never get off the ground.
You want your published data to be easy to use. By transforming your static datasets into data services ready for integration, tap helps you remove duplication of effort and enables innovation on top of your data, benefiting everyone from small community projects to large enterprise systems.
Case study: Scottish Energy Performance Certificate (EPC) data
The context
Energy Performance Certificates (EPCs) provide energy efficiency information and specifications, such as floor area and materials used, for a dwelling. We were building an energy savings calculator for properties across UK, for which we required EPC data based on address lookup.
The problem
While England and Wales has an API that returns EPC data from address lookup, Scotland publishes the data as CSV files only. The data files are a ZIP of 19 CSV files totalling 7.5 GB. We needed to be able to lookup certificates based on addresses performantly across this data.
The solution
Using tap we:
- Cleaned and standardised address data and field naming conventions.
- Deduplicated the data to only return latest certificate for an address.
- Enhanced compatibility by mimicking the England & Wales EPC API structure.
- Enabled performant geospatial queries through enrichment with ONS postcode lookup data and H3 indexing.
- Secured access with API key authentication while maintaining accessibility.
- Serve the complete dataset with various filtering with sub-100ms response times.
The API is publicly available via the epcdata.scot data portal.
The impact
epcdata.scot has served millions of requests from various organisations. It enables focusing on user value rather than the labourous process of data transformation and building out a data service around the EPC dataset.
Rather than forcing each user to solve the same data challenges, the API provides reusable building blocks that eliminate duplication and accelerate innovation. It supports use cases, such as energy savings calculators, lead generation, property research and climate policy.
Do you have any static data you want to turn into a service?
Talk to our teamLooks good? Missing something you need? We need your feedback.
Let us know if you need to get your data and applications talking, or you want to save money on complex, inefficient data platforms and integration tooling. We'd love to discuss and see how tap could help.
tap is built by DigitalSociety – we build bespoke digital tools for complex requirements in web applications, data engineering and cloud.