In the past quarter, we’ve been developing a new data warehouse system to improve how we handle and process our clients’ data. Put simply, a data warehouse is an organised and central store of data which has been transformed so that it is ready to be used by the business.
In this short blog I’ll list five key benefits to outline how this new approach has changed our way of working and allowed us to find efficiencies which we can pass on to our clients.
1) A central data hub
One of the key concepts of data warehousing is the centralisation of the data which is used by the business. This then means that all analysis can be driven by the same curated dataset – from a single version of the truth. Because the insights provided by your applications and the work done by your analysts all share this same source, you can ensure that there’s consistency across analyses. The central data can also be pre-validated, saving time which would otherwise have been spent by every analyst/application before they could make use of the data and ensuring that all conclusions drawn from it are valid and trustworthy.
2) Control over data access
It’s all good having one version of the truth, but it’s equally important that analysts are able to access it quickly and easily. If not, you will inevitably still encounter the issue of outdated extracts being ‘passed around’ the business. This is bad both in terms of keeping your data current and also from an information security perspective. We’ve addressed this by providing users with fast, easy access to the key datasets via an API – and removing all other routes. This way, there’s a single mechanism via which both analysts and applications access the data and through which all permissions can be controlled, simplifying the task of getting the current, validated and analysis-ready data and leaving no reason for old data files to go astray.
3) Improved security
It’s important to provide fast and easy access – but only to the right people! With users and applications requiring varying levels of access, we needed a way to manage this. By hiding all the data behind an API, we can then use OAuth2 to authorise only the appropriate level of access to each user or application. Again, this allows us to maintain a single version of each data set and apply filters to remove sensitive details, ensuring users only see what they are authorised to, without the need for duplication of the data.
4) Scalability of cloud-hosting
If you’re thinking about adopting a data warehouse, among the first considerations should be where to create it! For this, we have turned to cloud-hosting. This has come with several key benefits when compared to on-premise hosting:
-
Scalability – thanks to the flexible, subscription-based nature of cloud services, it’s simple to procure ‘virtual’ computer infrastructure where the processing power and storage capacity can be scaled to meet the requirements of each client implementation. With this approach, we’re able to save by paying only for what we need, avoiding the need to invest in on-premise hardware for which the specification must be carefully considered ahead of time, and which eventually becomes obsolete.
-
Security and data protection – from an information security perspective, each clients’ data can be both processed and stored in a location which is totally isolated from other clients.
5) Efficiency and agility
Because some data requirements are shared between our analysis products, the data warehouse also allows us to offer our clients additional analysis more easily (i.e. faster and at a more competitive price) where the data requirements are already partially covered. This ability to turn analysis round faster also allows our clients to make better use of their data in emergent situations on their networks (e.g. severe demand events). Furthermore, by maintaining data in a form which is ready for analysis, you can react quickly to any events occurring on the water network which might need responding to ASAP. For example, during a suspected supply interruption you can spend less time wrestling with data and more time investigating the cause and planning a response.
Looking forwards
We are excited to see what further benefits this new approach can bring in the future, as we continue to develop our data warehouse solution. If you’re interested in learning more about how we can help make sense of and derive insights from your data through our consultancy work, then please don’t hesitate to get in touch!
