Updated: Jan 18
Almost two decades ago, the strategic planning unit at a university in South Africa decided that they wanted to improve access to information. So they created a data warehouse.
At the time, financials and payroll were delivered by an aging mainframe based GL system. At month-end, we'd receive a pile of paper containing various reports. Two of us would go through it, manually separate it, place them in envelopes, label the envelopes and send them off via internal mail. This took 2-3 days. It was painful.
The data mart team chose the tech, created the data extracts, built the models and trained a range of users. It took a few months to build the reports that we needed - we had to painstakingly replicate the exact format that people were used to. But once the reports were developed, reporting had shifted - from paper based, once a month, to electronic, on-demand. A significantly better solution.
But that was two decades ago.
Facing the same challenges today, it is interesting to see the same waterfall style big data warehouse projects being established.
Can smaller DW projects, driven by use cases, deliver faster incremental value?
Definitely: Large project failure rates have shown no signs of decline
Definitely: Sophisticated tech for data prep and advanced analytics, including open source options (like our favourite, KNIME), are now mature and widely available
Definitely: Visualisation tools have come a long way too - e.g. we use Tableau and Power BI - and these are only two among various solid options
Maybe: you still need a clean, well governed set of data - garbage in, garbage out. So you still need an overarching data management approach.
Maybe: reporting, particularly when consolidating across disparate systems, is usually more efficient (faster, more accurate, reduced load on production systems, etc) via a data warehouse. The DW is not dead - but perhaps just not created in one big bang.
How are you approaching your use of data to better serve your customers?