Data Warehouses are not dead, yet
March 9th, 2008 by Peter Scott
One of the reasons that I left my old job to work with Rittman Mead Consulting was to get back to technology. Don’t get this wrong, I really did enjoy managing people and having responsibility for a BI practice, but that was at the expense of involvement in a lot of the delivery of systems; for every one day of project work, I tended to spend eight doing “other stuff”, and I was getting to miss the buzz of being at the sharp end of a project, well maybe not missing the 2:30am coding whilst eating a slice of pizza stuff (which is a young person’s game)
Recently, I have been set the challenge to grow the amount of data warehouse and data quality work I am involved in and perhaps at the same time help dispel the perception that Rittman Mead Consulting just do short engagements and training; in the three months or so I have been here I have spent just three days delivering training and 2.5 months on-site with a single customer developing a data warehouse, so that notion is a bit of a myth anyway. Maybe the new Rittman Mead web site will feature some of the longer term type of engagements we are involved in.
I suppose a good question to start with is “do people still develop data warehouses?” Mark brought up a similar idea the other week in his ‘future BI architecture’ piece which also attracted some good comments. Indirectly, I mentioned alternatives to DWs in my article on Real Time BI for the Evaluation Centre – I hope to either post the article here on the blog or perhaps on the Rittman Mead web site in the next few days. A lot of organisations have already invested in data warehouses, these will continue to need attention, either from the support perspective or for enhancement as new classes of information are added. Data warehouses still have a place in BI as way of facilitating the delivery of quality data to the reporting layer in an efficient way so I can’t quite see the death of the Data Warehouse just yet.
Tags: dw
March 9th, 2008 at 11:27 pm
What is this obssession with the “death of such and such technology” of late?
Does anyone honestly believe the acronym soup that characterizes the web2 architecture is seriously going to replace ANYTHING?
Because I have yet to see proof it even works, let alone replace anything….
March 10th, 2008 at 8:20 am
Noons – I doubt that data warehouses will go in my working lifetime (but then I am old… ;-) ) so I will still have a job for some while!
Maybe, we should really be speaking of evolution – DWs will change, become less monolithic, more federated, and support more types of data for query (such as image, free text – but getting the dimensionality of a photo defined is going to be a challenge!) And as ever the problem for people like us is to move shed loads of data quickly enough to be useful, and that is always going to be down database infrastructure and not some whizzy web query tool thing
March 10th, 2008 at 2:39 pm
Well, they just announced quantum memories now. Apparently, something like a TB in less than 1/8″square, non-volatile, with the same read/write times of DRAM.
I have difficulty even imagining the possibilities but if it becomes mainstream in say, 10 years, then it’s not widespread storing of photos I’m worried about: it’s full on HD video!
:-)
Worse: how do we index and x-ref all that so it can be queried usefully and come back in a usable time? Nope, classic indexing is not viable, we just can’t say “go build an index on those 3HB of videos”: our children will be able to use the index at current CPU and sort algorithm speeds…
I think Jim Gray’s “Personal Petabyte” is right on schedule. Once more he is proven right: we have to prebuild the catalogues and indexes and then just populate them as needed. So much for “no need for data modelling and logical db design”.
Darn, why did he have to go sailing!…