Comments on: End-to-end data quality http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/ Delivered Intelligence Wed, 10 Mar 2010 13:01:07 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Emil http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6601 Emil Wed, 29 Oct 2008 06:57:22 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6601 There are two things that are always contradict - from one side the customer to be happy with nice and correct reports and from the other side those correct reports to come from dirty data. The goal here is to find the intersection point. How do I do this. Some time I use the IFRS concept of materiality.. is the error material (Does the error lead to taking wrong decisions). If the answer is yes then the question is how did you corrected it up to now , the same way will be with the new system or some automation may be proposed to ease this process(e.g. OBIEE writeback option –see my previous post). If the answer is no then they can live happy with this error. But supplying erroneous reports even the customer accepted it is not good from professional point of view .Also it is a good practice to request the last IT Audit reports for the source systems and if somewhere audit controls findings and conclusions are mentioned this is a good starting point to discuss this issue. In all cases a very well prepared UAT (user acceptance criteria ) must be signed off even before project kicks off. There are two things that are always contradict – from one side the customer to be happy with nice and correct reports and from the other side those correct reports to come from dirty data. The goal here is to find the intersection point. How do I do this. Some time I use the IFRS concept of materiality.. is the error material (Does the error lead to taking wrong decisions). If the answer is yes then the question is how did you corrected it up to now , the same way will be with the new system or some automation may be proposed to ease this process(e.g. OBIEE writeback option –see my previous post). If the answer is no then they can live happy with this error. But supplying erroneous reports even the customer accepted it is not good from professional point of view .Also it is a good practice to request the last IT Audit reports for the source systems and if somewhere audit controls findings and conclusions are mentioned this is a good starting point to discuss this issue. In all cases a very well prepared UAT (user acceptance criteria ) must be signed off even before project kicks off.

]]> By: Peter Scott http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6600 Peter Scott Tue, 28 Oct 2008 15:09:51 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6600 @Tim - broadly the same as my suggestion to the customer. @Dave - I understand where you are coming from, and for some organizational cultures you can do this (it is the best way!), but in organisations that currently view data through spreadsheets backed by an extensive suite of macros to 'fix' the data it can be a trickier prospect to sell reports of bad data... data users are sometimes protected from problems for too long @Tim – broadly the same as my suggestion to the customer.
@Dave – I understand where you are coming from, and for some organizational cultures you can do this (it is the best way!), but in organisations that currently view data through spreadsheets backed by an extensive suite of macros to ‘fix’ the data it can be a trickier prospect to sell reports of bad data… data users are sometimes protected from problems for too long

]]> By: Tim Berry http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6599 Tim Berry Tue, 28 Oct 2008 14:35:32 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6599 Looking at the time they have they should instigate a central store and report MI fashion as to the spread of the content of the systems against what is valid, invalid and grey (potential for rule based cleanup). Usually there are placeholders in warehouses for non conformant data but these are usually filled larger than expected and the clean up investigation left until the warehouse is populated. I think that it is better to assess the source data prior to loading any target and that asessment needs to be planned so that volumes and actions can be considered and reassessed against one another. Therefore a central store that takes comparison and rule based checks is the way to go. Usually there isn't enough time but as these systems save more time after the extent of corruption is ascertained they are in the good position to instigate the action outside of the target system. Looking at the time they have they should instigate a central store and report MI fashion as to the spread of the content of the systems against what is valid, invalid and grey (potential for rule based cleanup).
Usually there are placeholders in warehouses for non conformant data but these are usually filled larger than expected and the clean up investigation left until the warehouse is populated.
I think that it is better to assess the source data prior to loading any target and that asessment needs to be planned so that volumes and actions can be considered and reassessed against one another.
Therefore a central store that takes comparison and rule based checks is the way to go. Usually there isn’t enough time but as these systems save more time after the extent of corruption is ascertained they are in the good position to instigate the action outside of the target system.

]]> By: Dave Katz http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6595 Dave Katz Mon, 27 Oct 2008 21:41:04 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6595 If the people who want the reports don't know much about the source, then you have to show them the 'garbage out'. I have found that showing the business users a few examples of reports based on dirty data tends to make the problem more concrete for them. Otherwise, discussions of data quality can seem very abstract. If the people who want the reports don’t know much about the source, then you have to show them the ‘garbage out’. I have found that showing the business users a few examples of reports based on dirty data tends to make the problem more concrete for them. Otherwise, discussions of data quality can seem very abstract.

]]> By: Peter Scott http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6592 Peter Scott Mon, 27 Oct 2008 16:38:46 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6592 @Emil - no need to apologise! You are right, there is no 100% solution and certainly not a one-size-fits-all one. The idea of write back is interesting, I can see it working for some cases. In the example I was thinking of we should really get the data fixed up in the sources - for various reasons of data governance (and third parties keeping their intellectual property) we do not have access direct (SQL) access to the application data structure structures. The other problem is that people who want the reports do not know much about the source.. I guess this is quite common where business users are interested in highly aggregated information and not raw fact The best we can do is report back the problems and hope they get fixed. @Emil – no need to apologise!

You are right, there is no 100% solution and certainly not a one-size-fits-all one.
The idea of write back is interesting, I can see it working for some cases. In the example I was thinking of we should really get the data fixed up in the sources – for various reasons of data governance (and third parties keeping their intellectual property) we do not have access direct (SQL) access to the application data structure structures. The other problem is that people who want the reports do not know much about the source.. I guess this is quite common where business users are interested in highly aggregated information and not raw fact

The best we can do is report back the problems and hope they get fixed.

]]> By: Emil http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6591 Emil Mon, 27 Oct 2008 14:24:36 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6591 Appologies Peter... Just missed to look at the author of the post.. Appologies Peter…

Just missed to look at the author of the post..

]]> By: Emil http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6590 Emil Mon, 27 Oct 2008 12:17:43 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6590 Hi Mark, From my experience in dealing with data quality matters there is no 100% solution. But in my projects what I have done I will gladly share it with you. Extending the Ralph Kimball idea to include a control dimention , I have come to a solution to log all errors encounterred in a dimentional objects ( facts and respective dimensions) then showing those error reports to the BI administartor or to respective business users responsible to check and correct errors. The error reports are not just reports but forms like where using writeback option of BI , procedures can be run to correct the errors and even to re-run the etl again. Automation is not possible but giving a nice user interface where to correct some/most of the errors is acceptable to most end-users. This is the practical solution of OBIEE , I have implemented ...of course all the analysis must be comleted and there lots of nice data cleansing methodolgies but at the end of the day , smth simple and understandable must be presented to the end-user and in most of the cases I had , this approach will be accepted... Best ragards, Emil. PS Hope to attend your training days once again ... if closer to EEC Hi Mark,

From my experience in dealing with data quality matters there is no 100% solution. But in my projects what I have done I will gladly share it with you. Extending the Ralph Kimball idea to include a control dimention , I have come to a solution to log all errors encounterred in a dimentional objects ( facts and respective dimensions) then showing those error reports to the BI administartor or to respective business users responsible to check and correct errors. The error reports are not just reports but forms like where using writeback option of BI , procedures can be run to correct the errors and even to re-run the etl again. Automation is not possible but giving a nice user interface where to correct some/most of the errors is acceptable to most end-users. This is the practical solution of OBIEE , I have implemented …of course all the analysis must be comleted and there lots of nice data cleansing methodolgies but at the end of the day , smth simple and understandable must be presented to the end-user and in most of the cases I had , this approach will be accepted…

Best ragards,
Emil.

PS Hope to attend your training days once again … if closer to EEC

]]> By: illiyaz http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/comment-page-1/#comment-6578 illiyaz Sat, 25 Oct 2008 20:43:20 +0000 http://www.rittmanmead.com/2008/10/25/end-to-end-data-quality/#comment-6578 Data Quality has been one of the oldest problems and i think if the users are well versed with the source system, they should be able to correct the data on the fly(using write back features) if it is possible.Also if a log of corrections is maintained(may be in a table), the inexperienced users can use the logtable to some extent in ascertaining the right values. Data Quality has been one of the oldest problems and i think if the users are well versed with the source system, they should be able to correct the data on the fly(using write back features) if it is possible.Also if a log of corrections is maintained(may be in a table), the inexperienced users can use the logtable to some extent in ascertaining the right values.

]]>