Take the postal code (or zip code) where does it fit in the hierarchy? Well the answer is might not fit at all. Postal codes were developed to help post offices deliver mail – and each postal authority did their own thing. The UK and the Netherlands have postal code systems that can identify a single street or even a cluster of houses of within a street. Other countries work on a code per town or group of nearby towns – so straight away we have a difference in grain; a few houses in the UK a few towns in France. In Germany postal codes relate to geographic areas but those areas are not aligned to the Bundesländer; on the other hand, France ties postal code to Department but there are anomalies notably where a river runs through a village and opposite banks share a postal code but are different Departments (and in one case, different regions). Some national postal codes are numeric, some area alphanumeric (like Canadian and UK ones), the length of the postcode varies between countries too.

Perhaps the sensible thing, especially if you are dealing with addresses from multiple countries, is to not use postal code as a level in geographical hierarchies. If you use them at all just make them as an attribute of the address and remember that they don’t always have geographical parents.

I think the key point about modelling geography is that just because you know how addresses work in your own country you can’t assume that they work like that in the country next door. If you have a requirement to report, for example, the efficiency of the postal service in delivering your goods by postal region you need to ensure that your reporting handles the anomalies and exceptions. As always, knowing your data is key to creating a correct model.

The CONTACT table in the source system had two columns storing the name: FIRST_NAME and LAST_NAME. There were two different processes that wrote entries in the CONTACT table… and one of them was faulty, writing the entire name in the FIRST_NAME column, though it was never corrected. So the following is a decent representation of what the data looked like:

SQL> create table CONTACT (contact_id NUMBER, first_name varchar2(50), last_name varchar2(50));

Table created.

Elapsed: 00:00:00.02
SQL> insert into CONTACT values (1, 'Bryson, Stewart W.', NULL);

1 row created.

Elapsed: 00:00:00.02
SQL> insert into CONTACT values (2, 'Mead, Jon', NULL);

1 row created.

Elapsed: 00:00:00.00
SQL> insert into CONTACT values (3, 'Mark', 'Rittman');

1 row created.

Elapsed: 00:00:00.00
SQL> select * from CONTACT;

CONTACT_ID | FIRST_NAME           | LAST_NAME
---------- | -------------------- | --------------------
         1 | Bryson, Stewart W.   |
         2 | Mead, Jon            |
         3 | Mark                 | Rittman

3 rows selected.

Elapsed: 00:00:00.00
SQL>

I needed to map the BMM such that I could return FIRST_NAME and LAST_NAME regardless of whether the entire name was concatenated into the FIRST_NAME, or whether it was correctly distributed across both columns. Additionally, the fact that the middle initial needed to be included with FIRST_NAME also proved a little troubling. At the end of the day, this is what I came up with in SQL:

SQL> SELECT CASE
  2           WHEN last_name IS null THEN trim(regexp_substr(first_name,'[^,]+$'))
  3           ELSE first_name
  4         END first_name,
  5         CASE
  6           WHEN last_name IS null THEN regexp_substr(first_name,'^([^,]+)')
  7           ELSE last_name
  8         END last_name
  9    FROM contact
 10  /

FIRST_NAME           | LAST_NAME
-------------------- | --------------------
Stewart W.           | Bryson
Jon                  | Mead
Mark                 | Rittman

3 rows selected.

Elapsed: 00:00:00.01
SQL>

To explain a bit, I’ll start with how I extracted the first name information from the FIRST_NAME column. I needed to start at the comma and then get the entire string until the end of the column. So I used the [^] structure in regular expressions, which basically says, return anything EXCEPT the character between the brackets and after the carrot (^). The plus (+) instructs the RegEx engine to return one or more instances of the previous structure. And at the end, the dollar sign ($) dictates that the entire string must run to the end of the column value. So taken all together, [^,]+$ instructs the RegEx engine to:

“Start at the first character that is not a comma, and return all non-comma characters all the way to the end of the column value.”

The only kludge introduced here is that the first non-comma character was actually a space, and to remove it, I simply used a TRIM. If some one has a way to do this without a TRIM, then I’d be glad to hear it.

To extract the last name information from the FIRST_NAME column, I used a similar mechanism, except that, instead of using the dollar sign ($) at the end, I put the carrot (^) at the beginning. It’s the same concept: it means that the expression returned has to begin at the start of the column value. So, the ^([^,]+ instructs the RegEx engine to:

“Start at the beginning of the column value, and return the whole string until a comma is encountered.”

Easy enough.

Now I want OBIEE to accept this SQL in the BMM. The only issue here is that OBIEE does not support regular expressions in it’s SQL language, so I have to use the EVALAUTE command to pass Oracle’s regular expression syntax back through to the database. So I’ll demonstrate how to do this, but first I’ll need to create a fact table to join to the CONTACT table in OBIEE.

SQL> CREATE TABLE activity (contact_id number, activity_date date, num_calls NUMBER);

Table created.

Elapsed: 00:00:00.04
SQL> INSERT INTO activity VALUES (1, SYSDATE-2, 10);

1 row created.

Elapsed: 00:00:00.00
SQL> INSERT INTO activity VALUES (2, SYSDATE-1, 20);

1 row created.

Elapsed: 00:00:00.00
SQL> INSERT INTO activity VALUES (3, sysdate, 30);

1 row created.

Elapsed: 00:00:00.00
SQL>
SQL> SELECT * FROM activity;

CONTACT_ID | ACTIVITY_DATE          |  NUM_CALLS
---------- | ---------------------- | ----------
         1 | 12/16/2009 10:21:34 AM |         10
         2 | 12/17/2009 10:21:34 AM |         20
         3 | 12/18/2009 10:21:34 AM |         30

3 rows selected.

Elapsed: 00:00:00.01
SQL>

To demonstrate the whole process in OBIEE, I first built the BMM to bring the data in how it is from the database:

To generate the correct data, using the regular expressions developed above, here is how I mapped the First Name attribute:

CASE
WHEN "bidw".""."STEWART"."CONTACT"."LAST_NAME" IS NULL
  THEN  Trim(BOTH ' ' FROM Evaluate('regexp_substr(%1,''[^,]+$'')', "bidw".""."STEWART"."CONTACT"."FIRST_NAME" ))
ELSE
  "bidw".""."STEWART"."CONTACT"."FIRST_NAME"
END

And here is how I mapped the Last Name attribute:

CASE
WHEN "bidw".""."STEWART"."CONTACT"."LAST_NAME" IS NULL
  THEN Evaluate('regexp_substr(%1,''^[^,]+'')', "bidw".""."STEWART"."CONTACT"."FIRST_NAME" )
ELSE
  "bidw".""."STEWART"."CONTACT"."LAST_NAME"
END

And finally… the results:

One point to keep in mind about fixing bad data on the source; it is just that, fixing DATA. We are not fixing bad applications or bad processes. Often fixing source applications is just not going to happen, especially with the case of legacy and third-party packaged applications. Likewise process can be hard to fix; if we rely on humans to key information in and the application is incapable of enforcing data rules then data entry problems will occur. For example take a university admissions system; here data entry is highly seasonal, first off a lot of prospective candidates (say about 10x the number who will start as students in the new year) then a reduced number of place offers which get further reduced by the candidates choosing another university or not reaching the entrance grades. This work flow often requires extra temporary staff just to key in details, they are not full-time employees, they do not know the significance of the data, they do not know about the reporting systems that give insight to the data, they are just there to get the data in as quickly as possible and with enough accuracy to let the process as whole work. It really does not matter to the process if the country name is not standardised providing the funding code is correct; even the course code may not matter that much providing the faculty is correct, especially if all faculty members follow a common scheme of study in the first year.
So, fixing data at source is most unlikely to be a one off exercise. True, a one-off data fix will improve the quality of our historic reporting, but what we are not doing is preventing data errors from occurring in the future. We must build into our ETL processes methods to continually monitor source data quality and build processes so that any errors detected can be corrected at source.

In the interim, the customer can work on data quality; they know what they need to report on (it’s in the project mandate!) and they know the sources of information (their operational systems) so they can start to verify that all of the required facts can be found in the source systems and more importantly look at the data content and assess “fitness for purpose”. If data defects are found then it may be possible to get them fixed before the serious construction of the ETL layer starts. Besides, the knowledge of source and target gives a good head start in the specification of ETL interfaces.

One particular issue they might meet, and one that is sadly far too common across many business sectors, is the use of operational systems that do not enforce data integrity. For whatever reasons there is just too much freedom in data entry and although it may not affect the operational system much it really can cause problems when you try to aggregate information on the BI system.

But how do we deal with this? Recently I joined in on a thread on one of LinkedIn BI groups where it was proposed that a “receive garbage, store garbage strategy was adopted” – in my opinion this might be OK for a mature BI system where users can understand that the reporting accurately reflects the source, but for a new venture into BI? To me, this seems to be too much a risk; it might be that the new BI users do not have sufficient exposure to the source systems to realise that the data is at fault on the source. We could prevent data that fails a quality threshold from loading on the BI system, but then we would show incomplete results which although correctly aggregated are misleading because of omission; at the end of the day load policy is a business choice. If we go with the “reject poor data” route we should seriously  think about providing a data quality dashboard on the reporting system to indicate the numbers of records that failed to be loaded and drill-down to the reasons why they failed.

So what do we do with data that fails the quality standard? Ideally, we should get it fixed at source. Auto-fixing on load is possible, but then we need to think about data governance and the possible ‘trust’ problems of the data being not aligned with the source. Maybe you could ‘standardise’ country names and other columns on loading; I’ve seen systems with ‘USA’, ‘U.S.A’, ‘U S A’, ‘US of A’, ‘America’,  and ‘US’ in the country data feed and that’s before we get to the mis-keying of ‘United’ to get ‘Untied’!  But maybe that sort of improvement in quality should also be available to operational systems users.

For this customer, I have suggested that they construct a source to BI target matrix and include some basic traffic light measures on the source data:

• How good is it?
• What sort of errors are present; missing items, typographical errors, missing or incorrect parents, inconsistent use of names, even data entered in the wrong fields.
• How important is it to be correct in the BI system; for example street address can not be aggregated in reporting and we may not be going to use BI to create mailing lists, but postal code (or a sub string of it) can be used to aggregate people by location areas.
• How important is it to be correct on the operational source – do we need to apply the corrections at source to improve the operational use of the system

But this type of quality review may not tackle the data problem that is probably hardest to deal with what is a correct fact? How do I know if house value of £20,000 is reasonable (it  could be in a shared ownership scheme) or £2,0000,000 or £20,000,000? We could set a validation range, but where is there that point that one penny more is obviously wrong, but the current value OK?

The past week I have been with a customer delivering a mixture of training and best practice advice. The training was based on my data warehouse design course, but heavily customised to fit their specific needs, and with complete new sections to look at data quality strategies and some specific approaches to managing a major BI development project. In reality, I guess that 40% of the material I presented came from existing Rittman Mead courses and the rest I created to fit the customer’s needs. I think this is a good way to work – I had always intended to write more extensively on data quality and the new slide set will be a useful add-in to our other training materials – and, of course, the customer gets the course they want.  I will write about some of the data quality issues we discussed in a future posting, a lot of this will be highly relevant to most BI / DW owners.

Next week I am working on some course customisations for another customer and then joining the others at our first ever Oracle BI Training Days – I have been working on this project for quite a while now and am really looking forward to seeing the fruits of our work. The whole team have put in a lot of effort; it should be a good event!

So what has data quality got to do with BI? Everything, For a BI system to be successful you need to fulfil three objectives:
• It must tell people what they need to know – by that I mean it must encompass enough detail to have real use
• it must tell people what they need know rapidly enough
• and it must tell people the truth
The last item is something that has to be designed in from the beginning of a BI project; performance and scope can be enhanced later but quality can’t, or not without having to replace already loaded data.
When I get involved on an ETL project I probably spend less than a quarter of my time on building ETL code and getting to it run, the majority of my time goes on data quality and finding and explaining anomalies. Where we can, we get data fixed at source, but sometimes that is not going to be possible; but data profiling allows us to formulate rules to handle the expected exceptions (whether these are auto-fix procedures or park it one side and let a human make the call rule) And then there is the unexpected exception, which of course must be handled.

Ignoring mock-database techniques such as reading spreadsheets using an ODBC driver and querying directly against the data the most common way of loading pure text files into Oracle is to describe the data in some form of control structure and then use SQL Loader or external tables to bring the data into a structure that can be queried. We may well need to do some work making sure that the data types are recognised correctly; this is often a case of making sure the that the file is appropriate for the NLS settings involved (or is that the NLS setting appropriate for the file) how many times do you see the 6-MAY-1955 failing to load because the database wants mai or even 05, or the wrong symbol is used for the decimal point. Some of these NLS induced difficulties may only affect a subset of the data for example some month abbreviations might be common across languages or perhaps a thousand separator is causing problems with numbers bigger than 999

But loading the data in only part of the battle, we need to check it for sense, to profile it for anomalies, and to try to impose some referential (or dimensional) integrity rules on it; if we load data relating to products and product type we need to verify that we already know about the product types (the parent keys) or come up with a mechanism that allows to load both parents and children; and what constitutes a duplicate record? Where we find problems we need to feedback to the data provider and workout how we can fix the source data if it is lacking or work with them on rules to fix-up the data once it is loaded. The key thing is to get this right before we start to propagate suspect data into a the data warehouse.

Quality in structured data is simple (from a rules viewpoint) for me: if it is not clean I won’t store it. That is, upstream of my data warehouse I have processes to clean data; it is de-duplicated, referentially checked, sense checked or whatever-else-is-appropriate checked before it is loaded and the nonconforming data is parked for investigation or rule-based fix-up. Dirty data is bad for reporting accuracy and user confidence, it is also problematic to remove once it has been propagated though a BI system. Tools exist to profile data, and database features help to clean up some of the problems with data hierarchy mismatches, for example Oracle dimension objects can be examined by a call to DBMS_DIMENSION.VALIDATE_DIMENSION (or before 10g, the similar DBMS_OLAP.VALIDATE_DIMENSION) and inspecting the exceptions table for ‘bad’ rows; this may not be the technique of choice for a data load process, but is a good method to validate that already loaded data complies to your sense of dimensionality.

But when we move to unstructured data we come across a fundamental problem… what is quality? By definition there are no clearly defined dimension hierarchies, there probably aren’t even tags to identify whether data is ‘reference’ (unstructured reference data sounds a bizarre concept!) or ‘fact’ let alone customer or product related. We are probably not concerned with duplicates as each unstructured item should be considered as unique, but do we need consider near duplicates to be versions (revisions) of the same item or new items? In practice the only sensible thing to today is store the data ‘as is’ maybe correcting spelling to some standard form (but that could well contravene governance rules) and rely on smart contextual indexing techniques to identify data items that possesses attribute.

Establishing context is hard

A long while back I worked on a project for a publisher of online classified ads that takes copy submitted electronically and assess it for compliance with standards of legality and decency. Apart from a complex rule base that required certain professions to supply validation of right to practice, the brunt of the system was about scanning for rude words, their phonetic equivalent and IM and web variants where numb3rs might appear in place of letters and vowels dropped. But this approach really worked with words in isolation, moving to spans of text and scanning for expressions becomes harder – just think about electronically trying to establish meaning from poetry where metaphors and non-standard word meanings abound. And then when the feed is not keyed text but machine recognised speech or scanned handwriting.

Often the biggest problems in data quality projects are nothing at all to do with the technology needed but are to do with people. And those people are not always the technologists involved directly with the work; often it is the information workers spread throughout the organisation. Fixing up these people issues needs a degree of charm and tact from the data architect running the data quality project, oh, and it needs a corporate big-hitter to sponsor the project and to enforce change.

• Common taxonomies need to be established
• Data stewards, or owners, (and these are business people, not IT staff) need to be identified and empowered to maintain specific types of data.
• Where regularity compliance is important (and isn’t that everywhere?) data quality processes need to be engineered so that only the right people can make changes and that might mean that the ownership of a item is split on attribute domains – say the financial attributes are owned by an accountant and the physical attributes by someone in supply chain.

As I said, sorting out the people is often the hard part, moving the clean data around is remarkably easy in comparison

However, for this customer, product is not simple, they have three ways to view a product depending on whether they are buying, selling or just counting it. Let me explain.

Assume my customer runs a burger bar (which of course they don’t) then they sell things such as double cheese burger which is two burger patties, lettuce, tomato, relish, mayo, cheese, dill pickle in a bun. They also buy from their supplier disks of ground meat, the burger patty; these are delivered in boxes of 50, 100, or 250. And finally the bar manager counts his stock in terms of individual burgers. In BI terms this sort of thing does not easily map into conventional hierarchies; for one thing there is a many to many relationship between counted stock items and items in a burger recipe – do we have one burger patty or two? and what about all of the other ingredients: this is not exactly the classic parent child rollup that is so easy to model in a data warehouse.

The relationship between goods supplied and counted stock is much more simple, each case size could be considered the child of the counted item. That is, 50 patty boxes and 250 patty boxes all rollup to “burger patty”. But this rollup also has an impact on the attribute stock cost – the cost of a single patty; obviously it is the cost of 1/50th of a carton of 50 patties, but what if the supplier discounts the larger boxes? what value is assigned to the item cost. And then, by extension, we try to calculate the cost of a cheese burger as the sum of the individual recipe items (plus the prep costs, fuel, staff etc) but what if we used burger patties from the big carton? Do our costs go down and profits increase?

This type of discussion takes a long time to work though, loads of “what ifs” and usually ends up with the accountants having to think things through. But the end result is that the reporting is more accurate than before, or at least I hope so