Recently, I sat down (virtually) with Kevin McGinley of Accenture to discuss the upcoming ODTUG Kscope 12. I was on the content selection committee, and immediately recognized how lucky ODTUG was to have Kevin coordinating this process. We had tough choices to make around content — this is always the case, as I’ve participated in this capacity before in the past. But Kevin always took us in the right direction, and after the process was over, I knew I wanted to have a discussion with him on the blog so our readers could see what awaits them at Kscope 12.

Kevin recently blogged about Kscope 12 on the ODTUG Blog, so perhaps that is a nice introduction to our interview here. I’d like to thank Kevin for taking a little time to do this interview, and I’d also like to thank Accenture for allowing him to appear here.

[Stewart Bryson] This is only your second Kscope, but already you are a winner of the Editor’s Choice award for your whitepaper at Kscope 11, and now, are the BI content lead for Kscope 12. What do you think it is about ODTUG and Kscope that you have connected with?

[Kevin McGinley] I was amazed by three things at Kscope 11.  First, the ODTUG community is a very warm, welcoming community of people who were very easy to engage with, both on a professional and personal level.  Second, I was pleased with the type of content presented at Kscope versus a larger conference like Open World.  The sessions feel very real, the presenters are very approachable, and the level of discussion/interaction is much higher.  Lastly, I was very impressed with the level of organization at Kscope.  The conference flowed very smoothly, there were a lot of interesting activities outside the core sessions, and the entertainment was top-notch.

[Stewart Bryson] For those folks who have never attended Kscope before, how would you describe the event, perhaps drawing comparisons or differences with other conferences?

[Kevin McGinley] As I alluded to above, Kscope is much more communal than a larger conference like Open World.  Open World is a mad dash against 40,000+ strangers to get from place to place.  You are exhausted by the end of the week, and the practical knowledge you take away can be limited.  Kscope is a more manageable pace, the practical knowledge you gain from the sessions is much higher, and there is greater emphasis on interaction and discussion.

[Stewart Bryson] Thinking specifically about the BI Stream, what would you say to Kscope Alumni about the BI Stream this year that might encourage them to give the conference another try?

[Kevin McGinley] I would say two things to this.  First, BI keeps growing at Kscope – we have about 50% more sessions than we did last year!  This is great because you get to offer more variety in the content and you also get to balance the “intro” audiences against the “technical” audiences – satisfying both.  Second, Kscope has a tremendous EPM presence – quite possibly the biggest EPM conference around – and with BI and EPM converging the way they are, this offers attendees a tremendous opportunity to start looking at how to maximize their Oracle investments in these two areas and expand the value they provide to their businesses.

[Stewart Bryson] What can you tell us about the content selection process? Did you have a particular focus or goal in mind when selecting and scheduling the presentations?

[Kevin McGinley] Because OBIEE 11g was introduced before Kscope 11, it had a very strong presence that year due to the sheer magnitude of the release.  It was necessary to insure that the ODTUG community was well informed about OBIEE 11g.  Now that OBIEE 11g has settled in the marketplace, we can explore/return to other areas like the packaged BI Applications, data integration with ODI and Golden Gate, EPM integration, more BI Publisher, and the recently announced Exalytics.  We tried to make sure we still covered relevant areas of OBIEE, but left room to cover more of the Oracle offerings around OBIEE, since it’s rarely used by itself in a vacuum.

[Stewart Bryson] Any particular BI sessions that you are looking forward to?

[Kevin McGinley] I see what you’re trying to do here, Stewart – you’re looking for me to plug your two presentations! In all seriousness, there are a lot of great sessions that I’m excited about.  I also love that we have a great balance between customer speakers, boutique consulting companies, large consulting companies, independents, and Oracle ACEs.  I think that’s important.  To answer your question, though, I’m really excited to hear from customers like JC Penny, Eaton Vance, General Dynamics, and Clark Construction covering topics like OBI/EPM integration, rolling-out mobility to executives, and project testing strategies.

[Stewart Bryson] Being involved with content selection can be very time-consuming. How supportive has Accenture been with your dedication to Kscope?

[Kevin McGinley] Accenture has been great.  I think no matter where you work, you’re often pretty busy, so it helps to have an employer who is supportive with the extra time required to make sure Kscope is a great experience for everyone.  Accenture really recognizes the value of a smaller, more intimate conference like Kscope – we host a similar conference for our Oracle customers – and encourages its employees to engage in the industry community where possible.

[Stewart Bryson] Personally, I think Kscope provides a great opportunity to step outside my usual focus on BI and see some sessions in other streams. Last year I attended sessions on Exadata, PL/SQL development, and APEX. Has anything outside of the BI stream caught your eye?

[Kevin McGinley] The great thing about BI is that it complements other tracks nicely.  You can’t get very far in BI without a data store of some sort, so both the Database track and the Essbase track offer sessions that would be attractive to BI attendees.  Both data stores require optimization for BI to perform, and each track has very practical sessions on how to accomplish this.  I’m excited about that.  Another track I find interesting is the EPM Business Content, a new track this year.  Geared more towards a director or senior manager, this track can really help a BI person understand how EPM can fit into their environment and drive additional value.

As you can see, Business Intelligence is in good hands at Kscope 12. Hopefully, we’ll see you there!

In the last post, I describe what I call the Model-Driven iteration, where we take thin requirements from the end-user in the form of a user story and generate the access and performance layer, or our star schema, logically using the OBIEE semantic model. Our first several iterations will likely be Model-Driven as we work with the end user to fine-tune the content he or she wants to see on the OBIEE dashboards. As user stories are opened, completed and validated throughout the project, end users are prioritizing them for the development team to work on. Eventually, there will come a time when an end user opens a story that is difficult to model in the semantic layer. Processes to correct data quality issues are a good example, and despite having the power of Exadata at our disposal, we may find ourselves in a performance hole that even the Database Machine can’t dig us out of. In these situations, we reflect on our overall solution and consider the maxim of Agile methodology: “refactoring”, or “rework”.

For Extreme BI, the main form of refactoring is ETL. The pessimist might say: “Well, now we have to do ETL development, what a waste of time all that RPD modeling was.” But is that the case? First off… think about our users. They have been running dashboards for some time now with at least a portion of the content they need to get their jobs done. As the die-hard Agile proponent will tell you… some is better than none. But also… the process of doing the Model-Driven iteration puts our data modelers and our ETL developers in a favorable position. We’ve eliminated the exhaustive data modeling process, because we already have our logical model in the Business Model and Mapping layer (BMM).

But we have more than that. We also have our source-to-target information documented in the semantic metadata layer. We can see that information using the Admin Tool, as depicted below, or we can also use the “Repository Documentation” option to generate some documented source-to-target mappings.

When embarking on ETL development, it’s common to do SQL prototyping before starting the actual mappings to make sure we understand the particulars of granularity. However, we already have these SQL prototypes in the nqquery.log file… all we have to do is look at it. The combination of the source-to-target-mapping and the SQL prototypes provide all the artifacts necessary to get started with the ETL.

When using ETL processing to “instantiate” our logical model into the physical world, we can’t abandon our Agile imperatives: we must still deliver the new content, and corresponding rework, within a single iteration. So whether the end user is opening the user story because the data quality is abysmal, or because the performance is just not good enough, we must vow to deliver the ETL Iteration time-boxed, in exactly the same manner that we delivered the Model-Driven Iteration. So, if we imagine that our user opens a story about data quality in our Customer and Product dimensions, and we decide that all we have time for in this iteration are those two dimension tables, does it make sense for us to deliver those items in a vacuum? With the image below depicting the process flow for an entire subject area, can we deliver it piecemeal instead of all at once?

The answer, of course, is that we can. We’ll develop the model and ETL exactly as we would if our goal was to plug the dimensions into a complete subject area. We use surrogate keys as the primary key for each dimension table, facilitating joining our dimension tables to completed fact tables. But we don’t have completed fact tables at this point in our project… instead we have a series of transaction tables that work together to form the basis of a logical fact table. How can we use a dimension table with a surrogate key to join to our transactional “fact” table that doesn’t yet have these surrogate keys?

We fake it. Along with surrogate keys, the long-standing best practice of dimension table delivery has been to include the source system natural key, as well as effective dates, in all our dimension tables. These attributes are usually included to facilitate slowly-changing dimension (SCD) processing, but we’ll exploit them for our Agile piecemeal approach as well. So in our example below, we have a properly formed Customer dimension that we want to join to our logical fact table, as depicted below:

We start by creating aliases to our transactional “fact” tables (called POS_TRANS_HYBRID and POS_TRANS_HEADER_HYBRID in the example above), because we don’t want to upset the logical table source (LTS) that we are already using for the pure transactional version of the logical fact table. We create a complex join between the customer source system natural key and transaction date in our hybrid alias, and the natural key and effective dates in the dimension table. We use the effective dates as well to make sure we grab the correct version of the customer entity in question in situations where we have enabled Type 2 SCD’s (the usual standard) in our dimension table.

This complex logic of using the natural key and effective dates is identical to the logic we would use in what Ralph Kimball calls the “surrogate pipeline”: the ETL processing used to replace natural keys with surrogate keys when loading a proper fact table. Using Customer and Sales attributes in an analysis, we can see the actual SQL that’s generated:

We can view this hybrid approach as an intermediate step, but there is also nothing wrong with this as a long-term approach if the users are happy and Exadata makes our queries scream. If you think about it… a surrogate key is an easy was of representing the natural key of the table, which is the source system natural key plus the unique effective dates for the entity. A surrogate key makes this relationship much easier to envision, and certainly code using SQL, but when we are insulated from the ugliness of the join with Extreme Metadata, do we really care? If our end users ever open a story asking for rework of the fact table, we may consider manifesting that table physically as well. Once complete, we would need to create another LTS for the Customer dimension (using an alias to keep it separate from the table that joins to the transactional tables). This alias would be configured to join directly to the new Sales fact table across the surrogate key… exactly how we would expect a traditional data warehouse to be modeled in the BMM. The physical model will look nearly identical to our logical model, and the generated SQL will be less interesting:

Now that I’ve described the Model-Driven and ETL Iterations, it’s time to discuss what I call the Combined Iteration, which is likely what most of the iterations will look like when the project has achieved some maturity. In Combined Iterations, we work on adding new or refactored RPD content alongside new or refactored ETL content in the same iteration. Now the project really makes sense to the end user. We allow the user community–those who are actually consuming the content–to dictate to the developers with user stories what they want the developers to work on in the next iteration. The users will constantly open new stories, some asking for new content, and others requesting modifications to existing content. All Agile methodologies put the burden of prioritizing user stories squarely on the shoulders of the user community. Why should IT dictate to the user community where priorities lie? If we have delivered fabulous content sourced with the Model-Driven paradigm, and Exadata provides the performance necessary to make this “real” content, then there is no reason for the implementors to dictate to the users the need to manifest that model physically with ETL when they haven’t asked for it. If whole portions of our data warehouse are never implemented physically with ETL… do we care? The users are happy with what they have, and they think performance is fine… do we still force a “best practice” of a physical star schema on users who clearly don’t want it?

So that’s it for the Extreme BI methodology. At the onset of this series… I thought it would require five blog posts to make the case, but I was able to do it in four instead. So even when delivering blog posts, I can’t help but rework as I go along. Long live Agile!

I’ll gradually introduce the different types of generic iterations that we engage in, focusing on what I call the “model-driven” iteration for this post. Our first few iterations are always model-driven. We begin when a user opens a user story requesting new content. For any request for new content, we require that all the following elements are including in the story:

1. A narrative about the data they are looking for, and how they want to see it. We are not looking for requirements documents here, but we are looking for the user to give a complete picture of what it is that they need.
2. An indication of how they report on this content today. In a new data warehouse environment, this would include some sort of report that they are currently running against the source system, and in a perfect world, this would involve the SQL that is used to pull that report.
3. An indication of data sets that are “nice to haves”. This might include data that isn’t available to them in the current paradigm of the report, or was simply too complicated to pull in that paradigm. After an initial inspection of these nice-to-haves and the complexity involved with including them in this story, the project manager may decide to pull these elements out and put them a separate user story. This, of course, depends on the Agile methodology used, and the individual implementation of that methodology.

First we assign the story to an RPD developer, who uses the modeling capabilities in the OBIEE Admin Tool to “discover” the logical dimensional model tucked inside the user story, and develop that logical model inside the Business Model and Mapping (BMM) layer. Unlike a “pure” dimensional modeling exercise where we focus only on user requirements and pay very little attention to source systems, in model-driven development, we constantly shift between the source of the data, and how best the user story can be solved dimensionally. Instead of working directly against the source system though, we are working against the foundation layer in the Oracle Next-Generation Reference Data Warehouse Architecture. We work from a top-down approach, first creating empty facts and dimensions in the BMM, and mapping them to the foundation layer tables in the physical layer.

To take a simple example, we can see how a series of foundation layer tables developed in 3NF could be mapped to a logical dimension table as our Customer dimension:

I rearranged the layout from the Admin Tool to provide an “ETL-friendly” view of the mapping. All the way to the right, we can see the logical, dimensional version of our Customer table, and how it maps back to the source tables. This mapping could be quite complicated, with perhaps dozens of tables. The important thing to keep in mind is that this complexity is hidden from not only the consumer of the reports, but also from the developers. We can generate a similar example of what our Sales fact table would look like:

Another way of making the same point is to look at the complex, transaction model:

We can then compare this to the simplified, dimensional model:

And finally, when we view the subject area during development of an analyses, all we see are facts and dimensions. The front-end developer can be blissfully ignorant that he or she is developing against a complex transactional schema, because all that is visible is the abstracted logical model:

When mapping the BMM to complex 3NF schemas, the BI Server is very, very smart, and understands how to do more with less. Using the metadata capabilities of OBIEE is superior to other metadata products, or superior to a “roll-you-own metadata” approach using database views, because of the following:

1. The generated SQL usually won’t involve self-joins, even when tables exists in both the logical fact table, and the logical dimension table.
2. The BI Server will only include tables that are required to facilitate the intelligent request, either because it has columns mapped to the attributes being requested, or because the table is a required reference table to bring disparate tables together. Any tables not required to facilitate the request will be excluded.

Since the entire user story needs to be closed in a single iteration, the user who opened the story needs to be able to see the actual content. This means that the development of the analyses (or report) and the dashboard are also required to complete the story. It’s important to get something in front of the end user immediately, but it doesn’t have to be perfect. We should focus on a clear, concise analyses in the first iteration, so it’s easy for the end user to verify that the data is correct. In future iterations, we can deliver high-impact, eye-catching dashboards. Equally important to closing the story is being able to prove that it’s complete. In Agile methodologies, this is usually referred to as the “Validation Step” or “Showcase”. Since we have already produced the content, then it’s easy to prove to the user that the story is complete. But suppose that we believed we couldn’t deliver new content in a single iteration. That would imply that we would have an iteration during our project that didn’t include actual end-user content. How would you go about validating or showcasing that content? How would we go about showcasing a completed ETL mapping, for instance, if we haven’t delivered any content to consume it?

What we have at the end of the iteration is a completely abstracted view of our model: a complex, transactional, 3NF schema presented as a star schema. We are able to deliver portions of a subject area, which is important for time-boxed iterations. The Extreme Metadata of OBIEE 11g allows us to remove this complexity in a single iteration, but it’s the performance of the Exadata Database Machine that allows us to build real analyses and dashboards and present it to the general user community.

In the next post, we’ll examine the ETL Iteration, and explore how we can gradually manifest our logical business model into a physical model over time. As you will see, the ETL iteration is an optional one… it will be absolutely necessary in some environments, and completely superflous in others.

If you want to export to UDML than you can use the commands: nqudmlexec.exe and  nqudmlgen.exe.
If you want to export to XUDML than you can use the commands: biserverxmlgen.exe and  
These commands can be found in this location: C:\oracle\middleware\Oracle_BI1\bifoundation\server\bin

Lets start with an example to export an RPD to XUDML.

1. Make a copy of your original RPD and put in a temporary folder , eg c:\testconnpool.
2. Generate the XUDML by executing the command biserverxmlgen
3. Go to the folder C:\oracle\middleware\Oracle_BI1\bifoundation\server\bin and type this command

biserverxmlgen -R C:\testconnpool\base.rpd -P Admin123 -O c:\testconnpool\test.xml -8

The parameters for the biserverxmlgen command are as follows:

- P repository_password
- R repository_pathname
- O output_script
-8 UTF-8


4. When this step is successful the XUDML is generated. Open the XUDML file named “test.xml” and check the content of the file.
<?xml version="1.0" encoding="UTF-8" ?>

As this post is about changing the connection pool we are going to look in the XUDML file and look for the tag <Connection Pool>

<ConnectionPool name="Master" parentName="&quot;Master&quot;" parentId="3023:5" parentUid="8" id="3029:7" uid="10" user="obiee" password="6C6F0BE01FC4111F3AC2236A364D912E9B70195D55A2B76F203F3DDAF9C76F072606B70FAE3A2C3FC913
843F794309A82CF208A690AD0045D3DCE7CCEEA3B72F 137949B8C699489A5D548D7D5F4828ADB4071CCB22E7FEECBCE71592C1AC13E3" timeout="300" maxConnDiff="10" maxConn="10" dataSource="master" type="Default" reqQualifedTableName="false" isSharedLogin="true" isConcurrentQueriesInConnection="false" isCloseAfterEveryRequest="false" outputType="xml" bulkInsertBufferSize="131072" tempTablePrefix="TT" transactionBoundary="1000" xmlaUseSession="false" isSiebelJDBSecured="false"><Description></Description></ConnectionPool>

So we were able to export the RPD to XUDML and to find the connection pool in this file. Now we want to change the password of this connection pool but this password is encrypted. When you try to change the encrypted password to  the new password  this will not work. Some extra steps are needed to change the password of the connection pool.

5. Go back to the  original copy of the RPD, open it and select Manage > Variables from the application menu.

6. Now we are going to create two static repository variables:

puser ‘obiee’
ppass ‘mastertest’

7. Next, go the connection pool in the rpd and replace the user and the password with the static variables. Replace the user name with valueof(puser) and replace the password with valueof(ppass) and save the RPD.

8. Now rerun the XUDML command
biserverxmlgen -R C:\testconnpool\base.rpd -P Admin123 -O c:\testconnpool\test2.xml -8

9. Open the XUDML file “test2.xml” and look for the variable tag ppass
<Variable name="puser" id="3031:65" uid="126">
<Description></Description>
<Expr><![CDATA['obiee']]></Expr>
</Variable>
<Variable name="ppass" id="3031:66" uid="128">
<Description></Description>
<Expr><![CDATA['mastertest']]></Expr>
</Variable>

Change the content of this variable with the desired new password; in this case you need to replace ‘mastertest’ by the new password and save the changes.

Now generate the new RPD by using the command biserverxmlexec. Go to the folder C:\oracle\middleware\Oracle_BI1\bifoundation\server\bin and execute this command:

biserverxmlexec -I c:\testconnpool\test2.xml -O base2.rpd

The tool will prompt for the repository password.

The parameters for the biserverxmlexec command are as follows:

-P      : Password for repository
-I      : Input script file name
-B      : Base repository name
-O      : Output repository name

A new RPD is generated and the password for the connection pool has been changed.

I have long felt that BI is only done for one of three reasons – the law says we must report things, it saves us money, or it makes us money; so if knowing something sooner gives us competitive advantage then surely that is a good thing. Knowing sooner is not enough though; it is also about being able to act on the information to facilitate a change in the organization that enhances return (or lowers costs). To my mind we are moving from the traditional “let’s look at this in aggregate” stance to a world where we ask “what is the significance of this newly observed fact”. This type of analysis requires a body of data to create a reference model and access to smart statistical tools to allow us to make judgments based on probabilities. Making such decisions based on dynamic events is not just for stock markets and bankers, the same principles apply in many sectors. I know of some restaurant chains that have investigated using centrally monitored sales across all outlets to dynamically adjust staff levels based on likely demand – staff are sent home, brought in, moved between outlets based on a predictive model that uses past trading patterns across many outlets.

As usual, most of the building blocks we need to do this are available to us, we just need a bit of creativity to join them together into an architecture. For this kind of use I feel that messaging should be core to the data capture – we want to look at single items of “fact” and do some statistical analysis on them before adding them to the data warehouse (or what ever form our data repository takes) so that the new fact can become part of the base data set we use to analyze the next fact to arrive. Micro-batch loading of log based change data is probably less suited here as we are:

1. adding to the latency by using discrete loads at fixed intervals and
2. the processing of many items at a time complicates the statistical analysis and alerting phases (after all if we get 2453 credit card transactions in a batch only a few will be potentially fraudulent).

After we capture a message from the source system we can pass the information through a chain of processes to analyze the information, propagate alerts based on the statistical significance of the item and add the data to the data store so that it becomes part of the knowledge. This last stage of adding the message to the data will probably need to be in a micro-batch mode rather than one-row-at-a-time-as-it-arrives – the latencies of adding fact to a conformed OLAP system (database, cubes, whatever) are such that single row additions will just take too much time, even if our target is an in-memory system. Here the art of the designer is to balance the availability of data, the time to reprocess the OLAP structures, the desire to keep the system up to date. It is always worth noting that for many data domains having to-the-second data is not that important as any new rows are unlikely to change the statistical results, however some subject domains will need access to all of the recent information including that which has not yet made it to the data warehouse, and here the creativity comes in.

One of the sources for federation is a classic, batch-loaded EDW, with ETL processes that load conformed dimension tables, followed by fact tables that store the measures and calculations for the enterprise. Oracle Warehouse Builder (OWB), the ETL tool built inside the Oracle Database, is a standard choice for data warehouses built on the Oracle Database, and below, I show a sample process flow of what that batch load might look like:

Logical table sources (LTS’s) are a key feature within the OBIEE semantic model but are often misunderstood. Each LTS represents a single location for data to exist for either a logical fact table, or logical dimension table. A logical table in the BMM can have multiple LTS’s for any of the following reasons:

1. Including different table sources into a single logical table at different levels of granularity. Tables containing data pre-aggregated at a different level in a hierarchy is a common example of this scenario, and is known as “vertical fragmentation”.

2. Including different table sources into a single logical table at the same level of granularity. Having data exist in two different locations, but wanting them to be combined in particular situations, is a common example of this scenario, and is known as “horizontal fragmentation”.

Using horizontal fragmentation in OBIEE, we can map a single logical fact table to multiple LTS’s. For example, suppose we had a physical fact table in our EDW called SALES_FACT. To represent that fact table in the semantic model, we would create a logical fact table in the BMM — called “Sales Fact Realtime” in this example — and create an LTS that maps to the SALES_FACT table. We would also map another LTS which presents this data in the source system as well. As the source system is transactional and likely exists in third-normal form (3NF), the LTS that maps to the transactional schema would likely not be a simple one-to-one relationship. In 3NF, we would likely have to join multiple tables in our source system to represent the logical fact table Sales Fact Realtime:

We would have to do something comparable with the Customer Dimension:

With the two LTS’s, we still need to configure the horizontal fragmentation. For this implementation, I have configured a repository variable called RV_REALTIME_THRESHOLD_DT, with an initialization block that keeps the value consistently at TRUNC(SYSTDATE). I use this variable as the threshold between reporting against the EDW schema and the source system schema.

Once I have the variable available, I can configure the fragmentation on the fact table to use the threshold to determine the appropriate source for a particular record. This is less complicated with the EDW LTS… simple fragmentation configured for all rows with a transaction date less than the threshold date:

Whereas only the source system contains the newer rows needed for layering in real-time data… both the EDW and the source system contain historic data, albeit the EDW data is likely transformed to a certain degree. So we have to configure fragmentation using the RV_REALTIME_THRESHOLD_DT variable, but we also have to use that variable as a filter on the source system LTS to make sure we don’t over allocate the data.

What’s the result of all this complex mapping among different LTS’s in the BMM? OBIEE understands that each source schema is completely segmented, and the tables in each LTS never join to tables in the other LTS… but they do union. OBIEE will construct a complete query against the transactional schema, in this example, joining between the CUSTOMER_DEMOG_TYPES, CUSTOMERS, POS_TRANS and POS_TRANS_HEADER tables. Additionally, OBIEE will construct another complete query against the EDW schema, in this case, only the tables SALES_FACT and CUSTOMER_DIM. The BI Server then logically unions the results between the two source schemas into a single result set that is returned whenever a user builds a report against the logical tables Customer Dim and Sales Fact Realtime. So I run the following report against my fragmented Sales Fact Realtime:

The interesting part is how OBIEE does the logical union. When the EDW and the transactional schema exist in separate databases, the BI Server issues two different database queries and combines them into a single result set in its own memory space. However, if the schemas exist within the same database, as the Oracle Next-Generation Reference Architecture recommends, then the BI Server is able to issue a single query, transforming the logical union into an actual physical union in the SQL statement, as demonstrated in the statement below. Notice that the SQL threshold has been applied, and the UNION was constructed with a single SQL statement pushed down from the BI Server to the Oracle Database holding the Foundation and Presentation and Access layers in our Oracle architecture:

WITH
SAWITH0 AS (select T44105.AMOUNT as c1,
     T44042.CUSTOMER_LAST_NAME as c2,
     T48199.CALENDAR_MONTH_NUMBER as c3,
     T48199.CALENDAR_YEAR as c4,
     T48199.SQL_DATE as c5
from
     GCBC_EDW.DATE_DIM T48199 /* CONFORMED_DATE_DIM */ ,
     GCBC_EDW.CUSTOMER_DIM T44042,
     GCBC_EDW.SALES_FACT T44105
where  ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T48199.DATE_KEY ) ),
SAWITH1 AS (select T43971.SAL_AMT as c1,
     T43901.CUST_LAST_NAME as c2,
     T48199.CALENDAR_MONTH_NUMBER as c3,
     T48199.CALENDAR_YEAR as c4,
     T48199.SQL_DATE as c5
from
     GCBC_EDW.DATE_DIM T48199 /* CONFORMED_DATE_DIM */ ,
     GCBC_CRM.CUSTOMERS T43901,
     GCBC_POS.POS_TRANS T43971,
     GCBC_POS.POS_TRANS_HEADER T43978
where  ( T43901.CUST_ID = T43978.CUST_ID
         and T43971.TRANS_ID = T43978.TRANS_ID
         and T48199.DATE_KEY =  TRUNC(T43978.TRANS_DATE)
         and T43978.TRANS_DATE >= TO_DATE('2011-05-16 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') 
       )),
SAWITH2 AS ((select concat(D0.c4, D0.c3) as c2,
     D0.c5 as c3,
     D0.c2 as c4,
     D0.c1 as c5
from
     SAWITH0 D0
union all
select concat(D0.c4, D0.c3) as c2,
     D0.c5 as c3,
     D0.c2 as c4,
     D0.c1 as c5
from
     SAWITH1 D0)),
SAWITH3 AS (select sum(D3.c5) as c1,
     D3.c2 as c2,
     D3.c3 as c3,
     D3.c4 as c4
from
     SAWITH2 D3
group by D3.c2, D3.c3, D3.c4)
select distinct 0 as c1,
     D2.c2 as c2,
     D2.c3 as c3,
     D2.c4 as c4,
     D2.c1 as c5
from
     SAWITH3 D2
order by c2, c4, c3

But OBIEE is also capable of doing the fragmentation equivalent of “partition pruning.” When the BI Server has enough information to know that the entire result set will come from a single source, then the SQL will be issued against only one of the LTS’s. For instance, if I click on one of the “SQL Date” attributes in the above report which will apply a filter on the fragmentation column, the BI Server will know that the result set only comes from the EDW:

WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
     concat(T48199.CALENDAR_YEAR, T48199.CALENDAR_MONTH_NUMBER) as c2,
     T48199.DATE_KEY as c3,
     T48199.SQL_DATE as c4,
     T44042.CUSTOMER_LAST_NAME as c5
from
     GCBC_EDW.DATE_DIM T48199 /* CONFORMED_DATE_DIM */ ,
     GCBC_EDW.CUSTOMER_DIM T44042,
                   GCBC_EDW.SALES_FACT T44105
where  ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY
         and T44042.CUSTOMER_LAST_NAME = 'Carr'
         and T44105.SALES_DATE_KEY = T48199.DATE_KEY
         and T48199.SQL_DATE = TO_DATE('2009-07-03' , 'YYYY-MM-DD')
         and concat(T48199.CALENDAR_YEAR, T48199.CALENDAR_MONTH_NUMBER) = '200907' )
group by T44042.CUSTOMER_LAST_NAME,
         T48199.DATE_KEY,
         T48199.SQL_DATE,
         concat(T48199.CALENDAR_YEAR, T48199.CALENDAR_MONTH_NUMBER))
select distinct 0 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5,
     D1.c1 as c6
from
     SAWITH0 D1
order by c2, c5, c4, c3

Before closing this section of the real-time discussion, I want to take a minute to identify the strengths and weaknesses of this approach. As far as strengths go, we have several items that register with this solution. First off… this is a low-latency solution. When using the Oracle Next-Generation Reference Architecture, we have the latency of streaming, or “GoldenGating,” the content from the source system to the DW database. With clients we’ve had in the past, this can run anywhere from a few seconds to several minutes, depending on the solution implemented. Additionally, there is no complex logical or physical data modeling and supporting ETL to deliver this solution, as there is with the EDW with a Real-Time Component, which we will explore in the next posting.

As far as weaknesses go, there will be a fair amount of complex RPD semantic-layer modeling. Obviously, the degree of difficulty depends on a number of factors: number of source systems integrated, number of subject areas, complexity of reports delivered, etc. Also, increased complexity of RPD modeling may introduce performance degradation as OLTP schemas have to be transformed “on the fly” to star schemas by the BI Server. But keep in mind… we are typically only doing this for at most a day’s worth of data, so with proper database tuning, this content can usually perform quite well.

Next up: EDW with a Real-Time Component

The subject matter will be “Agile Data Warehousing on Oracle Exadata and OBIEE 11g”. This is a subject I’ve been devoting a lot of time to lately, both in project delivery and in speaking. With two full data warehouse delivery projects on Exadata under my belt, and several other partial projects, the Database Machine is absolutely a paradigm shift. But the real tipping point comes when these DW capabilities are combined with a powerful metadata layer, such as exists in OBIEE 11g. Over the last few years, I’ve adjusted and re-adjusted long-standing beliefs about how data warehouses should be built and delivered. While I’ll talk about what makes Exadata and OBIEE different, my main focus is demonstrating how to use the features to deliver BI in accordance with standard Agile concepts. I also have a series of blog posts planned to dive into this subject in detail.

If you’re interested in homework, I’ll be discussing the Oracle Next-Generation Data Warehouse Reference Architecture, Exadata Smart-Scan, and the OBIEE Semantic Model. Additionally, I’ll spend some time on the Agile Manifesto, the generic agile development movement, and what effect they have on DW delivery methodologies.