My talk will be about Realtime Data Warehousing – it is an overview of reasons, techniques and pitfalls, but I do cover a lot of material in that hour. Of course, Change Data Capture (CDC) will be a major part of the talk; Oracle has so many options here including their recently acquired GoldenGate product set. As always, the slides will be here on the Rittman Mead site soon after I speak.

My colleague, Stewart Bryson has also had some recent thoughts about change data capture over on the TDWI group at LinkedIn.com (group membership needed); he was quite preceptive (and on the money, in my opinion) with his comment “I would hesitate to let technical limitations dictate user requirements. In today’s BI/DW market, there are very few technical limitations that cannot be solved one way or another.”

One of points I will make in my Realtime DW talk, and perhaps I need a few more slides to do it justice, is the need to profile the change you capture on the source system. Often there is a lot of “noise” that looks like change but you have no real interest in it at the data warehouse. Not all systems are “well behaved”; I have seen systems that always update a record even if nothing has changed and even systems that update each column as separate statement with its own commit.  Of course, even systems that don’t have those vices can still have columns that have no DW significance being updated and see those changes being filtered out on the data warehouse after we had already done a lot of work (processing, network bandwidth and the like) to get the data there.

The more I do this kind of work I feel there is a need to switch CDC on on the live source for a while and see the typical patten of change that occurs in a day, week, period whatever and then make decisions on how to handle this defensively downstream. Do we need to exclude certain columns that are just “noise”? What will be the impact of multiple, rapidly-occurring commits on how we handle SCD-2 dimensions? Of course we can predict what will see and come up with a proposed solution but the real source often has a few surprises up its sleeve – once a customer gave me a sequence of order statuses that an order passed through in its life-cycle except that on the actual source system the order sequence was not the same as their documentation and that would impact our reporting.

As a principle, I hate the idea of having to update a fact. A fact has happened, it is not going to change. I suppose to be more accurate a “change” is a new event, a new fact, a new fact occurring at a different time. So how to model this? – well instinctively I would go for a table that is only inserted (preferably appended to – think set based!) containing whatever dimensions are needed (don’t forget the ‘when’ dimension) PLUS an ‘EVENT’ dimension (one row per expected status) and the measures (how many, how much etc). To report on this we need to rotate the table so that the events that belong to single item appear in the same row. Before Oracle 11g we would need to construct some SQL using a mix of case statements and analytic functions to rotate the data. But now we have a potentially better way the, Oracle 11g Pivot operator.

Here we define a set of dimensions for the row (similar to the dimensions in a Group BY clause), the aggregation operators for the pivoted measures – which of course could include MIN() or MAX() for the cases when want to pivot DATE types. We also need to define the dimensions we want to pivot by, and here we can actually choose multiple dimensions; this again is somewhat similar to the GROUP BY of traditional SQL. Remember though when we pivot we sometimes only expect to ‘aggregate’ a single row – if we want to pivot order date and dispatch date then we probably have just one of each!.

So how does it look? Well the Oracle 11g documentation describes the syntax and gives some examples – here I am showing a slightly more complicated case where we are pivoting by two dimensions, each with a known set of code values. This example is based on two of the examples in the Oracle 11g Data Warehousing Guide

	SELECT * FROM	(
		SELECT product, channel, quarter, quantity_sold FROM sales_view
		) PIVOT (SUM(quantity_sold) as SUMQ, SUM(amount_sold) as SUMS
			FOR (channel, quarter) IN
			((5, '02') AS CATALOG_Q2,
		 	(4, '01') AS INTERNET_Q1,
		 	(4, '04') AS INTERNET_Q4,
		 	(2, '02') AS PARTNERS_Q2,
		 	(9, '03') AS TELE_Q3
			) );

The query returns a column for the product and for each of the specified pairs of channel and quarter a column for each measure. So we get columns for:

PRODUCT, CATALOG_Q2_SUMQ, CATALOG_Q2_SUMS, INTERNET_Q1_SUMQ, INTERNET_Q1_SUMS, INTERNET_Q4_SUMQ, INTERNET_Q4_SUMS, PARTNERS_Q2_SUMQ, INTERNET_Q4_SUMS, TELE_Q3_SUMQ, and TELE_Q3_SUMS

Note how the the measure name is concatenated to the alias in the in list.
As you can see we don’t need to specify each combination of channel and quarter – just the ones we want in our pivoted view. We also don’t use a GROUP BY clause – we specify the columns we want to see (both the dimensions and the aggregations) and Oracle implicitly groups by all of the columns not in aggregated functions.

In my example I used SELECT * to wrap the inline pivot, in practice I would explicitly select the columns and perhaps alias them to more meaningful names than the concatenated ones generated by Oracle. I would also expose the pivot as database view and thus access it from OWB or OBIEE where it appears to be just another table or view.

Another point to note is that you might see null values in the pivoted measures and these can be due to one of two reasons: the value stored for that combination of dimensions (in our case channel and quarter) is actually NULL, or that the combination does not exist. If you need to (and you may not need to) you can differentiate by using a COUNT measure; if the count is zero then the combination does not exist in the source table, if one or more then the source has NULLs stored for the combination.

We used a similar pivot view to the one above to monitor stock movements in a warehouse – in this case we needed to track individual batches of product from multiple potential suppliers, so in addition to the product dimension we had dimensional columns for batch id (a degenerate dimension) and supplier. The view was then exposed to OWB to allow us to include the aggregated result set in our ETL process – we needed to calculate some additional measures based on the difference between two of the pivoted columns. The Pivot operator greatly simplified our ETL for this fact – we could easily write an ETL process with a straight aggregation then pivot the results with CASE statements or DECODES or whatever – but that would have been less clear and also increased the number of “moving parts”.

We have had no problems with performance with our data set – 80 million rows pivoted on Exadata to just a few seconds. But it was not too slow on our non-exadata development machine either.

When I engage with clients that have a data warehouse operating in NOARCHIVELOG mode, their usual reasoning for this decision is a perceived performance gain. This makes sense on the surface… because NOARCHIVELOG prevents the generation of all that unwanted and unneeded REDO, right?

Not exactly. There is misconception about what NOARCHIVELOG actually means, and hopefully, I can clear that up with a demonstration. I have a database in NOARCHIVELOG, and I’ll test to see whether my statements generate REDO:

SQL> SELECT log_mode
  2    FROM v$database;

LOG_MODE
------------
NOARCHIVELOG

1 row selected.

Elapsed: 00:00:00.00
SQL>
SQL> CREATE TABLE target.sales
  2      AS SELECT *
  3           FROM sh.sales
  4          WHERE 1=0;

Table created.

Elapsed: 00:00:00.59
SQL>
SQL> SET autotrace on statistics
SQL>
SQL> INSERT INTO target.sales
  2         SELECT *
  3           FROM sh.sales;

918843 rows created.

Elapsed: 00:00:02.92

Statistics
----------------------------------------------------------
       1897  recursive calls
      40779  db block gets
       7062  consistent gets
       1585  physical reads
   38832896  redo size
        742  bytes sent via SQL*Net to client
        958  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          2  sorts (memory)
          0  sorts (disk)
     918843  rows processed

SQL>
SQL> ROLLBACK;

Rollback complete.

Elapsed: 00:00:01.32
SQL>
SQL> INSERT /*+ APPEND */ INTO target.sales
  2         SELECT *
  3           FROM sh.sales;

918843 rows created.

Elapsed: 00:00:06.00

Statistics
----------------------------------------------------------
       1042  recursive calls
       5581  db block gets
       2874  consistent gets
       1052  physical reads
      92108  redo size
        732  bytes sent via SQL*Net to client
        975  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          5  sorts (memory)
          0  sorts (disk)
     918843  rows processed

SQL>
SQL> ROLLBACK;

Rollback complete.

Elapsed: 00:00:00.00
SQL>  

The regular insert statement generated 38M of REDO in a NOARCHIVELOG database. Interesting. And the INSERT /*+ APPEND */ statement generated only 92K. Though it would appear that neither of these statements actually executed in NOLOGGING mode, the truth is that the APPEND statement did. All statements generate a little bit of REDO, because updates to the data dictionary are always logged.

So why do regular inserts generate REDO on a NOARCHIVELOG database? There is a myth in the Oracle world that NOARCHIVELOG means that no REDO is generated, but that is not the case. Choosing NOARCHIVELOG mode simply means that we are foregoing the option to use media recovery (restoring datafiles, rolling forward). Think about it: REDO is not simply for media recovery, it’s also for crash recovery. If all REDO generation was suspended, Oracle wouldn’t be able to open after a simple server crash. In NOARCHIVELOG mode, there are situations where we can suspend most of the REDO generated, and one of those situations involves using the INSERT /*+ APPEND */ statement. So why would the database allow these NOLOGGING operations? Because direct-path operations write blocks directly into datafiles, bypassing the buffer cache. We wouldn’t have to rely on the online REDO logs to recover those transactions, and so Oracle allows us to minimize the REDO generated.

So if you have your database in NOARCHIVELOG mode for performance reasons, but you are using ETL tools that don’t support true direct-path writes on Oracle (a lot of the third-party tools don’t), or you are using cursor-based, row-by-row load scenarios, the same amount of REDO is generated if the database was in ARCHIVELOG mode. The only thing gained from operating in this manner is the privilege of having to shut down the database whenever a backup is needed.

Perhaps another myth that gets perpetuated is that we can’t have the best of both worlds, but in fact we can. We can minimize the amount of REDO generated, we can operate in ARCHIVELOG mode, we can backup our database in online mode, and we would be able to restore from that backup. The solution: NOLOGGING tables and indexes. I’ll put the database in ARCHIVELOG mode, and rerun the test case above with one small change: I’ll change the table to be NOLOGGING:

SQL> startup mount
ORACLE instance started.

Total System Global Area  422670336 bytes
Fixed Size                  1336960 bytes
Variable Size             343935360 bytes
Database Buffers           71303168 bytes
Redo Buffers                6094848 bytes
Database mounted.
SQL> alter database
  2  archivelog;

Database altered.

SQL> alter database
  2  open;

Database altered.

SQL> SELECT log_mode
  2    FROM v$database;

LOG_MODE
------------
ARCHIVELOG

1 row selected.

Elapsed: 00:00:00.06
SQL>
SQL> ALTER TABLE target.sales
  2        nologging;

Table altered.

Elapsed: 00:00:01.02
SQL>
SQL> SET autotrace on statistics
SQL>
SQL> INSERT INTO target.sales
  2         SELECT *
  3           FROM sh.sales;

918843 rows created.

Elapsed: 00:00:02.47

Statistics
----------------------------------------------------------
      15560  recursive calls
      33573  db block gets
      13861  consistent gets
       6260  physical reads
   38289752  redo size
        740  bytes sent via SQL*Net to client
        958  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
        154  sorts (memory)
          0  sorts (disk)
     918843  rows processed

SQL>
SQL> ROLLBACK;

Rollback complete.

Elapsed: 00:00:01.45
SQL>
SQL> INSERT /*+ APPEND */ INTO target.sales
  2         SELECT *
  3           FROM sh.sales;

918843 rows created.

Elapsed: 00:00:03.51

Statistics
----------------------------------------------------------
          1  recursive calls
       4628  db block gets
       1718  consistent gets
         59  physical reads
       8072  redo size
        732  bytes sent via SQL*Net to client
        975  bytes received via SQL*Net from client
          4  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
     918843  rows processed

SQL>
SQL> ROLLBACK;

Rollback complete.

Elapsed: 00:00:00.03
SQL> 

We get the exact same behavior with a NOLOGGING table in ARCHIVELOG mode than we did with NOARCHIVELOG mode. But is having the database in ARCHIVELOG mode of any value when all of our ETL processes are NOLOGGING? We can perform an online backup, but would we even be able to restore from that backup if we have transactions that executed as NOLOGGING?

The answer is “yes” and “yes”. We just need one small change to our backup strategy: a well-placed incremental backup.

To increase the performance of our incremental backup, we need to create a block change tracking file. The database keeps a list of all changed blocks so that RMAN will know exactly what to backup during an incremental:

SQL> alter database enable block change tracking
  2  using file '/oracle/oradata/bidw1/change_blocks.bct';

Database altered.

Elapsed: 00:00:02.16
SQL> select * from
  2  v$block_change_tracking;

STATUS       | FILENAME                                 |      BYTES
------------ | ---------------------------------------- | ----------
ENABLED      | /oracle/oradata/bidw1/change_blocks.bct  |   11599872

1 row selected.

Elapsed: 00:00:00.01
SQL>  

We start by taking the initial incremental level 0 backup:

RMAN> backup incremental
2> level 0 database
3> plus archivelog;

Starting backup at 11-FEB-10
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=45 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=18 RECID=47 STAMP=710646180
input archived log thread=1 sequence=19 RECID=48 STAMP=710646955
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_annnn_TAG20100211T015555_5q7bhw0c_.bkp tag=TAG20100211T015555 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02
Finished backup at 11-FEB-10

Starting backup at 11-FEB-10
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental level 0 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/oracle/oradata/bidw1/system01.dbf
input datafile file number=00002 name=/oracle/oradata/bidw1/sysaux01.dbf
input datafile file number=00003 name=/oracle/oradata/bidw1/undotbs01.dbf
input datafile file number=00004 name=/oracle/oradata/bidw1/users01.dbf
input datafile file number=00005 name=/oracle/oradata/bidw1/example01.dbf
input datafile file number=00007 name=/oracle/oradata/bidw1/target01.dbf
input datafile file number=00006 name=/oracle/oradata/bidw1/tdrep01.dbf
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd0_TAG20100211T015557_5q7bhz1o_.bkp tag=TAG20100211T015557 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:05:26
channel ORA_DISK_1: starting incremental level 0 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_ncsn0_TAG20100211T015557_5q7btbnf_.bkp tag=TAG20100211T015557 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:09
Finished backup at 11-FEB-10

Starting backup at 11-FEB-10
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=20 RECID=49 STAMP=710647302
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_annnn_TAG20100211T020143_5q7btr8x_.bkp tag=TAG20100211T020143 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 11-FEB-10

RMAN>

Now I’ll load the SALES table with another INSERT /*+ APPEND */ to make sure we have a NOLOGGING operation since our last backup.

SQL> insert /*+ APPEND */
  2  into target.sales
  3  select * from
  4  sh.sales;

918843 rows created.

Elapsed: 00:00:21.06

Statistics
----------------------------------------------------------
       2780  recursive calls
       6081  db block gets
       2434  consistent gets
       5442  physical reads
     136036  redo size
       1536  bytes sent via SQL*Net to client
       1155  bytes received via SQL*Net from client
          6  SQL*Net roundtrips to/from client
         10  sorts (memory)
          0  sorts (disk)
     918843  rows processed

SQL> commit;

Commit complete.

Elapsed: 00:00:00.07
SQL> 

This is the step in our process that requires a slight change to our backup and recovery strategy: we should get an incremental level 1 backup as soon as the load is complete. This will physically backup all blocks that have been affected by the load, and we wouldn’t need to logically apply the REDO logs that are missing the NOLOGGING operations. Since we have changed block tracking, this step will be extremely fast, and I recommend that the ETL process flow or main driving script execute the backup as the very last step in the batch load.

RMAN> backup incremental
2> level 1 database
3> plus archivelog;

Starting backup at 11-FEB-10
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=30 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=18 RECID=47 STAMP=710646180
input archived log thread=1 sequence=19 RECID=48 STAMP=710646955
input archived log thread=1 sequence=20 RECID=49 STAMP=710647302
input archived log thread=1 sequence=21 RECID=50 STAMP=710648694
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_annnn_TAG20100211T022455_5q7d67t6_.bkp tag=TAG20100211T022455 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 11-FEB-10

Starting backup at 11-FEB-10
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental level 1 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/oracle/oradata/bidw1/system01.dbf
input datafile file number=00002 name=/oracle/oradata/bidw1/sysaux01.dbf
input datafile file number=00003 name=/oracle/oradata/bidw1/undotbs01.dbf
input datafile file number=00004 name=/oracle/oradata/bidw1/users01.dbf
input datafile file number=00005 name=/oracle/oradata/bidw1/example01.dbf
input datafile file number=00007 name=/oracle/oradata/bidw1/target01.dbf
input datafile file number=00006 name=/oracle/oradata/bidw1/tdrep01.dbf
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd1_TAG20100211T022457_5q7d6cgv_.bkp tag=TAG20100211T022457 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting incremental level 1 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_ncsn1_TAG20100211T022457_5q7d6t16_.bkp tag=TAG20100211T022457 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 11-FEB-10

Starting backup at 11-FEB-10
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=22 RECID=51 STAMP=710648715
channel ORA_DISK_1: starting piece 1 at 11-FEB-10
channel ORA_DISK_1: finished piece 1 at 11-FEB-10
piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_annnn_TAG20100211T022515_5q7d6vg7_.bkp tag=TAG20100211T022515 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 11-FEB-10

RMAN>

Now, let’s see if we can restore:

RMAN> startup mount

Oracle instance started
database mounted

Total System Global Area     422670336 bytes

Fixed Size                     1336960 bytes
Variable Size                356518272 bytes
Database Buffers              58720256 bytes
Redo Buffers                   6094848 bytes

RMAN> restore database;

Starting restore at 11-FEB-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /oracle/oradata/bidw1/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /oracle/oradata/bidw1/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /oracle/oradata/bidw1/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /oracle/oradata/bidw1/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /oracle/oradata/bidw1/example01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /oracle/oradata/bidw1/tdrep01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /oracle/oradata/bidw1/target01.dbf
channel ORA_DISK_1: reading from backup piece /oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd0_TAG20100211T015557_5q7bhz1o_.bkp
channel ORA_DISK_1: piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd0_TAG20100211T015557_5q7bhz1o_.bkp tag=TAG20100211T015557
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:06:37
Finished restore at 11-FEB-10

RMAN> recover database;

Starting recover at 11-FEB-10
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /oracle/oradata/bidw1/system01.dbf
destination for restore of datafile 00002: /oracle/oradata/bidw1/sysaux01.dbf
destination for restore of datafile 00003: /oracle/oradata/bidw1/undotbs01.dbf
destination for restore of datafile 00004: /oracle/oradata/bidw1/users01.dbf
destination for restore of datafile 00005: /oracle/oradata/bidw1/example01.dbf
destination for restore of datafile 00006: /oracle/oradata/bidw1/tdrep01.dbf
destination for restore of datafile 00007: /oracle/oradata/bidw1/target01.dbf
channel ORA_DISK_1: reading from backup piece /oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd1_TAG20100211T022457_5q7d6cgv_.bkp
channel ORA_DISK_1: piece handle=/oracle/flash_recovery_area/BIDW1/backupset/2010_02_11/o1_mf_nnnd1_TAG20100211T022457_5q7d6cgv_.bkp tag=TAG20100211T022457
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15

starting media recovery
media recovery complete, elapsed time: 00:00:03

Finished recover at 11-FEB-10

RMAN> alter database open;

database opened

RMAN> 

So that’s it for the Three “R”’s. I had a lot of fun revisiting the “operations” side of the house, and logging in as SYSDBA again. It’s amazing how it all just came back to me… I didn’t have to look at the manuals at all. Okay… maybe once.

There are numerous approaches to crafting sustainable ETL; as a matter of fact, Peter Scott wrote a post by that very title. Jon Mead contributed this one about resuming ETL processes. Just a note of clarification: what Jon describes as “resuming” is what I am describing here as “code-controlled restarting”: building smart ETL process flows and instrumented mappings so that a record is kept of what’s already been run. This is a required component, and I recommend coding best practices such as these into all ETL processes. But the restartability feature I’m focusing on is “data management restartability”, which deals with controlling data sets after failures. So the feature that I’m plugging in for the Restartability phase is Oracle’s Flashback functionality.

Flashback provides the capability to revert the entire database, or smaller portions of it, to a particular point in time. For Oracle, a “point in time” is always referenced by the System Change Number(SCN), an internal clock for the Oracle Database. It auto-increments every time a transaction commits, but other sources, such as the SMON process, can increment the SCN as well. The current SCN can be viewed in many of the data dictionary tables, as well as using the DBMS_FLASHBACK server package.

SQL> select current_scn,
  2  dbms_flashback.get_system_change_Number
  3  from v$database;

CURRENT_SCN | GET_SYSTEM_CHANGE_NUMBER
----------- | ------------------------
    2536238 |                  2536238

1 row selected.

SQL> 

We can convert from SCN’s to timestamps and back again, but this conversion is not exact. The Oracle documentation states that the functions are precise to about 3 seconds, which is evident from this example:

SQL> select SCN_TO_TIMESTAMP(2536238) scn
  2  from dual;

SCN
------------------------------
02/09/2010 12.47.26000000000

1 row selected.

SQL> select TIMESTAMP_TO_SCN('02/09/2010 12.47.26000000000') ts
  2  from dual;

        TS
----------
   2536237

1 row selected.

SQL>

Even though we access both Flashback Database and Flashback Table with the same general syntax and specify SCN’s for both incarnations, the technical implementation under the hood is drastically different. Flashback Table is completely an UNDO operation, and is really not a new feature at all. Oracle has always used the UNDO space (rollback segments before that) to manage the state of tables as of a particular SCN to allow the robust multi-versioning that keeps reads and writes from blocking one another. Flashback Table is just an “opening” of the multi-version API, in a manner of speaking, so that any SCN can be viewed as long as sufficient UNDO exists.

Flashback Database, on the other hand, doesn’t use UNDO at all, instead using new instance file structures called flashback logs in conjunction with a little bit of archived redo. Flashback logs contain prior versions of changed blocks, and we use the version of the block just prior to the SCN of interest and put them back in the datafiles, followed by redo log recovery to get the database to the exact point of the SCN.

So what part does Oracle’s Flashback technology play with data warehouse fault tolerance, specifically in the area of Restartability? Some aspect of the load will likely need to be “undone” before we can continue, and this is where Flashback fits in neatly, as demonstrated in the following examples.

I created copies of the CUSTOMERS, PRODUCTS and SALES tables from the SH schema and inserted the rows from there as well. Before I start, I need to enable row movement on the new SALES table. This would need to be implemented for all tables in the data warehouse that are a consideration for Flashback Table:

SQL> alter table target.sales enable row movement;

Table altered.

SQL>
SQL> SELECT count(*) FROM target.products;

  COUNT(*)
----------
       72

1 row selected.

SQL> SELECT count(*) FROM target.customers;

  COUNT(*)
----------
     55500

1 row selected.

SQL> SELECT count(*) FROM target.sales;

  COUNT(*)
----------
    918843

1 row selected.

SQL> 

Next I’ll get ready to execute my code. First, I’ll create what’s called a “restore point” in the database. This allows me to give an intelligent name to a particular SCN and is similar to tagging a release in Subversion. Before each new step in the process, I’ll create a restore point so that each phase has a tagged, referenceable SCN. As I’m using the concept of a unique, sequence-generated number for each batch that runs (Jon calls it an “execution ID” in his posting above), I’ll work that number into the name of my restore points.

SQL> create restore point dw_load_1001;

Restore point created.

SQL>

Next… I do the processing that moves the necessary files into place (if any), prepares and loads the ODS tables, etc. After that… I move into the load of the dimensional model itself:

SQL> create restore point load_customers_1001;

Restore point created.

SQL> exec dw_load.load_customers;
Number of records loaded: 0

PL/SQL procedure successfully completed.

SQL> create restore point load_products_1001;

Restore point created.

SQL> exec dw_load.load_products;
Number of records loaded: 72

PL/SQL procedure successfully completed.

SQL> create restore point load_sales_1001;

Restore point created.

SQL> exec dw_load.load_sales;
5 indexes and 0 local index partitions affected on table TARGET.SALES
Number of records loaded: 699999
Rebuild processes for unusable indexes on 28 partitions of table TARGET.SALES executed
No matching unusable global indexes found

PL/SQL procedure successfully completed.

SQL> 

So the data warehouse load ran without error, so I can assume that it was successful, right? In looking back over the log, I see that no rows were actually loaded into the CUSTOMERS table. After researching the issue, I discover that the Change Data Capture process on the source system is experiencing errors, and there were no rows published to the CUSTOMERS change set. Since the load didn’t technically fail, the process continued to the load of the fact table, and it’s very likely that many of the rows in the fact table have the wrong surrogate key from the CUSTOMERS table.

In describing my triage approach from earlier postings, the “aftermath” is exactly what I’m trying to avoid. In my experience, ETL load failures and the subsequent aftermath (investigations, data corrections, and reloads) cause more downtime than any other hardware or software related issues. But with the approach I’ve put into place, this aftermath shouldn’t concern me, because now I can simply “undo” it (pun intended).

SQL> flashback table target.sales to restore point load_sales_1001;

Flashback complete.

SQL> select count(*) from target.sales;

  COUNT(*)
----------
    918843

1 row selected.

SQL> create restore point new_load_customers_1001;

Restore point created.

SQL> exec dw_load.load_customers;
Number of records loaded: 99

PL/SQL procedure successfully completed.

SQL> create restore point new_load_sales_1001;

Restore point created.

SQL> exec dw_load.load_sales;
5 indexes and 0 local index partitions affected on table TARGET.SALES
Number of records loaded: 699999
Rebuild processes for unusable indexes on 28 partitions of table TARGET.SALES executed
No matching unusable global indexes found

PL/SQL procedure successfully completed.

SQL> 

Instead of flashing back, I could try to sort out the issue. For instance, if I’m attaching the unique execution ID to every row in the fact table, either directly, or through an AUDIT dimension table, then I could probably identify the rows for this run. But why would I do this when the Flashback functionality is already available to me?

My test case above was a simple one; I was able to proceed just by flashing back a single table before restarting the process. However, in a large enterprise data warehouse, the effort involved in a typical aftermath could be staggering depending on how many fact tables are involved, how many dimension tables track history with SCD Type 2 changes, etc. Combine that with the possible need to flashback ODS tables, history tables, persistent staging tables, etc. I’ve seen numerous situations where the exact ramifications are tough to quantify: we know what broke, but we have no idea what needs to be fixed. Perhaps there was a hardware failure in the middle of an ETL load, and it’s hard to identify just exactly which tables were loaded and which ones weren’t. In this case, what I really need is the ability to do a complete “do-over”: put everything back the way it was prior to the beginning of the load, and just restart everything.

Enter Flashback Database. So I’ll demonstrate what’s required to enable this feature, and then I’ll replay the test case above and solve it from this angle.

I first need to put my database in Archive Log Mode, as archived redo is a required component of the feature:

SQL> startup mount
ORACLE instance started.

Total System Global Area  422670336 bytes
Fixed Size                  1336960 bytes
Variable Size             335546752 bytes
Database Buffers           79691776 bytes
Redo Buffers                6094848 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     25
Next log sequence to archive   27
Current log sequence           27
SQL> 

Next, I need to configure the Flash Recovery Area, which is a file system on the server where the database will create the flashback logs:

SQL> alter system set db_recovery_file_dest_size=3G;

System altered.

SQL> Alter system set db_recovery_file_dest='/oracle/flash_recovery_area';

System altered.

SQL>

Finally, I need to set the flashback_retention_target parameter, which instructs the Flash Recovery Area on our needs for retention. This parameter is actually in minutes… thanks for the consistency Oracle. After that, I just enable flashback and open the database:

SQL> alter system set db_flashback_retention_target=2880;

System altered.

SQL> alter database flashback on;

Database altered.

SQL> alter database open;

Database altered.

SQL> 

So, Flashback Database should be ready to use. I’ll take a quick look and see if the database thinks it’s ready:

SQL> select oldest_flashback_scn,
  2         oldest_flashback_time,
  3         startup_time
  4    from v$flashback_database_log
  5         cross join v$instance;

OLDEST_FLASHBACK_SCN | OLDEST_FLASHBACK_TIME  | STARTUP_TIME
-------------------- | ---------------------- | ----------------------
             2912097 | 02/10/2010 12:10:30 AM | 02/10/2010 12:09:08 AM

1 row selected.

Elapsed: 00:00:00.15
SQL>  

Now I’ll flashback the entire database to the very first restore point I created: dw_load_1001:

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  422670336 bytes
Fixed Size                  1336960 bytes
Variable Size             343935360 bytes
Database Buffers           71303168 bytes
Redo Buffers                6094848 bytes
Database mounted.
SQL> flashback database to restore point dw_load_1001;

Flashback complete.

SQL> alter database open resetlogs;

Database altered.

SQL> SELECT count(*) FROM target.products;

  COUNT(*)
----------
       72

1 row selected.

SQL> SELECT count(*) FROM target.customers;

  COUNT(*)
----------
     55500

1 row selected.

SQL> SELECT count(*) FROM target.sales;

  COUNT(*)
----------
    918843

1 row selected.

SQL>

So the immediate downside of this approach is that it requires the involvement of the operations team because the database has to be in mount mode, and the data warehouse is not available during this slight outage. However, when compared with the time it might take to sort out and correct massive aftermath scenarios, this seems to be the preferable choice. Is the data warehouse really “available” if data corrections and data reloads are occurring? I would rather involve the operations team for a quick, concrete fix so the reload can complete as soon as possible.

The next “R” is Restoring, though it really involves putting the pieces in place for a scalable Backup strategy. And “Backup” doesn’t start with an “R”.

Resuming is the ability to continue effortlessly after an error. The important thing is that there should be no aftermath from the error: our process should pause gracefully until the error is corrected. The Oracle Database has offered out of the box functionality for resuming since version 9i in the form of Resumable Space Allocation. Resumable operations are supported for SELECT queries, DML and DDL, and can be enabled at either the system or the session level. To enable at the system level, the RESUMABLE_TIMEOUT database parameter should have a non-zero value.

SQL> alter system set resumable_timeout=3600;

System altered.

SQL>

To enable resumable operations at the session level, the statement follows this basic syntax, with the TIMEOUT and NAME clauses being optional:

ALTER SESSION ENABLE RESUMABLE <TIMEOUT n> <NAME string>;

The TIMEOUT value is specified in seconds, and if omitted, the default value of 7200 is used, or 2 hours. The NAME clause gives the resumable session a user-friendly name for when we are monitoring for resumable sessions (as we will see later) to see which of our processes is suspended. Enabling resumable operations for the session level requires that the RESUMABLE permission has been granted:

SQL> grant resumable to stewart;

Grant succeeded.

SQL>

Resumable operations can also be enabled with the Oracle utilities… such as SQL-Loader, Export/Import and Datapump. The command-line parameters RESUMABLE, RESUMABLE_NAME and RESUMABLE_TIMEOUT exist to mimic the functionality mentioned above.

Now for a demonstration. I’ll create a situation that is ripe for a space allocation error: I’ll put an empty copy of the SALES fact table from the SH schema in a tablespace with only 250K of space:

SQL> create tablespace target datafile '/oracle/oradata/bidw1/target01.dbf' size 250K;

Tablespace created.

SQL> create table target.sales tablespace target as select * from sh.sales where 1=0;

Table created.

SQL>

Now I’ll load some records into the table, which should cause it to suspend. To prepare my session, I need to enable resumable operations. Since I always instrument my code, I’ll register my process with the database. After that, I have an easy way to guarantee consistency when referring to processes. Now, I can use the registered name for my resumable session as well:

SQL> exec dbms_application_info.set_module('SALES fact load','insert some rows');

PL/SQL procedure successfully completed.

SQL>
SQL> DECLARE
  2     l_module VARCHAR2(48) := sys_context('USERENV','MODULE');
  3  BEGIN
  4     EXECUTE IMMEDIATE
  5     'alter session enable resumable timeout 18000 name '''||l_module||'''';
  6  END;
  7  /

PL/SQL procedure successfully completed.

SQL>

I start loading the records in hopes of a suspended session:

SQL> insert into target.sales select * from sh.sales;

So now, I open up another session, and I start another transaction against the TARGET.SALES table, just to pile on the TARGET tablespace:

SQL> exec dbms_application_info.set_module('SALES fact load2','insert more rows');

PL/SQL procedure successfully completed.

SQL>
SQL> DECLARE
  2     l_module VARCHAR2(48) := sys_context('USERENV','MODULE');
  3  BEGIN
  4     EXECUTE IMMEDIATE
  5     'alter session enable resumable timeout 18000 name '''||l_module||'''';
  6  END;
  7  /

PL/SQL procedure successfully completed.

SQL> insert into target.sales select * from sh.sales;

I’ll have a look in the DBA_RESUMABLE view (there is also a USER_RESUMABLE version) for my suspended sessions. Even though I could get all the following information with a single SQL statement, I broke it up for better visibility on the blog:

SQL> select name, start_time, suspend_time, status from dba_resumable;

NAME              | START_TIME           | SUSPEND_TIME         | STATUS
----------------- | -------------------- | -------------------- | ------------
SALES fact load2  | 02/06/10 10:33:33    | 02/06/10 10:33:33    | SUSPENDED
SALES fact load   | 02/06/10 10:29:03    | 02/06/10 10:29:03    | SUSPENDED

2 rows selected.

Elapsed: 00:00:00.07
SQL> select name, sql_text from dba_resumable;

NAME              | SQL_TEXT
----------------- | -----------------------------------------------
SALES fact load2  | insert into target.sales select * from sh.sales
SALES fact load   | insert into target.sales select * from sh.sales

2 rows selected.

SQL> select name, error_msg from dba_resumable;

NAME              | ERROR_MSG
----------------- | ------------------------------------------------------------------------
SALES fact load2  | ORA-01653: unable to extend table TARGET.SALES by 8 in tablespace TARGET
SALES fact load   | ORA-01653: unable to extend table TARGET.SALES by 8 in tablespace TARGET

2 rows selected.

SQL>

The Oracle Database also publishes server alerts concerning suspended transactions using the Server-Generated Alerts infrastructure. This infrastructure uses the AWR toolset, the server package DBMS_SERVER_ALERT for getting and setting metric threshholds, and the queue table ALERT_QUE to hold alerts that have been published from AWR. Custom processes could be written to mine ALERT_QUE for these alerts, but the easiest way to configure and view server alerts is using Oracle Enterprise Manager (OEM). On the Alerts section of the main OEM page, we can see three different alerts generated by the Oracle Database:

If we click on the “Session Suspended” link, we can see the multiple alerts generated in this category:

Another alert generated indirectly by the suspended transaction is the “Configuration” class event caused by our session “waiting” to proceed. The Oracle wait event interface can show us information about the suspend waits on the system:

SQL> SELECT event,
  2         SUM(time_waited) time_waited,
  3         SUM(total_waits) total_waits,
  4         AVG(average_wait) average_wait
  5    FROM gv$session_event
  6   WHERE lower(event) LIKE '%suspend%'
  7   GROUP BY event
  8   ORDER BY time_waited ASC
  9  /

EVENT                                          | TIME_WAITED | TOTAL_WAITS | AVERAGE_WAIT
---------------------------------------------- | ----------- | ----------- | ------------
statement suspended, wait error to be cleared  |      305373 |        1377 |       221.78

1 row selected.

SQL>

To free up the space issue, I’ll enable autoextend on the TARGET tablespace. Then, I’ll take a look and see if anything has changed:

SQL> alter database datafile '/oracle/oradata/bidw1/target01.dbf'
  2  autoextend on next 10M maxsize 1000M;

Database altered.

SQL> select status, resume_time, name from dba_resumable;

STATUS       | RESUME_TIME          | NAME
------------ | -------------------- | -----------------
NORMAL       | 02/06/10 10:56:49    | SALES fact load2
NORMAL       | 02/06/10 10:56:49    | SALES fact load

2 rows selected.

SQL>

The Resumable Space Allocation features includes the AFTER SUSPEND trigger, which allows the specification of a system-wide trigger that will fire whenever a transaction is suspended. The typical use for this functionality is alerting as suspended operations don’t write anything to the alert log.

UPDATE: I made a mistake here… suspended transactions do in fact cause entries in the alert log, and so does the RESUME process detailed below.

There are some features in the DBMS_RESUMABLE package that may make sense when writing an AFTER SUSPEND trigger:

SQL> desc dbms_resumable
PROCEDURE ABORT
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SESSIONID                      NUMBER                  IN
FUNCTION GET_SESSION_TIMEOUT RETURNS NUMBER
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SESSIONID                      NUMBER                  IN
FUNCTION GET_TIMEOUT RETURNS NUMBER
PROCEDURE SET_SESSION_TIMEOUT
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SESSIONID                      NUMBER                  IN
 TIMEOUT                        NUMBER                  IN
PROCEDURE SET_TIMEOUT
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 TIMEOUT                        NUMBER                  IN
FUNCTION SPACE_ERROR_INFO RETURNS BOOLEAN
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 ERROR_TYPE                     VARCHAR2                OUT
 OBJECT_TYPE                    VARCHAR2                OUT
 OBJECT_OWNER                   VARCHAR2                OUT
 TABLE_SPACE_NAME               VARCHAR2                OUT
 OBJECT_NAME                    VARCHAR2                OUT
 SUB_OBJECT_NAME                VARCHAR2                OUT

SQL>

This package adds functionality for writing custom processes in the AFTER SUSPEND trigger. The SPACE_ERROR_INFO function returns specifics about the table and tablespace affected by the space error. A series of checks could be coded enabling specific actions depending on which objects were affected. A suspended process can be ended prematurely with the ABORT procedure, or more time can be added using the SET_TIMEOUT procedure. I actually had one client explain how she had written an AFTER SUSPEND trigger that compiled information about the tablespace affected so that an “ALTER DATABASE… RESIZE…” command could be issued to add more space to the affected datafile. I didn’t have the heart to tell her that she had basically written a feature that already existed in the database: AUTOEXTEND.

So what are the best practices to take away from this? Quite simply… all ETL mappings and flows, as well as database maintenance processes, should use Resumable Space Allocation, preferably using the NAME clause in conjunction with DBMS_APPLICATION_INFO. Setting a RESUMABLE_TIMEOUT value at the system level can be scary, because a single suspended transaction could cause locks that reverberate all the way through the system. But is this really a concern in a BI/DW environment? Are there any processes in our batch load window or with any of our operational maintenance processes that we wouldn’t want to enable for resumable operations, no matter how many processes back up waiting for them to complete? It could spell bad news if we used any kind of synchronous replication technology to move data to the DW instance, but short of that, I can’t think of any. Please let me know if you have alternative viewpoints.

I’ve never found much reason to use the AFTER SUSPEND trigger though. Data warehouses should have production-type monitoring running already, just like other production systems. OEM is more than satisfactory for basic monitoring and alerting, and with the Server-Generated Alerts introduced in 10g, forms a complete product for Oracle environments. But regardless of which monitoring solution is used, it should be able to issue simple queries against the database and alert based on the results of those queries. A select against the DBA_RESUMABLE table provides all the information required to send out an alert, and with features such as AUTOEXTEND, I just can’t see a requirement for the ability to issue procedural code because a transaction is suspended.

UPDATE: as pointed out above, since suspended transactions do in fact show up in the alert log, this is good news for integrating Resumable Space Allocation into an existing environment. Assuming that there’s proper alert log monitoring with paging functionality already in place, implementing resumable operations can simply use that infrastructure already in place.

Keep your eyes open for the next of the “Three R’s” in BI/DW fault tolerance: Restarting.

Fault tolerance isn’t disaster recovery exactly… though there’s a lot of overlap. Instead, fault tolerance is the ability to recover from errors, and those errors can result from hardware issues, software issues, general systems issues (network latency, out-of-space errors), and human mistakes. The main point is that BI/DW environments present unique challenges, both for operations and for the development team. I’m not preposing that the divide between transactional and reporting systems is necessarily vast… we still need redundant storage systems and dependable backup strategies. I am preposing, however, that one-size-fits-all approaches to fault-tolerance is problematic, and applying standards that evolved in support of transactional systems may not provide the best protection for BI/DW environments.

The operational teams (DBAs, Unix Admins, Storage Admins, etc.) and the development teams (source system extraction, ETL) have to work closer in a BI/DW than perhaps they do in OLTP environments. Of course, OLTP developers have to write scalable code… but I think that’s within their control for the most part. ETL developers are thrashing around millions or billions of rows of data, and because of this, everything needs to be well-oiled: undo spaces need to be available, temp space needs to be plentiful, standard operational jobs such as backup and recovery or statistics gathering need to keep the batch load window in mind, etc. Whereas OLTP code is exclusively SQL… ETL code is packed full of DDL: partition-exchange loads, index and constraint maintenance, table truncates, the whole gamut.

So when working with millions or billions of rows, we need to eliminate errors as best we can. Sounds simple enough, but the truth is that errors are going to happen, and there’s nothing we can do to wipe them out completely. But we can mitigate. So we need to introduce a triage process: catching and correcting errors as early as possible so that their damage is minimal. In essence: don’t let simple errors turn into weekend-long data correction issues, where millions of rows need to be updated or deleted. Let’s work smarter, not harder, using every solution available to use, including features present in the Oracle Database, best practices in ETL development, and possible modifications to our backup and recovery strategies.

I should note that, when speaking of BI/DW environments, I still have the batch load paradigm squarely in mind. Although the line in the sand is certainly moving in one constant direction, most data warehouses are still loaded with either batch or mini-batch processing. However, being a fan of near-real-time data warehouse techniques (as my colleague Peter Scott has written from time to time… only reporting from the source system itself is truly real-time), I’ll be sure to point out how some of these techniques differ the closer we get to the actual transaction.

I currently have three postings in mind that correlate to the Three R’s of Data Warehouse Fault Tolerance. Be on the lookout for the first installment coming soon: Resuming.

If you’ve been following the wider analytic database market over the past few years (Curt Monash’s DBMS2 website is a good place to start), you’ll probably be aware of products such as Sybase IQ and Vertica Analytic Database that store data in columns as opposed to rows. Interestingly, a lot of former ex-Oracle data warehousing server tech people ended up at Vertica, including Lilian Hobbs (author of “Oracle 10g Database Data Warehousing”) , and over the past couple of years vendors such as these have achieved some success in the marketplace with their column store approach. In fact I ran an interview with Mike Stonebraker, CTO of Vertica, on this blog a few months ago (prior to the release of Exadata 2) where he set out the case for column-based storage and the “shared nothing” architecture that his product uses.

Fast forward to the weeks before Open World, and Oracle announce the Sun Database Machine and version 2 of the Exadata Storage software that comes with it. Apart from Flash Cache (primarily aimed at OLTP environments) the major innovation from my perspective, and a bit of a volte-face from Oracle, was this halfway-house approach to column-based storage which they termed “Hybrid Columnar Compression”. The idea behind this is as follows:

• The Oracle database is still primarily a row-based engine, but there is a new type of segment compression that compresses data in columns
• This is only available if you are using Exadata Storage Server – Kevin Closson says that this is due to technical (as opposed to marketing) reasons
• It comes in two forms; Warehouse Compression (add COMPRESS FOR QUERY to your table definition script) and Archive Compression (add COMPRESS FOR ARCHIVE to your table script).
• Compress for query is optimized for DSS-style queries (the same market as Vertica/Sybase IQ), whilst Compress for Archive burns up more CPU but achieves greater compression than Warehouse compression.
• It all works through a feature introduced in Exadata 2 / Oracle Database 11gR2 called “Compression Units”.

Arup Nanda has written a good article for Oracle Magazine that goes through the technology and explains how rows relate to compression units, and compression units relate to blocks. Basically, whereas blocks store data in rows, each column next to each other in the row (more or less, row chaining can affect this), so that blocks conceptually look like this:

The compression units used by Hybrid Columnar Compression typically span several data blocks and inside, organize data into sets of columns, with each compression unit holding the data for the columns for several rows.

These compression blocks offer two potential advantages to data warehousing customers; firstly they compress better because columns are more likely to contain repeating values (genders, cities, account flags etc) than rows (where individual columns are often unrelated, at least in terms of the values they contain), and because you therefore pack more data in per block, it takes less blocks, and less disk I/O, to get hold of the data you are interested in – the same idea that makes regular compression attractive to DW users. Arup’s article goes into the syntax and the concepts in more detail, if you’re interested, and also has some test cases to show the kinds of benefits you can expect (though of course you’ll need Exadata 2 to try them out). Also, beware of the locking issues that compression units can bring – as each unit contains several rows of data, a lock on one row in the compression unit will lock the other rows as well, which makes the feature not really suitable for OLTP environments.

For me though, the interesting test of this will be to see how hybrid columnar compression compares to the “pure-columnar” approach used by vendors such as Vertica. I would imagine Vertica would argue that whilst its great that these compression units store their data organized by column, their approach would be superior for DSS customers as their equivalent of blocks stores just the data for individual columns, not all the columns for a set of rows. As such, their approach may well require even less disk I/O as you won’t be pulling back all the (compressed) columns that are stored with the columns you require (as you do with Oracle’s compression units), though I’m not close enough to either of the technologies to know if this is the case.

I doubt we’ll ever really see an apples-to-apples benchmark test of both technologies side-by-side (if only because most vendors’ license terms prohibit publishing benchmarks), but for Oracle customers I guess this is by-the-by; even if columnar compression isn’t as DSS-efficient as pure column-based storage, it’s still a huge boost to queries and it keeps it all in the familiar, manageable Oracle environment, a benefit not to be dismissed if you have thousands of databases to manage and you’re aiming for vendor and hardware consolidation. For customers already sold on Exadata and SmartScan and for whom this is if anything a bit of a bonus, it’ll be interesting to try this out on your own data and see how much benefit it brings.

First off, as you typically use it with RAC (though it does work with single nodes, thanks Greg for the clarification) and requires 11gR2, it’s not the easiest thing to set up in a test environment, so I’m going off white papers and presentations for the time being. Conceptually, it’s a different approach to vendors like QlikView and Microsoft (with PowerPivot) who are using the local memory on users’ PC, with Oracle instead using the aggregated memory of nodes in a RAC cluster. As such it’s something an administrator would set up rather than users, though once its set up it should just take effect in the background, making queries run faster.

As the name suggests, In-Memory Parallel Query is based of the parallel execution feature in the database that divides up the task of scanning a large table into several processes that each scan a part of the table. It’s typically used in conjunction with partitioning and servers with lots of CPUs, and prior to 11gR2, these parallel server processes (co-ordinated by the query co-ordinator, linked to your login session) would in most cases read the required data (broken into “granules”) direct from disk bypassing the buffer cache using direct path I/O, so as not to age everything out with what would presumably be a large amount of incoming data. Only objects smaller than 2% of the size of the buffer cache for an individual database instance would get read into the buffer cache, as illustrated in this diagram from Maria Colgan & Thierry Cruanes’s Open World Presentation, “Extreme Performance with Oracle Database 11g and In-Memory Parallel Execution”, which shows traditional PQ working with a small partitioned table and a four-node RAC cluster. Note how individual partitions (objects) are going into each node’s buffer cache.

In addition, if (as above) the database concerned was in a RAC cluster, then over time all of the partitions would get copied across all of the node buffer caches, via cache fusion, as queries came in for particular nodes and which required data in other nodes’ buffer cache, as this diagram shows:

All-in-all, because of the 2% limit (or more precisely, the value of the _PARALLEL_MIN_TABLE_THRESHOLD parameter) most parallel query operations ignored the buffer cache (and therefore the available RAM on the database servers) entirely and did all their work via direct reads from disk. Given the increases in available memory though, there was clearly an opportunity to change things around and start taking advantage of it. This is where “In-Memory Parallel Query” comes in.

There’s a new parameter in 11gR2 called PARALLEL_DEGREE_POLICY, which enables a bunch of new parallel query features introduced with this release. One thing it does is allows Oracle to decide whether, and how much degree of parallelism to use on a query, removing the need for you to set this manually per table, per query or whatever, with Oracle instead deciding based on the circumstances whether to use parallel query. Setting this parameter to AUTO also enables Parallel Statement Queuing, which gives Oracle the option to put a query on hold rather than have it either overwhelm the system, or more likely have its DOP scaled back so as not to do this. What this parameter also does when set to AUTO is enabled In-Memory Parallel Query.

With this feature enabled, Oracle will decide on a query-by-query basis whether to try and load the objects accessed by a statement into the aggregated buffer cache of your database server(s). There’s a complex set of rules that Oracle uses to determine whether to cache these set of tables;

• the size of the objects (too small, it’s not worth it; too big, it won’t fit, but if they are big enough and fit within memory, they’ll be considered)
• how often the objects are updated
• and so on

If the objects meet the criteria, and you are using RAC, then they will be “fragmented” and distributed amongst the nodes’ buffer caches. Importantly, Oracle will know which node has got which fragment (a process called “affiinitization”), and once a fragment has been mapped all subsequent accesses of this data will happen on its particular node. If a particular node, executing its particular PX server process, actually needs data cached in another node’s buffer cache, the PX Server process on the other node will send back just the results required, not the object itself, with no need for objects to be copied from node to node via cache fusion. If the data required isn’t anywhere in the cache, it’ll be retrieved via direct path I/O so as not to inadvertently clear everything else out of the cache.

So that’s the theory of how it works. As I said, it needs 11gR2 and typically you’d use it as part of a RAC setup in order to maximize the amount of memory available and cache a meaningful amount of database objects. Given that it’s not RAC dependent though I guess you could probably try it out and demonstrate it on a laptop (my 8GB Macbook may well be up to it, there’s a challenge for me), but anyway I’ll be interesting to try it out once I come across on of our customers lucky enough to have all the prerequisites in place. In the meantime, the Oracle white paper “Parallel Execution Fundamentals in Oracle Database 11gR2″ by Hermann Baer and Maria Colgan servers as a good introduction to all of the new PX features in 11gR2, if you want to read about them in more detail.

[Update 19th January : Updated article to remove comments about the feature being dependent on RAC. Thanks, Greg]

OBIEE, at the BI Server level. lets you define query limits that either warn or stop users from exceeding certain elapsed query times or number of rows returned. Assuming you define a “standard” group for most OBIEE users, you might want to stop them from displaying reports (requests) that return more than 50,000 rows, whilst you might want to warn them if their query takes over five minutes to run.

(note that this is different to Discoverer, which used to maintain statistics and warn you if the query was predicted to take over a certain time, rather than warn you if it took over a certain time).

You can also restrict the times of day that users can run queries, which you might do to stop users running queries before 9am, assuming managers come in early and want to get their figures quickly.

Whilst these are quite handy, you might want to exercise more control over how resources are allocated to these groups, and if you’re using an Oracle database as the back-end database, a feature you might therefore want to use is the Oracle Database Resource Manager. The Oracle Database Resource Manager (or DBRM for short) allows you to define consumer groups and resource plans which can then be used to:

• Guarantee certain sessions a minimum amount of processing resource regardless of the total load on the system
• Allocate percentages of CPU time to different users and applications
• Restrict or limit the degree of parallelism for queries
• Restrict the total number of active sessions for a particular group
• Manage runaway sessions, by either stopping them running or switching them to a lower-priority group
• Prevent the execution of queries predicted to take over a certain amount of time
• Automatically allocate sessions to particular resource plans depending on attributes of the session
• Limit the amount of time that a session can be idle, including if it is blocking another session from starting

In an OBIEE context, the way that you would set this up depends on whether users connect using their own Oracle login, or through a shared login. If they use their own Oracle database login, this will be passed through to the Oracle database holding the source data and the resource manager will apply the relevant resource plan, based on which consumer group they belong to.

More likely though is the situation where users access the database through a shared login which works via a connection pool. In this situation, it’s the shared login that is sent through to the Oracle database which will again allocate it to a resource group and apply the resource plan accordingly. DBRM therefore, from a database perspective, has the following three major elements:

1. Resource Consumer Groups, which are groups of sessions that share the same characteristics and that have resource plan allocated to them
2. Resource Plans, containers for sets of Resource Plan Directives, and
3. Resource Plan Directives, which are instructions to Oracle on how to allocate database resources

DBRM resource plan directives are therefore allocated to resource consumer groups, not individual users, with all of this being managed by an Oracle PL/SQL package called DBMS_RESOURCE_MANAGER. So how does this work in practice?

Well, Tim Hall’s website (http://www.oracle-base.com) is usually the place I go to for concise definitions of new Oracle features, and he’s written a series of articles about DBRM since its inception back in Oracle 8i:

• Resource Manager in Oracle Database 8i (a good initial introduction to the feature)
• Resource Manager Enhancements in Oracle 9i (added automatic resource group switching, an extended cover to UNDO_POOL)
• Resource Manager Enhancements in Oracle Database 10g (more fine-grained control over long-running sessions)
• Resource Manager Enhancements in Oracle Database 11g (pre-defined resource groups plus minor enhancements)

There’s also a more general white paper from Oracle entitled “Best Practices for a Data Warehouse on Oracle Database 11g” that mentions Resource Manager in the wider context of an Oracle Data Warehouse, and of course the Oracle Documentation. For the purposes of working through the feature though, it’s probably worth thinking about a scenario where this might come in useful.

In this scenario, we have three main groups of users:

1. Standard DW users, who we will give a guaranteed 30% of CPU time to, who won’t be able to run queries in parallel, and who won’t be allowed to run queries that are predicted to take more than 5 minutes to run
2. Advanced DW users, who will be given a guaranteed 50% of CPU, who can use parallel query up to a DOP of 4, but will be bounced down to the Standard DW user group if their queries actually take more than 1 minute
3. Executives, who will be guaranteed 20% of CPU time and again, won’t have restrictions on what they do

These will become the main “Resource Consumer Groups” as far as DBRM is concerned. Note that for the above groups, the CPU restrictions will only take place when CPU usage reaches 100% (allowing us to ration a scarce resource only when necessary), and CPU resources from groups that aren’t using them will be allocated to those that need them if we’re running under 100% usage. So it’s not too restrictive an approach and one that will only really kick-in when resources are scarce. Note also that the 11g release of DBRM also allows disk I/O to be taken into account, which would allow us to include the amount of disk activity in resource plans when disk I/O becomes the bottleneck (or we could just go and buy an Exadata machine, I guess…)

This will all be set up in a resource plan called “Daytime”, that will only apply during normal working hours (out of hours, users can do what they want).

Using Oracle 11.1.0.7 on Linux x86, assuming we’ll go with the shared logon approach, the first step is to create the shared logons, one for each of the groups.

CREATE USER standard_dw_user IDENTIFIED BY password
DEFAULT TABLESPACE users
TEMPORARY TABLESPACE temp;

GRANT CONNECT TO standard_dw_user;

CREATE USER advanced_dw_user IDENTIFIED BY password
DEFAULT TABLESPACE users
TEMPORARY TABLESPACE temp;

GRANT CONNECT TO advanced_dw_user;

CREATE USER executives IDENTIFIED BY password
DEFAULT TABLESPACE users
TEMPORARY TABLESPACE temp;

GRANT CONNECT TO executives;

Then I grant “select any table” to the three users (don’t do this at home, kids) so that they can access the SH schema:

GRANT SELECT ANY TABLE TO standard_dw_user;

GRANT SELECT ANY TABLE TO advanced_dw_user;

GRANT SELECT ANY TABLE TO executive;

Now in 11g there’s an option to create a simple resource plan that just allocates consumer groups a share of CPU resources. This is easier to set up than the full resource plan I’ll create later on, and to do this you just issue the following PL/SQL call.

BEGIN
  DBMS_RESOURCE_MANAGER.CREATE_SIMPLE_PLAN(SIMPLE_PLAN => 'DAYTIME',
   CONSUMER_GROUP1 => 'STANDARD_DW_GROUP', GROUP1_PERCENT => 30,
   CONSUMER_GROUP2 => 'ADVANCED_DW_GROUP', GROUP2_PERCENT => 50,
   CONSUMER_GROUP3 => 'EXECUTIVE_DW_GROUP', GROUP3_PERCENT => 20);
END;
/

This has the effect of creating the three consumer groups, and setting up the three directives in the new DAYTIME resource plan.

Next we need to firstly, grant permission for each user to use its respective consumer group, and then set these groups as the users’ default.

BEGIN
  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'standard_dw_user',
    CONSUMER_GROUP => 'standard_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'advanced_dw_user',
    CONSUMER_GROUP => 'advanced_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'executive',
    CONSUMER_GROUP => 'executive_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('standard_dw_user', 'standard_dw_group');

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('advanced_dw_user', 'advanced_dw_group');

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('executive', 'executive_dw_group');

END;
/

Finally we now need to turn on the resource plan.

ALTER SYSTEM SET RESOURCE_MANAGER_PLAN = daytime;

Now I can start sessions as these users, and then run the following SELECT statement as a DBA user.

SELECT username, resource_consumer_group
FROM   v$session
WHERE  username in ('EXECUTIVE','STANDARD_DW_USER');

USERNAME                       RESOURCE_CONSUMER_GROUP
------------------------------ --------------------------------
STANDARD_DW_USER               STANDARD_DW_GROUP
EXECUTIVE                      EXECUTIVE_DW_GROUP

which shows that the users have been allocated to the correct group. Now, when CPU usage reaches 100%, resources will be allocated according to this plan, splitting CPU use according to the percentages in the simple resource plan, and allocating spare resources to another plan when a particular group isn’t using up all its allocation.

As I mentioned earlier, you can get cleverer and put in directives to restrict access to parallel query, shift down to a more constrained group if query times go past a certain level, restrict access to I/O and so on if you wish. This is done through what’s called a “complex resource plan” and it’s a bit more complicated to set up.

The first thing you need to do when working with a complex resource plan is to create a “pending area”. According to the manuals, a pending area is “a staging area where you can create a new resource plan, update an existing plan, or delete a plan without affecting currently running applications. When you create a pending area, the database initializes it and then copies existing plans into the pending area so that they can be updated”. We’ll also take the opportunity to delete the previous resource plan, after we set up the pending area.

conn / as sysdba

SET ECHO ON

BEGIN
  DBMS_RESOURCE_MANAGER.CLEAR_PENDING_AREA();
  DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA();
END;
/

BEGIN
     DBMS_RESOURCE_MANAGER.DELETE_PLAN_CASCADE(PLAN=>'DAYTIME');
END;
/

We’ll now set up a new plan, and the three different resource groups;

BEGIN
  DBMS_RESOURCE_MANAGER.CLEAR_PENDING_AREA();

  DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA();

  DBMS_RESOURCE_MANAGER.CREATE_PLAN(
   PLAN    => 'DAYTIME_PLAN',
   COMMENT => 'Complex Daytime Plan');

  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP (
   CONSUMER_GROUP => 'STANDARD_DW_GROUP',
   COMMENT        => 'Standard DW Users');

  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP (
   CONSUMER_GROUP => 'ADVANCED_DW_GROUP',
   COMMENT        => 'Advanced DW Users');

  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP (
   CONSUMER_GROUP => 'EXECUTIVE_DW_GROUP',
   COMMENT        => 'Executive DW Users');

END;
/

Now there are a fair few more parameters available for the CREATE_PLAN procedure, mostly concerned with the method by which we allocate resources (”emphasis” or “ratio”), but for now we’ll go with the defaults.

Next we define the plan directives.

BEGIN

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'STANDARD_DW_GROUP',
   COMMENT                  => 'Standard DW group',
   MGMT_P1                  => 30,
   PARALLEL_DEGREE_LIMIT_P1 => 0,
   SWITCH_TIME              => 300,
   SWITCH_GROUP             => 'CANCEL_SQL',
   SWITCH_ESTIMATE          => TRUE
   );

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'ADVANCED_DW_GROUP',
   COMMENT                  => 'Advanced DW group',
   MGMT_P1                  => 45,
   PARALLEL_DEGREE_LIMIT_P1 => 4,
   SWITCH_TIME              => 60,
   SWITCH_GROUP             => 'STANDARD_DW_GROUP',
   SWITCH_ESTIMATE          => FALSE
   );

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'EXECUTIVE_DW_GROUP',
   COMMENT                  => 'Executive DW group',
   MGMT_P1                  => 20
   );

 DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'OTHER_GROUPS',
   COMMENT                  => 'other groups',
   MGMT_P1                  => 5
   );

END;
/

Note how I’ve had to allocate some resource to any users falling outside these groups, in this complex resource plan.

Finally we validate, and then submit, the pending area.

BEGIN
 DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA();
 DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA();
END;
/

Then we have to grant permission to our users to use these plan directives, and set them as the user’s default, as we did with the simple plan.

BEGIN
  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'standard_dw_user',
    CONSUMER_GROUP => 'standard_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'advanced_dw_user',
    CONSUMER_GROUP => 'advanced_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP(
    GRANTEE_NAME   => 'executive',
    CONSUMER_GROUP => 'executive_dw_group',
    GRANT_OPTION   => FALSE);

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('standard_dw_user', 'standard_dw_group');

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('advanced_dw_user', 'advanced_dw_group');

  DBMS_RESOURCE_MANAGER.SET_INITIAL_CONSUMER_GROUP('executive', 'executive_dw_group');

END;
/

And finally, turn on the new resource plan.

ALTER SYSTEM SET RESOURCE_MANAGER_PLAN = daytime_plan;

Then we can log in, in separate sessions, as these three users, and check that they’ve been allocated to the correct consumer groups.

SELECT username, resource_consumer_group
FROM   v$session
WHERE  username in ('EXECUTIVE','STANDARD_DW_USER','ADVANCED_DW_USER');

USERNAME                       RESOURCE_CONSUMER_GROUP
------------------------------ --------------------------------
ADVANCED_DW_USER               ADVANCED_DW_GROUP
EXECUTIVE                      EXECUTIVE_DW_GROUP
STANDARD_DW_USER               STANDARD_DW_GROUP

So let’s try and trigger some of the resource management. Now the hard CPU limits will be hard to test, as they only kick on when CPU usage on the machine gets to 100%, and even then when there is no spare capacity in a consumer group that’s not using all it’s allocation, but we should be able to get a “standard DW user” to have their query cancelled if it’s going to take more than five minutes.

I create a copy of the SH schema, insert the contents of the SALES table into itself a couple of times, then write a SELECT statement that aggregates the table using a cartesian join – something that will take ten minutes or so to run.

conn standard_dw_user/password

SELECT sum(amount_sold)
FROM   sh_copy.sales
,      sh_copy.products
,      sh_copy.customers
,      sh_copy.times
,      sh_copy.promotions
,      sh_copy.channels
/

ERROR at line 2:
ORA-00040: active time limit exceeded - call aborted

So the Database Resource Manager has aborted the query, even before it executes, because the predicted time for the query was greater than five minutes.

If we now log in as the advanced DW user, we can see consumer group switching in action.

conn advanced_dw_user/password

select sum(amount_sold)
from   sh_copy.sales
,      sh_copy.products
,      sh_copy.customers
/

Leaving the session running switching back to the SYS user, I can see their consumer group initially being the advanced user group:

SELECT username, resource_consumer_group
FROM   v$session
WHERE  username in ('EXECUTIVE','STANDARD_DW_USER','ADVANCED_DW_USER');

USERNAME                       RESOURCE_CONSUMER_GROUP
------------------------------ --------------------------------
ADVANCED_DW_USER               ADVANCED_DW_GROUP

After a minute though, I check again and see that they have indeed switched down to the lower priority group.

SQL> /

USERNAME                       RESOURCE_CONSUMER_GROUP
------------------------------ --------------------------------
ADVANCED_DW_USER               STANDARD_DW_GROUP

Using this technique you can “manage” power users who write runaway queries, allowing the query to continue but switching them away from taking up all of the system’s resources.

So, wrapping this all up, how to we make use of it with OBIEE. The trick to it is two things; firstly, we need to amend the resource directive to make it work properly with connection pools, otherwise one particular OBIEE user exceeding time limits will push all the other users of that connection pool into this lower-priority consumer group, and secondly we need to define multiple connection pools that make use of each resource group.

Firstly I need to go back and amend the directives to use SWITCH_TIME_IN_CALL rather than SWITCH_TIME (which bounces any new sessions by a user back to the original resource group rather than leaving them in the ones the previous session was switched in to), like this:

BEGIN

 DBMS_RESOURCE_MANAGER.CLEAR_PENDING_AREA();
 DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA();

 DBMS_RESOURCE_MANAGER.DELETE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'STANDARD_DW_GROUP'
  );

 DBMS_RESOURCE_MANAGER.DELETE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'ADVANCED_DW_GROUP'
  );

 DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'STANDARD_DW_GROUP',
   COMMENT                  => 'Standard DW group',
   MGMT_P1                  => 30,
   PARALLEL_DEGREE_LIMIT_P1 => 0,
   SWITCH_TIME_IN_CALL      => 300,
   SWITCH_GROUP             => 'CANCEL_SQL',
   SWITCH_ESTIMATE          => TRUE
   );

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE (
   PLAN                     => 'DAYTIME_PLAN',
   GROUP_OR_SUBPLAN         => 'ADVANCED_DW_GROUP',
   COMMENT                  => 'Advanced DW group',
   MGMT_P1                  => 45,
   PARALLEL_DEGREE_LIMIT_P1 => 4,
   SWITCH_TIME_IN_CALL      => 60,
   SWITCH_GROUP             => 'STANDARD_DW_GROUP',
   SWITCH_ESTIMATE          => FALSE
   );

  DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA();
  DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA();

END;
/

Now we switch over to the BI Administration tool, and define some standard, advanced and executive users that belong to standard, advanced and executive security groups.

Now we take copies of the standard connection pool, and name it after each of the consumer groups, deleting the original one.

Then we edit each connection pool, changing the shared login to correspond to the users we created earlier (remember to check the “use qualified names” checkbox so that it still reads from the correct database schema once connected).

Now set permissions on each of the connection pools, so that only the relevant BI Server security group can connect through them.

Then when users log in, they’ll be assigned to the correct connection pool (as it’s the only one, for that data source, they’ll have permissions on). When the user then comes to run the query, they’ll run as the correct database user and get assigned to the correct consumer group, and then if they hit a resource directive issue, they will either get bumped down a group or, as in the case below, have their query aborted as it exceeds the allowed time estimate.

Full details of Oracle Database Resource Manager can be found in the online docs here.

operation$...	PK	...	C1
 I		87		'J' -- a newly inserted row
UO		1		NULL
UN		1		'X' -- NULL value becomes 'X'
UO		99		'Y'
UN		99		'Z' -- 'Y' becomes 'Z'
UO		42		'A'
UN		42		NULL -- 'A' becomes NULL
UO		39		NULL
UN		39		NULL -- data value remains unchanged (not necessarily NULL)

As I mentioned last time we have no real interest in delete records (in fact my source system does not generate deletes); we are using CDC to populate a data warehouse and the fact that a record had existed is of interest to us. As I also mentioned we are not using supplementary logging so we can’t just apply the I and UN records in chronological order

A point I did not emphasise last time is that change data capture does just that: it captures changes, in fact every commit of a change. This means that we may see changes that are of no interest to our data warehouse. It also means we may see ‘non-changes’ where a record is “updated” and then committed without changes to data values being made. I would suggest that the best way to filter out these records (if filtering is appropriate for your business) is as part of the downstream ETL and not of the change capture process.

So how do we go about the building of a change capture view. The first thing is to realise that we can do this with analytic functions, the second thing is that we will need a lot of them (several for each column being processed) and this can have a major (negative) impact on query performance as we carry out many window sort operations on the same dataset.

Let’s start with the update row type. For an update we need a row to exist, therefore any change should be built on a prior version of the row. If we consider the first update for a row in the subscriber view it must either be for a newly inserted row or one we already hold in target table. Subsequent updates in the same CDC subscriber view need to be applied in the order they occurred. As the CDC identifies changes by the row key, a SCN and a RSID value we already have the bones of ordering our changes to apply, we only need to add in a way of finding the original value in the target table if it exists, again analytics come to our rescue.

Whenever I write a query using analytics I try to work out the how to partition and order the data to achieve my goal. With the updates I need to process pairs of UO and UN records for the same key value and change number and order them so that the UN records comes last. We then need to look for changes between the UO and UN record. To my thinking this is a simple LEAD() or LAG() to bring the before and after versions onto the same row and a case statement to determine if the column has become null in the captured change.

select * from (
SELECT OPERATION$,
  CSCN$,
  COMMIT_TIMESTAMP$,
  RSID$,
  ORDER_ID,

/* Now for the changing columns */
  ORDER_STATUS,
  CASE
    WHEN lag(ORDER_STATUS) over (partition BY ORDER_ID, rsid$ order by OPERATION$ DESC ) IS NULL
    AND ORDER_STATUS                                                            IS NULL
    THEN NULL -- No change to value
    ELSE
      CASE
        WHEN ORDER_STATUS IS NULL
        THEN 2  -- 2 = change from NOT NULL to NULL
        ELSE 1  -- 1 = change from NULL to NOT NULL
      END
  END c_ORDER_STATUS
/* repeat similar logic for each of the other columns of interest. */

  FROM CDC_ORDERS
  WHERE operation$ <> 'I' -- only looking at UO and UN operations
  )
WHERE operation$ = 'UN' - we only need to process the final states of each change
)

We now need to union this set of rows to the most recent stored version of the row; this is either the version in our data store with the most recent timestamp or the insert record in our CDC view. Picking the most recent value the data store can be achieved by using the row_number function:

	SELECT OPERATION$, CSCN$,COMMIT_TIMESTAMP$,RSID$,ORDER_ID, ORDER_STATUS, L_ORDER_STATUS ... from (
	SELECT 'X' OPERATION$, -- set a constant
	  -1 CSCN$, -- set to a constant so we can simply filter it out later -- we don't want to re-insert this record!
	  COMMIT_TIMESTAMP$,
	  RSID$,
	  ORDER_ID,

/* For each column of interest */
	  ORDER_STATUS,
	  1 c_ORDER_STATUS, -- set a constant
/* Repeat the above block */

	row_number() over (partition by order_id, order by COMMIT_TIMESTAMP$ DESC,RSID$ DESC) RN  -- most recent version will be 1
	from  ODS_ORDERS
	where ORDER_ID in (select order_id from CDC_ORDERS) -- we only want order_id values that are in the CDC view
	) where RN = 1 -- only select the most recent

or for the ‘I’ record in our CDC subscriber view window.

	SELECT OPERATION$,
	  CSCN$,
	  COMMIT_TIMESTAMP$,
	  RSID$,
	  ORDER_ID,

	/* Now for the changing columns */
	  ORDER_STATUS,
	  1 c_ORDER_STATUS -- set a constant
	/* repeat similar logic for each of the other columns of interest. */

	  FROM CDC_ORDERS
	  WHERE operation$ = 'I' -- only looking at I operations

As the previous value for a given PK will be in one of two places (but not both) then a simple UNION ALL would suffice to provide all of the rows we need to build the history of changes.
The next stage of the processing is take the three UNION ALL sources and then using analytics “copy down” previous values to fill in blanks. Here I use the LAST_VALUE function to look back over an ordered window.

SELECT
	OPERATION$,
    CSCN$,
    COMMIT_TIMESTAMP$,
    RSID$,
    ORDER_ID,
    CASE
      WHEN c_ORDER_STATUS IS NOT NULL  -- there is a new value of ORDER_STATUS
      THEN ORDER_STATUS
      ELSE -- look at the last change for this column
        CASE LAST_VALUE(c_ORDER_STATUS ignore nulls) over (partition BY ORDER_ID order by CSCN$, RSID$)
          WHEN 1 -- changed to a non null at the last change
          THEN LAST_VALUE(ORDER_STATUS ignore nulls) over (partition BY ORDER_ID order by CSCN$, RSID$)
          WHEN 2 -- became NULL at the last change
          THEN NULL
/*
we could use LAST_VALUE(ORDER_STATUS ) over (partition BY ORDER_ID order by CSCN$, RSID$) but this would add a sort to query plan
*/
        END
    END ORDER_STATUS,
/* similar code for the remaining columns all selected from my UNION ALL VIEW of ODS_ORDER, CDC_ORDER where operation = 'I' and CDC_ORDER where operation is not 'I'

The final things to deal with are: the COMMIT_TIMESTAMP$ column is a DATE and we may get multiple rows for a given key and date if multiple commits occur in the same second; as far as I am concerned here, multiple commits are (in spirit) the same change so we could take the last row in any given second, again we use the row_number function for this

ROW_NUMBER() OVER (PARTITION BY ORDER_ID, COMMIT_TIMESTAMP$ ORDER BY RSID$ DESC) RN

and not reinserting the “seed” rows we took from ODS_ORDERS … but as we set the CSCN$ to be -1 we just filter on CSCN$ values to be greater than zero in our insert.

That’s basically it – a huge view with many, many analytics – but it performs quite well providing you are not processing too large a window.