Comments on: Moving Global Electronics Data using Sunopsis http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/ Delivered Intelligence Wed, 17 Mar 2010 13:47:41 -0400 http://wordpress.org/?v=2.8.4 hourly 1 By: Mark Rittman » Putting Together a BI Suite Sample Dataset, Part 1 http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1993 Mark Rittman » Putting Together a BI Suite Sample Dataset, Part 1 Tue, 05 Dec 2006 22:13:11 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1993 [...] I’ve more or less finishing the opening chapter of the book I’m working on, and I’ve reached the part now where I need a set of sample data for the examples that are going to follow. Without giving too much away (the contracts are being signed now), I need a set of data that I can model and then transform, load into an Oracle database and then report on using BI Suite Standard Edition and Enterprise Edition. To properly show off the Enterprise Edition features, I also want to create a set of Microsoft SQL Server data and an Analysis Services cube that I can then transform using Sunopsis Data Conductor, plus I want data in a range of XML, Excel and flat file data sources to bring into the various databases. All of this has to integrate together, and help me create a number of scenarios to show off the features of Discoverer, Answers, Dashboards and Delivers (and ideally, the SOA features in the Fusion Middleware platform) - a fairly ambitious set of objectives. [...] [...] I’ve more or less finishing the opening chapter of the book I’m working on, and I’ve reached the part now where I need a set of sample data for the examples that are going to follow. Without giving too much away (the contracts are being signed now), I need a set of data that I can model and then transform, load into an Oracle database and then report on using BI Suite Standard Edition and Enterprise Edition. To properly show off the Enterprise Edition features, I also want to create a set of Microsoft SQL Server data and an Analysis Services cube that I can then transform using Sunopsis Data Conductor, plus I want data in a range of XML, Excel and flat file data sources to bring into the various databases. All of this has to integrate together, and help me create a number of scenarios to show off the features of Discoverer, Answers, Dashboards and Delivers (and ideally, the SOA features in the Fusion Middleware platform) – a fairly ambitious set of objectives. [...]

]]> By: mark http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1992 mark Tue, 05 Dec 2006 20:17:04 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1992 Vincent You can use any expression that's valid on the platform where you're doing the transformation (usually the target platform). For example, you can do datatype conversions, CASE statements, more or less anything that you'd normally use with the "expression" operation in OWB. regards Mark Vincent

You can use any expression that’s valid on the platform where you’re doing the transformation (usually the target platform). For example, you can do datatype conversions, CASE statements, more or less anything that you’d normally use with the “expression” operation in OWB.

regards

Mark

]]> By: Vincent at ITToolbox blogs http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1991 Vincent at ITToolbox blogs Tue, 05 Dec 2006 03:13:20 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1991 What about business rules that are more complex than a straight through mapping, such as data type conversion (string to date) or conditional mapping with IF or CASE statements, defaulting NULL values, concatenating or trimming fields. Can these functions be added at the column mapping stage? What about business rules that are more complex than a straight through mapping, such as data type conversion (string to date) or conditional mapping with IF or CASE statements, defaulting NULL values, concatenating or trimming fields. Can these functions be added at the column mapping stage?

]]> By: mark http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1990 mark Sat, 02 Dec 2006 21:26:02 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1990 Stewart, Thanks for the comments. In terms of which product Oracle will keep, or if indeed they will close off one product and just keep the other - I just don't know what Oracle's plans here would be, certainly nothing has been said in public as to what the long-term plans are. Officially of course, Oracle are still in the process of acquiring Sunopsis, so we probably can't expect anything official until the second half of 2007. In terms of which one do I think should be kept? Well, I don't think it's feasable long-term to keep two ETL products going, otherwise Oracle's sales people will end up spending all their time trying to explain why you should use one product rather than the other to customers. When it comes to making the choice though - well, I like the way that OWB development closely mirrors ETL development using SQL and the Oracle platform - it's "close enough to the metal" for someone with decent Oracle development skills to build an efficient ETL process. With Sunopsis though, I like the way it splits the ETL process into business rules and the actual implementation, dividing the work out in this way would make sense on a lot of projects I've worked on. So it's not an easy decision to make. WRT the comment about bulk collect and forall - I did think of this at the time of writing, as you say a lot of this optimisation is handled for you automatically now, however I suspect whoever wrote the knowledge module for Sunopsis wouldn't be aware of this, and anyway the automatic optimisation will only be relevant for 10g. Good point though. Stewart,

Thanks for the comments. In terms of which product Oracle will keep, or if indeed they will close off one product and just keep the other – I just don’t know what Oracle’s plans here would be, certainly nothing has been said in public as to what the long-term plans are. Officially of course, Oracle are still in the process of acquiring Sunopsis, so we probably can’t expect anything official until the second half of 2007.

In terms of which one do I think should be kept? Well, I don’t think it’s feasable long-term to keep two ETL products going, otherwise Oracle’s sales people will end up spending all their time trying to explain why you should use one product rather than the other to customers. When it comes to making the choice though – well, I like the way that OWB development closely mirrors ETL development using SQL and the Oracle platform – it’s “close enough to the metal” for someone with decent Oracle development skills to build an efficient ETL process. With Sunopsis though, I like the way it splits the ETL process into business rules and the actual implementation, dividing the work out in this way would make sense on a lot of projects I’ve worked on. So it’s not an easy decision to make.

WRT the comment about bulk collect and forall – I did think of this at the time of writing, as you say a lot of this optimisation is handled for you automatically now, however I suspect whoever wrote the knowledge module for Sunopsis wouldn’t be aware of this, and anyway the automatic optimisation will only be relevant for 10g. Good point though.

]]> By: Stewart Bryson http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1989 Stewart Bryson Fri, 01 Dec 2006 16:23:03 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1989 One more thing: you mentioned that the PL/SQL routines might be faster if BULK COLLECTS and FORALLS were used. Actually... that's not exactly the case anymore with 10g. Dynamic cursors (which Sunopsis is doing, according to your code excerpt) now do a FORALL with a limit of 100 by default. You can read about it at: http://asktom.oracle.com/pls/ask/f?p=4950:8:2368310651016297924::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12226527910608 One more thing: you mentioned that the PL/SQL routines might be faster if BULK COLLECTS and FORALLS were used. Actually… that’s not exactly the case anymore with 10g. Dynamic cursors (which Sunopsis is doing, according to your code excerpt) now do a FORALL with a limit of 100 by default.

You can read about it at:
http://asktom.oracle.com/pls/ask/f?p=4950:8:2368310651016297924::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:12226527910608

]]> By: Stewart Bryson http://www.rittmanmead.com/2006/11/30/moving-global-electronics-data-using-sunopsis/comment-page-1/#comment-1988 Stewart Bryson Fri, 01 Dec 2006 16:06:05 +0000 http://www.rittmanmead.com/wp2/2006/11/30/moving-global-electronics-data-using-sunopsis/#comment-1988 Just the fact that it doesn't do row-by-row processing (which Informatica and the other big boys do) is encouraging. I don't know of any other ETL tool that supports the MERGE statement on Oracle, so that is another big plus. Also, writing your own knowledge modules seems promising, but it depends how difficult it is. It is a bit discouraging that the MERGE requires a primary key. There are simply too many cases in data warehousing where MERGING without a primary key is preferable. Actually... I can't think of any situations where MERGING on the primary key is of any use. When surrogate keys are the normal primary key on dimension tables, but the natural key is the more meaning attribute, this seems to be a rather arbitrary coding choice that wasn't too well thought out. But the ability to do loads using DML is an incredible upside. Most tools introduce lowest common denominator load routines that usually amount to row-by-row processing. Sounds like Sunopsis might have the paradigm right with the knowledge modules. I know that Oracle will sift through the two products and try to incorporate the best features of both, but do you think--from an initial assessment--that Sunopsis or OWB will be the target of the other products surviving features? Which do you think it should be? Thanks as always for your contribution. I usually just get loaded between flights. Just the fact that it doesn’t do row-by-row processing (which Informatica and the other big boys do) is encouraging. I don’t know of any other ETL tool that supports the MERGE statement on Oracle, so that is another big plus. Also, writing your own knowledge modules seems promising, but it depends how difficult it is.

It is a bit discouraging that the MERGE requires a primary key. There are simply too many cases in data warehousing where MERGING without a primary key is preferable. Actually… I can’t think of any situations where MERGING on the primary key is of any use. When surrogate keys are the normal primary key on dimension tables, but the natural key is the more meaning attribute, this seems to be a rather arbitrary coding choice that wasn’t too well thought out.

But the ability to do loads using DML is an incredible upside. Most tools introduce lowest common denominator load routines that usually amount to row-by-row processing. Sounds like Sunopsis might have the paradigm right with the knowledge modules.

I know that Oracle will sift through the two products and try to incorporate the best features of both, but do you think–from an initial assessment–that Sunopsis or OWB will be the target of the other products surviving features? Which do you think it should be?

Thanks as always for your contribution. I usually just get loaded between flights.

]]>