Oracle Endeca Week : How Do You Develop Endeca Latitude Applications?

February 23rd, 2012 by

In this final posting in Endeca week, we’re going to look at the process of creating applications using Endeca Latitude, or as it’s now known, Oracle Endeca Information Discovery. Although the individual steps are different, the process overall isn’t all that different to creating a data mart or an Essbase cube. Details and tutorials are a bit thin on the ground, but hopefully with this last post in the blog series, you’ll at least get to see the overall process and the tools that you’ll use. Before we start though, here’s a recap of this week’s posts:

Traditional BI development techniques tend to spend a lot of time at the start, and at the end, of the process. You spend a fair bit of time developing a dimensional model, aggregation rules and strategies, with the actual mechanics of the ETL bit hopefully then being simple. Once you’re done, you spend the rest of the time developing reports, analyses, dashboards and so on. With Endeca Latitude, the data modeling part, and the dashboard building part, are relatively simple, and it’s the ETL bit that takes the time.

A typical flow of tasks for a Latitude project is as follows:

  1. First you use Endeca Latitude Integration Suite to load data into the MDEX engine (this is the bit that takes the time)
  2. Then, you use the configuration web service to configure the MDEX record schema and MDEX engine features
  3. Next, you create the Latitude Studio application, and then
  4. Finally, you use the administration web service to set up monitoring and backups

In this blog post, we’re going to look at the first and third steps – using the Endeca Latitude Data Integrator tool, part of Information Integration Suite, to get data into an MDEX engine, and then using Studio to create a search and analysis web application. This is only a high-level look, but we’re also working on a more step-by-step demo that hopefully will run on OTN or a similar site – watch this space.

So the first task that you’ll need to perform is obtain your data, and then use Endeca Latitude Data Integrator to ETL it into an MDEX engine. If you’ve used tools such as ODI or OWB, you’ll be familiar with the concept of a graphical ETL tool, and Latitude uses an OEM’d version of the open-source CloverETL tool for this task. Using Data Integrator, you either import or create references to your source data, define metadata for it such as the column structure, then create mappings, or “edges” between the source data and either transformation operators, or operators that load data into the MDEX engine. In the screenshot below, you can see a typical “graph” loading data from some source files into an MDEX engine, using the bulk-load writer transformation.

Loading up the MDEX engine is like loading an Essbase cube, in that all the data you need to use in the analysis will need to be in the MDEX, and depending on how you configure it, either all or part of it will be held in the in-memory cache, with the data being persisted to an column-store, compressed on-disk database. Once you’ve got some data together, you can then start up the Endeca Studio Server and log into Endeca Studio.

Endeca Studio is a web-based environment that presents you initially with a blank canvas, to which you can add pages and Studio components. These components communicate with MDEX via web services, and you add them from the Control Panel > Add Component in Studio, like this:

Selecting the Components option brings up a dialog listing the various Studio components you can add. Typically, you’d divide the canvas into two columns, the left-hand one thinner than the right-hand, and add search, breadcrumb and guided (facteted) navigation components to this side.

Then, you’d start adding visual components such as word clouds, record lists, metrics bars and so on to the main part of the screen. In the example below, we’re using LQL, the Latitude Query Language, to define the query for a chart component:

Once you’ve done the hard work of getting your unstructured, semi-structured and unstructured data into MDEX, parsed and combined it, creating the Studio application is the easy bit. Typically, you’d add some navigation elements to the left-hand side, such as a search bar, breadcrumbs (selected criteria) area, and the guided navigation control that’s common to most Studio applications:

Finally, once you’re done, the application would look something like this, with navigation down the side, visualizations on the rest of the page, and multiple pages of information. As you click on the guided navigation items, the controls on the right are automatically filtered, in a similar way to Qlikview, or the new high-speed visualizations provided with Exalytics.

So that’s an overview of the Endeca Studio and MDEX development process. If you’re interested in downloading the Endeca Latitude software, it’s now called Oracle Endeca Information Discovery and you can find it for Windows x64 and Linux x64 over at http://edelivery.oracle.com – start with Oracle Endeca Information Discovery Quick Start (2.2.2) for Microsoft Windows x64 (64-bit) and the product documentation, and you can set up the same Quickstart demo that I’ve been running here.

That’s all for Endeca, at least for now – next week, we’ll be taking an in-depth look at Oracle Exalytics BI Machine.

Comments

  1. Stuart Wallace Says:

    Hi,

    Just wanted to say thanks for Endeca Week. A great series of blogs that have positioned the product well.

    I guess I have a number of concerns about how this will all fit together in the Oracle BI picture. As you suggested in Day 3 there are a number of possible development routes. I guess CloverETL is ripe for development as a Knowledge Module but as for everything else, well, let’s just hope customers are presented with a clear and justifiable business case for each component.

    Cheers,

    Stuart

  2. Deepak P Says:

    hello,

    Thanks for this awesome starter on EID.

    I downloaded and installed Endeca Quickstart from edelivery.oracle.com few weeks back(now this installable is not present here). anyway I was able to install sucessfully but i was unable to get the sample application files which as per the instruction from in the quickstart guide can be downloaded by –
    To add the Latitude Sample Application files to an existing Latitude 2.2.x installation:
    1. Log in to your EDeN account.
    2. On the EDeN homepage, click Downloads.
    3. On the Tools and Utilities page, find the Product Downloads section, then click View and
    download purchased products.
    4. On the Product Downloads page, click Endeca Latitude 2.
    5. In the Current Releases table, click Installation and Quick Start.
    6. From the Installation and Quick Start page, download the Quick Start Project
    (Latitude_2.2.x_QuickStart.zip).
    7. Extract this folder to C:\.
    This creates the following directory structure:
    C:\Endeca\Latitude\2.2.x\QuickStart\studio.

    Now my problem is EDeN site is not working so I am unable to find this zip file “Latitude_2.2.x_QuickStart.zip”. Can you please provide some direction from where I can get this sample application file?
    Thanks in advance for response.

    Deepak P

  3. Andrew Says:

    Hi Deepak

    You will find it under the documentation download on the oracle site under the e-delivery.Oracle.com.

    Find Endeca Information Discovery (2.2.2) Media Pack for Microsoft Windows x64 (64-bit) –>
    Oracle Endeca Information Discovery (2.2.2) Documentation V30398-01

    Alternatively email me at andrew.murphy@c3businesssolutions.com and will send to you.

  4. Mohit Says:

    Hello,

    Thanks for the elaborated post. I have worked on Endeca 5.1 and now have to use Endeca Latitude for one of my projects.
    I have a question that previously we used to create the pipeline through developer studio and provide the data source in the pipeline itself. But now after Latitude in place, is this still valid? As you mentioned, giving the data source is done in Latitude studio, so I was wondering where does the pipeline stand here?
    In case there is any document that would help me understand the changes, I would be more then happy to receive on my email address.
    Thanks a ton!

  5. Phil Franklin Says:

    Latest release is now 2.3 with an updated Quickstart app, and some good improvements to the Query language and Studio front end. Developer Studio can no longer be used as the data design and load tool with Oracle Endeca Information Discovery (formerly Latitude). You now use the Integrator (a full featured ETL tool) and web services. In fact most ETL tools that have a web services component can be used to configure and load data into the engine.
    The edelivery.oracle.com site is where product lives but you will need to be registered to access stuff

  6. Mark Rittman Says:

    Thanks Phil. We’ve covered the new Oracle Endeca Information Discovery 2.3 release including the updated development process, in this new three-part blog series:

    1) Creating an Oracle Endeca Information Discovery 2.3 Application Part 1 : Scoping and Design
    http://www.rittmanmead.com/2012/06/creating-an-oracle-endeca-information-discovery-2-3-application-part-1-scoping-and-design/

    2) Creating an Oracle Endeca Information Discovery 2.3 Application Part 2 : Preparing and Loading Data
    http://www.rittmanmead.com/2012/06/creating-an-oracle-endeca-information-discovery-2-3-application-part-2-preparing-and-loading-data/

    3) Creating an Oracle Endeca Information Discovery 2.3 Application Part 3 : Creating the User Interface
    http://www.rittmanmead.com/2012/06/creating-an-oracle-endeca-information-discovery-2-3-application-part-3-creating-the-user-interface/

    thanks, Mark

  7. Jason Says:

    Mark,
    Jason Here . I am a data architect / developer and is in the process of making my hand dirty with endeca and is stuck with one major issue. The data load from endeca integrator to studio takes for ever (1 to 2 hours) I got a 64 bit dual core 2.5MH each processer with 2 gb ram (windows 7) . BTW I am not doing anything fancy yet. I am just doing the exercises from the endeca tutorial. I can see that the log messages write the progression of the load and once the load is done I can see the data in studio.
    What would be the minimum configuration needed for me as a developer?

    Any helpful pointers will be useful

    Thank you

  8. Mark Rittman Says:

    Jason – it sounds like memory is the issue. You’d need at least 4GB on the laptop to run EID properly, with ideally 8GB available if possible.

  9. Jason Says:

    Mark,
    Thank You. That was the issue. I upgraded the Ram to 8 GB now it takes under 2 min. You are the Man!!.

    Thanks Again for your help.

  10. Pramodh Says:

    Hi all,

    I just wanna knw that does Endeca has good scope or not. Because I am gonna work on that tool. so

  11. Shehan Says:

    Hi all,
    I’m going to upgrade JDK 1.4 to 1.6, but without changing my endeca 5.1 version. Will it generate any other errors? Is it compatible? Please help me on this thing asap.
    Thanks and Best Regards,
    Shehan.

  12. Farnaz Mostowfi Says:

    Shehan,
    Which Endeca Application are you using?
    All version of Oracle Endeca Information Discovery uses JDK 1.6 and I dont think they can run on 1.4.

  13. Shehan Says:

    Hi,

    We are using only search engine part of the endeca. I hope that you got what I mean. In anycase can you please reply with your email address?Then I can directly contact you. By the way thanks a lot for helping me.

    Thanks and Best Regards,
    Shehan.

  14. Shehan Says:

    Hi Farnaz,

    Hope you got what I ment by that previous question. Could you please help me on this thing?

    Thanks and Regards,
    Shehan.

Website Design & Build: tymedia.co.uk