Rittman endeca

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
A Developer Introduction to Oracle Endeca Information Discovery 2.3
Mark Rittman, Technical Director, Rittman Mead
June 2012
Developing a Hybrid Search/Analytic Application
1Monday, 11 June 12

Mark Rittman
• Mark Rittman, Co-Founder of Rittman Mead
• Oracle ACE Director, specialising in Oracle BI&DW
• 14 Years Experience with Oracle Technology
• Regular columnist for Oracle Magazine
• Author of forthcoming Oracle Press book on OBIEE 11g
• Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
• Email : mark.rittman@rittmanmead.com
• Twitter : @markrittman
2Monday, 11 June 12

Who Are Endeca? Why Did Oracle Acquire Them?
• Recent (2012) acquisition by Oracle, aimed at strengthening ERP and E-Commerce offerings
• Endeca was best known for search and guided navigation features for e-commerce sites
‣ HMV.com
‣ Autotrader.com
‣ Sony e-commerce sites
‣ etc
• Key feature is searching across arbitrary
selections of attributes, fast response,
supports discovery and exploration
‣ “Faceted Search”
• Enabled through a hybrid search/analytic
database called the Endeca Server
Endeca “Faceted
Search” feature
3Monday, 11 June 12

What is Oracle Endeca Information Discovery?
• As well as their search/guided navigation product, Endeca also had a BI platform
• Originally called Endeca Latitude, now rebranded Oracle Endeca Information Discovery (EID)
• A BI platform built around the same Endeca Server, and focused on search/discovery of data
• Complementary to OBIEE and Essbase
• Separately licensed to BI Foundation,
long-term plans for tech platform integration
4Monday, 11 June 12

Information Discovery vs. Reporting & Analysis
• Data volume, variety and growth presents issues in finding answers to business questions
‣ Unstructured data, social network data, call centre logs as well as regular structured data sets
• Datasets change, don’t always fit dimensional models, and arrive quickly
• Users want self-service access to data with minimal setup time
• Reporting and Analysis is great for accurate answers to known questions ...
• ... Data discovery provides fast answers to new questions
• Guiding principle : Quickly explore all relevant data
Quickly Explore All Relevant Data
•Relationships
undefined or
unknown
•No up-front data
modelling
•Rapid, iterative
change
•Advanced search
•Contextual
navigation
•Analytics
•Structured
•Semi-structured
•Unstructured
•Even messy data is OK
•Not in the data
warehouse, yet
5Monday, 11 June 12

“Search-First” Interface
• Traditional BI tools (OBIEE, Discoverer etc)
are focused on reporting and delivering
• Oracle EID takes a “search first” approach,
building on e-commerce and web search
user experiences
• Search + Contextual Navigation
+ Visual Analysis
• Split-second response times to
support and encourage data exploration
Value Search
across all
attributes
Breadcrumb list
showing all filters
applied so far
Guided
Navigation,
free-form filtering
across all
atttributes
6Monday, 11 June 12

Data Discovery Scenarios
• Product Quality Reports / Warranty Analysis
• Online TV Shopping Sales Sentiment Analysis
• Police Crime Report & Evidence Analysis
Type of Customer
•Police forces, CID etc
•Intelligence agencies (MI5 etc)
•Private investigators, journalists
Traditional BI Issues
•By the time you’ve fitted it to a
conformed dimensional model,
you’ve missed the opportunity
•Evidence spread across disparate
documents, sources and formats
Data Discovery Solution
•Minimal up-front data modeling
•No requirement for a common model
•Support for unstructured, semi-
structured and structured sources
•Search is the primary user interface
Type of Customer
•Manufacturers, retailers
•Consumer bodies
•Insurance companies
•Most useful information is in free-
form documents and reports
•Success comes from correlating
information from many sources
•Focus on reporting and numbers
•Easy linking of disparate sources
that only share limited commonality
•All data considered, with ability to
parse and detect meaning in docs
•Unlimited exporing across all attribs.
Type of Customer
•Online and TV-based retailers
•E-commerce operations
•B2C companies with vocal,
online customer base
•Sales reporting only covers what
you’ve sold. not why you’ve sold it
•Consumer sentiment is found on
blogs, Facebook and Twitter, not
easily brought into BI datasets
•Combine unstructured social
networking feeds with sales data
•Content acquisition from non-
traditional sources
•Analyze consumer sentiment
7Monday, 11 June 12

Oracle Endeca Information Discovery Platform Overview
• A suite of tools to capture, load and then analyze all types of data
• Built around the Oracle Endeca Server,
a hybrid search/analytic database
• Full ETL capabiities plus web-based
dashboard and data exploration tools
• Additional tools can be licenced for
content acquisition and text enrichment /
sentiment analysis
Oracle Endeca Server
Oracle Endeca Information
Discovery Integrator
Oracle Endeca Information
Discovery Studio
Call centre
transcripts
Spreadsheets
and budget data
Documents, PDFs
and other unstructured
sources
RDBMS Data Master Data Social Media
and Web feeds
8Monday, 11 June 12

Oracle Endeca Server : A Hybrid Search/Analytic Database
• Key to these capabilities is the Oracle Endeca Server and its datastores (databases)
• Proprietary database engine focused on search and analytics
• Data organized as records, made up of attributes stored as key/value pairs
• No over-arching schema,
no tables, self-describing attributes
• Every record can have its own unique
set of attributes, with the overall data model
emerging over time as data is loaded
• Endeca Server hallmarks:
‣ Minimal upfront design
‣ Support for “jagged” data
‣ Administered via web service calls
‣ “No data left behind”
‣ “Load and Go”
9Monday, 11 June 12

Inside an Endeca Server Datastore
Shared attributes common to all data sources
eg. Time, City, Brand, Key Account
Sales transaction data
from a data mart or data warehouse
External data from market
research or customer surveys /
demographics
Forecast data
from Essbase
Supporting data
and comments from
social media, e.g.
Facebook, Twitter
Attributes unique to the sales data mart / warehouse
e.g. Salesperson, Customer, Channel, Response Type
Attributes unique to just this data source,
for example Customer Segment, Category, Socio Class
“Jagged” data, where each row may
use different sets of attributes e.g. customer
demographic and behavioral attributes
10Monday, 11 June 12

Endeca Server Search and Document Analysis Capabilities
• Search and unstructured data support adds powerful document analysis capabilities
‣ Keyword search, boolean search, parametric search, wildcard search, dimension search and
dimension filters
‣ Dimension precedence rules
‣ Numeric range, geospatial, date/time and security filters
‣ Spell correction/suggestion,
and “do you mean”-type alternative presentation
‣ Find similar, and 1 and 2-way synonyms
‣ Stemming and lemmatisation
‣ Keyword-in-context snippeting
‣ Results clustering, relevance ranking, sorting and paging
‣ Multi-language support

Oracle EID Integrator and Studio
• Data is loaded into Oracle Endeca Server datastores using Oracle EID Integrator
‣ Data Integration (ETL) tool build on open-source CloverETL tool (Eclipse framework)
‣ Oracle EID functionality provided through components that call Endeca Server web services
• User Interface created and delivered using Oracle EID Studio, 100% web-based
‣ Create dashboards made up of search, navigation and data analysis components
‣ Also provides Endeca Server / Studio admin features

Endeca Information Discovery Platform & Software Requirements
• 64-bit application, benefits from multiple CPUs/cores and lots of RAM
• Oracle Information Discovery 2.3 available for Windows and Linux platforms
‣ Windows Server 2008 SP1 64-bit
‣ Oracle Linux / RHEL 5 64-bit
• Downloadable from Oracle Software Delivery Cloud (Edelivery) (full or trial license),
OTN later on (distribution restrictions due to embedded third-party components)
• Endeca Server licensed per-CPU, Oracle EID named-user plus (per-CPU maybe later)

Oracle EID and Oracle BI Foundation
• Complementary technology to OBIEE and Essbase
• Extends BI capabilities to unstructured and semi-structured data
• Provides an alternative “search and contextual navigation UI” for users
• Oracle Endeca Server enables “load and go” data analysis with minimal up-front modeling
Query, reporting
and analysis
Multi-dimensional
OLAP
Production
Reporting
BI
Foundation
Suite
•Interactive dashboard, ad-hoc queries,
drilling enabled by a ROLAP engine
•Multi-dimensional engine, what-if?
analysis, scenario modeling, planning
•High-volume, pixel-perfect layout,
printing
OBIEE
Essbase
BI Publisher
Data Discovery •Search and contextual navigation for
structured and unstructured data
Endeca Information
Discovery
Endeca
Information
Discovery
BI Foundation Suite + Endeca Information Discovery = Complete BI Solution

An OEID 2.3 Development Scenario
• Data to be sourced from structured and semi-structured sources
• Sales data supplemented by customer survey responses
• Loaded into the search/analysis-focused Endeca Server
• Analyzed using a search/guided navigation-led dashboard
• Based on the Quickstart demo/dataset provided with Oracle EID 2.3
Structured,
Semi-Structured
& Unstructured
Data Sources
Oracle
Endeca
Information
Discovery
Integrator
Oracle
Endeca
Server
Oracle
Endeca
Information
Discovery
Studio
End-User Analytic/
Search Application

High-Level Oracle EID Development Process (Simplified)
1. Create the Endeca Server datastore (if required)
2. Use Integrator to create the initial set of records, assign a spec (unique ID) and review
3. Add additional data sources, including unstructured and semi-structured
4. Configure datastore attributes, place into groups and provide user-friendly names
5. Configure search interfaces as required, to enable record search
6. Create the Studio user interface, prototype and repeat steps 3-6 as required
7. Deploy out to user community

The Oracle EID Integrator Development Environment
Project elements
(graphs, in/out
data, params etc)
Properties,
metadata etc for
selected object
Graph, containing a
a single component
(Web Service call to
create a new Endeca
Server datastore)
Additional graph
components
(readers, writers,
joiners + OEID
components)
Buttons to
run graphs, run
in debug mode
etc

Integrator Graphs, Components and Edges
• Data mappings in Integrator are called graphs
• Graphs contain components that are joined by edges that connect to component ports
• Edges are assigned metadata, which define the colums and expressions used to move data
• Graphs are then then run and execute row-based, pipelining data through the process
Input Filter Sort
Join
Input
Load
Component
Edge
Port
Graph

Information Discovery-Specific Components in Integrator
• Integrator comes with a number of specific to Oracle EID, for example:
‣ Create a new datastore, or reset existing ones
‣ Incrementally load datastore records, or bulk-load them
‣ Add attributes via key/value pairs
‣ Tag and enrich text
‣ Import and export datastore configurations
• All except bulk-load call Endeca Server web service APIs
‣ Bulk-load uses a separate, direct-load interface OEID-Specific
Components

Creating a New Endeca Server Datastore
• The Endeca Server runs as a server process, with datastores being created, then attached
• New datastores are either created using the endeca-cmd Java command-line utility
• Or through a graph component that calls the same Endeca Server web service API
‣ Uses a configuration file to obtain
server host name, datastore name, port etc
for the Endeca Server being called
cd c:OracleEndecaServer2.3.0endeca-cmd
endeca-cmd create-ds bikestore
Endeca Server web
service API call to
create a datastore, using
datastore name in
parameter file

Loading the Initial Record Set
• The first stage in creating the dataset is to load the initial set of records
• In this example, loaded from a relational fact table export, contains measures and keys
• Records require a spec(ification), a unique key that uniquely identifies each record
In the example below, the spec is created using a transformation component that concatenates
the SalesOrderID and SalesOrderLineID
Writer component,
writing records to the
Endeca Server data
store
Transform (Reformat)
component,
concatenating input
columns to create
record spec (PK)
Reader component,
reading fact table
rows (records) from
a flat file export

Transformations and Expressions
• Data flowing through graphs can be transformed at various stages, for example
‣ As it flows through an edge
‣ During a join
• Expression syntax is CTL1/CTL2
‣ Clover Transformation Language
‣ Similar to Java expressions, source
and target-technology neutral
‣ Original CTL1 version now being
superceded by CTL2
Expression used to
create the spec
column,
concatenation two
input columns

Creating Metadata Definitions, Assigning to Edges and Sources
• Metadata defines how sources are accessed, and how data flow through edges
• Created from the original source/target object, additional expressions can be added as required
• Assigned to edges by right-clicking on them and choosing Select Metadata >
Drag file into it’s
Metadata outline
folder, to create
initial metadata
definition
Refine the metadata
definition, amending
datatypes, selecting
delimiter etc
Apply the metadata
definition to edges
on the graph

Viewing the Initial Record Set
• The initial record load can then be viewed in Studio
• One record per row of input data, at this point mostly measures and dimension keys
• Each input column becomes an attribute, with values stored internally in key/value pairs
• Later on, we will enhance these records by
‣ Adding additional sources,
for more useful attribute values
‣ Configuring the attribute datatypes
and names, grouping them and
enabling them for search
Initial set of
record attributes
An example
record, showing
values for loaded
attributes

Loading Additional Structured and Semi-Structured Sources
• Additional sources can be loaded using the same graph, joining on common IDs
• Unstructured/semi-structured sources with at least a matching ID can be brought in
• Continue adding sources until full set of required data is loaded into the datastore
Joiner, joining the
combined sales +
employee data with
survey data, on
common
SalesOrderNumber
Joiner, joining the
sales fact and
employee data on a
common
EmployeeKey
column

Text Parsing, Whitelisting, Enrichment and Sentiment Analysis
• Text attributes can be analyzed, enriched and parsed using standard and additional features
‣ Build in features include text tagging component
and various CTL functions
• Oracle Endeca Text Enrichment module
‣ Text analysis capabilities for extraction
of people, pleaces, themes etc
‣ Automatic summarized abstracts etc
‣ Uses Salience Engine from Lexalytics, Inc.
‣ Additional cost option
• Oracle Endeca Text Enrichment with
Sentiment Analysis module
‣ As above, but with ability to calculate
aggregate sentiment scores
‣ Determine positive or negative sentiment
in comments, surveys, feedback
Whitelist text
tagger, using the
whitelist of terms in
the
SurveyWhiteList file
Whitelist of terms to
be tagged using the
Text Tagger - Whitelist
component

Oracle EID Integrator
Demonstrations

Creating the Studio Interface
• Once you have some data loaded, you can start to create the Studio web-based interface
• Create the initial Studio pages early, use as part of the prototyping process
• Over time, add more visualizations and data views to aid discovery
• Once the model and attribute set are fairly stable, you can perform additional configuration
‣ Rename and relabel attributes, arrange into groups, configure for searching
Flat list of
attributes, using
names inherited
from file metadata
Individual
datastore record,
using default
attribute names
from file metadata
Studio
application menu
User details, and tabs for
Studio dashboard pages

Typical Studio Development Tasks
• Adding value and record search components, and guided navigation
• Adding tables and cross tabs, and configuring EQL queries for their data sets
• Adding charts, maps and other visualizations

Creating Pages, Configuring Layouts and Adding Components
• Studio uses pages with layouts to organize content, similar to OBIEE’s dashboards
• Components are then added to the pages, to create the dashboard content
Components that can be
added to Studio pages
Guided Navigation
component being
dropped onto page
layout template
Buttons to add new
pages, new components
and configure layouts
Preferences and
Configuration buttons
for the component

Performing Value Searches (Type-ahead Searching of Records)
• By default, the Search Box component provides
type-ahead value search within records
• Lists all occurences of a particular value,
broken down by attribute
• Does not require any prior configuration
• Useful for identifying which attributes should
then be included in search interfaces in order to
enable record searches

Creating and Enabling Search Interfaces
• Search interfaces create sets of attributes that are then available for record searches
• When values are returned, they are used to filter the record set
• Requires search interfaces to be set up and enabled using
an Endeca Server web service API and a configuration file/source
Reader
component,
reading in the
Search Index
configuration file
A second graph data
flow, that sets up and
enables the required
search interfaces, again
using the WS API
Graph data flow that
configures the selected
attributes as searchable,
using an Endeca Server
WS API

Selecting and Enabling Search Interfaces in Studio
• Studio components, such as Search Box, have configuration setting pages
• To enable record search, one or more search interfaces need to be selected and enabled
• Also provides ability to disable and enable type-ahead value search
Create a new
search box
configuration, and
select search
interfaces that it
will use
Before record search
can be used, it needs to
be configured to use one
or more
“search interfaces”
Enable / disable type-
ahead value-search

Performing a Record Search
• Record searches filter the record set by the specified search term, within the attributes in the
selected search interface
• Record search is only performed with the search button is pressed
• Default search interface is used, unless another one is specifically selected
Select search
interface
Press Search
button
Record set is filtered
by the search term
Search term(s) appear in
the Breadcrumb list

Configuring Attribute Naming and Grouping through Studio
• Attribute metadata can be updated to provide display names, change sort order etc
• Attributes can also be placed into groups, hidden from guided navigation menus
• Configurable through Studio, with the
ability to export the new configuration
for use in Integrator
Studio Control
Panel, including
menu item for
managing
datastore
Attribute Settings
List of
unassigned
attributes,
currently in the
“Other” group
List of attribute groups,
and their current
members
Selecting an
attribute group to
add the selected
attributes to

Configuring Attribute Naming and Grouping through a Graph
• Attribute configuration is applied through another Endeca Server web service API
• Exported configuration from Studio can then be used in a graph to programatically apply
configuration to other / reset data stores
Graph ETL process to
load attribute metadata
(name, datatype etc)
from a flat file and apply
to datastore using an
Endeca Server WS API
Graph ETL process to
create attribute groups
and apply them to the
datastore, using the
Endeca Server WS API

Results Tables, Cross Tabs and Data Explorer Components
• Results Table component displays selected attribures for a set of records
• Cross Tab component displays record set in crosstab matrix
• Data Explorer component displays attribute values for an individual record

Defining the Data Source for a Cross Tab Component
• Cross tab components (and optionally, table components) need to be given a data source
• Datasources for these components are defined using EQL (Endeca Query Language)
‣ Similar to SQL, assumes a single table of records, implied SELECT with GROUP BY
clauses, and other variations suited to the Endeca Server record / attribute data model
EQL Statement
defining Cross Tab
data set

EQL Statement Types and Examples
• RETURN statement, returns a result set to the calling component
• DEFINE statement, creates a temporary table with the result set, used to create views
Return SalesTotal
AS
SELECT SUM(FactSales_SalesAmount) as TotalSales
GROUP BY DimDate_FiscalYear, DimDate_FiscalQuarter,
DimSalesTerritory_SalesTerritoryCountry
DEFINE RegionTotals
AS
SELECT
SUM(Amount) AS Total
GROUP BY Region

Configuration and Viewing of Cross Tab Component
• Once EQL query has been defined, component can then be configured and deployed on page
List of available
dimension
attributes and
metrics
Attributes assigned to
row and column
dimensions, and the
metrics area
Cross Tab component
after it has been
successfully configured

Charting and Other Data Visualization Components
• Other visualization components can be used, including
‣ Charts
‣ Maps
‣ Metrics Bars
‣ Tag Clouds
‣ Alerts
• All use EQL queries
for their data source,
apart from charts
which use views

Creating EQL Views
• Views are stored EQL queries, used for configuring charts
• Provide an abstraction layer over the datastore, easier for end-users to work with
• Can be used to create aggregates and subqueries, useful for complex analyses and
multi-step calculations
• Administered using Studio, can
also be applied using a graph
(both call the same Endeca Server
web service API)
Oracle Endeca Server
Employees
View
Products
View
Transactions
View
Physical
Model
View (Abstraction)
Model
User Interface
and Visualizations

Creating Views and Using with Chart Components
• Views are typically created using the View Manager is Studio, then exported for use in graphs
• Once defined, they can be used (amongst other things) as data sources for charts
The view definition,
expressed as an EQL
DEFINE statement
List of available views,
already defined
List of previously-
defined views, available
as datasets for the
component
Chart configuration
options

The Finished Studio Application

Oracle EID Studio
Demonstrations

Getting Started with Oracle Endeca Information Discovery
• Endeca Development Series on the Rittman Mead Blog
‣ http://www.rittmanmead.com/category/oracle-endeca/
• Getting Started with Oracle Endeca Information Discovery screencast series
‣ http://www.oracle.com/technetwork/middleware/endeca/learnmore/index.html
• Use Rittman Mead to deliver a fixed-price Data Discovery Pilot
‣ Fixed-price of £5k + VAT + expenses, agreed scope initial pilot deployment
‣ Up to four structured and one un/semi-structured source
‣ Delivered in one week with an initial Studio web application
‣ Contact us at info@rittmanmead.com
for more details!

More Information
• Thank you for attending this presentation
• More information can be found at http://www.rittmanmead.com
• Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
• Look out for our book, “Oracle Business Intelligence Developers Guide” due Q3 2012
• Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)

A Developer Introduction to Oracle Endeca Information Discovery 2.3
Mark Rittman, Technical Director, Rittman Mead
June 2012
Developing a Hybrid Search/Analytic Application

Rittman endeca

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Rittman endeca

Similar to Rittman endeca (20)

Recently uploaded

Recently uploaded (20)

Rittman endeca