Apache Lens provides a unified OLAP cube abstraction to perform operational and exploratory analytics over both real-time and batch data stored in multiple data systems. It addresses the complexity and data silos that arise from having multiple disparate analytics systems by providing a single metadata layer and consistent mechanism for discovering and querying all data across different storage tiers and systems. Lens is currently an Apache incubator project with two releases and supports data stores like Hive, JDBC, and is deployed at companies like InMobi and Flipkart.

1. Apache LENS : Unified OLAP on
Realtime and Batch Data
HADOOP SUMMIT
San Jose 2015
Sharad Agarwal (Flipkart)
Jothi Padmanabhan (InMobi)

2. Motivation

3. Most enterprises
have more than one
Analytics Data Systems

4. Catering to diverse requirements
Flexible
questions
Fast response
Fresh data

5. Different kind of analysis
Variety
No of
Queries
Operational
Exploratory

6. Example Scenarios

7. Example 1
Operational and Exploratory

8. Analyse user behaviour in Mobile
Advertising domain
User Activity
timestamp
user-id
location-id
device-id
device-
orientation
served-ad-id
clicks
downloads
revenue
User
user-id
age
gender
interests
Location
location-id
city
country
Device
device-id
manufacturer
os
model

9. Operational Analytics
User Activity (by demog, geo, device)
Click activity of users by city and age
Download activity by gender and country for iOS/
Android
Exploratory Analytics
Download conversion by device orientation
(landscape/portrait)

10. Activity
ETL
Subset
Activity
User
All
Activity
Location
User
DWH Store
Batch Store (Hive)
Device
Location
Operational
Analytics
Exploratory
Analytics
Frequently used Data
Low latency response
All Data
High latency response
Device

11. Example 2
Reacting to Realtime-Data

12. Sale Days in e-commerce
Look at current trends to mark related
items for offers
Logistics decisions

13. Orders
Offers
Realtime Store
Logistics
Realtime
Dashboards
Fresh Data
only for recent time
window
Product
Raw Data Streams
Streaming
pipelines

14. Solving varied scenarios
leads to
multiple disparate Systems
and
complexity

15. Complexity is a silent
killer!
Data Inconsistencies
High engineering and operation cost
Moving data across systems is non trivial
Confusion among users
Multiple definitions of data
Different way of access
Data Silos - Data Discovery

16. What is desired ?
Easy and Consistent mechanism to
discover and query all data
Cost and performance trade-off knobs
for different queries

17. Apache LENS
Cube abstraction
Common view across
Multiple Tiers
Multiple Storages

18. OLAP Data Model
CUBE DIMENSION
Dimension
Table
Fact Table

19. Tiered Data Layout -
Fact
Raw Fact (mr), Dim-Cuts (dr)
Agrr Fact1 mr1<=mr, da1<dr
Agrr Fact2 mr2<=mr1,
da2<dr1
Measures, Dim-cuts
Query
Interactivity…
Query
Flexibility

20. Activity
ETL
All
Activity Location
User
Batch Store (Hive)
Device
Weekly
Rollup
Monthly
Rollup
Multiple Tiers of storage for
performance
Unified
View
LENS
CUBE

21. Traditional DWH
Exploratory
Frequently used Data
Low latency response
Fresh Data
Low latency response
Realtime store
All Data
High latency response
Batch store
Batch
ETL
Streaming
Aggregations
Batch
ETL
Unified
View
LENS
CUBE
Operational

22. Architecture

23. Lens Capabilities
OLAP Cube Abstraction
Data Discovery via single metadata layer
Query Life Cycle Management
Data Optimisation via Query Analytics
Fast Workload based experimentation
with newer systems: Spark, Tez, AWS
Redshift etc.

24. Integrates with Apache
Zeppelin for Data
exploration

29. Current Status
Incubated in Apache in Nov 2014
Two releases - 2.1 being latest
Supports stores
Hive
JDBC compliant
Deployed at
InMobi
Flipkart

30. Lens Roadmap
Authorization
Scheduler service
Make it suitable to integrate with BI
tools
Automatic Roll up suggestions
New Drivers : Elastic search, Spark SQL
Administrator console

31. http://lens.incubator.apache.org
Mailing Lists
lens-dev@lens.incubator.apache.org
lens-user@lens.incubator.apache.org
Stay Involved