Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Fifth elephant-grill
Similar to Fifth elephant-grill (20)
Recently uploaded
Recently uploaded (20)
Fifth elephant-grill
- 1. Grill Unified analytics platform Amareshwari Sriramadasu
- 2. Amareshwari Sriramadasu • Architect in Data platform team at Inmobi • Working in Hadoop and eco systems since 2007 • Apache Hadoop – PMC • Apache Hive– Committer • Worked with Yahoo! earlier
- 3. Analytics at Inmobi - Problem areas and Motivation Introduction to Grill OLAP Model Query examples Grill design Agenda
- 4. Digital advertising at Inmobi Courtesy: http://www.liesdamnedlies.com/ Owns & Sells Real estate on digital inventory Has reach to users Wants to target Users Brings money Market place Consumer
- 5. Analytics Use cases • Understanding Trends & Inference • Forecasting and Anamoly detection Data scientists • Feedback to improve Ad Relevance in Real Time Engineering systems • Troubleshooting of issues Developers • Publisher/Advertiser specific analytics(dashboards) Advertisers and publishers • Tracking specific accounts Account managers • Inventory sizing and estimation Business/Product analysts
- 6. • Canned/Dashboard queries • Adhoc queries • Interactive/Batch queries • Scheduled queries • Infer insights through ML algorithms Analytics use cases
- 7. Analytics Warehouses at Inmobi and Scale • Billions of Ad Requests/Impressions per day • 170 TB Hadoop Warehouse • 5 TB SQL Columnar Datawarehouse • 70 TB Hbase cluster
- 8. Why both Hadoop and SQL warehouse? Canned Adhoc Response Times IO (Input Records) Adhoc Canned Adhoc Query Engine Query Dashboard queries are mostly canned and Interactive Adhoc queries can be Interactive or batch depending on the data volumes and query complexity
- 9. • Disparate user experience • Disparate data storage and execution engines • Schema management across storages • Not leveraging ‘SQL on Hadoop’ community Problems
- 10. Analytics at Inmobi - Problem areas and Motivation Introduction to Grill OLAP Model Query examples Grill design Agenda
- 11. GRILL Analytics As Service Unify the Catalog and Query layer for Adhoc/Canned Batch/Interactive Reports on single Interface
- 13. Data Layout – Fact data Dim-cuts, Measures Cost
- 14. Data Layout – Dimension data … Subsetm (am < am-1) … Subset2 (a2 < a1) Subset1 (a1 < ax) All attributes (ax) Cost
- 15. Data Layout – Snowflake Aggr Factk : measures (mak <= ma(k-1)), dims (dak < da(k-1)) ….. Aggr Fact2 : measures (ma2 <= ma1), dimensions (da2 < da1) Aggr Fact1 : measures (ma1 <= mr), dimensions (da1 < dr) Raw Fact: measures (mr), dimensions(dr) Dim2_1 Dim3 Dim2 Dim4_1 Dim4 Dim1
- 16. Associates structure to data Provides Metastore and catalog service – Hcatalog Provides pluggable storage interface Accepts SQL like queries HQL is widely adopted language by systems like Shark, Impala Has strong apache community Data warehouse features like cubes, facts, dimensions Logical table associated with multiple physical storages Pluggable execution engine Query lifecycle management Query quota management Scheduling queries WhatdoesHiveprovide WhatismissinginHive Apache Hive to the rescue
- 17. Analytics at Inmobi - Problem areas and Motivation Introduction to Grill OLAP Model Query examples Grill design Agenda
- 18. Data Model Cube Fact Table • Physical Fact tables Dimension Dimension Table • Physical Dimension tables Storage
- 19. Data Model - Cube Expressions • Any expression with reachable fields Dim-attributes • Simple dim-attribute • Referenced dim-attribute • Hierarchical dim-attribute • Timed dim-attribute Measure • Column Measure Cube Measures Dim- attributes Expressions Note : Some of the concepts are borrowed from http://community.pentaho.com/projects/mondrian/
- 20. Data Model - Dimension Expressions • Any expression with reachable fields Attributes • Simple dim-attribute • Referenced dim-attribute • Hierarchical dim-attribute Dimension Attributes Expressions
- 21. Data Model - Relationships Cube Dimension Dimension Fact table Cube Fact table Storage Dimension Table Dimension Dimensio n table Storage
- 22. Data Model Aggrk Fact Table ….. Aggr2 Fact Table Aggr1 Fact Table Raw FactTable Dim2_table1 Dim2_table2 Dim3_table1 Dim2_table2Dimtable1 Dimtable2 CUBE Dimension Dimension Dimension
- 23. Analytics at Inmobi - Problem areas and Motivation Introduction to Grill OLAP Model Query examples Grill design Agenda
- 24. CUBE SELECT [DISTINCT] select_expr, select_expr, ... FROM cube_table_reference WHERE [where_condition AND] TIME_RANGE_IN(colName , from, to) [GROUP BY col_list] [HAVING having_expr] [ORDER BY colList] [LIMIT number] Queries on OLAP cubes
- 25. • SELECT ( city.name ), ( city.stateid ) FROM c2_citytable city LIMIT 100 • SELECT ( city.name ), ( city.stateid ) FROM c1_citytable city WHERE (city.dt = 'latest') LIMIT 100 cube select name, stateid from city limit 100 Example query
- 26. Example query • SELECT (citytable.name), sum((testcube.msr2)) FROM c2_testfact testcube INNER JOIN c1_citytable city ON ((testcube.cityid)= (city.id)) WHERE (( testcube.dt='2014-03- 10-03') OR (testcube.dt='2014-03-10-04') OR (testcube.dt='2014-03-10-05') OR (testcube.dt='2014-03-10-06') OR (testcube.dt='2014-03-10-07') OR (testcube.dt='2014-03- 10-08') OR (testcube.dt='2014-03-10-09') OR (testcube.dt='2014-03-10-10') OR (testcube.dt='2014-03-10-11') OR (testcube.dt='2014-03-10-12') OR (testcube.dt='2014-03- 10-13') OR (testcube.dt='2014-03-10-14') OR (testcube.dt='2014-03-10-15') OR (testcube.dt='2014-03-10-16') OR (testcube.dt='2014-03-10-17') OR (testcube.dt='2014-03- 10-18') OR (testcube.dt='2014-03-10-19') OR (testcube.dt='2014-03-10-20') OR (testcube.dt='2014-03-10-21') OR (testcube.dt='2014-03-10-22') OR (testcube.dt='2014-03- 10-23') OR (testcube.dt='2014-03-11') OR (testcube.dt='2014-03-12-00') OR (testcube.dt='2014-03-12 -01') OR (testcube.dt='2014-03-12-02') )AND (city.dt = 'latest') GROUP BY(city.name) cube select city.name, msr2 from testcube where timerange_in(dt, '2014-03-10-03’, '2014-03-12-03’)
- 27. Analytics at Inmobi - Problem areas and Motivation Introduction to Grill OLAP Model Query examples Grill design Agenda
- 28. Implements an interface • explain • execute • executeAsynchronously • fetchResults • Specify all storages it can support Pluggable execution engine
- 29. OLAP Cube QL query Rewrite query for available execution engine’s supported storages Get cost of the rewritten query from each execution engine Pick up execution engine with least cost and fire the query Cube query with multiple execution engines
- 30. Grill Server
- 31. Grill – current state Server Query Service Metastore Service Metrics Query statistics(In progress) Scheduled queries(In progress) Query caching(In progress) Client Java Client CLI JDBC Client Execution Engine Hive Driver JDBC Driver
- 32. • Normalize query cost • Load balancing across execution engines • Alter meta hooks in StorageHandler • Authentication and authorization • Machine learning through Grill • Query quota management Grill roadmap
- 33. Github source for grill • https://github.com/InMobi/grill Github source for Hive • https://github.com/InMobi/hive Documentation • http://inmobi.github.io/grill Mailing lists • grill-users@googlegroups.com • grill-dev@googlegroups.com References
- 35. Backup
- 36. Data Model – Storage Storage • Name • End point • Properties • Ex : ProdCluster, StagingCluster, Postgres1, HBase1, HBase2
- 37. Data Model – Fact Table Fact table Cube Fact table Storage FactTable • Columns • Cube that it belongs • Storages on which it is present and the associated update periods
- 38. Data Model – Dimension table DimensionTable • Columns • Dimension to which it belongs • Storages on which it is present and associated snapshot dump period, if any. Dimension Table Dimension Dimension table Storage
- 39. Data Model – Storage tables and partitions Storagetable • Belongs to fact/dimension • Associated storage descriptor • Partitioned by columns • Naming convention – storage name followed by fact/dimension name • Partition can override its storage descriptor • Fact storage table Fact table • Dimension storage table Dimension table
- 40. Resolve candidate tables and storages Automatically resolve joins, aggregations Allows SQL over Cube QL Queries can span multiple storages Accepts multiple time ranges in query All Hive QL features Query features
- 41. Adhoc querying system Internal Dashboards Customer facing Dashboards and Reporting Analytics systems at Inmobi
Editor's Notes
- Inmobi provides marketplace, where it buys the space on mobile from publishers and sells it to advertisers, meanwhile it acquires users.
- Inmobi has 130TB hadoop warehouse and 5TB SQL warehouse. Let us see an example of reporting page. This is the dashboard a publisher sees.
- Conventional columnar databases (RDBMS) systems lend themselves well for interactive SQL queries over reasonably small datasets in the order of 10-100s of GB, while hadoop based warehouses operate well over large datasets in the order of TBs and PBs and scales fairly linearly. Though there have been some improvements recently in storage structures in the Hadoop warehouses such as ORC, queries over hadoop still typically adopts a full scan approach. Choosing between these different data stores based on cost of storage, concurrency, scalability and performance is fairly complex and not easy for most users.
- Individually all the systems we just saw work really great! They provide best time responses to user queries. Disparate user experience because of multiple reporting systems Involves a learning curve for systems and their api Disparate data storage systems causing inability to scale Altering schema involves different systems Data discovery Cannot leverage data in other systems Not leveraging community around Cannot experiment with new storage/execution engine out of the box
- Column Measure : name, type, default aggregate, format string, start date, end date Expression Measure : Associated Expression Simple Dimension: name, type, start date, end date Referenced Dimension : Referencing table and column Hierarchical Dimension :hierarchy Expression Dimension : Associated expression
- Column Measure : name, type, default aggregate, format string, start date, end date Expression Measure : Associated Expression Simple Dimension: name, type, start date, end date Referenced Dimension : Referencing table and column Hierarchical Dimension :hierarchy Expression Dimension : Associated expression
- Column Measure : name, type, default aggregate, format string, start date, end date Expression Measure : Associated Expression Simple Dimension: name, type, start date, end date Referenced Dimension : Referencing table and column Hierarchical Dimension :hierarchy Expression Dimension : Associated expression
- The grammar is subset of HQL Resolve candidate dimension tables and the storage tables . Resolve the candidate fact tables which can answer the query, pick the ones from top of the pyramid. Resolve fact storage tables for the queried time range. Automatically resolve joins using the relationships between cubes and dimension. Automatically add aggregate functions to measures. Add expression to group by clause, if projected; and project group by clause, if it is not.
- Beta version in prod
- Beta version already in production
- Resolve candidate dimension tables and the storage tables . Resolve the candidate fact tables which can answer the query, pick the ones from top of the pyramid. Resolve fact storage tables for the queried time range. Automatically resolve joins using the relationships between cubes and dimension. Automatically add aggregate functions to measures. Add expression to group by clause, if projected; and project group by clause, if it is not.
- Adhoc querying system Adhoc and Batch queries Scheduled queries Based on Hadoop Mapreduce Provides UI and custom api Data is stored in HDFS Dashboard system Canned reports Interactive and adhoc queries Provides UI and Custom api Data is stored in columnar DWH Customer facing system Face to the outside world (Advertisers and publishers) Interactive and adhoc queries Provides UI and custom api Data is stored in relational DB, Postgres