Partners 2013 LinkedIn Use Cases for Teradata Connectors for Hadoop


Published on

Published in: Business, Technology
1 Comment
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Then, through the lens of big data = “complexity” rather than “volume” – we’re seeing technology evolve that supports:new types of programming at scale such as MapReduce and graph processing engines.Better/different ways of dealing with unstructured dataLess schema dependence – have the flexibility to load data quickly, store cheaply, and process later as needed
  • Then, through the lens of big data = “complexity” rather than “volume” – we’re seeing technology evolve that supports:new types of programming at scale such as MapReduce and graph processing engines.Better/different ways of dealing with unstructured dataLess schema dependence – have the flexibility to load data quickly, store cheaply, and process later as needed
  • Then, through the lens of big data = “complexity” rather than “volume” – we’re seeing technology evolve that supports:new types of programming at scale such as MapReduce and graph processing engines.Better/different ways of dealing with unstructured dataLess schema dependence – have the flexibility to load data quickly, store cheaply, and process later as needed
  • Release of a SQL-H solution for the Teradata database with Teradata Database 14.10.Teradata SQL-H provides dynamic SQL access to Hadoop data in Teradata. With TeradataSQL-H, users can join Hadoop data with Teradata tables.Teradata SQL-H is important to customers because it enables analysis of Hadoop data in Teradata. It also allows standard ANSI SQL access to Hadoop data, leverages existing BI tool investments, as well as lowers costs by making data analysts self-sufficient.
  • Partners 2013 LinkedIn Use Cases for Teradata Connectors for Hadoop

    2. 2. Agenda • UDA > Need for high volume data movement > Challenges with high volume data movement • TDCH & SQL-H > Data movement between Teradata and Hadoop > Architecture, key features, and various packages • LinkedIn POC > Architecture big picture > Use cases > POC environment > POC results and learning > Wish list of enhancements • Next steps and Q&A Copyright © Teradata 2013
    3. 3. TERADATA UNIFIED DATA ARCHITECTURE Data Scientists Quants Business Analysts Engineers LANGUAGES VIEWPOINT Customers / Partners Executives MATH & STATS DATA MINING Aster Connector for Hadoop Operational Systems BUSINESS INTELLIGENCE Aster Teradata Connector DISCOVERY PLATFORM Front-Line Workers SQL-H Teradata Connector for Hadoop SQL-H Teradata Studio Smart Loader for Hadoop CAPTURE | STORE | REFINE IMAGES TEXT WEB & SOCIAL MACHINE LOGS Copyright © Teradata 2013 SUPPORT INTEGRATED DATA WAREHOUSE Data Fabric AUDIO & VIDEO APPLICATIONS CRM SCM ERP
    4. 4. Data Movement Challenges • Data Movement supposed to be Easy - So businesses can spend more time on analytics - But it is not easy as businesses would like in reality - Challenges are even greater with massively parallel systems • Data Movement between Teradata and Hadoop > Two massively parallel systems > Any high volume data movement – Should exploit as much underlying parallelism as appropriate – Single–threaded or single-node processing architecture will not cut it – Move data along the path in compressed form for as long as possible > Various popular Hadoop data formats – Should be supported to avoid the need for staging & intermediate files – Automatic data type/format conversion to minimize manual work by users > Constraints in a production environment – Should be accommodated as much as possible – E.g., limitation on concurrent sessions imposed by mixed workload control Copyright © Teradata 2013
    5. 5. Can technologies work with each other? Copyright © Teradata 2013
    6. 6. Big Data Army Together Copyright © Teradata 2013
    8. 8. TDCH Architecture TD Export Import Tools Hive Sqoop HCat Pig MapReduce Hadoop I/O Format DB I/O Format Teradata Teradata I/O Format … File I/O Format Text Sequence RC Hadoop DFS Teradata DB Copyright © Teradata 2013
    9. 9. TDCH Technical Highlights • Build on MapReduce - For execution and deployment scalability - Proven scalability for up to thousands of nodes - For integration with various data formats - Text, Sequence, RC, ORC (soon), Avro (soon), … - For integration with other MR based tools - Sqoop, Hive, Hcatalog, Pig (future), and possibly others • Built for Usability & Integration > Simple command-line interface > Simple application programming interface for developers > Metadata-based data extraction, load, and conversion – Built-in support for reading/writing data in Hive and Hcat tables – Built-in serialization/de-serialization support for various file formats Copyright © Teradata 2013
    10. 10. TDCH Export Methods (Hadoop  Teradata) Various Implementations • Batch-insert - Each mapper starts a session to insert data via JDBC batch execution • Multiple-fastload - Each mapper starts a separate fastload job to load data via JDBC fastload • Internal-fastload - Each mapper starts a session to load data via JDBC fastload protocol but all sessions are operating as a single fastload job Copyright © Teradata 2013
    11. 11. TDCH Import Methods (Teradata  Hadoop) Various Implementations • Split-by-value - Each mapper starts a session to retrieve data in a given value range from a source table in Teradata • Split-by-hash - Each mapper starts a session to retrieve data in a given hash value range from a source table in Teradata • Split-by-partition - Each mapper starts a session to retrieve a subset of partitions from a source table in Teradata if the source table is already a partitioned table - If the source table is not a partitioned table, a partitioned staging table will be created with a partition key that is the same as the distribution key • Split-by-amp - Each mapper gets data from an individual amp - TD 14.10 required; this method makes use of table operators Copyright © Teradata 2013
    12. 12. Various Packages for End Users > Teradata Connector for Hadoop – For users who would like to use a simple command line interface > Sqoop Connectors for Teradata – For users who would like to use the Sqoop command line interface – Sqoop connector for TD from Hortonworks uses TDCH under the cover – New Sqoop connector for TD from Cloudera uses TDCH under the cover > TD Studio Smart Loader for Hadoop – For users who would like to use the Teradata Studio GUI – TD Studio Smart Loader for Hadoop uses TDCH under the cover for data movement between Terada and Hadoop Copyright © Teradata 2013
    13. 13. TERADATA SQL-H
    14. 14. Teradata SQL-H • SQL-H > Build on table operators > Enable dynamic SQL access to Hadoop data > Can list existing Hadoop database and files > SQL requests parsed and executed by Teradata > Can join data in Hadoop with tables in Teradata • Why is this important? > Enables analysis of Hadoop data in Teradata > Allow standard ANSI SQL access to Hadoop data > Lowers costs by making data analysts self-sufficient > Leverage existing BI tool investments just like Aster SQL-H does • Released with Teradata Database 14.10 Copyright © Teradata 2013
    15. 15. Teradata SQL-H Example SELECT CAST(Price AS DECIMAL (8,2)) ,CAST(Make AS VARCHAR(20)) ,CAST(Model AS VARCHAR(20)) FROM LOAD_FROM_HCATALOG( USING SERVER('') PORT('9083') USERNAME ('hive') DBNAME('default') TABLENAME('CarPriceData') COLUMNS('*') TEMPLETON_PORT('1880') ) as CarPriceInfo;  The SQL-H Table Operator query is launched on the Teradata side.  Data conversion is conducted within Teradata after the data has been transferred from Hadoop. Copyright © Teradata 2013
    16. 16. Connectors Designed for Two Different Audiences Teradata Hadoop ETL Tools BI Tools HCat Pig Sqoop Teradata Tools SQL-H Hive TD Connector for Hadoop (TDCH) Teradata SQL MapReduce Text Sequence HDFS Teradata DB TDCH: Scalable, high performance bi-directional data movement Copyright © Teradata 2013 RC
    17. 17. The Challenge with Hadoop • Hadoop > Excellent scalability > Rapidly evolving ecosystem > Not yet as enterprise-ready as one would like – Lacking support for effective performance management • Challenge (and opportunity) > Enterprise tools and apps to fill the gap > Provide the instrumentation and functionality – For fine-grain (parallel systems) performance management • TDCH is improving > with richer instrumentation and functionality > to fill the performance management gap as much as possible Copyright © Teradata 2013
    18. 18. LinkedIn Overall Data Flow Hadoop Site (Member Facing Products) Activity Data Kafka Camus Member Data Espresso / Voldemort / Oracle DWH ETL Product, Sciences, Enterprise Analytics Changes Databus External Partner Data Lumos Ingest Utilities Teradata Enterprise Products Core Data Set Derived Data Set Computed Results for Member Facing Products Copyright © Teradata 2013
    19. 19. LinkedIn Data System - Hadoop Most data in Avro format Access via Pig & Hive Most High-volume ETL processes run here Specialized batch processing Algorithmic data mining Copyright © Teradata 2013
    20. 20. LinkedIn Data System - Teradata Interactive Querying (Low Latency) Integrated Data Warehouse Hourly ETL Well-modeled Schemas Workload Management Standard BI Tools Copyright © Teradata 2013
    21. 21. LinkedIn Use Cases • Hadoop is the main platform for data staging, data exploration, click stream ETL, and machine learning; • Teradata is the main platform for data warehouse, BI and relational data discovery; • Hadoop holds multi-PB data; TD holds hundred-TB data; • Data need to flow between Hadoop and Teradata; • Analytical processes and applications need to leverage the most appropriate platform to deliver the data intelligence: > Are all the data needed there? (1-week/3-month/3-year…) > Which programming interfaces are available? (SQL/HQL/Pig…) > How fast I need/How slow I can tolerate? > How to share the results? Who will consume them? Copyright © Teradata 2013
    22. 22. LinkedIn TDCH POC Environment Copyright © Teradata 2013
    23. 23. LinkedIn Use Cases - Export • Copy Hourly/Daily Clickstream Data from HDFS to TD • Copy Scoring & Machine Learning Result from HDFS to TD > Challenges: Big volume and tight SLA > Steps: 1. Converted data files from Avro to many ^Z-delimited *.gz files via Pig first (flatten map/bag/tuple, and remove special unicode chars) 2. Quickly load *.gz files using Teradata Connector into the staging table with the help of internal.fastload protocol 3. TDCH execute INSERT DML to copy records from the staging table into the final fact table > Other Options: 1. Combine many *.gz into a few, download to NFS, load via TPT 2. Download many *.gz via webHDFS to NFS, load via TPT Copyright © Teradata 2013
    24. 24. LinkedIn Use Cases - Export Copyright © Teradata 2013
    25. 25. LinkedIn Use Cases - Import • Publish dimension and aggregate tables from TD to HDFS > Challenges: Heavy query workload on TD and tight SLA. Traditional JDBC data dump does not yield high throughput to extract all the dimensional tables within the limited window. > Steps: 1. Full dump for small to medium size dimensional tables 2. Timestamp-based incremental for big dimensional tables Then use M/R job to merge the incremental file with the existing dimensional file on HDFS Save the new dimensional file using LiAvroStorage() as #LATEST copy, and retire the previous version 3. Date-partition-based incremental dump for aggregate tables > Other Options: 1. Home-grown M/R job to extract using and write to LiAvroStorage() directory 2. Write Custom TPT OUTMOD Adapter to convert EXPORT operator’s data parcel to Avro, upload via webHDFS Copyright © Teradata 2013
    26. 26. High Level Findings is not tested Network latency plays a big factor to E2E speed # sessions is subject to TD workload rules Copyright © Teradata 2013
    27. 27. LinkedIn POC Benchmark Reference Test Data Set: > About 275M rows and 250 bytes/row > 2700MB in TD with BLC and 2044MB as GZip text in HDFS > 64664MB as uncompressed text • Import uses split-by-hash & Export uses internal-fastload * Import will spend the first a couple of minutes to spool data * Export M/R job may combine the specified # of mappers to smaller # # Mappers 32 64 128 960s 758s 330s 67MB/sec 85MB/sec 190MB/sec # Mappers Export Import Time Throughput Import 15 28 52 Import Time 970s 870s 420s Throughput 67MB/sec 75MB/sec 154MB/sec Copyright © Teradata 2013
    28. 28. LinkedIn POC Findings • TDCH is very easy to setup and use • TDCH provides good throughput for JDBC-based bulk data movement (import and export) • TDCH simplifies the bulk data exchange interface, so more robust ETL system can rely on TDCH • Network latency can be a big performance factor • In production environment, it is not practical to execute TDCH with too many mappers (e.g. over 150+) for TDCH • Depends on the data set, using too many mappers will not result in performance gain (because the overhead is high) • Many factors can impact E2E performance, debug is hard > Some mappers can run for much longer than the others even with the similar number of records to process > Multiple mappers can run on the same DD – is that wrong? Copyright © Teradata 2013
    29. 29. LinkedIn Business Benefits/Results • LinkedIn can have simpler methods to move and access data seamlessly throughout their environment using the Teradata Connector for Hadoop. • This leads to reduced costs and operation complexity because > Command is invoked from Hadoop gateway machine, so the security verification is taken care by SSH session and Kerberos token already. > The data synchronization between HDFS and Teradata is faster with the help of both systems’ parallel capability. > Less ETL jobs are needed for the move data movement, hence easier to support and troubleshoot. Copyright © Teradata 2013
    30. 30. Sample Code - Export hadoop com.teradata.hadoop.tool.TeradataExportTool -D -D -D -D"-Xmx1G" -D mapreduce.job.max.split.locations=256 -libjars $LIB_JARS -url jdbc:teradata://DWDEV/database=DWH,CHARSET=UTF8 -username DataCopy -password $DEV_DATACOPY_PASS -classname com.teradata.jdbc.TeraDriver -queryband "App=TDCH;ClientHost=$HOSTNAME;PID=$$;BLOCKCOMPRESSION=NO;" -fileformat rcfile -jobtype hive -method internal.fastload -sourcepaths /user/$USER/tdch/example_table_name.rc -debughdfsfile /user/$USER/tdch/mapper_debug_info -nummappers 32 -targettable dwh.tdch_example_table_name -sourcetableschema "COL_PK BIGINT, SORT_ID SMALLINT, COL_FLAG TINYINT, ORDER_ID INT, ORDER_STATE STRING, ..., ORDER_UPDATE_TIME TIMESTAMP" Copyright © Teradata 2013
    31. 31. Sample Code - Import hadoop com.teradata.hadoop.tool.TeradataImportTool -D -D -D -D"-Xmx1G" -D mapred.output.compress=true -D -libjars $LIB_JARS -url jdbc:teradata://DWDEV/database=DWH,CHARSET=UTF8 -username DataCopy -password $DEV_DATACOPY_PASS -classname com.teradata.jdbc.TeraDriver -queryband "App=TDCH;ClientHost=$HOSTNAME;PID=$$;" -fileformat textfile -jobtype hdfs –method -targetpaths /user/$USER/tdch/example_table_name.text -debughdfsfile /user/$USER/tdch/mapper_debug_info -nummappers 32 -sourcetable dwh.tdch_example_table_name Copyright © Teradata 2013
    32. 32. Mapper Level Performance Info -debughdfsfile option (sample output) mapper id is: task_201307092106_634296_m_000000 initialization time of this mapper is: 1382143274810 elapsed time of connection created of this mapper is: 992 total elapsed time of query execution and first record returned is:221472 total elapsed time of data processing and HDFS write operation is:296364 end time of this mapper is: 1382143848579 mapper id is: task_201307092106_634273_m_000025 initialization time of this mapper is: 1382143015468 elapsed time of connection created of this mapper is: 463 total elapsed time of data processing and send it to Teradata is:701876 end time of this mapper is: 1382143720637 Copyright © Teradata 2013
    33. 33. Track Mapper Session in DBQL TDCH injects task attempt id into QueryBand Select SubStr( RegExp_SubStr(QueryBand, '=attempt_[[:digit:]]+_[[:digit:]]+_.*[[:digit:]]+'), 10 ) MR_Task, SessionID, QueryID, min(StartTime), min(FirstStepTime), min(FirstRespTime), sum(NumResultRows), cast(sum(SpoolUsage) as bigint) SpoolUsage, sum(TotalIOCount), max(MaxIOAmpNumber) from DBC.DBQLogTbl where StatementGroup not like 'Other' and NumResultRows > 0 and UserName = 'DataCopy' and CollectTimeStamp >= timestamp '2013-10-17 09:10:11' and QueryBand like '%attemp_id=attempt_201310161616_118033_%' group by 1,2,3 order by 1, SessionID, QueryID; * This feature does not work for internal-fastload yet Copyright © Teradata 2013
    34. 34. Adjust TCP Send/Receive Window • 1ms ~ 70ms network round trip time (traceroute) can be the indicator of suboptimal latency, which will significantly affect TDCH throughput • If the network just has bad latency without dropping many packets, increase the TCP window buffer from its default value 64K to 6MB~8MB can improve TDCH performance • The result varies based the network and data set’s size > When data set size for each mapper is small, no visible improvement is observed > When data set size for each mapper is big, on a network with high latency, the TDCH throughput can improve 30% ~ 100% with big TCP window > TCP window size has impact on both Import and Export jobs (Import) jdbc:teradata://tdpid/database=DBC,TCP=RECEIVE8192000 (Export) jdbc:teradata://tdpid/database=DBC,TCP=SEND8192000 Copyright © Teradata 2013
    35. 35. Wish List based on LinkedIn POC • Compression over the wire/network protocol > If JDBC and FASTLOAD session can compress records and then transmit, the TDCH speed can be 3~10 times faster. > Otherwise, a pair of data import/export proxy agents can help to buffer, consolidate and compress the network traffic • Split-By-Partition enhancement > TDCH can create many partitions for stage table to avoid data skew (e.g. # partitions = # AMPs) > But it can then effectively loop these partitions through 32 or 64 sessions without further consuming too much spool • Avro format support with simple capability of mapping element/attribute from map/record/array… to columns • Auto map TD data types to RC & Avro primitive types • Easier way to use special chars as parameter in command • More meaningful error message for mapper failure • Better and granular performance trace and debug info Copyright © Teradata 2013
    36. 36. What is Next? • Turn learning into TDCH enhancements > Data formats: – Support Avro and Optimized RC > Metadata access: – Use dictionary tables through views without extra privileges > Performance management: – Instrument for fine-grain monitoring and efficient trouble shooting • Start proof-of-concept work with SQL-H > Was in the original plan but ran out of time > Will start the SQL-H POC after release upgrade to TD 14.10 Copyright © Teradata 2013
    37. 37. Acknowledgement • Many have contributed to this effort … • LinkedIn: > Eric Sun, Jerry Wang, Mark Wagner, Mohammad Islam • Teradata: > Bob Hahn, Zoom Han, Ariff Kassam, David Kunz, Ming Lei, Paul Lett, Mark Li, Deron Ma, Hau Nguyen, Xu Sun, Darrick Sogabe, Rick Stellwagen, Todd Sylvester, Sherry Wang, Wenny Wang, Nick Xie, Dehui Zhang, … Copyright © Teradata 2013
    38. 38. Email: Email: PARTNERS Mobile App InfoHub Kiosks Copyright © Teradata 2013