Downloaded 45 times
![© 2015 Progress Software Corporation. All rights reserved.22
Geek Speak
$ sqoop help import
usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]
Common arguments:
--connect <jdbc-uri> Specify JDBC connect string
--connect-manager <jdbc-uri> Specify connection manager class to use
--driver <class-name> Manually specify JDBC driver class to use
--hadoop-mapred-home <dir>+ Override $HADOOP_MAPRED_HOME
--help Print usage instructions
-P Read password from console
--password <password> Set authentication password
--username <username> Set authentication username
--verbose Print more information while working
--hadoop-home <dir>+ Deprecated. Override $HADOOP_HOME](https://image.slidesharecdn.com/webinar-marketing-data-lakes-160421180649/75/Building-a-marketing-data-lake-22-2048.jpg)
Uploaded bySumit Sarkar
Building a marketing data lake
The document discusses marketing data lakes and their significance in managing large-scale marketing data effectively. It outlines industry trends, benefits of data lakes, and techniques for ingesting data directly from SaaS applications using tools such as Apache Sqoop and Falcon. It emphasizes the transition from ETL to ELT processes in leveraging big data for strategic decision-making in marketing.
More Related Content
![Journey to Marketing Data Lake [BRK1098]](https://cdn.slidesharecdn.com/ss_thumbnails/oraclemme-final-170418005539-thumbnail.jpg?width=640&height=640&fit=bounds)
![Data APIs Don't Discriminate [API World Stage Talk]](https://cdn.slidesharecdn.com/ss_thumbnails/apiworldstage2017-170927153840-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Building a marketing data lake
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Building a marketing data lake
- 1. How Big DataISVs get marketing data into lakes Sumit Sarkar Chief Data Evangelist Progress DataDirect Gary Angel Advisory Digital Analytics Center of Excellence Principal EY
- 2. © 2015 ProgressSoftware Corporation. All rights reserved.2 Audio Bridge Options & Question Submission
- 3. How Big DataISVs get marketing data into lakes Sumit Sarkar Chief Data Evangelist Progress DataDirect Gary Angel Advisory Digital Analytics Center of Excellence Principal EY
- 4. © 2015 ProgressSoftware Corporation. All rights reserved.4 Agenda What is a Marketing Data Lake? Industry trends around accessing marketing data in SaaS applications How to ingest data with Apache Sqoop and Apache Falcon directly from SaaS applications How big data vendors can embed SaaS connectivity
- 5. © 2015 ProgressSoftware Corporation. All rights reserved.5 What is a Marketing Data Lake?
- 6. © 2015 ProgressSoftware Corporation. All rights reserved.6 A data lake is a large-scale storage repository and processing engine. A data lake provides "massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs” - SAS Institute What is a Marketing Data Lake?
- 7. © 2015 ProgressSoftware Corporation. All rights reserved.7 Benefits of a Marketing Data Lake? Some of the benefits of a data lake include: Store data in all shapes and sizes Flexible analytics with “schema on read” Query data using SQL or big data programming frameworks Eliminate data silos
- 8. © 2015 ProgressSoftware Corporation. All rights reserved.8 Why Marketing Data? CMOs will outspend CIOs on technology by 2017 (Gartner) Oracle spent $3B on a martech aquisition spree to gain CMO mindshare. Expect more collaboration between CMO and CIO (CIO.com) Modern Marketing Data Warehouse Webinar ~500 registrations (Progress)
- 9. © 2015 ProgressSoftware Corporation. All rights reserved.9 Industry trends around accessing marketing data in SaaS applications
- 10. © 2015 ProgressSoftware Corporation. All rights reserved.10 It’s easy to forget that it’s still about solving real business problems. Relevant data Transaction / behavior history Manage Data Perform Analytics Drive Decisions Insights continuous feedback loop Appropriate data sources Answers to business questions Strategy (Thinking) Moves Right to Left Implementation Moves Left to Right Before you think data, think decisions!
- 11. © 2015 ProgressSoftware Corporation. All rights reserved.11 Our marketing data is almost all in the cloud CRM Web Behavior Mobile Behavior Search Buys Display Buys Owned Social Public SocialMeta-Data And it’s almost all complex, stream data – which means APIs that only give aggregations aren’t too useful
- 12. © 2015 ProgressSoftware Corporation. All rights reserved.12 Detail is important because this digital data is true big data The relationship between events is critical
- 13. © 2015 ProgressSoftware Corporation. All rights reserved.13 We’re almost never solving for one problem with a big data system Reporting Analytics Summarized Data Segmented Data Detail Data We can’t just aggregate / We can’t not aggregate Dashboarding Campaign Optimization Customer Drill-down Attribution, CLTV, Experience, Personalization Targeting Forecasting
- 14. © 2015 ProgressSoftware Corporation. All rights reserved.14 Segmentation is a one important technique to aggregate and join Customer segmentation Visit type identification RFM models KPDs and metrics Measu remen t Found ation Customers v. prospects Owned products Persona Product focused Shopping focused Social focused Customer service Measurement of success specific to each segment and visit Recency and frequency for every segment and visit type Additional metrics that help identify drivers of success Segmentation allows for effective aggregation of the meaning and outcome of streamed event data: Measurement foundation
- 15. © 2015 ProgressSoftware Corporation. All rights reserved.15 End-to-End Strategies ReportCubeParkFull Detail ReportParkFull Detail Most organizations do some combination of at least 1 & 2 Direct to Detail (2) has many advantages if it can be made performant (more flexible reporting and much less maintenance) Semi-Detail (3) is designed to capture most of the advantages of (2) when (2) isn’t performantReport Semi- Detail ParkFull Detail 1 2 3
- 16. © 2015 ProgressSoftware Corporation. All rights reserved.16 How to ingest data with Apache Sqoop and Apache Falcon directly from SaaS applications
- 17. © 2015 ProgressSoftware Corporation. All rights reserved.17 What is Apache Sqoop? Apache Sqoop Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. Sqoop successfully graduated from the Incubator in March of 2012 and is now a Top-Level Apache project http://sqoop.apache.org/
- 18. © 2015 ProgressSoftware Corporation. All rights reserved.18 What is Apache Falcon? Apache Falcon Falcon is a feed processing and feed management system aimed at making it easier for end consumers to onboard their feed processing and feed management on hadoop clusters. https://falcon.apache.org/ Note: Falcon uses Sqoop for import/export operation. Sqoop will require appropriate database driver to connect to the relational database. Please refer to the Sqoop documentation for any Sqoop related question. Please make sure the database driver jar is copied into oozie share lib for Sqoop.
- 19. © 2015 ProgressSoftware Corporation. All rights reserved.19 Data in SaaS Applications is Siloed, Protected by Proprietary APIs Designed for Process Integration, not Data Integration
- 20. © 2015 ProgressSoftware Corporation. All rights reserved.20 How to ingest data directly from SaaS applications
- 21. © 2015 ProgressSoftware Corporation. All rights reserved.21 JDBC access to SaaS data Progress DataDirect JDBC Connector Schema Manager Apache Sqoop Salesforce.com Schema User Defined Schema Driver uses SOAP API Bulk API Metadata API
- 22. © 2015 ProgressSoftware Corporation. All rights reserved.22 Geek Speak $ sqoop help import usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS] Common arguments: --connect <jdbc-uri> Specify JDBC connect string --connect-manager <jdbc-uri> Specify connection manager class to use --driver <class-name> Manually specify JDBC driver class to use --hadoop-mapred-home <dir>+ Override $HADOOP_MAPRED_HOME --help Print usage instructions -P Read password from console --password <password> Set authentication password --username <username> Set authentication username --verbose Print more information while working --hadoop-home <dir>+ Deprecated. Override $HADOOP_HOME
- 23. © 2015 ProgressSoftware Corporation. All rights reserved.23 Why ISVs are turning to a single interface for SaaS? Get JDBC interface on top of any API Data Source API Eloqua Web Services API (REST/SOAP) Bulk and non-Bulk APIs No query language Oracle Service Cloud Web Services APIs (REST/SOAP) ROQL Google Analytics Hypercube (query limits of 10 metrics grouped by max of 7 dimensions) Veeva CRM SOAP, BULK, Metadata APIs SOQL
- 24. © 2015 ProgressSoftware Corporation. All rights reserved.24 As the Market Switches from ETL to ELT, Data Access is critical ETLELT Extract Transform Load View Operational Systems Staging Area Data Warehouse Analytics Apps Operational Systems Extract & Load Big Data Warehouse Transform & View Analytics, Data Prep, and even traditional DW
- 25. © 2015 ProgressSoftware Corporation. All rights reserved.25 How big data vendors are embeding SaaS connectivity
- 26. © 2015 ProgressSoftware Corporation. All rights reserved.26 Progress DataDirect Embed Sales & Marketing Connectors into the Data Access Layer
- 27. © 2015 ProgressSoftware Corporation. All rights reserved.27 Ingest data across 200+ data sources (beyond marketing data sources) Big Data/NoSQL Apache Hadoop Hive Cloudera Hortonworks Pivotal HD MapR EMR Pivotal HAWQ Cloudera Impala MongoDB Spark SQL Cassandra SAP HANA Data Warehouses Amazon Redshift SAP Sybase IQ Teradata Pivotal Greenplum Relational Oracle DB Microsoft SQL Server IBM DB2 MySQL PostgreSQL IBM Informix SAP Sybase Pervasive SQL Progress OpenEdge Progress Rollbase SaaS/Cloud Salesforce.com Database.com FinancialForce Veeva CRM ServiceMAX Any Force.com App Hubspot Marketo Microsoft Dynamics CRM Microsoft SQL Azure Oracle Eloqua Oracle Service Cloud Google Analytics EDI/XML/Text EDIFACT EDIG@S EANCOM X12 IATA Healthcare EDI: X12, HIPAA, ICD-10, HL7 Custom EDI Flat files: CSV, TSV, dBase, Clipper, Foxpro, Paradox Text Files Any SDK SequeLink Socket Server Customer Engineering
- 28. © 2015 ProgressSoftware Corporation. All rights reserved.28 Single API for data lake ingestion from SaaS sources Ingest data against a single API (JDBC) Get a single dedicated partner Connect to unlimited data with a single API Get unlimited support
- 29. How Big DataISVs get marketing data into lakes Sumit Sarkar Chief Data Evangelist Progress DataDirect Gary Angel Advisory Digital Analytics Center of Excellence Principal EY
Editor's Notes
- #2 How Big Data ISVs get marketing data into lakes Marketing data is driving significant new Big Data investments from CIO and CMO offices. The latest Big Data trend is storing that data in lakes for analytics, providing massive storage for any type of data to be used for 360 customer views, predictive lead scoring, personalization, or sentiment analysis. However, marketing data is increasingly stored in the cloud creating a connectivity challenge. Big Data vendors provide facilities to transfer core business data between relational database systems and Data Lakes, such as Apache Sqoop. But what about cloud data sources where existing Apache Sqoop connection managers do not work well with cloud SaaS APIs, each with a proprietary REST or SOAP API? The key to accelerating adoption of big data technology is providing easy access to disparate cloud data sources such as Salesforce, Oracle CX, Marketo, Eloqua, Google Analytics or Adobe Omniture. Competitive advantage then results from having embedded connectivity within your technology for data ingestion to an organization’s most important data, customer data. Join this informative and entertaining webinar as we explore: What is a Marketing Data Lake? Industry trends around accessing marketing data in SaaS applications How to ingest data with Apache Sqoop and Apache Falcon directly from SaaS applications How big data vendors can embed SaaS connectivity Speaker(s): Sumit Sarkar, Data Connectivity Evangelist, Progress Software Gary Angel, Advisory Digital Analytics Center of Excellence Principle, Ernst & Young Asset(s): Follow-up asset sent in email Mike Johnson’s blog: https://www.progress.com/blogs/are-you-ready-to-go-fishing-in-a-data-lake
- #3 Give Attendees a closer look at the control panel and how they can participate. Join Audio: 2 ways to do so, 1) to use VoIP, click on “Mic & Speakers”, or 2) to use your telephone, click on “telephone” and dial-in using the numbers and information provided 2) All lines are muted for today’s webinar. We do plan to have a live Q&A session at the end of the presentations. However if you have a question at any time during this webinar, simply submit your questions via the “Question” section of the webinar interface located to the right of your screen – we will collect all questions through this “Question Window”. Final Note: we are recording today’s webinar and will posted to PartnerLink
- #4 Why ISVs? Strata: big data vendors, data prep, data pipelines, data management, etc Data Lakes are part of the solution.
- #7 Last webinar was around building a Marketing Data Warehouse. Data Warehouse is “Schema on Write” architecture and typically loaded with ETL tools Data Lakes are loaded with raw data (no “T”) and create the “Schema on Read” on business demand
- #8 The kinds of data from which you can derive value are unlimited. You can store all types of structured and unstructured data in a data lake, from CRM data, to social media posts. You don’t have to have all the answers upfront. Simply store raw data—you can refine it as your understanding and insight improves. You have no limits on how you can query the data. You can use a variety of tools to gain insight into what the data means. You don’t create any more silos. You gain a democratized access with a single, unified view of data across the organization. http://info.zaloni.com/hubfs/Architecting_Data_Lakes_Zaloni.pdf By Ben Sharma and Alice LaPlante
- #9 Source: http://www.cio.com/article/2825086/cio-role/is-the-cio-cmo-transition-of-power-becoming-a-reality.html
- #11 http://info.zaloni.com/hubfs/Architecting_Data_Lakes_Zaloni.pdf By Ben Sharma and Alice LaPlante
- #18 Traditionally positioned for RDBMS via JDBC. There are specialized connectors for sources such as MySQL or Postgres; and generic JDBC for any third party.
- #19 Note: Falcon uses Sqoop for import/export operation. Sqoop will require appropriate database driver to connect to the relational database. Please refer to the Sqoop documentation for any Sqoop related question. Please make sure the database driver jar is copied into oozie share lib for Sqoop. Commercial data lake management solutions are available from many of Hadoop vendors (Cloudera Navigator), as well as standalone from companies such as Zaloni and Podium Data.
- #23 bash-4.1$ sqoop import --connect "jdbc:datadirect:sforce:SecurityToken=3jZ0x4NcgClYDhxJqMa3c744://test.salesforce.com;User=ids.integration@hp.com.fltesta;Password=informatica@123;DatabaseName=sandbox" --query 'SELECT TOP 10 t.* FROM Case as t WHERE $CONDITIONS' -m 1 --target-dir /sample/table/q50 --driver com.ddtek.jdbc.sforce.SForceDriver --verbose
- #24 R&D challenges building SQL connectivity across cloud sources such as Marketo Not all SaaS APIs expose a standard query language. In those cases, the engineering team looks at each object individually. Each object may be exposed with a different API with unique rules for invoking, searching filtering, etc. It required a significant effort to provide a standard experience querying across the entire data model. Handling full join capabilities. In cases where the SaaS APIs do not support a query language with JOIN capability, the engineering team has to perform that operation. This requires a translation from SQL to efficiently call Marketo APIs to return the minimal amount of data prior to performing the join. When joining two very large objects, the data access layer may use up considerable resources on the application server or desktop. Therefore, deployment of the data access layer to an elastic cloud service such as DataDirect Cloud makes a lot of sense for two reasons: Faster performance and use fewer memory/CPU resources on the client application server or desktop Leverage the superior bandwidth between DataDirect Cloud and Marketo where pre-joined datasets get exchanged. How to handle data models? Is it static or dynamic? How are changes detected and communicated to the client? Each SaaS data source is different and in the case of Marketo, certain objects are better queried through views and others through tables. Handling this matrix of data models and objects across all SaaS sources was certainly a challenge.
- #28 350+ ISVs 10,000 DEUs We’re excited to get MongoDB data into the hands of more people through open data standards
- #29 Develop against open standards Avoid vendor lock-in by adopting open industry standards. DataDirect is the leader in data connectivity standards having co-founded the ODBC specification and serves on the JDBC Expert Group, OData Technical Committee and ANSI SQL Committee. Connect to unlimited data with a single API Access the full breadth of data sources using a single, decoupled, code base and API for the data access layer protecting you from changes in metadata, error handling, and API or protocol revisions. Get a single dedicated partner Deliver full support for the breadth of data sources in all shapes and sizes, with constant vigilance for the next security vulnerability (POODLE, FREAK, LOGJAM) in your data access layer.Focus your engineering resources on your core business. Get unlimited support We live for your next big customer. Make sure your POC is a success with 24/7 partner support and access to expertise from our engineering teams, partnerships and leading technology companies such as Microsoft, Oracle, and IBM through our TSANet multi vendor support channel.