Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kinesis and RDS

2,600 views
2,058 views

Published on

Companies are dealing with increasingly large data sets and looking for ways to significantly improve the scale and cost of Big Data analysis with AWS. This hands-on session shows you how you can achieve that. With hundreds of pre-built connectors, you will learn how to get your on-premise and cloud data into Redshift in minutes, not days, and at a significantly reduced costs using Informatica Cloud Integration. With fully certified support for large scale RDS deployments and Informatica’s Vibe Data Stream solution for automated streaming data collection for Kinesis, Informatica offers a comprehensive cloud integration solution for Big Data analytics with AWS. The ability to seamlessly migrate Informatica’s PowerCenter to Amazon Cloud (EC2) offers customers a Cloud migration path, with even higher performance and lower costs.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,600
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
113
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kinesis and RDS

  1. 1. Power Big Data Analytics with Informatica Cloud Integration Ron Lunasin, Informatica Cloud Product Management Ajay Gandhi, Informatica Cloud Product Marketing Alan Lundberg, Informatica Vibe Data Stream Marketing March 2014
  2. 2. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 2
  3. 3. Leader in Cloud and Hybrid IT Integration 3 Gartner MQ for Integration Platform as a Service, Jan 2014 The Forrester Wave: Hybrid2 Integration, Q1 2014
  4. 4. Hybrid IT Architecture is the New Normal
  5. 5. Informatica Cloud Platform Cloud Integration and Data Management Cloud Data Quality Cloud Master Data Management Cloud Process Automation Cleanse and De-Dupe Visualize Relationships Improve User Experience Leverage Existing Systems Cloud Integration Secure Sandbox Cloud Test Data Management You Need More Than Just Integration
  6. 6. Hundreds of Connectors
  7. 7. • 100% Cloud • Developer and App User Collaboration • Productivity for Advanced Integration Use Cases • Vibe Integration Packages Visual Productivity for Advanced Cloud and Hybrid Integration Informatica Cloud Designer
  8. 8. Introducing Vibe Integration Packages (VIP’s) Redefining Hybrid IT Integration Agility VIPs=Pre-built parameterized integration workflows Built by developers for app users and other developers/partners App users configure VIPs using wizards to build custom integrations VIPs work with Cloud and PowerCenter VIPs can be distributed via Informatica Marketplace VIPs are easily embedded into 3rd-party apps via APIs
  9. 9. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 9
  10. 10. Common Customer Use Cases • Reduce costs by extending DW rather than adding HW • Migrate completely from existing DW systems • Respond faster to business; provision in minutes • Improve performance by an order of magnitude • Make more data available for analysis • Access business data via standard reporting tools • Add analytic functionality to applications • Scale DW capacity as demand grows • Reduce HW & SW costs by an order of magnitude Traditional Enterprise DW Companies with Big Data SaaS Companies
  11. 11. Using the Cloud Isn’t an “All or Nothing” Choice 11
  12. 12. Integrating AWS With Existing On-Premises IT 12
  13. 13. Use Cloud To Make On-Premises Apps Better 13 Backup Analytics
  14. 14. Cloud Apps That Integrate With On-Premises Apps 14 AWS serves application content & data Integration to data centers for financial transactions
  15. 15. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 15
  16. 16. Support for AWS Database and Big Data Services 16 Pre-built Cloud & PowerCenter Connectors for RDS and Redshift Vibe Data Streaming for Kinesis InformaticaCloud.com/Amazon-Redshift
  17. 17. 2 Informatica Cloud Architecture Overview- Redshift 4Secure Agent Your Company or VPC Amazon Redshift 31 Amazon RDS
  18. 18. Map Once. Deploy Anywhere. ON PREMISE HADOOP 3rd PARTY APPLICATIONS CLOUD
  19. 19. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 19
  20. 20. ©2013 Informatica. Proprietary and Confidential 20 Redshift Upsert – Manual Coding Way Extract the data from source1 Put into flat files and compress2 3 Transfer Compressed Files To S3 4 Wait for S3 Consistency 5 Copy Data From S3 Into Staging Table6 Inner Join With Target Table To Delete Rows To Be Updated Insert Updated Rows From Staging Table 7 Delete Staging Table 8 9 Delete Files From S3 Create Staging Table in Redshift 10 Or, Do It In 3 Simple Steps…
  21. 21. Redshift Upsert – Informatica Cloud Way 1 2 3 Choose Upsert Operation Map Your Fields Run Or Schedule!
  22. 22. Informatica Cloud Amazon Redshift Architecture Firewall Informatica Cloud Secure Agent Metadata Mappings Build mapping and execute job 1 1 Retrieve Account Data 2 2 3 Put Account Data into Flat File 4 Transfer compressed Flat File to S3 5 Initiate copy from S3 6 Load data into Amazon Redshift 6 3 54
  23. 23. DEMO! 23
  24. 24. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 24
  25. 25. 2626 Data / Sensor Diversity…
  26. 26. 27 How to make sense of it all…
  27. 27. Streaming Collection: Vibe Data Stream (VDS) 28 VibeDataStreamBus Publish/Subscribe Leverage High Performance Messaging Infrastructure. Publish with Ultra Messaging for global distribution without additional staging or landing. Cloudera, Pivotal, Hortonworks, MapR Targets Web Servers, Operations Monitors, rsyslog, SLF4J, etc. Handhelds, Smart Meters, etc. Discrete Data Messages Sources VDS Node VDS Node VDS Node VDS Node VDS Node Management and Monitoring Internet of Things, Sensor Data VDS Node Real Time Analysis, Stream Processing No SQL Databases: HBASE, Cassandara, Riak, MongoDB
  28. 28. 29 Transactions, OLTP, OLAP Social Media, Web Logs Machine Device, Scientific Documents and Emails Vibe Data Stream Vibe Data Stream Vibe Data Stream
  29. 29. AWS Kinesis + Informatica – Framework for Deeper Insight 30 Level 2 Reduce time-to-information & time-to-decision Operational pattern matching, alerts, Real-time analytics Level 3 Create Visibility & Insight to Understand the Business Impact Operational KPIs, Alignment of IT & Business, Drill down Service Delivery Applications OSS / BSS Applications Network Applications Level 1 Instrument for Problems & Opportunities Detection, response, correlation & extrapolation of trends Vibe Data Stream
  30. 30. Use Cases – Solving the Difficult Problems 31 Detect Patterns Exception Monitoring Process Monitoring • Deviations from norm (Monitoring, Fraud, Error) • Trending up/down to exceed a threshold • SLA monitoring • 3 events within 5 milliseconds • A then B then C occurs • Geospatial processing • Are process workflows operating properly? • Are manual processes completed on time? • Detect Missing Work and Queued Work
  31. 31. Informatica in “Lambda Architectures” Adapted from “Runaway Complexity in Big Data”, Nathan Marz, Sept. 25/2012 Transactions, OLTP, OLAP Social Media, Web Logs Machine Device, Scientific Documents and Emails Batch Layer Batch View Big Data Analytics + Real Time Streams Speed Layer Real Time View Serving Layer Merged View • Stream Processing • Filter / Classify • Correlate
  32. 32. • Solution approach that complements and augments traditional BI and reporting • Combines approaches and techniques from various technology areas, including: • End-to-end and comprehensive data Integration • Event processing and event-driven architectures • Rapid data provisioning via a common data access layer • Access to LIVE data in operational systems • Access to all types of data including unstructured data AWS Kinesis + Informatica = “Real Time Operational Intelligence” Sense Reason Respond Visualize
  33. 33. Architectural Implications Batch processing Data structured, homogenous High Volume and variety Distributed SystemsCentralized Database-centric Client Server Systems Prioritize Modeling events as enterprise objects / assets Real Time Yesterday Today Events treated as 2nd class citizens
  34. 34. Informatica User Interactions Developers / OEMs Developer IDE TemplatesAnalyst MyRulePoint Portal SDKs User Tool Business User
  35. 35. 36 Streaming Collection: Vibe Data Stream Dev Benefits • Central Monitoring Console for Deployment • Fault Tolerant • High Availability • Vertical & Horizontal Scaling • Ease of Configuration
  36. 36. Streaming Collection: Topology View 37
  37. 37. Informatica on Amazon Kinesis • Solving tough infrastructure problems.. • .. So you stay focused on solving tough business problems.. • Coming soon on Amazon.. • Stay tuned.. • Drop an email with your use cases, needs to me at alundberg@informatica.com 38
  38. 38. Today’s Agenda • Informatica Cloud Integration and Data Management • Common Customer Use Cases • Informatica for AWS DB and Big Data Services • Informatica Cloud + Redshift Demonstration • Informatica Vibe Data Stream for Kinesis • Next Steps and Q&A 39
  39. 39. Next Steps • Visit us at Booth# 107 to see more demos! • Get started with Informatica Cloud • InformaticaCloud.com • Learn more about our Redshift Connector • InformaticaCloud.com/Amazon-Redshift 40
  40. 40. Q & A 41 • Ron Lunasin, Informatica Cloud Product Management • Ajay Gandhi, Informatica Cloud Product Marketing • Alan Lundberg, Informatica Vibe Data Stream Marketing @infacloud InformaticaCloud.com

×