Uploaded byImpetus Technologies
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAnalytix Webinar
The document discusses the evolution and implementation of a 'smart enterprise big data bus' designed to enhance data synchronization and analytics within modern enterprises. It highlights the challenges posed by siloed applications in the rapidly evolving big data landscape, particularly with Hadoop as a central component. The implementation of this solution, including various technologies and architectures, aims to facilitate real-time responses and efficient data integration across enterprise systems.
More Related Content
Similar to Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAnalytix Webinar
Smart Enterprise Big Data Bus for the Modern Responsive Enterprise- StreamAnalytix Webinar
- 1. Smart Enterprise BigData Bus ForThe Modern Responsive Enterprise Anand Venugopal Sr. Director - StreamAnalytix Larry Pearson V.P. of Marketing Recorded version available at http://bit.ly/1FND9fe
- 2. The Journey SoFar… • Impetus Technologies – Leading Big Data Solutions Provider • StreamAnalytix – Enterprise Class Streaming Analytics platform • Early Access Program: Aug 2014 • Pilot Projects: October 2014 – February 2015 • GA Launch: February 2015 • Today – Sharing one of the key insights and use-case patterns from pilot Recorded version available at http://bit.ly/1FND9fe
- 3. Smart Enterprise BigData Bus Agenda Why ? Case Study Emerging Big Data Landscape and its Challenges What ? Solution Characteristics Components Required How ? Implementation Using StreamAnalytix Technology Stack Q&A Summary Follow up Recorded version available at http://bit.ly/1FND9fe
- 4. Case Study -How It Began Enterprise Data Hub/Lake Hadoop App-1 App-2 App-3 Enterprise NoSQL Healthcare giant is speeding up critical business processes by using a streaming analytics platform for real-time data synchronization between their Hadoop platform and their enterprise NoSQL database Recorded version available at http://bit.ly/1FND9fe
- 5. Emerging Enterprise BigData Landscape • Hadoop Stack – Becoming the Center of the Enterprise Data Universe • Business Critical Applications Are Still ''Silos'' • Real-time Streaming Analytics Adoption Rapidly Increasing Recorded version available at http://bit.ly/1FND9fe
- 6. Hadoop Active Archive ETL Single Source of Truth BI + Analytics Streams/ Transactions Hadoop– It Makes Coffee Too! Recorded version available at http://bit.ly/1FND9fe
- 7. Hadoop Architecture View:Cloudera Recorded version available at http://bit.ly/1FND9fe
- 8. Hadoop Architecture View:Hortonworks Recorded version available at http://bit.ly/1FND9fe
- 9. Hadoop Architecture View:MapR Recorded version available at http://bit.ly/1FND9fe
- 10. What’s Missing FromThese Pictures? Recorded version available at http://bit.ly/1FND9fe
- 11. How Hadoop ConnectsTo The Rest Of It All ! Enterprise Data Hub/Lake Hadoop ERP CRM Current EDW/ ADW App Server 1 App Server 2 Hub and Spoke Architecture The Integration Challenge – Developing, Maintaining, Managing the Hub-spoke System HadoopActive Archive ETL Single Source of Truth BI + Analytics Streams/ Transactions Recorded version available at http://bit.ly/1FND9fe
- 12. Emerging Enterprise BigData Landscape • Hadoop Stack – Becoming the Center of the Enterprise Data Universe • Business Critical Applications Are Still ''Silos'' • Real-time Streaming Analytics Adoption Rapidly Increasing Recorded version available at http://bit.ly/1FND9fe
- 13. Siloed Enterprise Applications MayTake Many Years To Integrate With Hadoop Provisioning Transaction Processing Billing Customer Service Recorded version available at http://bit.ly/1FND9fe
- 14. The Data IntegrationChallenge Is Here To Stay Enterprise Data Hub/Lake Hadoop ERP CRM Current EDW/ ADW App Server 1 App Server 2 Hub and Spoke Architecture The Integration Challenge – Developing, Maintaining, Managing the Hub-spoke System Recorded version available at http://bit.ly/1FND9fe
- 15. Emerging Enterprise BigData Landscape • Hadoop Stack – Becoming the Center of the Enterprise Data Universe • Business Critical Applications Are Still ''Silos'' • Real-time Streaming Analytics Adoption Rapidly Increasing Recorded version available at http://bit.ly/1FND9fe
- 16. • Growth ofInternet of Things (IoT) and Sensor/ Machine Data Sources • Context-sensitive Customer Service Sales Web Site Billing Customer Service The Modern Enterprise Expected To Be ''Real-time'' Or ''Near Real-time'' Recorded version available at http://bit.ly/1FND9fe
- 17. • Mobile LocationBased Offers • Internet Advertisements • Call-center Agent Interactions The Modern Enterprise Expected To Be ''Real-time'' Or ''Near Real-time'' Sales Web Site Billing Customer Service Recorded version available at http://bit.ly/1FND9fe
- 18. Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop Real-timeResponses Need Hi-speed Any-to-Any Data Synchronization & Analytics Over Streaming Data Developing and managing all the peer-to-peer data interfaces and the hub-spoke model together would be very hard Recorded version available at http://bit.ly/1FND9fe
- 19. Emerging Enterprise BigData Landscape • Hadoop Stack – Becoming the Center of the Enterprise Data Universe • Business Critical Applications Are Still ''Silos'' • Real-time Streaming Analytics Adoption Rapidly Increasing Recorded version available at http://bit.ly/1FND9fe
- 20. Smart Enterprise BigData Bus Agenda Why ? Case Study Emerging Big Data Landscape and its Challenges What ? Solution Characteristics Components Required How ? Implementation Using StreamAnalytix Technology Stack Q&A Summary Follow up Recorded version available at http://bit.ly/1FND9fe
- 21. Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop AHi-speed Big Data Bus Architecture Would Be An Efficient Any-to-Any Data Synchronization Mechanism Recorded version available at http://bit.ly/1FND9fe
- 22. Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop AHi-speed Big Data Bus Architecture Would Be An Efficient Any-to-Any Data Synchronization Mechanism Recorded version available at http://bit.ly/1FND9fe
- 23. Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop AHi-speed Big Data Bus Architecture Would Be An Efficient Any-to-Any Data Synchronization Mechanism Recorded version available at http://bit.ly/1FND9fe
- 24. Provisioning Machine Data Processing Billing Customer Service Enterprise Data Hub/Lake Hadoop TheHi-speed Big Data Bus Would Also Need To Support ''On The Wire'' Computation For Data Analytics, Transforms, CEP Transformation Analytics, Alerting Recorded version available at http://bit.ly/1FND9fe
- 25. Capability List forthe ''Smart Enterprise Big Data Bus'' • Ingest • Parse • Filter • Transform • Move • Store • Read • Synchronize • Analyse • Predict • Alert • Visualise AT SCALE, AND FAST ! Provisioning Machine Data Processing Billing Enterprise Data Hub/Lake Hadoop Transformation Analytics, AlertingCustomer Service Recorded version available at http://bit.ly/1FND9fe
- 26. Is this real?? Case Studies Recorded version available at http://bit.ly/1FND9fe
- 27. Is this real?? Case Studies Read-Write Adapters Stream Processing Services provided by the ''Smart Enterprise Big Data Bus'' include UI for Work-flow Orchestration, Management and Monitoring ''Stations'' in the Data Transit System Reliable, Fault-tolerant, Elastic Scalable Distributed Stream Processing and Transport Fabric Recorded version available at http://bit.ly/1FND9fe
- 28. ESB (vs. SmartEnterprise Big Data Bus) • Were architected for a different workload in a different era • Designed for light weight remote service invocations – not as a heavy throughput full peer-to-peer data transfer mechanism • No compute / analytics capability on the wire • Expensive vertical scaling vs. distributed elastic scale-out with commodity hardware • Monolithic workflows vs. independent control and elastic scalability of each stage in a workflow based on compute needs Recorded version available at http://bit.ly/1FND9fe
- 29. Smart Enterprise BigData Bus Agenda Why ? Case Study Emerging Big Data Landscape and its Challenges What ? Solution Characteristics Components Required How ? Implementation using StreamAnalytix Technology Stack Q&A Summary Follow up Recorded version available at http://bit.ly/1FND9fe
- 30. Smart Enterprise BigData Bus Implementation • Kafka • Rabbit MQ • Apache Storm • Operators • RT Dashoards • Websockets • CEP • Filter • Indexer • NoSQL • HDFS, Hbase • PMML AT SCALE, FAST, EASY ! Recorded version available at http://bit.ly/1FND9fe
- 31. StreamAnalytix Architecture BlockDiagram Recorded version available at http://bit.ly/1FND9fe
- 32. StreamAnalytix – EnterpriseBig Data Bus – Creation Recorded version available at http://bit.ly/1FND9fe
- 33. Data Fabric -2 Recorded version available at http://bit.ly/1FND9fe
- 34. Monitoring Screen Recorded versionavailable at http://bit.ly/1FND9fe
- 35. Real-Time Dashboard Recorded versionavailable at http://bit.ly/1FND9fe
- 36. Sample Deployment/Hardware Specification Externalsystems Data Store Integration layer Processing and Analytics layer Storm RHEL 4 cores, 16GB RAM vm06dev222 x 1 ElasticSearch RHEL 8 cores, 32GB RAM vm06dev218 x 1 PostgreSQL RHEL 4 cores, 8GB RAM vm06dev222 x 1 Meta Information Store Zookeeper x 1 Kafka+ZK RHEL 4 cores, and 8GB RAM vm06dev220 x 1 Free Switch x 2 Graphite x 1 Web UI Layer RHEL 4 cores, and 2GB RAM vm07eng29, vm07eng31 IR(Platform) x 1 RHEL 4 cores, and 2GB RAM vm07dev97 RHEL 4 cores, 16GB RAM vm06dev222 RHEL 4 cores, 4GB RAM vm06dev224 Tomcat(Query Server) RHEL 4 cores, 8GB RAM vm06dev218 x 1 Tomcat(Admin+Log UI) RHEL 4 cores, 16GB RAM vm06dev222 x 1 Couchbase RHEL 8 cores, 32GB RAM vm06dev224 x 1 RabbitMQ RHEL 4 cores, and 4GB RAM vm06dev224 x 1
- 37. www.streamanalytix.com Email: inquiry@streamanalytix.com ? Recorded versionavailable at http://bit.ly/1FND9fe