IBM's big data solutions like InfoSphere BigInsights and InfoSphere Streams can help organizations gain insights from large, diverse data. BigInsights provides an enhanced Hadoop platform for analyzing structured and unstructured data at scale. Streams enables real-time analysis of high-volume streaming data. The document discusses how these solutions helped clients like Vestas optimize investments using 3 petabytes of data and an Asian telco reduce costs and improve customer experience from 5 billion daily records.
Use dependency injection to get Hadoop *out* of your application codeDataWorks Summit
Hadoop MapReduce provides transparent parallelization but often results in specialized code bases that interact with low-level data formats. We present a means of using dependency injection to manage data flows in MapReduce which in turn supports reusable, Hadoop-agnostic application code that interacts with high-level business domain objects. An example is provided that applies Dependency Injection to the Hadoop WordCount example and shows how the same code invoked from the WordCount MapReduce job can be reused in a real-time context. We then discuss Opower’s application of this pattern to employ the same core calculations in both batch processing and in servicing real-time requests from end users. This topic will be of interest to those interested in reusing core batch calculations in real-time contexts. It also provides a means forward for organizations moving to Hadoop that have existing code components that they would like to employ in batch MapReduce computations.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, ClouderaMongoDB
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Sudhir hadoop and Data warehousing resume Sudhir Saxena
Overall 5.8 Years of Professional IT experience in Data Warehousing and Business Intelligence.
Having one year onsite experience in Mexico and USA with USAA Client, USA.
Having good Knowledge in Bigdata, Hadoop, Pig, Hive, Sqoop, Hbase, Python, Spark, Scala.
Semantic Data Management: a research area of the chair for technologies and management of digital transformation from the university of Wuppertal, Germany.
For more information, see here: https://www.tmdt.uni-wuppertal.de/de
Use dependency injection to get Hadoop *out* of your application codeDataWorks Summit
Hadoop MapReduce provides transparent parallelization but often results in specialized code bases that interact with low-level data formats. We present a means of using dependency injection to manage data flows in MapReduce which in turn supports reusable, Hadoop-agnostic application code that interacts with high-level business domain objects. An example is provided that applies Dependency Injection to the Hadoop WordCount example and shows how the same code invoked from the WordCount MapReduce job can be reused in a real-time context. We then discuss Opower’s application of this pattern to employ the same core calculations in both batch processing and in servicing real-time requests from end users. This topic will be of interest to those interested in reusing core batch calculations in real-time contexts. It also provides a means forward for organizations moving to Hadoop that have existing code components that they would like to employ in batch MapReduce computations.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, ClouderaMongoDB
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Sudhir hadoop and Data warehousing resume Sudhir Saxena
Overall 5.8 Years of Professional IT experience in Data Warehousing and Business Intelligence.
Having one year onsite experience in Mexico and USA with USAA Client, USA.
Having good Knowledge in Bigdata, Hadoop, Pig, Hive, Sqoop, Hbase, Python, Spark, Scala.
Semantic Data Management: a research area of the chair for technologies and management of digital transformation from the university of Wuppertal, Germany.
For more information, see here: https://www.tmdt.uni-wuppertal.de/de
Big Data & Analytics continues to redefine business. Data has transitioned from an underused asset to the lifeblood of the organisation, and a critical component of business intelligence, insight and strategy.
Big Data Scotland is the largest annual data analytics conference held in Scotland: it is supported by ScotlandIS and The Data Lab and free for delegates to attend. The conference is geared towards senior technologists and business leaders and aims to provide a unique forum for knowledge exchange, discussion and cross-pollination.
The programme will explore the evolution of data analytics; looking at key tools and techniques and how these can be applied to deliver practical insight and value. Presentations will span a wide array of topics from Data Wrangling and Visualisation to AI, Chatbots and Industry 4.0.
Key Topics
• Tools and techniques
• Corporate data culture, business processes, digital transformation
• Business intelligence, trends, decision making
• AI, Real-time Analytics, IoT, Industry 4.0, Robotics
• Security, regulation, privacy, consent, anonymization
• Data visualisation, interpretation and communication
• CRM and Personalisation
Extracting value from Big Data is not easy. The field of technologies and vendors is fragmented and rapidly evolving. End-to-end, general purpose solutions that work out of the box don’t exist yet, and Hadoop is no exception. And most companies lack Big Data specialists. The key to unlocking real value lies with thinking smart and hard about the business requirements for a Big Data solution. There is a long list of crucial questions to think about. Is Hadoop really the best solution for all Big Data needs? Should companies run a Hadoop cluster on expensive enterprise-grade storage, or use cheap commodity servers? Should the chosen infrastructure be bare metal or virtualized? The picture becomes even more confusing at the analysis and visualization layer. The answer to Big Data ROI lies somewhere between the herd and nerd mentality. Thinking hard and being smart about each use case as early as possible avoids costly mistakes in choosing hardware and software. This talk will illustrate how Deutsche Telekom follows this segmentation approach to make sure every individual use case drives architecture design and the selection of technologies and vendors.
Driven by data - Why we need a Modern Enterprise Data Analytics PlatformArne Roßmann
In order to turn data into opportunities, you need to build a modern data analytics platform. But because literally everything changes so fast, built-in flexibility is paramount.
This presentation covers:
- how to leverage all your data to generate insights
- the capabilities needed to build a flexible platform
- how to incorporate sustainability requirement
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...Capgemini
Rip and replace isn't a good approach to IT change. When looking at Hadoop, MPP, in-memory and predictive analytics the challenge is making them co-exist with current solutions.
Learn how Capgemini’s Pivotal CoE utilizes Cloud Foundry and PivotalOne to help businesses adopt new technologies without losing the value of current investments.
Presented by Michael Wood of Pivotal and Steve Jones, Global Director, Strategy, Big Data and Analytics, Capgemini, at EMC World 2014.
Operational Analytics Using Spark and NoSQL Data StoresDATAVERSITY
NoSQL data stores have emerged for scalable capture and real-time analysis of data. Apache Spark and Hadoop provide additional scalable analytics processing. This session looks at these technologies and how they can be used to support operational analytics to improve operational effectiveness. It also looks at an example of how operational analytics can be implemented in NoSQL environments using the Basho Data Platform with Apache Spark:
•The emergence of NoSQL, Hadoop and Apache Spark
•NoSQL Use Cases
•The need for operational analytics
•Types of operational analysis
•Key requirements for operational analytics
•Operational analytics using the Basho Data Platform with Apache Spark.
Agile, Automated, Aware: How to Model for SuccessInside Analysis
The Briefing Room with David Loshin and Embarcadero
Live Webcast October 27, 2015
Watch the archive: https://bloorgroup.webex.com/bloorgroup/onstage/g.php?MTID=eea9877b71c653c499c809c5693eae8fe
Data management teams face some tough challenges these days. Organizations need business-driven visibility that enables understanding and awareness of enterprise data assets – without worrying about definitions and change management. But with information architectures evolving into a hybrid mix of data objects and data services built over relational databases as well as big data stores, serving up accurately defined, reusable data can become a complex issue.
Register for this episode of The Briefing Room to learn from veteran Analyst David Loshin as he explains the importance of agile, automated workflows in today’s enterprise. He’ll be briefed by Ron Huizenga of Embarcadero, who will discuss how his company’s ER/Studio suite approaches data modeling and management from a modern architecture standpoint. He will explain that unifying the way information is represented can not only eliminate the need for costly workarounds, but also foster collaboration between data architects, developers and business users.
Visit InsideAnalysis.com for more information.
Hadoop 2.0 - Solving the Data Quality ChallengeInside Analysis
The Briefing Room with Dr. Claudia Imhoff and RedPoint Global
Live Webcast on July 22, 2014
Watch the archive:
https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7bb4cbc33402c3b5f649343052cb9a6d
Whether data is big or small, quality remains the critical characteristic. While traditional approaches to cleansing data have made strides, nonetheless, data quality remains a serious hurdle for all organizations. This is especially true for identity resolution in customer data, but also for a range of other data sets, including social, supply chain, financial and other domains. One of the most promising approaches for solving this decades-old challenge incorporates the power of massive parallel processing, a la Hadoop.
Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Claudia Imhoff, who will explain how Hadoop 2.0 and its YARN architecture can make a serious impact on the previously intractable problem of data quality. She’ll be briefed by George Corugedo of RedPoint Global, who will show how his company’s platform can serve as a super-charged marshaling area for accessing, cleansing and delivering high-quality data. He’ll explain how RedPoint was one of the first applications to be certified for running on YARN, which is the latest rendition of the now-ubiquitous Hadoop.
Visit InsideAnlaysis.com for more information.
Organizations have been collecting, storing, and accessing data from the beginning of computerization. Insights gained from analyzing the data enable them to identify new opportunities, improve core processes, enable continuous learning and differentiation, remain competitive, and thrive in an increasingly challenging business environment.
The well-established data architecture, consisting of a data warehouse, fed from multiple operational data stores, and fronted by BI tools, has served most organizations well. However, over the last two decades, with the explosion of internet-scale data, and the advent of new approaches to data and computational processing, this tried-and-true data architecture has come under strain, and has created both challenges and opportunities for organizations.
In this green paper, we will discuss modern approaches to data architecture that have evolved to address these challenges and provide a framework for companies to build a data architecture and better adapt to increasing demands of the modern business environment. This discussion of data architecture will be tied to the Data Maturity Journey introduced in EQengineered’s June 2021 green paper on Data Modernization.
Microsoft and Hortonworks Delivers the Modern Data Architecture for Big DataHortonworks
Joint webinar with Microsoft and Hortonworns on the power of combining the Hortonworks Data Platform with Microsoft’s ubiquitous Windows, Office, SQL Server, Parallel Data Warehouse, and Azure platform to build the Modern Data Architecture for Big Data.
Big Data International Keynote Speaker Mark van Rijmenam shared his vision on Hadoop Data Lakes during a Zaloni Webinar. What are the Hadoop Data Lake trends for 2016, what are the data lake challenges and how can organizations benefit from data lakes.
Introduce the Big-Data data characteristic, big-data process flow/architecture, and take out an example about EKG solution to explain why we are run into big data issue, and try to build up a big-data server farm architecture. From there, you can have more concrete point of view, what the big-data is.
Exploring the Wider World of Big Data- Vasalis KapsalisNetAppUK
Every second of every day you hear about Electronic systems creating ever increasing quantities of data. Systems in markets such as finance, media, healthcare, government and scientific research feature strongly in the Big Data processing conversation. While extracting business value from Big Data is forecast to bring customer and competitive advantage and benefits. In this session hear Vas Kapsalis, NetApp Big Data Business Development Manager, discuss his views and experience on the wider world of Big Data.
Red hat Open Source Day 2017, Milan - "From Mainframe to Container, a Cloud s...Kiratech
La presentazione, tenuta da Giulio Covassi, CEO di Kiratech, intende fare un excursus storico dall'invenzione dei primi computer al mainframe, dall'inizio dell'informatica moderna con Unix e Linux al Cloud Computing e alla virtualizzazione, fino ain container, all'intelligenza artificiale e il machine Learning.
One key area of Oracle OpenWorld 2016 was data in various shapes. Big Data, streaming data and traditional transactional data. The power of SQL to access and unleash all data - even data in NoSQL databases. The advent of the citizen data scientist. Streaming data analysis in real time on vast and fast and vast data, data discovery. And the new Oracle Database 12cR2 release. Forms, APEX, SQL and PL/SQL.
Big Data & Analytics continues to redefine business. Data has transitioned from an underused asset to the lifeblood of the organisation, and a critical component of business intelligence, insight and strategy.
Big Data Scotland is the largest annual data analytics conference held in Scotland: it is supported by ScotlandIS and The Data Lab and free for delegates to attend. The conference is geared towards senior technologists and business leaders and aims to provide a unique forum for knowledge exchange, discussion and cross-pollination.
The programme will explore the evolution of data analytics; looking at key tools and techniques and how these can be applied to deliver practical insight and value. Presentations will span a wide array of topics from Data Wrangling and Visualisation to AI, Chatbots and Industry 4.0.
Key Topics
• Tools and techniques
• Corporate data culture, business processes, digital transformation
• Business intelligence, trends, decision making
• AI, Real-time Analytics, IoT, Industry 4.0, Robotics
• Security, regulation, privacy, consent, anonymization
• Data visualisation, interpretation and communication
• CRM and Personalisation
Extracting value from Big Data is not easy. The field of technologies and vendors is fragmented and rapidly evolving. End-to-end, general purpose solutions that work out of the box don’t exist yet, and Hadoop is no exception. And most companies lack Big Data specialists. The key to unlocking real value lies with thinking smart and hard about the business requirements for a Big Data solution. There is a long list of crucial questions to think about. Is Hadoop really the best solution for all Big Data needs? Should companies run a Hadoop cluster on expensive enterprise-grade storage, or use cheap commodity servers? Should the chosen infrastructure be bare metal or virtualized? The picture becomes even more confusing at the analysis and visualization layer. The answer to Big Data ROI lies somewhere between the herd and nerd mentality. Thinking hard and being smart about each use case as early as possible avoids costly mistakes in choosing hardware and software. This talk will illustrate how Deutsche Telekom follows this segmentation approach to make sure every individual use case drives architecture design and the selection of technologies and vendors.
Driven by data - Why we need a Modern Enterprise Data Analytics PlatformArne Roßmann
In order to turn data into opportunities, you need to build a modern data analytics platform. But because literally everything changes so fast, built-in flexibility is paramount.
This presentation covers:
- how to leverage all your data to generate insights
- the capabilities needed to build a flexible platform
- how to incorporate sustainability requirement
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...Capgemini
Rip and replace isn't a good approach to IT change. When looking at Hadoop, MPP, in-memory and predictive analytics the challenge is making them co-exist with current solutions.
Learn how Capgemini’s Pivotal CoE utilizes Cloud Foundry and PivotalOne to help businesses adopt new technologies without losing the value of current investments.
Presented by Michael Wood of Pivotal and Steve Jones, Global Director, Strategy, Big Data and Analytics, Capgemini, at EMC World 2014.
Operational Analytics Using Spark and NoSQL Data StoresDATAVERSITY
NoSQL data stores have emerged for scalable capture and real-time analysis of data. Apache Spark and Hadoop provide additional scalable analytics processing. This session looks at these technologies and how they can be used to support operational analytics to improve operational effectiveness. It also looks at an example of how operational analytics can be implemented in NoSQL environments using the Basho Data Platform with Apache Spark:
•The emergence of NoSQL, Hadoop and Apache Spark
•NoSQL Use Cases
•The need for operational analytics
•Types of operational analysis
•Key requirements for operational analytics
•Operational analytics using the Basho Data Platform with Apache Spark.
Agile, Automated, Aware: How to Model for SuccessInside Analysis
The Briefing Room with David Loshin and Embarcadero
Live Webcast October 27, 2015
Watch the archive: https://bloorgroup.webex.com/bloorgroup/onstage/g.php?MTID=eea9877b71c653c499c809c5693eae8fe
Data management teams face some tough challenges these days. Organizations need business-driven visibility that enables understanding and awareness of enterprise data assets – without worrying about definitions and change management. But with information architectures evolving into a hybrid mix of data objects and data services built over relational databases as well as big data stores, serving up accurately defined, reusable data can become a complex issue.
Register for this episode of The Briefing Room to learn from veteran Analyst David Loshin as he explains the importance of agile, automated workflows in today’s enterprise. He’ll be briefed by Ron Huizenga of Embarcadero, who will discuss how his company’s ER/Studio suite approaches data modeling and management from a modern architecture standpoint. He will explain that unifying the way information is represented can not only eliminate the need for costly workarounds, but also foster collaboration between data architects, developers and business users.
Visit InsideAnalysis.com for more information.
Hadoop 2.0 - Solving the Data Quality ChallengeInside Analysis
The Briefing Room with Dr. Claudia Imhoff and RedPoint Global
Live Webcast on July 22, 2014
Watch the archive:
https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7bb4cbc33402c3b5f649343052cb9a6d
Whether data is big or small, quality remains the critical characteristic. While traditional approaches to cleansing data have made strides, nonetheless, data quality remains a serious hurdle for all organizations. This is especially true for identity resolution in customer data, but also for a range of other data sets, including social, supply chain, financial and other domains. One of the most promising approaches for solving this decades-old challenge incorporates the power of massive parallel processing, a la Hadoop.
Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Claudia Imhoff, who will explain how Hadoop 2.0 and its YARN architecture can make a serious impact on the previously intractable problem of data quality. She’ll be briefed by George Corugedo of RedPoint Global, who will show how his company’s platform can serve as a super-charged marshaling area for accessing, cleansing and delivering high-quality data. He’ll explain how RedPoint was one of the first applications to be certified for running on YARN, which is the latest rendition of the now-ubiquitous Hadoop.
Visit InsideAnlaysis.com for more information.
Organizations have been collecting, storing, and accessing data from the beginning of computerization. Insights gained from analyzing the data enable them to identify new opportunities, improve core processes, enable continuous learning and differentiation, remain competitive, and thrive in an increasingly challenging business environment.
The well-established data architecture, consisting of a data warehouse, fed from multiple operational data stores, and fronted by BI tools, has served most organizations well. However, over the last two decades, with the explosion of internet-scale data, and the advent of new approaches to data and computational processing, this tried-and-true data architecture has come under strain, and has created both challenges and opportunities for organizations.
In this green paper, we will discuss modern approaches to data architecture that have evolved to address these challenges and provide a framework for companies to build a data architecture and better adapt to increasing demands of the modern business environment. This discussion of data architecture will be tied to the Data Maturity Journey introduced in EQengineered’s June 2021 green paper on Data Modernization.
Microsoft and Hortonworks Delivers the Modern Data Architecture for Big DataHortonworks
Joint webinar with Microsoft and Hortonworns on the power of combining the Hortonworks Data Platform with Microsoft’s ubiquitous Windows, Office, SQL Server, Parallel Data Warehouse, and Azure platform to build the Modern Data Architecture for Big Data.
Big Data International Keynote Speaker Mark van Rijmenam shared his vision on Hadoop Data Lakes during a Zaloni Webinar. What are the Hadoop Data Lake trends for 2016, what are the data lake challenges and how can organizations benefit from data lakes.
Introduce the Big-Data data characteristic, big-data process flow/architecture, and take out an example about EKG solution to explain why we are run into big data issue, and try to build up a big-data server farm architecture. From there, you can have more concrete point of view, what the big-data is.
Exploring the Wider World of Big Data- Vasalis KapsalisNetAppUK
Every second of every day you hear about Electronic systems creating ever increasing quantities of data. Systems in markets such as finance, media, healthcare, government and scientific research feature strongly in the Big Data processing conversation. While extracting business value from Big Data is forecast to bring customer and competitive advantage and benefits. In this session hear Vas Kapsalis, NetApp Big Data Business Development Manager, discuss his views and experience on the wider world of Big Data.
Red hat Open Source Day 2017, Milan - "From Mainframe to Container, a Cloud s...Kiratech
La presentazione, tenuta da Giulio Covassi, CEO di Kiratech, intende fare un excursus storico dall'invenzione dei primi computer al mainframe, dall'inizio dell'informatica moderna con Unix e Linux al Cloud Computing e alla virtualizzazione, fino ain container, all'intelligenza artificiale e il machine Learning.
One key area of Oracle OpenWorld 2016 was data in various shapes. Big Data, streaming data and traditional transactional data. The power of SQL to access and unleash all data - even data in NoSQL databases. The advent of the citizen data scientist. Streaming data analysis in real time on vast and fast and vast data, data discovery. And the new Oracle Database 12cR2 release. Forms, APEX, SQL and PL/SQL.
Presentation for Taxonomy Bootcamp 2015 by Naomi Oorbeck & Jessica DuVerneay. Covers how taxonomy improved digital products, when to use a lightweight approach, planning & scoping lightweight work, and an overview of key skills and approaches to taxonomy development.
Reference Architecture: EMC Hybrid Cloud with VMwareEMC
This Reference Architecture introduces an EMC Enterprise Private Cloud solution for an on-premises infrastructure as a service (IaaS) offering that enables IT to deliver private cloud-based services to their business. It describes the main features and functionality of the solution and the solution architecture and key components.
모든 것을 연결하여 최적의 인프라 기저를 형성하고 그 위에 빅데이터 저장 및 분석의 기반을 만들어 인공지능을 얹은 글로벌 시장의 리더와 붙을 수 있는 플랫폼을 소개합니다.
멀티벤더 지향의 기본 개념위에 메이저 금융S사의 지원으로 이미 많은 시스템 및 메이저 기업의 IoT데이터를 수집하여 빅데이터 분석을 진행중이며, S사의 로봇 기술 지원 및 향후 Drone의 제어, 데이터의 머신러닝등을 통하여 이미 글로벌 시장을 향하여 나아가고 있습니다.
많은 분들의 관심과 협력 부탁 드립니다.
문의 및 파트너쉽 : contact@littleworld.net
Đây là sự kiện mà ITEC hợp tác cùng cộng đồng AWS Việt Nam tổ chức:
Diễn giả: Bùi Kiên Cường và Quân Phương trình bày.
Thời gian: 22-09-2016, tại Hatch!Nest Hà Nội.
Rio Cloud Computing Meetup 25/01/2017 - Lançamentos do AWS re:Invent 2016Filipe Barretto
Palestra realizada no Rio Cloud Computing Meetup, apresentando os principais lançamentos durante o AWS re:Invent 2016, divulgados nas palestras do Andy Jassy, CEO da AWS, e do Werner Vogels, CTO da Amazon.
2017 GRESB Real Estate Results - The NetherlandsGRESB
2017 GRESB Real Estate Results presentation for The Netherlands, presented on 12 October in De Bilt, hosted by Sweco Capital Consultants
SLides 3-16: Sweco
Monitorización del rendimiento, análisis de configuraciones, seguridad y compresión de un entornos Azure u On-premises, a través de los análisis de logs.
A Strategic View of Enterprise Reporting and Analytics: The Data FunnelInside Analysis
The Briefing Room with Colin White and Jaspersoft
Slides from the Live Webcast on June 12, 2012
As the corporate appetite for analytics and reporting grows, companies must find a way to secure a strategic view of their information architecture. End users with varying degrees of expertise need a wide range of data and reports delivered in a timely fashion. As the audience for analytics expands, that puts pressure on IT infrastructure and staff. And now with the promise of Hadoop and MapReduce, the organization's desire for business insight becomes even more significant.
In this episode of The Briefing Room, veteran Analyst Colin White of BI Research will explain the value of being strategic with enterprise reporting. White will be briefed by Karl Van den Bergh of Jaspersoft, who will tout his company's “data funnel” concept, which is designed to strategically manage an organization's information architecture. By aligning information assets along this funnel, IT can effectively address the spectrum of analytical needs – from simple reporting to complex, ad hoc analysis – without over-taxing personnel and system resources.
Startup pitch presented by co-founder and CEO Jaco Els. Cubitic offers a predictive analytics platform that allows developers to build custom solutions for analytics and visualisation on top of a machine learning engine.
Corporate Senior Vice President, Noriyuki Toyoki, shares Fujitsu’s vision of the increasingly prevalent role technology takes in our daily lives. Everything you ever wanted to know about big data, smart grids, supercomputing and how they can support society through disaster recovery, healthcare ICT and food production - to create a human centric intelligent society.
About ActuateOne for Utility Analytics
Water and Energy Utilities are under tremendous pressure to demonstrate progress in asset optimization, grid optimization and performance gains across traditional business drivers such as customers, revenue protection, utility regulatory compliance and financials. ActuateOne for Utility Analytics provides a comprehensive portfolio of software and utility analytics industry expertise to ensure today’s utility leaders and customers always have access to the right information, insight and collaborative capabilities for accurate and informed decisions. Delivered through a single platform, ActuateOne for Utility Analytics ignites any utility or grid Analytics initiative with integrated asset optimization dashboards, grid optimization dashboards, utility compliance reports as well as Transformer Management Scorecards, Substation & Equipment Management Scorecards and Utility KPI Dashboards which help today’s Utility enhance performance and maximize grid performance.
On May 19, 2020, analyst firm IDC; vendors Intel, MemVerge, Net
App and Penguin Computing; and end-user Credit Suisse, introduced a new category called Big Memory. Big Memory hardware and software together transform scarcity and volatility into abundance, persistence, and high-availability.
AI for Manufacturing (Machine Vision, Edge AI, Federated Learning)byteLAKE
This is the extended presentation about byteLAKE's and Lenovo's Artificial Intelligence solutions for Manufacturing.
Topics covered: AI strategy for manufacturing, Edge AI, Federated Learning and Machine Vision.
It's the first publication in the upcoming series: AI for Manufacturing. Highlights: AI-assisted quality monitoring automation, AI-assisted production line monitoring and issues detection, AI-assisted measurements, Intelligent Cameras and many more. Reach out to us to learn more: welcome@byteLAKE.com.
Presented during the world's first Federated Learning conference (Jun'20). Recording: https://youtu.be/IMqRIi45dDA
Related articles:
- Revolution in factories: Industry 4.0.
https://medium.com/@marcrojek/revolution-in-factories-industry-4-0-conference-made-in-wroclaw-2020-translation-ae96e5e14d55
- Cognitive Automation helps where RPAs fall short.
https://medium.com/@marcrojek/cognitive-automation-helps-where-rpas-fall-short-a1c5a01a66f8
- Machine Vision, how AI brings value to industries.
https://medium.com/@marcrojek/machine-vision-how-ai-brings-value-to-industries-e6a4f8e56f42
Learn more:
- https://www.bytelake.com/en/cognitive-services/
- https://www.lenovo.com/ai
- https://federatedlearningconference.com/
This presentation on Open Source and Cloud Technologies was given by Vizuri SVP Joe Dickman at the 2012 Destination Marketing Technology Forum in Raleigh, NC. For more information please visit our website at www.vizuri.com or email solutions@vizuri.com.
2. Please note
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,
and the workload processed. Therefore, no assurance can be given that an individual user
will achieve results similar to those stated here.
1
3. IBM Montpellier Client Center
Our Client Center partners with
clients to meet their IT infrastructure
goals and improve their overall
business by demonstrating the
capabilities of the IBM solutions.
Smarter Computing Design: Benchmarks & Proofs of Concept:
System Briefings Energy, Cities, Cloud, Water, –PureSystems
Business Resilience –System z
Software Briefings –Power Systems
Enterprise Architecture Design
–HPC
Demonstrations z Key Workload Initiatives –System x & Blade
Advanced Technical Skills –Storage
Industry Showcases Solution Testing
ISV Solution Centers:
BP, ISV & CSI Support SAP, Oracle, Siebel WW GDPS Solution Testing
WW Financial Services CoE Software zTEC
New Technology Introduction
Talk & Teach Design Prove
2
4. Innovation Lab – Resources & Skills
Smarter Cities Innovation through R&D Collaborative Projects supported by CAS
France in partnership with Labs & Clients (funded by Governments or European
Commission) and Client projects
Big Data / BAO & Smarter Cities offerings
Customer Briefings & Workshops: Architecture, Design Session, PoC
Presales technical support: RFP, sizing, pilot support, architecture Showcases Xavier Vasques Virginie Radisson Marie Angèle Grilli Olivier Hess
Manager Business Leader Project Manager CTO
Smarter Energy & Cities: Innovation with a vision of improving Energy
Smarter cities
Consumption through the use of IT and BAO with Universities/company
Montpellier Water Management COE: The use of numerical simulations-HPC for
Water Manager as Deep Thunder from IBM Research first implemented in IBM
Europe by our IBM Montpellier team.
Christophe Menichetti Elsa Fabres Colin Dumontier Romain Chailan
BAO/Big Data Data Analytics & IOC HPC/Water Specialist PhD Student
Specialist Specialist
Saniya Ben Hassen Jean-Philippe Durney Denis Gras
BAO IT Architect BAO/Big Data IT Smarter Cities
Architect IT Architect
Promote and develop innovative assets around Data
applied to Smarter Planet/Cities issues in order to
engage customer and collaborative R&D projects
3
5. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
Big Data in action
> Our Customer Center Showcase experience
4
4
6. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
Big Data in action
> Our Customer Center Showcase experience
5
5
8. Our data rich world is exploding…
4.6
IT: Logs & 30 billion RFID
transactions
billion
tags today camera
Twitter process (1.3B in 2005 phones
7 TBs of world
data every day wide
900
million
GPS
devices
Facebook processes
sold
10 TBs of annually
World Data Centre for Climate data every day
keeps 220 TBS of Web data
by 2013
and 9 PBs of auxiliary
supporting data 2
billion
Capital market people
data volumes grew on the
76 million smart Web by
1,750%, 2003-06 meters in 2009… 2011
200M by 2014 Text, Blog,
Weblog 7
7
9. The Big Data Opportunity
Extracting insight from an immense volume, variety and velocity
of data, in context, beyond what was previously possible.
Variety: Manage the complexity of
multiple relational and non-
relational data types and
schemas
Velocity: Streaming data and large
volume data movement
Volume: Scale from terabytes to
zettabytes (1B TBs)
8
8
8
10. Bring Together a Large Volume and Variety of Data to Find New Insights
Multi-channel customer
sentiment and experience a
analysis
Detect life-threatening
conditions at hospitals in
time to intervene
Predict weather patterns to plan
optimal wind turbine usage, and
optimize capital expenditure on
asset placement
Make risk decisions based on
real-time transactional data
Identify criminals and threats
from disparate video, audio,
and data feeds
9
9
11. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
Big Data in action
> Our Customer Center Showcase experience
10
10
12. Big Data : why is it possible Now ?
Traditional approach : Data to Function Traditional approach
Application server and Database
User request Query Data server are separate
Database Data can be on multiple servers
Application Analysis Program can run on
server
server
multiple Application servers
Network is still a the middle
Send result return Data Data have to go through the network
process Data
Data
•Big Data Approach
Big Data approach : Function to Data Analysis Program runs where are
Query & the data : on Data Node
Send Function to process Data Only the Analysis Program are have
process on Data Data to go through the network
User request Data
nodes Analysis Program need to be
Data
nodes
Master Data
nodes MapReduce aware
node nodes Highly Scalable :
Data
Data 1000s Nodes
Data Petabytes and more
Data
Send Consolidate result
11
11
13. Big Data : why is it possible Now ?
Traditional approach : Data to Function Example :
User request Query Data How many hours Clint Eastwood
Database appears in all the movies he has done ?
Application
server All movies need to be parsed to find
server
Clint face
Send result return Data
Traditional approach : All movies are
process Data going to be sent through the Network
Data
Big Data approach : Function to Data
Query &
• Big Data Approach : Only the
Send Function to process Data Analysis Program and Clint picture are
process on Data Data sent through the Network
User request Data
nodes
Data
nodes
Master Data
nodes
node nodes
Data
Data
Data
Data
Send Consolidate result
12
12
14. Merging the Traditional and Big Data Approaches
Traditional Approach Big Data Approach
Structured & Repeatable Analysis Iterative & Exploratory Analysis
IT
Business Users
Delivers a platform to
Determine what enable creative
question to ask discovery
IT Business Users
Structures the Explores what
data to answer questions could be
that question asked
Monthly sales reports Brand sentiment
Profitability analysis Product strategy
Customer surveys Maximum asset utilization
13
13
15. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
Big Data in action
> Our Customer Center Showcase experience
14
14
16. IBM Big Data platform
Analyse unstructured Big
Data
Analyze structutred Big
Data
Analytic Applications Content Analytics
Cognos BI Reporting Exploration / Functional Industry Predictive
Reporting BIContent Index for contextual collaborative
Reporting / Content
SPSS Visualization App App Analytics Analytics
Analytics insights
Reporting
Create Reports on BigInsights , Analyze
In Streams Simplify your warehouse
Unlock Big Data Big Data Platform PureData Analytics, PureData
Operational Analytics
Infosphere Data Explorer
Visualization Application Systems Deliver deep insight with advanced
Gather, extract and explore data using
best of breed visualization
& Discovery Development Management in-database analytics and operational
analytics
Analyze Raw Rata Accelerators
InfoSphere BigInsights
Infosphere Streams (RT) Index Big Data
Data Explorer
Speed time to value with analytic and Hadoop Stream Data Content Analytics
application accelerators Content
System Computing Warehouse Management Index for contextual collaborative
insights
Reduce costs with Hadoop
PlatForm Computing , GPFS
Cost-effectively analyze Manage Big Data
petabytes of structured and
Gardium, Information Server
unstructured information
Information Integration & Governance Govern data quality and manage
information lifecycle
insights
Analyze Streaming Data
InfoSphere Streams
Cloud | Mobile | Security
Analyze streaming data and large data
bursts for real-time insights 15
17. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
InfoSphere BigInsights
16
16
18. What’s so Special About Open Source Hadoop?
Storage Scalable
• Distributed • New nodes can be added on the fly
• Reliable
• Commodity gear Affordable
• Massively parallel computing on
commodity servers
Flexible
• Hadoop is schema-less – can absorb
MapReduce any type of data
• Parallel Programming
Fault Tolerant
• Fault Tolerant
• Through MapReduce software
framework
17
17
19. Basic Hadoop principles: HDFS and MapReduce
Hadoop Distributed File System = HDFS : where Hadoop stores the data
– This file system spans all the nodes in a cluster
Hadoop computation model
– Data stored in a distributed file system spanning many inexpensive computers
– Bring function to the data
– Distribute application to the compute resources where the data is stored
– Scalable to thousands of nodes and petabytes of data
public static class TokenizerMapper Hadoop Data Nodes
extends Mapper<Object,Text,Text,IntWritable> {
private final static IntWritable
one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text val, Context
StringTokenizer itr =
new StringTokenizer(val.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
}
context.write(word, one); 1. Map Phase
}
} (break job into small parts)
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWrita
private IntWritable result = new IntWritable();
public void reduce(Text key, Distribute map 2. Shuffle
Iterable<IntWritable> val, Context context){
int sum = 0;
for (IntWritable v : val) { tasks to cluster (transfer interim output
sum += v.get();
. . .
for final processing)
MapReduce Application 3. Reduce Phase
(boil all output down to
Shuffle a single result set)
Result Set Return a single result set
18
18
20. InfoSphere BigInsights
Platform for volume, variety, velocity -- V3
Enhanced Hadoop foundation
Analytics for V3
Enterprise Edition
Text analytics & tooling
Licensed
Usability
Web console Business process accelerators (“Apps”)
Integrated install Text analytics
Spreadsheet-style analysis tool
Spreadsheet-style tool
Enterprise class
RDBMS, warehouse connectivity
Ready-made “apps”
Integrated Web-based console
Enterprise Class Basic Edition Flexible job scheduler
Storage, security, cluster Performance enhancements
Free download
management Eclipse-based tooling
Integration Integrated install LDAP authentication
Online InfoCenter
Connectivity to DB2, Netezza, JDBC ....
BigData Univ.
databases, SPSS, Cognos, Unica, Apache
Hadoop
coremetrics, Streams, Datastage
Breadth of capabilities 19
19
21. InfoSphere BigInsights – A Full Hadoop Stack
Open Source Components IBM Specific Components
20
20
22. Vestas optimizes
capital investments
based on 3 Petabytes
of information.
Capabilities Utilized:
InfoSphere BigInsights
InfoSphere Warehouse
• Model the weather to optimize
placement of turbines, maximizing
power generation and longevity.
• Reduce time required to identify
placement of turbine from weeks to
hours.
• Incorporate 3 PB of structured and
semi-structured information flows.
• Data volume expected to grow to 6 PB.
21
21
2
23. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
InfoSphere Streams
>
22
22
24. IBM InfoSphere Streams for companies who need to…
Real-time delivery
Deal with Terabytes of data each
second ICU Environment
Monitoring
Monitoring
Work with application, sensor and Algo Powerful Telco churn
Analytics predict
internet data, video/audio Trading
Cyber Smart
Security Government / Grid
Deliver insight in microseconds to Law enforcement
analytical applications
Support complex scenarios using Millions of
events per Microsecond
C++ or Java code Latency
second
Integrate with existing analytics
& data warehousing investments
Traditional /
Non-traditional
data sources
23
23
25. Stream Computing – Analyze Data in Motion
Traditional Computing Stream Computing
Historical fact finding Current fact finding
Find and analyze information stored on disk Analyze data in motion – before it is stored
Batch paradigm, pull model Low latency paradigm, push model
Query-driven: submits queries to static data Data driven – bring the data to the query
24
24
26. Big Data in Real-Time with InfoSphere Streams
Filter / Sample
Modify Annotate
Fuse
Classify
25
25
27. Asian Telco reduces
billing costs and
improves customer
satisfaction
Capabilities:
Stream Computing
Analytic Accelerators
Real-time mediation and analysis of
5B CDRs per day
Data processing time reduced from
12 hrs to 1 min
Hardware cost reduced to 1/8th
Proactively address issues
(e.g. dropped calls) impacting customer
satisfaction.
26
26
2
28. Most Use Cases Combine Technologies
Variety
Volume
Combination of Streams filters
Non-traditional/ incoming data
internet data
with traditional
data
InfoSphere BigInsights
InfoSphere Streams
Traditional
Data Reuse Warehouse
Analytic models
IBM Data Warehouse Velocity
Persistent Data In-Motion Data
27
27
29. Big Data Patterns
Common Big Data and Warehouse patterns
Separate unstructured & structured analysis Common analysis of structured and unstructured
data
App /BI App / BI App / BI
Visualization Visualization Visualization / Exploration
Exploration Exploration
BigInsights Warehouse
BigInsights Warehouse
Unstructured Structured
Unstructured Structured
Warehouse and BigInsights partitioning Warehouse batch offload
App / BI App / BI
App / BI
Visualization Visualization
Visualization
Exploration Exploration
Exploration
Warehouse BigInsights BigInsights
Warehouse
Structured
Structured 28
30. Big Data Patterns
Common Big Data and Warehouse patterns
In motion, at rest analysis with BigInsights In motion and at rest applications
Real time App Analytic App / Real time App / BI
/ BI BI
Streams BigInsights
Streams BigInsights
Warehouse
Warehouse
Streaming data Streaming data
In motion, at rest analysis of structured and In motion, structured at rest analysis
unstructured data
Real time App Real time App Analytic App / BI
Analytic App / BI
/ BI / BI
BigInsights Warehouse Streams BigInsights Warehouse
Streams
Streaming data Unstructured Structured Streaming data Structured
data data data
29
31. AGENDA
Big Data Challenges
> Why the interest is growing?
Big Data Technologies
> What is Big Data ?
IBM Big Data Solutions
> IBM Big Insights and IBM Streams
Big Data in action
> Our Customer Center Showcase experience
30
30
32. Big Data Use Cases and customer outcomes
Findings from the research collaboration of IBM Institute for Business Value and Saïd Business School, University of Oxford
Big data objectives Big data sources
Customer-centric outcomes New business model
Respondents were
Operational optimization Employee collaboration asked which data
Risk / financial management sources are currently
being collected and
Top functional objectives identified by organizations with active big analyzed as part of
data pilots or implementations. Responses have been weighted active big data efforts
and aggregated. within their organization.
31
33. Operations / Performance Data is Exploding
A typical enterprise with 5000 servers, running 125 applications across 2 to 3
data centers generates in excess of 1.3 TB of data per day
Data Ratio
Only 3% of the data generate is operations Metric Data Unstructured Data
oriented metric data. 3%
97% is made up of unstructured/semi
structured data 97%
Workloads are running on heterogeneous
platforms.
32
34. Log Analysis: Problem Characteristics
Several thousand log files collected daily, data collected over several years
Infrastructure (Servers, Networks, Storage), Middleware (App Server, Web Server, Database Server,
Messaging Server), Apps
Value in collocating and co-analyzing the above data
Millions of files, petabytes of data in total, terabytes produced per day.
The relationships between logs (links shown below) have to be discovered
Large percentage of storage in an enterprise is for log data
Analysis of log data has many challenges
One replica stops
responding... Collection and parsing of data
App
2
App
Server Interpretation of logs
App Load
2 Balancer
Replicated
Database SMEs flooded with common bugs
...causing a fraction of database calls to time out...
Lack of a joined up view.
...which leads to intermittent failures in the
application. Reactive rather than proactive
33
35. Central Lab Platform – Before
The consequence of scattered Infrastructures for hands-on classes are
high costs and business transformation roadblocks
34
34
36. Central Lab Platform – After
The scattered infrastructures were transformed into a
centralized consolidated hands-on Cloud Platform
35
35
37. Central Lab Platform Cloud Architecture
SELF-SERVE SERVICE SERVICE DYNAMIC
PORTAL REQUEST PROVISIONING INFRASTRUCTURE
Class
Manager Teacher & Students
Management
CLP Cloud Management Front-end Internet
access
Web Portal
Planning VPN
Reporting
Invoicing
Reservation
CLP Application engine Setup
manager
Shared CLP
Resources
TA CLP
TPM
Daily repl.
Workflows
TA DB CLP DB & Scripts
36
39. Big Data Project Trends & Directions
2 Majors Front End objective to demonstrate Big Data Benefit
Navigating Enterprise Information: “Leverage Big Data Business Value”
• 360° Operational View : To accelerate incident resolution
• 360° Business View : To provide metric and Insight
– Cloud Data Center utilization : Data Center Business View
– Training Labs : Data Center’s Customer Business View
Predictive Incident Alerting : “Act Proactively on Incident”
• Create Predictive Models based on log history to alert before Incident
arrived
• Reduce number of Incident Tickets
38
40. How support Team Work today : Many applications / Information dispersion
39
41. Navigating Enterprise Information: 360° Operational View
About | Help | Profile | Logout - Durney
Power System System X System Z Storage Software
Sort by: Date Relevance Title
Search: 153494
Your query has been expanded. Show Expansions
0 documents selected. Select/deselect all on this page Global Status
Documentation Service Warn Error Down Up
Top 76 Results
Ticket Citrix 0 0 0 10
Creator ID Assignee Status Priority Course code Class # Contact Network 0 0 0 180
Lab Setup Guide (4)
Nick Yabut 153494 Jean-Philippe Durney Open 2 AN14GB H65X Martin Elliff Storage 0 0 0 170
Courses Exercices (3)
need to rebuild LPAR2 for this course (sys5442_lpar2), but can't log into the class NIM server Phone # Master
Production documentation (10)
nim151 (10.6.151.35). It appears to be off-line and it is not showing on the managed system. Cell # servers 2 0 0 4
Best Practice (3) Emailmartin.elliff@mail.com Nim 4 0 56 87
Citrix (3) Sametime ID inst151 TPM 2 0 0 1
Provisionning (10)
Storage (4) Course Schedule
Open Tickets
TSM (3)
Overview (4) Sev 1 Sev 2 Sev 3
Processes (5) 5
Tech Choices (12) 4
How To (15)
3
more | all
2
Lotus Notes 1
Re: AN14 scripts on LPAR 10nov.2012 0
I have copied a tar file with all the script for the an14
course on you nim server "sys3862_nim1" in Ticket on AN14 h-24 h-12 h-6 h-1
/home/an14. ... AN14 scripts on LPAR Sent by: ID Assignee Status Priority Course code Class # Contact
Jeffrey Emmanuel D ...
153301 Pascal Seignez Closed 2 AN14GB H65X Martin Elliff
Re: Ticket #123078 course AN14 ref 8849/E9D4/9416
26fév.2012 Access
AN14 ref 8849/E9D4/9416 Hello We have sent 3 IBM CLP class information: AN14GB / H65X (Jan. 21, 2013, 12:00 PM)sys5442 -- We have found that when a device
course kits : IBM CLP class ... is deleted from any of the LPARs (rmdev -dl hdisk2), cfgmgr has to be run twice to bring the device back online. I'm Top 11 Results
sure this is not standard behaviour. Can you explain why this is happening?
mime.htm (Ticket #123078 Updated (IBM Problem Tracking & ...
25fév.2012
AN14 class number E9D4 - customer sent message 150000 Pascal Seignez Closed 2 AN14GB 9023 Amin Ezzy HMC (4)
on St to request for 4 more additional ... AN14 class TPMHMC (3)
number E9D4, and also applid for two more kits ST: AN14G 9023 all students could not log in to the HMCs, username / password error
because the students ... Course HMC (1)
HMC Power down 03Jan.2013 NIM (2)
Pour le cours an14, ZRGV, l'instructeur demande •nim_master
pourquoi les lpars sont en AIX 7. … 60906 2nd Level Support Closed 2 AN14GB 2861 Martin Elliff
•nim151
Customer has issues on course AN14GB/2861. Storage (2)
more | all CLP Servers (3)
Citrix (1)
59861 Jean Midot Closed 2 AN140 VYRM Ben Gibbs Admin Tools (5)
Unable to log in to citrix (elabs), UID and PW not working : error : invalid credentials for all one example : UID :
stud148_1 pw : dayheat_67 more | all
40
42. Navigating Enterprise Information: 360° Business View
About | Help | Profile | Logout - Durney
Power System System X System Z Storage Software
Sort by: Date Relevance Title
Search:
Your query has been expanded. Show Expansions
Show Metrics
• Number of Running courses versus Number of Logged Students (typically Extract through Big Data Log
Analysis)
• Cumulative time usage per course/session
• Servers, Storage usage
• Electric consumption
Create Correlated views
• Electric Consumption versus number of courses running
• Consolidate view per by Global Training Partner
Analyze operation
• Number of Ticket per Courses Brand, per Course, per Geo
• Average Resolution time per Incident type
• Top 10 incident per frequency
• Top 10 incident per Geo
• Top 10 course per Geo
41
41
43. To learn more and deeper
IBM Tivoli Product to monitor and analyse machine logs:
> IBM Log Analytics
Download the presentation
on Pulse2013 site
Session 1844 :
Problem Determination and
Resolution in Minutes Using
Unstructured Data Analytics
Martin O’Brien - Product
Manager
Geetha Adinarayan - Client
Best Practices Lead
42