SlideShare a Scribd company logo

Cosmos, Big Data GE Implementation

FIWARE
FIWARE
FIWAREFIWARE

Cosmos, Big Data GE Implementation

1 of 52
Download to read offline
Cosmos, Big Data GE implementation 
Building your first application using FI-WARE 
Open APIs for Open Minds
Big Data: 
What is it and how 
much data is there 
1
What is big data? 
2 
> small 
data
What is big data? 
3 
http://commons.wikimedia.org/wiki/File:Interior_view_of_Stockholm_Public_Library.jpg 
> big data
How much data is there? 
4
Data growing forecast 
5 
2.3 
3.6 
12 
19 
11.3 
39 
0.5 
1.4 
Global 
users 
(billions) 
Global networked 
devices 
(billions) 
Global broadband 
speed 
(Mbps) 
Global traffic 
(zettabytes) 
http://www.cisco.com/en/US/netsol/ns827/networking_solutions_sub_solution.html#~forecast 
2012 
2012 
2012 
2012 
2017 
2017 
2017 
2017

Recommended

Fiware io t_ul20_cpbr8
Fiware io t_ul20_cpbr8Fiware io t_ul20_cpbr8
Fiware io t_ul20_cpbr8FIWARE
 
FIWARE Developers Week_Bootcamp_presentation
FIWARE Developers Week_Bootcamp_presentationFIWARE Developers Week_Bootcamp_presentation
FIWARE Developers Week_Bootcamp_presentationFIWARE
 
Building Your Own IoT Platform using FIWARE GEis
Building Your Own IoT Platform using FIWARE GEisBuilding Your Own IoT Platform using FIWARE GEis
Building Your Own IoT Platform using FIWARE GEisFIWARE
 
How to Install & Configure Your Own Identity Manager GE
How to Install & Configure Your Own Identity Manager GEHow to Install & Configure Your Own Identity Manager GE
How to Install & Configure Your Own Identity Manager GEFIWARE
 
The 'Serverless' Paradigm, OpenWhisk and FIWARE
The 'Serverless' Paradigm, OpenWhisk and FIWAREThe 'Serverless' Paradigm, OpenWhisk and FIWARE
The 'Serverless' Paradigm, OpenWhisk and FIWAREFIWARE
 
Intro to fog and openstack jp
Intro to fog and openstack jpIntro to fog and openstack jp
Intro to fog and openstack jpSatoshi Konno
 
How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2Fernando Lopez Aguilar
 

More Related Content

What's hot

What is new in icinga2
What is new in icinga2What is new in icinga2
What is new in icinga2Icinga
 
Icinga2 Hacking Session 2014-10-10
Icinga2 Hacking Session 2014-10-10Icinga2 Hacking Session 2014-10-10
Icinga2 Hacking Session 2014-10-10Icinga
 
Icinga2 - Upcoming API for Icinga2
Icinga2 - Upcoming API for Icinga2Icinga2 - Upcoming API for Icinga2
Icinga2 - Upcoming API for Icinga2Icinga
 
Orion Context Broker Exercises
Orion Context Broker ExercisesOrion Context Broker Exercises
Orion Context Broker ExercisesFermin Galan
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGIMike Pittaro
 
Как мы взломали распределенные системы конфигурационного управления
Как мы взломали распределенные системы конфигурационного управленияКак мы взломали распределенные системы конфигурационного управления
Как мы взломали распределенные системы конфигурационного управленияPositive Hack Days
 
Implementing a build manager in Ada
Implementing a build manager in AdaImplementing a build manager in Ada
Implementing a build manager in AdaStephane Carrez
 
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery BlueHat Security Conference
 
Arduino、Web 到 IoT
Arduino、Web 到 IoTArduino、Web 到 IoT
Arduino、Web 到 IoTJustin Lin
 
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrate
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrateToshaan Bharvani – Icinga2 and Ansible, how to manage and migrate
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrateIcinga
 
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade ServerlessKatyShimizu
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamCodemotion
 
Test Driven Documentation with Spring Rest Docs JEEConf2017
Test Driven Documentation with Spring Rest Docs JEEConf2017Test Driven Documentation with Spring Rest Docs JEEConf2017
Test Driven Documentation with Spring Rest Docs JEEConf2017Roman Tsypuk
 
Monitoring OSGi Applications with the Web Console - Carsten Ziegeler
Monitoring OSGi Applications with the Web Console - Carsten ZiegelerMonitoring OSGi Applications with the Web Console - Carsten Ziegeler
Monitoring OSGi Applications with the Web Console - Carsten Ziegelermfrancis
 
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...mfrancis
 
Swift hardware hacking @ try! Swift
Swift hardware hacking @ try! SwiftSwift hardware hacking @ try! Swift
Swift hardware hacking @ try! SwiftSally Shepard
 
Shindig in 2 hours
Shindig in 2 hoursShindig in 2 hours
Shindig in 2 hourshanhvi
 
How to Use Cryptography Properly: Common Mistakes People Make When Using Cry...
How to Use Cryptography Properly:  Common Mistakes People Make When Using Cry...How to Use Cryptography Properly:  Common Mistakes People Make When Using Cry...
How to Use Cryptography Properly: Common Mistakes People Make When Using Cry...All Things Open
 
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech Talk
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech TalkHacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech Talk
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech TalkRed Hat Developers
 

What's hot (20)

What is new in icinga2
What is new in icinga2What is new in icinga2
What is new in icinga2
 
Icinga2 Hacking Session 2014-10-10
Icinga2 Hacking Session 2014-10-10Icinga2 Hacking Session 2014-10-10
Icinga2 Hacking Session 2014-10-10
 
Icinga2 - Upcoming API for Icinga2
Icinga2 - Upcoming API for Icinga2Icinga2 - Upcoming API for Icinga2
Icinga2 - Upcoming API for Icinga2
 
Orion Context Broker Exercises
Orion Context Broker ExercisesOrion Context Broker Exercises
Orion Context Broker Exercises
 
OpenStack API's and WSGI
OpenStack API's and WSGIOpenStack API's and WSGI
OpenStack API's and WSGI
 
Как мы взломали распределенные системы конфигурационного управления
Как мы взломали распределенные системы конфигурационного управленияКак мы взломали распределенные системы конфигурационного управления
Как мы взломали распределенные системы конфигурационного управления
 
Finatra v2
Finatra v2Finatra v2
Finatra v2
 
Implementing a build manager in Ada
Implementing a build manager in AdaImplementing a build manager in Ada
Implementing a build manager in Ada
 
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery
BlueHat v17 || Detecting Compromise on Windows Endpoints with Osquery
 
Arduino、Web 到 IoT
Arduino、Web 到 IoTArduino、Web 到 IoT
Arduino、Web 到 IoT
 
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrate
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrateToshaan Bharvani – Icinga2 and Ansible, how to manage and migrate
Toshaan Bharvani – Icinga2 and Ansible, how to manage and migrate
 
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time stream
 
Test Driven Documentation with Spring Rest Docs JEEConf2017
Test Driven Documentation with Spring Rest Docs JEEConf2017Test Driven Documentation with Spring Rest Docs JEEConf2017
Test Driven Documentation with Spring Rest Docs JEEConf2017
 
Monitoring OSGi Applications with the Web Console - Carsten Ziegeler
Monitoring OSGi Applications with the Web Console - Carsten ZiegelerMonitoring OSGi Applications with the Web Console - Carsten Ziegeler
Monitoring OSGi Applications with the Web Console - Carsten Ziegeler
 
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...
Leveraging the strength of OSGi to deliver a convergent IoT Ecosystem - O Log...
 
Swift hardware hacking @ try! Swift
Swift hardware hacking @ try! SwiftSwift hardware hacking @ try! Swift
Swift hardware hacking @ try! Swift
 
Shindig in 2 hours
Shindig in 2 hoursShindig in 2 hours
Shindig in 2 hours
 
How to Use Cryptography Properly: Common Mistakes People Make When Using Cry...
How to Use Cryptography Properly:  Common Mistakes People Make When Using Cry...How to Use Cryptography Properly:  Common Mistakes People Make When Using Cry...
How to Use Cryptography Properly: Common Mistakes People Make When Using Cry...
 
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech Talk
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech TalkHacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech Talk
Hacking the Mesh: Extending Istio with WebAssembly Modules | DevNation Tech Talk
 

Viewers also liked

FIWARE Developers Week_ Introduction to Managing Context Information at Large...
FIWARE Developers Week_ Introduction to Managing Context Information at Large...FIWARE Developers Week_ Introduction to Managing Context Information at Large...
FIWARE Developers Week_ Introduction to Managing Context Information at Large...FIWARE
 
FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...
 FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M... FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...
FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...FIWARE
 
Developing your first application using FIWARE
Developing your first application using FIWAREDeveloping your first application using FIWARE
Developing your first application using FIWAREFIWARE
 
Io t idas_intro_ul20_nobkg
Io t idas_intro_ul20_nobkgIo t idas_intro_ul20_nobkg
Io t idas_intro_ul20_nobkgFIWARE
 
FIWARE IoT Proposal & Community
FIWARE IoT Proposal & CommunityFIWARE IoT Proposal & Community
FIWARE IoT Proposal & CommunityFIWARE
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013SATOSHI TAGOMORI
 
IoT Agents (Introduction)
IoT Agents (Introduction)IoT Agents (Introduction)
IoT Agents (Introduction)dmoranj
 
FIWARE IoT Ready Programme
FIWARE IoT Ready ProgrammeFIWARE IoT Ready Programme
FIWARE IoT Ready ProgrammeFIWARE
 
Developing your first application using FI-WARE
Developing your first application using FI-WAREDeveloping your first application using FI-WARE
Developing your first application using FI-WAREFermin Galan
 
FIWARE Developers Week_IoT basic exercises
FIWARE Developers Week_IoT basic exercisesFIWARE Developers Week_IoT basic exercises
FIWARE Developers Week_IoT basic exercisesFIWARE
 
Fiware Developers Week Iot exercises (Advanced)
Fiware Developers Week Iot exercises (Advanced)Fiware Developers Week Iot exercises (Advanced)
Fiware Developers Week Iot exercises (Advanced)dmoranj
 
Fiware IoT_IDAS_intro_ul20_v2
Fiware IoT_IDAS_intro_ul20_v2Fiware IoT_IDAS_intro_ul20_v2
Fiware IoT_IDAS_intro_ul20_v2FIWARE
 

Viewers also liked (13)

FIWARE Developers Week_ Introduction to Managing Context Information at Large...
FIWARE Developers Week_ Introduction to Managing Context Information at Large...FIWARE Developers Week_ Introduction to Managing Context Information at Large...
FIWARE Developers Week_ Introduction to Managing Context Information at Large...
 
FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...
 FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M... FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...
FIWARE Developers Week_IoT Agents with Thinking Things and OMA lightweight M...
 
Developing your first application using FIWARE
Developing your first application using FIWAREDeveloping your first application using FIWARE
Developing your first application using FIWARE
 
Io t idas_intro_ul20_nobkg
Io t idas_intro_ul20_nobkgIo t idas_intro_ul20_nobkg
Io t idas_intro_ul20_nobkg
 
Fluentd and WebHDFS
Fluentd and WebHDFSFluentd and WebHDFS
Fluentd and WebHDFS
 
FIWARE IoT Proposal & Community
FIWARE IoT Proposal & CommunityFIWARE IoT Proposal & Community
FIWARE IoT Proposal & Community
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013
 
IoT Agents (Introduction)
IoT Agents (Introduction)IoT Agents (Introduction)
IoT Agents (Introduction)
 
FIWARE IoT Ready Programme
FIWARE IoT Ready ProgrammeFIWARE IoT Ready Programme
FIWARE IoT Ready Programme
 
Developing your first application using FI-WARE
Developing your first application using FI-WAREDeveloping your first application using FI-WARE
Developing your first application using FI-WARE
 
FIWARE Developers Week_IoT basic exercises
FIWARE Developers Week_IoT basic exercisesFIWARE Developers Week_IoT basic exercises
FIWARE Developers Week_IoT basic exercises
 
Fiware Developers Week Iot exercises (Advanced)
Fiware Developers Week Iot exercises (Advanced)Fiware Developers Week Iot exercises (Advanced)
Fiware Developers Week Iot exercises (Advanced)
 
Fiware IoT_IDAS_intro_ul20_v2
Fiware IoT_IDAS_intro_ul20_v2Fiware IoT_IDAS_intro_ul20_v2
Fiware IoT_IDAS_intro_ul20_v2
 

Similar to Cosmos, Big Data GE Implementation

Big data, just an introduction to Hadoop and Scripting Languages
Big data, just an introduction to Hadoop and Scripting LanguagesBig data, just an introduction to Hadoop and Scripting Languages
Big data, just an introduction to Hadoop and Scripting LanguagesCorley S.r.l.
 
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011Jonathan Seidman
 
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...rhatr
 
FIWARE Tech Summit - FIWARE Big Data Ecosystem Cosmos
FIWARE Tech Summit - FIWARE Big Data Ecosystem CosmosFIWARE Tech Summit - FIWARE Big Data Ecosystem Cosmos
FIWARE Tech Summit - FIWARE Big Data Ecosystem CosmosFIWARE
 
Windows Azure HDInsight Service
Windows Azure HDInsight ServiceWindows Azure HDInsight Service
Windows Azure HDInsight ServiceNeil Mackenzie
 
Web Services Hadoop Summit 2012
Web Services Hadoop Summit 2012Web Services Hadoop Summit 2012
Web Services Hadoop Summit 2012Hortonworks
 
Building Hadoop Data Applications with Kite
Building Hadoop Data Applications with KiteBuilding Hadoop Data Applications with Kite
Building Hadoop Data Applications with Kitehuguk
 
Developing a Redis Module - Hackathon Kickoff
 Developing a Redis Module - Hackathon Kickoff Developing a Redis Module - Hackathon Kickoff
Developing a Redis Module - Hackathon KickoffItamar Haber
 
Leveraging Hadoop in your PostgreSQL Environment
Leveraging Hadoop in your PostgreSQL EnvironmentLeveraging Hadoop in your PostgreSQL Environment
Leveraging Hadoop in your PostgreSQL EnvironmentJim Mlodgenski
 
Hydra - Getting Started
Hydra - Getting StartedHydra - Getting Started
Hydra - Getting Startedabramsm
 
Big data using Hadoop, Hive, Sqoop with Installation
Big data using Hadoop, Hive, Sqoop with InstallationBig data using Hadoop, Hive, Sqoop with Installation
Big data using Hadoop, Hive, Sqoop with Installationmellempudilavanya999
 
Working with Hive Analytics
Working with Hive AnalyticsWorking with Hive Analytics
Working with Hive AnalyticsManish Chopra
 
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of AltiscaleDebugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of AltiscaleData Con LA
 
containerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdcontainerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdDocker, Inc.
 
Get started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseGet started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseHenk van der Valk
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsLynn Langit
 
Above the cloud: Big Data and BI
Above the cloud: Big Data and BIAbove the cloud: Big Data and BI
Above the cloud: Big Data and BIDenny Lee
 
Practical introduction to dev ops with chef
Practical introduction to dev ops with chefPractical introduction to dev ops with chef
Practical introduction to dev ops with chefLeanDog
 

Similar to Cosmos, Big Data GE Implementation (20)

Big data, just an introduction to Hadoop and Scripting Languages
Big data, just an introduction to Hadoop and Scripting LanguagesBig data, just an introduction to Hadoop and Scripting Languages
Big data, just an introduction to Hadoop and Scripting Languages
 
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011
Data Analysis with Hadoop and Hive, ChicagoDB 2/21/2011
 
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...
Building Google-in-a-box: using Apache SolrCloud and Bigtop to index your big...
 
FIWARE Tech Summit - FIWARE Big Data Ecosystem Cosmos
FIWARE Tech Summit - FIWARE Big Data Ecosystem CosmosFIWARE Tech Summit - FIWARE Big Data Ecosystem Cosmos
FIWARE Tech Summit - FIWARE Big Data Ecosystem Cosmos
 
Windows Azure HDInsight Service
Windows Azure HDInsight ServiceWindows Azure HDInsight Service
Windows Azure HDInsight Service
 
Web Services Hadoop Summit 2012
Web Services Hadoop Summit 2012Web Services Hadoop Summit 2012
Web Services Hadoop Summit 2012
 
Building Hadoop Data Applications with Kite
Building Hadoop Data Applications with KiteBuilding Hadoop Data Applications with Kite
Building Hadoop Data Applications with Kite
 
Developing a Redis Module - Hackathon Kickoff
 Developing a Redis Module - Hackathon Kickoff Developing a Redis Module - Hackathon Kickoff
Developing a Redis Module - Hackathon Kickoff
 
Leveraging Hadoop in your PostgreSQL Environment
Leveraging Hadoop in your PostgreSQL EnvironmentLeveraging Hadoop in your PostgreSQL Environment
Leveraging Hadoop in your PostgreSQL Environment
 
Hydra - Getting Started
Hydra - Getting StartedHydra - Getting Started
Hydra - Getting Started
 
Big data using Hadoop, Hive, Sqoop with Installation
Big data using Hadoop, Hive, Sqoop with InstallationBig data using Hadoop, Hive, Sqoop with Installation
Big data using Hadoop, Hive, Sqoop with Installation
 
Working with Hive Analytics
Working with Hive AnalyticsWorking with Hive Analytics
Working with Hive Analytics
 
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of AltiscaleDebugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
Debugging Hive with Hadoop-in-the-Cloud by David Chaiken of Altiscale
 
containerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdcontainerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerd
 
Get started with Microsoft SQL Polybase
Get started with Microsoft SQL PolybaseGet started with Microsoft SQL Polybase
Get started with Microsoft SQL Polybase
 
מיכאל
מיכאלמיכאל
מיכאל
 
Apache Hive
Apache HiveApache Hive
Apache Hive
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce Fundamentals
 
Above the cloud: Big Data and BI
Above the cloud: Big Data and BIAbove the cloud: Big Data and BI
Above the cloud: Big Data and BI
 
Practical introduction to dev ops with chef
Practical introduction to dev ops with chefPractical introduction to dev ops with chef
Practical introduction to dev ops with chef
 

More from FIWARE

Behm_Herne_NeMo_akt.pptx
Behm_Herne_NeMo_akt.pptxBehm_Herne_NeMo_akt.pptx
Behm_Herne_NeMo_akt.pptxFIWARE
 
Katharina Hogrebe Herne Digital Days.pdf
 Katharina Hogrebe Herne Digital Days.pdf Katharina Hogrebe Herne Digital Days.pdf
Katharina Hogrebe Herne Digital Days.pdfFIWARE
 
Christoph Mertens_IDSA_Introduction to Data Spaces.pptx
Christoph Mertens_IDSA_Introduction to Data Spaces.pptxChristoph Mertens_IDSA_Introduction to Data Spaces.pptx
Christoph Mertens_IDSA_Introduction to Data Spaces.pptxFIWARE
 
Behm_Herne_NeMo.pptx
Behm_Herne_NeMo.pptxBehm_Herne_NeMo.pptx
Behm_Herne_NeMo.pptxFIWARE
 
Evangelists + iHubs Promo Slides.pptx
Evangelists + iHubs Promo Slides.pptxEvangelists + iHubs Promo Slides.pptx
Evangelists + iHubs Promo Slides.pptxFIWARE
 
Lukas Künzel Smart City Operating System.pptx
Lukas Künzel Smart City Operating System.pptxLukas Künzel Smart City Operating System.pptx
Lukas Künzel Smart City Operating System.pptxFIWARE
 
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptx
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptxPierre Golz Der Transformationsprozess im Konzern Stadt.pptx
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptxFIWARE
 
Dennis Wendland_The i4Trust Collaboration Programme.pptx
Dennis Wendland_The i4Trust Collaboration Programme.pptxDennis Wendland_The i4Trust Collaboration Programme.pptx
Dennis Wendland_The i4Trust Collaboration Programme.pptxFIWARE
 
Ulrich Ahle_FIWARE.pptx
Ulrich Ahle_FIWARE.pptxUlrich Ahle_FIWARE.pptx
Ulrich Ahle_FIWARE.pptxFIWARE
 
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptx
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptxAleksandar Vrglevski _FIWARE DACH_OSIH.pptx
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptxFIWARE
 
Water Quality - Lukas Kuenzel.pdf
Water Quality - Lukas Kuenzel.pdfWater Quality - Lukas Kuenzel.pdf
Water Quality - Lukas Kuenzel.pdfFIWARE
 
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptx
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptxCameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptx
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptxFIWARE
 
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptx
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptxFiWareSummit.msGIS-Data-to-Value.2023.06.12.pptx
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptxFIWARE
 
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptx
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptxBoris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptx
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptxFIWARE
 
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....FIWARE
 
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdf
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdfAbdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdf
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdfFIWARE
 
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdf
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdfFGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdf
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdfFIWARE
 
HTAG_Skalierung_Plattform_lokal_final_versand.pptx
HTAG_Skalierung_Plattform_lokal_final_versand.pptxHTAG_Skalierung_Plattform_lokal_final_versand.pptx
HTAG_Skalierung_Plattform_lokal_final_versand.pptxFIWARE
 
EU Opp_Clara Pezuela - German chapter.pptx
EU Opp_Clara Pezuela - German chapter.pptxEU Opp_Clara Pezuela - German chapter.pptx
EU Opp_Clara Pezuela - German chapter.pptxFIWARE
 
OSIH.pptx
OSIH.pptxOSIH.pptx
OSIH.pptxFIWARE
 

More from FIWARE (20)

Behm_Herne_NeMo_akt.pptx
Behm_Herne_NeMo_akt.pptxBehm_Herne_NeMo_akt.pptx
Behm_Herne_NeMo_akt.pptx
 
Katharina Hogrebe Herne Digital Days.pdf
 Katharina Hogrebe Herne Digital Days.pdf Katharina Hogrebe Herne Digital Days.pdf
Katharina Hogrebe Herne Digital Days.pdf
 
Christoph Mertens_IDSA_Introduction to Data Spaces.pptx
Christoph Mertens_IDSA_Introduction to Data Spaces.pptxChristoph Mertens_IDSA_Introduction to Data Spaces.pptx
Christoph Mertens_IDSA_Introduction to Data Spaces.pptx
 
Behm_Herne_NeMo.pptx
Behm_Herne_NeMo.pptxBehm_Herne_NeMo.pptx
Behm_Herne_NeMo.pptx
 
Evangelists + iHubs Promo Slides.pptx
Evangelists + iHubs Promo Slides.pptxEvangelists + iHubs Promo Slides.pptx
Evangelists + iHubs Promo Slides.pptx
 
Lukas Künzel Smart City Operating System.pptx
Lukas Künzel Smart City Operating System.pptxLukas Künzel Smart City Operating System.pptx
Lukas Künzel Smart City Operating System.pptx
 
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptx
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptxPierre Golz Der Transformationsprozess im Konzern Stadt.pptx
Pierre Golz Der Transformationsprozess im Konzern Stadt.pptx
 
Dennis Wendland_The i4Trust Collaboration Programme.pptx
Dennis Wendland_The i4Trust Collaboration Programme.pptxDennis Wendland_The i4Trust Collaboration Programme.pptx
Dennis Wendland_The i4Trust Collaboration Programme.pptx
 
Ulrich Ahle_FIWARE.pptx
Ulrich Ahle_FIWARE.pptxUlrich Ahle_FIWARE.pptx
Ulrich Ahle_FIWARE.pptx
 
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptx
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptxAleksandar Vrglevski _FIWARE DACH_OSIH.pptx
Aleksandar Vrglevski _FIWARE DACH_OSIH.pptx
 
Water Quality - Lukas Kuenzel.pdf
Water Quality - Lukas Kuenzel.pdfWater Quality - Lukas Kuenzel.pdf
Water Quality - Lukas Kuenzel.pdf
 
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptx
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptxCameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptx
Cameron Brooks_FGS23_FIWARE Summit_Keynote_Cameron.pptx
 
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptx
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptxFiWareSummit.msGIS-Data-to-Value.2023.06.12.pptx
FiWareSummit.msGIS-Data-to-Value.2023.06.12.pptx
 
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptx
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptxBoris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptx
Boris Otto_FGS2023_Opening- EU Innovations from Data_PUB_V1_BOt.pptx
 
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....
Bjoern de Vidts_FGS23_Opening_athumi - bjord de vidts - personal data spaces....
 
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdf
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdfAbdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdf
Abdulrahman Ibrahim_FGS23 Opening - Abdulrahman Ibrahim.pdf
 
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdf
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdfFGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdf
FGS2023_Opening_Red Hat Keynote Andrea Battaglia.pdf
 
HTAG_Skalierung_Plattform_lokal_final_versand.pptx
HTAG_Skalierung_Plattform_lokal_final_versand.pptxHTAG_Skalierung_Plattform_lokal_final_versand.pptx
HTAG_Skalierung_Plattform_lokal_final_versand.pptx
 
EU Opp_Clara Pezuela - German chapter.pptx
EU Opp_Clara Pezuela - German chapter.pptxEU Opp_Clara Pezuela - German chapter.pptx
EU Opp_Clara Pezuela - German chapter.pptx
 
OSIH.pptx
OSIH.pptxOSIH.pptx
OSIH.pptx
 

Recently uploaded

Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Mateusz Kwasniewski
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?GleecusTechlabs1
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfDomotica daVinci
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfDomotica daVinci
 
Journey of Television in World & in India
Journey of Television in World & in IndiaJourney of Television in World & in India
Journey of Television in World & in IndiaAdarshAgarwal66
 
Heltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfHeltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfMarielaL5
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Enhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for PartnersEnhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for PartnersThousandEyes
 
My sample product research idea for you!
My sample product research idea for you!My sample product research idea for you!
My sample product research idea for you!KivenRaySarsaba
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingshrey Ansh
 
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptxssuser796efb
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveDomotica daVinci
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIMemory Fabric Forum
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfkeyaramicrochipusa
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfThomas Poetter
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxMemory Fabric Forum
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfDomotica daVinci
 

Recently uploaded (20)

Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdf
 
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdfZ-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
Z-Wave Fan coil Thermostat Heltun_HE-HT01_User_Manual.pdf
 
Journey of Television in World & in India
Journey of Television in World & in IndiaJourney of Television in World & in India
Journey of Television in World & in India
 
Heltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfHeltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdf
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
COE AI Lab Universities
COE AI Lab UniversitiesCOE AI Lab Universities
COE AI Lab Universities
 
Enhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for PartnersEnhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for Partners
 
My sample product research idea for you!
My sample product research idea for you!My sample product research idea for you!
My sample product research idea for you!
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons working
 
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx
2) Presentation_Overview_ISO_16140-3_Method_verification_20210322.pptx
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-Wave
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AI
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdf
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptx
 
zigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdfzigbee motion sensor user manual NAS-PD07B2.pdf
zigbee motion sensor user manual NAS-PD07B2.pdf
 

Cosmos, Big Data GE Implementation

  • 1. Cosmos, Big Data GE implementation Building your first application using FI-WARE Open APIs for Open Minds
  • 2. Big Data: What is it and how much data is there 1
  • 3. What is big data? 2 > small data
  • 4. What is big data? 3 http://commons.wikimedia.org/wiki/File:Interior_view_of_Stockholm_Public_Library.jpg > big data
  • 5. How much data is there? 4
  • 6. Data growing forecast 5 2.3 3.6 12 19 11.3 39 0.5 1.4 Global users (billions) Global networked devices (billions) Global broadband speed (Mbps) Global traffic (zettabytes) http://www.cisco.com/en/US/netsol/ns827/networking_solutions_sub_solution.html#~forecast 2012 2012 2012 2012 2017 2017 2017 2017
  • 7. It is not only about storing big data but using it! 6 http://commons.wikimedia.org/wiki/File:Interior_view_of_Stockholm_Public_Library.jpg > big data > tools
  • 8. How to deal with it: The Hadoop reference 7
  • 9. Hadoop was created by Doug Cutting at Yahoo!... … based on the MapReduce patent by Google 8
  • 10. Well, MapReduce was really invented by Julius Caesar 9 Divide et impera* * Divide and conquer
  • 11. An example How much pages are written in latin among the books in the Ancient Library of Alexandria? 10 LATIN REF1 P45 GREEK REF2 P128 EGYPT REF3 P12 LATIN pages 45 EGYPTIA N LATIN REF4 P73 LATIN REF5 P34 EGYPT REF6 P10 GREEK REF7 P20 GREEK REF8 P230 45 (ref 1) still reading… Mappers Reducer
  • 12. An example How much pages are written in latin among the books in the Ancient Library of Alexandria? 11 GREEK REF2 P128 still reading… EGYPTIA N LATIN REF4 P73 LATIN REF5 P34 EGYPT REF6 P10 GREEK REF7 P20 GREEK REF8 P230 GREEK 45 (ref 1) Mappers Reducer
  • 13. An example How much pages are written in latin among the books in the Ancient Library of Alexandria? LATIN pages 73 EGYPTIA N 12 LATIN REF4 P73 LATIN REF5 P34 GREEK REF7 P20 GREEK REF8 P230 LATIN pages 34 45 (ref 1) +73 (ref 4) +34 (ref 5) Mappers Reducer
  • 14. An example How much pages are written in latin among the books in the Ancient Library of Alexandria? GREEK GREEK 13 GREEK REF7 P20 GREEK REF8 P230 idle… 45 (ref 1) +73 (ref 4) +34 (ref 5) Mappers Reducer
  • 15. An example How much pages are written in latin among the books in the Ancient Library of Alexandria? idle… idle… idle… 14 45 (ref 1) +73 (ref 4) +34 (ref 5) 152 TOTAL Mappers Reducer
  • 18. What is Cosmos? • Cosmos is Telefónica's Big Data platform • Dynamic creation of private computing clusters as a 17 service • Infinity, a cluster for persistent storage • Cosmos is Hadoop ecosystem-based • HDFS as its distributed file system • Hadoop core as its MapReduce engine • HiveQL and Pig for querying the data • Oozie as remote MapReduce jobs and Hive launcher • Plus other proprietary features • Infinity protocol (secure WebHDFS) • Cygnus, an injector for context data coming from Orion CB
  • 20. What can be done with Cosmos? 19 What Locally (ssh’ing into the Head Node) Remotely (connecting your app) Clusters operation Cosmos CLI REST API I/O operation ‘hadoop fs’ command REST API (WebHDFS, HttpFS, Infinity protocol) Querying tools (basic analysis) Hive CLI JDBC, Thrift* MapReduce (advanced analysis) ‘hadoop jar’ command Oozie REST API
  • 21. Clusters operation: Getting your own roman legion 20
  • 22. Using the RESTful API (1) 21
  • 23. Using the RESTful API (2) 22
  • 24. Using the RESTful API (3) 23
  • 25. Using the Python CLI 24 • Creating a cluster $ cosmos create --name <STRING> --size <INT> • Listing all the clusters $ cosmos list • Showing a cluster details $ cosmos show <CLUSTER_ID> • Connecting to the Head Node of a cluster $ cosmos ssh <CLUSTER_ID> • Terminating a cluster $ cosmos terminate <CLUSTER_ID> • Listing available services $ cosmos list-services • Creating a cluster with specific services $ cosmos create --name <STRING> --size <INT> --services <SERVICES_LIST>
  • 26. How to exploit the data: Commanding your roman legion 25
  • 27. 1. Hadoop filesystem commands 26 • Hadoop general command $ hadoop • Hadoop file system subcommand $ hadoop fs • Hadoop file system options $ hadoop fs –ls $ hadoop fs –mkdir <hdfs-dir> $ hadoop fs –rmr <hfds-file> $ hadoop fs –cat <hdfs-file> $ hadoop fs –put <local-file> <hdfs-dir> $ hadoop fs –get <hdfs-file> <local-dir> • http://hadoop.apache.org/docs/current/hadoop-project-dist/ hadoop-common/CommandsManual.html
  • 28. 2. WebHDFS/HttpFS REST API 27 • List a directory GET http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTSTATUS • Create a new directory PUT http://<HOST>:<PORT>/<PATH>?op=MKDIRS[&permission=<OCTAL>] • Delete a file or directory DELETE http://<host>:<port>/webhdfs/v1/<path>?op=DELETE [&recursive=<true|false>] • Rename a file or directory PUT http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=RENAME&destination=<PATH> • Concat files POST http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CONCAT&sources=<PATHS> • Set permission PUT http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETPERMISSION [&permission=<OCTAL>] • Set owner PUT http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETOWNER [&owner=<USER>][&group=<GROUP>]
  • 29. 2. WebHDFS/HttpFS REST API (cont.) • Create a new file with initial content (2 steps operation) PUT http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE [&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>] [&permission=<OCTAL>][&buffersize=<INT>] HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE... Content-Length: 0 PUT -T <LOCAL_FILE> http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=CREATE... 28 • Append to a file (2 steps operation) POST http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=APPEND[&buffersize=<INT>] HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND... Content-Length: 0 POST -T <LOCAL_FILE> http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND...
  • 30. 2. WebHDFS/HttpFS REST API (cont.) • Open and read a file (2 steps operation) GET http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN [&offset=<LONG>][&length=<LONG>][&buffersize=<INT>] HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN... Content-Length: 0 GET http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=OPEN... • http://hadoop.apache.org/docs/current/hadoop-project-dist/ hadoop-hdfs/WebHDFS.html • HttpFS does not redirect to the Datanode but to the HttpFS server, hidding the Datanodes (and saving tens of public IP addresses) • The API is the same • http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ 29 index.html
  • 31. 3. Local Hive CLI • Hive is a querying tool • Queries are expresed in HiveQL, a SQL-like language • https://cwiki.apache.org/confluence/display/Hive/Language 30 Manual • Hive uses pre-defined MapReduce jobs for • Column selection • Fields grouping • Table joining • … • All the data is loaded into Hive tables
  • 32. 3. Local Hive CLI (cont.) • Log on to the Master node • Run the hive command • Type your SQL-like sentence! $ hive $ Hive history file=/tmp/myuser/hive_job_log_opendata_XXX_XXX.txt hive>select column1,column2,otherColumns from mytable where column1='whatever' and columns2 like '%whatever%'; Total MapReduce jobs = 1 Launching Job 1 out of 1 Starting Job = job_201308280930_0953, Tracking URL = http://cosmosmaster-gi:50030/jobdetails.jsp?jobid=job_201308280930_0953 Kill Command = /usr/lib/hadoop/bin/hadoop job - Dmapred.job.tracker=cosmosmaster-gi:8021 -kill job_201308280930_0953 2013-10-03 09:15:34,519 Stage-1 map = 0%, reduce = 0% 2013-10-03 09:15:36,545 Stage-1 map = 67%, reduce = 0% 2013-10-03 09:15:37,554 Stage-1 map = 100%, reduce = 0% 2013-10-03 09:15:44,609 Stage-1 map = 100%, reduce = 33% … 31
  • 33. 4. Remote Hive client • Hive CLI is OK for human-driven testing purposes • But it is not usable by remote applications • Hive has no REST API • Hive has several drivers and libraries • JDBC for Java • Python • PHP • ODBC for C/C++ • Thrift for Java and C++ • https://cwiki.apache.org/confluence/display/Hive/HiveClie 32 nt • A remote Hive client usually performs: • A connection to the Hive server (TCP/10000) • The query execution
  • 34. 4. Remote Hive client – Get a connection 33 private Connection getConnection( String ip, String port, String user, String password) { try { // dynamically load the Hive JDBC driver Class.forName("org.apache.hadoop.hive.jdbc.HiveDriver"); } catch (ClassNotFoundException e) { System.out.println(e.getMessage()); return null; } // try catch try { // return a connection based on the Hive JDBC driver, default DB return DriverManager.getConnection("jdbc:hive://" + ip + ":" + port + "/default?user=" + user + "&password=" + password); } catch (SQLException e) { System.out.println(e.getMessage()); return null; } // try catch } // getConnection https://github.com/telefonicaid/fiware-connectors/tree/develop/resources/hive-basic-client
  • 35. 4. Remote Hive client – Do the query 34 private void doQuery() { try { // from here on, everything is SQL! Statement stmt = con.createStatement(); ResultSet res = stmt.executeQuery("select column1,column2," + "otherColumns from mytable where column1='whatever' and " + "columns2 like '%whatever%'"); // iterate on the result while (res.next()) { String column1 = res.getString(1); Integer column2 = res.getInteger(2); // whatever you want to do with this row, here } // while // close everything res.close(); stmt.close(); con.close(); } catch (SQLException ex) { System.exit(0); } // try catch } // doQuery https://github.com/telefonicaid/fiware-connectors/ tree/develop/resources/hive-basic-client
  • 36. 4. Remote Hive client – Plague Tracker demo https://github.com/telefonicaid/fiware-connectors/tree/develop/resources/plague-tracker 35
  • 37. 5. MapReduce applications • MapReduce applications are commonly written in Java • Can be written in other languages through Hadoop Streaming • They are executed in the command line $ hadoop jar <jar-file> <main-class> <input-dir> <output-dir> • A MapReduce job consists of: • A driver, a piece of software where to define inputs, outputs, formats, etc. and the entry point for launching the job • A set of Mappers, given by a piece of software defining its behaviour • A set of Reducers, given by a piece of software defining its behaviour 36 • There are 2 APIS • org.apache.mapred  old one • org.apache.mapreduce  new one • Hadoop is distributed with MapReduce examples • [HADOOP_HOME]/hadoop-examples.jar
  • 38. 5. MapReduce applications – Map /* org.apache.mapred example */ public static class MapClass extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { /* use the input value, the input key is the offset within the file and it is not necessary in this example */ String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); /* iterate on the string, getting each word */ while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); /* emit an output (key,value) pair based on the word and 1 */ output.collect(word, one); 37 } // while } // map } // MapClass
  • 39. 5. MapReduce applications – Reduce /* org.apache.mapred example */ public static class ReduceClass extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { 38 int sum = 0; /* iterate on all the values and add them */ while (values.hasNext()) { sum += values.next().get(); } // while /* emit an output (key,value) pair based on the word and its count */ output.collect(key, new IntWritable(sum)); } // reduce } // ReduceClass
  • 40. 5. MapReduce applications – Driver 39 /* org.apache.mapred example */ package my.org import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; public class WordCount { public static void main(String[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setMapperClass(MapClass.class); conf.setCombinerClass(ReduceClass.class); conf.setReducerClass(ReduceClass.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } // main } // WordCount
  • 41. 6. Launching tasks with Oozie • Oozie is a workflow scheduler system to manage Hadoop jobs • Java map-reduce • Pig and Hive • Sqoop • System specific jobs (such as Java programs and shell scripts) • Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) 40 of actions. • Writting Oozie applications is about including in a package • The MapReduce jobs, Hive/Pig scritps, etc (exeutable code) • A Workflow • Parameters for the Workflow • Oozie can be use locally or remotely • https://oozie.apache.org/docs/4.0.0/index.html#Developer_Do cumentation
  • 42. 6. Launching tasks with Oozie – Java client OozieClient client = new OozieClient("http://130.206.80.46:11000/oozie/"); // create a workflow job configuration and set the workflow application path Properties conf = client.createConfiguration(); conf.setProperty(OozieClient.APP_PATH, "hdfs://cosmosmaster-gi: 41 8020/user/frb/mrjobs"); conf.setProperty("nameNode", "hdfs://cosmosmaster-gi:8020"); conf.setProperty("jobTracker", "cosmosmaster-gi:8021"); conf.setProperty("outputDir", "output"); conf.setProperty("inputDir", "input"); conf.setProperty("examplesRoot", "mrjobs"); conf.setProperty("queueName", "default"); // submit and start the workflow job String jobId = client.run(conf); // wait until the workflow job finishes printing the status every 10 seconds while (client.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) { System.out.println("Workflow job running ..."); Thread.sleep(10 * 1000); } // while System.out.println("Workflow job completed");
  • 43. Useful references 42 • Hive resources: • HiveQL language  https://cwiki.apache.org/confluence/display/Hive/LanguageManual • How to create Hive clients  https://cwiki.apache.org/confluence/display/Hive/HiveClient • Hive client example  https://github.com/telefonicaid/fiware-connectors/ tree/develop/resources/hive-basic-client • Plague Tracker demo  https://github.com/telefonicaid/fiware-livedemoapp/ tree/master/cosmos/plague-tracker • Plague Tracker instance  http://130.206.81.65/plague-tracker/ • Hadoop filesystem commands: • http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ CommandsManual.html • WebHDFS and HttpFS REST APIs: • http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html • http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/index.html • Oozie • https://oozie.apache.org/docs/4.0.0/index.html#Developer_Documentation
  • 44. Cosmos place in FI-WARE: Typical scenarios 43
  • 45. General IoT platform SENSOR 2 THINGS IoT Backend Device Management 44 CKAN DATA PROCESSING COSMOS (BIG DATA) DATA QUERYING SUBS OPEN DATA CONTEXT BROKER measures / commands IoT/Sensor Open Data T-T Accounting & Payment & Billing IDM & Auth SHORT TERM HISTORIC REAL TIME PRCSSING BI ETL BLNK RULES DEFINITION BLNK OPERATIONAL DASHBOARD KPI GOVERNANCE OPEN DATA PORTALS CEP GIS Service Orchrestation Context Adapters City Services
  • 46. Real time context data persistence (architecture) https://github.com/telefonicaid/fiware-connectors/tree/develop/flume https://forge.fi-ware.eu/plugins/mediawiki/wiki/fiware/index.php/How_to_persist_Orion_data_in_Cosmos 45
  • 47. Real time context data persistence (detail) 46
  • 48. Real time context data persistence (examples) • Information coming from city sensors • Presence  map gradients, aglomerations… • Services usage  distributions, top users (if available), top POIs, unused resources… • Information generated by smartphones • Geolocation  routes, map gradients, 47 aglomerations… • Issues reporting  top neighbourhooods in incidents, crimilality, noises, garbage, plagues… • Any other real time information • Depending on your app, this could be product likes, product consumption, user-2-user feedback…  recommendations, advertisement…
  • 49. Roadmap: More functionalities and integrations 48
  • 50. Roadmap • Integrate the clusters creation with the cloud portal • No more REST API work • Streaming analysis capabilities • Not all the analysis can wait for a batch processing • Geolocation analysis capabilities • An important source of data nowadays 49 • Integrate with CKAN • As a source of batch data • Integrate with the Marketplace • Selling datasets • Selling analysis results • Selling applications and algorithms
  • 52. Thanks !  http://fi-ppp.eu  http://fi-ware.eu  Follow @Fiware on Twitter! 51