Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From measurement to knowledge with sofia2 Platform

222 views

Published on

BrightTalk webinar given last June, 21st and available in https://summits.brighttalk.com/webinar/big-data-and-real-time-analytics-in-the-iot-from-measurement-to-knowledge/
Covering:
- IoT Analytics business cases
- IoT & Analytics platform
- Examples and tecnical description.

Published in: Technology
  • Hello! Who wants to chat with me? Nu photos with me here http://bit.ly/helenswee
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

From measurement to knowledge with sofia2 Platform

  1. 1. From measurement to knowledge Big data and real time analytics in the IoT http://twitter.com/SOFIA2_Platform http://about.sofia2.com plataformasofia2@indra.es http://sofia2.com
  2. 2. 2 Raquel López IoT Expert Manager Sofia2 IoT Platform Presentation raquel-lopez-alarcon rlalarcon@minsait.com @rlalarcon
  3. 3. 3 1. Business Approach to IoT 2. IoT Analytics Platform 3. The Technical Part…. We will talk about…
  4. 4. 4 Business Approach to IoT Section 1
  5. 5. 5 IoT as a Business Opportunity More than 10 billion of connected devices Plus 10-20 billions to add in the next 10 years Only 1% o fthe data is actually used (as alerts, event triggering….)  Devices generate a continuous flow of data whose acquisition and analysis will allow us a better understanding of reality and an improvement and faster decision making.  We will have a volume of data without coming, which will allow us to extract an extensive knowledge of the client and with this to offer the customer custom solutions to measure and participate in their day to day Internet Of Things terminology was used for the first time on 1999 referring to identifiable objects (things) and its vistual representation in an infrastructure. Business Processes Transformation New Business Models
  6. 6. 6 Why using IoT? The Goal Making better and faster business decisions by the combination of real-time process and analysis of the information.
  7. 7. 7 IoT Mainstays Integration & Connectivity Process and Analytics IoT Flow Social flow General SystemsDevices … RRSSAPIs G.Analytics PIWIK Y.Finance … “Interoperability between IoT systems is critical. Of the total potential economic value the IoT enables, interoperability is required for 40 percent on average and for nearly 60 percent in some settings.” McKinsey “Currently, most IoT data are not used. For example, on an oil rig that has 30,000 sensors, only 1 percent of the data are examined. That’s because this information is used mostly to detect and control anomalies—not for optimization and prediction, which provide the greatest value.” McKinsey
  8. 8. 8 Types of IoT Analytics Descriptive Predictive Prescriptive Pushes and Alerts Embedded Models Streaming Realtime Analysis What question to answer? What method to apply? Combining both perspectives we can get many applicable use-cases
  9. 9. 9 IoT Analytics usecases Forbes
  10. 10. 10 IoT analytics platform Section 2
  11. 11. 11 IoT Platform Capabilities
  12. 12. 12 IoT Bussiness Solution Referential Model
  13. 13. 13 Administration, Security, Monitoring, Control Business apps Custom apps Sensors and devices Storage Publish People Third Party Systems INGEST Bulk, Event Ingestion PROCESS Preparation, Anaytics VISUALIZE IoT Platform Information Flow
  14. 14. 14 IoT Broker  Light communication Protocols: REST, MQTT, WS, etc…  Development APIs & SDKs  Real time processing. Rules and flows  Semantic model Real Time Repository Big Data Repository Staging Area Notebooks: • Algorithm Execution • Schedulling • Data Scientist collaborative environment • Exports and Visualization. Machine Learning: • Algorithm Execution • Parametrization • Publication • Machine Learning inference API Manager: Service Publication and lifecycle management Dashboards & synoptics Diifferent options to configure dashboards for realtime & analytics visualization of the data Open Data Portal: Opendata publication in different standard formats. Data Links: ODBC & JDBC Access to data Data Flow  Visual ETL style model.  Bacth ingestión.  Monitorization  Stadistics & graphics  Heterogeneous sources of information IoT Platform Functional Capabilities
  15. 15. 15 Platform Modules Discovery Apps & VerticalIoT Flow Social flow General SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW SystemsSDKs Java Javascript Python Android iOS .Net R C(C++ … Devices … RRSSAPIs G.Analytics PIWIK Y.Finance … IoT Gateway IoT Broker Ontology Stream Process Sofia2 Storage Real Time Database Notebooks API Manager DataLink HTML5 & API MANAGEMENT CONFIG DB Sofia2 Control Panel Security Historical Database Staging Area ODBC JDBC Dashboards Ontology APIs VerticalSystemsAppsExplotationVisual Plugins REST Ext APIs Open Data Portal Synoptics CEP Ontology Flows MQTT … APIs Rules Bots ML Engine SPACES DEVICES Devices SEMANTIC Custom Integration DataFlow BigData Ingestion
  16. 16. 16 The technical part… Section 3
  17. 17. 17 Energy Segmentation Context / complexity • Electricity consumption information retrieval • Segments for the organization of consumption patterns / Period of the customer set. • Consumer recovery and trend analysis. Technological solution • Collection and storage of sources: Ingestion in HDFS. • Data exploration: Understanding its distribution, quality, volumetry. They are data of hourly curves of 6 months. • Cleaning of data and selection of relevant information. • Indicators generation by CUP and Period. • Inference of new data to complete dataset. • Application of K-means algorithm and elbow technique for selection of K. • Analysis of results
  18. 18. 18 Energy Segmentation Apps & VerticalIoT Flow Social flow General SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW SystemsSDKs Java Javascript Python Android iOS .Net R C(C++ … Devices … RRSSAPIs G.Analytics PIWIK Y.Finance … Stream Process Sofia2 Storage Real Time Database Notebooks API Manager HTML5 & API MANAGEMENT CONFIG DB Sofia2 Control Panel Historical Database Staging Area JDBC Dashboards Ontology APIs VerticalSystemsApps Ext APIs CEP Ontology Flows APIs Rules ML Engine SPACES DEVICES SEMANTIC DataFlow 1 2 3 4 5
  19. 19. 19 INGEST / STORE ANALYZE PUBLISH & VIEW Capabilities Visual ETL modelling. Java extensible. Integrated in Sofia2 Control Panel (HTML5) Debug and Preview. Sources & destinations: Kafka, Hadoop, JMS, JDBC, AmazonS3… Alerts and Notifications Deployment integrated on Control Panel. 18 available processors for ETL modelling: Split, hasher, remover, merger… Charts and execution statistics visualization Energy Segmentation
  20. 20. 20 Energy Segmentation INGEST / STORE ANALYZE PUBLISH & VIEW Optimized for and agile and real time access to the information. Real Time Data Base Optimized not the storage and management of big Data, not so often accessed. Historical Data Base Capabilities One repository per each moment on the lifecycle of the information. Support to different technologies depending on the information access pattern. Standard support and SQL & NO-SQL databases Access to the information with optimized response times. Horizontal Scalable. Different repositories are integrated between them and with other modules of the platform Each entity of the data model have a configured time-window, in order to identify the data stored as historic or real- time information Information will be persisted in the real- time database until it would be automatically moved to the historical database. Any information (real-time or historical) will be accessible from any of the other modules of the platform: Integration, Machine Learning, APIManager Differential Staging Repository For structured, semi-structured and unstructured data, as a temporary holder in case of transform processess requiring persistence.
  21. 21. 21 Energy Segmentation INGEST / STORE ANALYZE PUBLISH & VIEW Capabilities Interpreter Management (sh, jdbc, md, angular…) Shared and multiuser environment Data Scientist IDE. Multilenguaje execution on the same notebook (SparkSQL, R, Hive, Scala, Python) Instant visualization of results with incorporated charts Invocable from other platform moduels such as Realtime Processing Module. Schedulled execution of Web environment integrated in the Control Panel of the platform Export and visualization
  22. 22. 22 Energy Segmentation INGEST / STORE ANALYZE PUBLISH & VIEW Capabilities Configurable Gadgets with integration and querying capabilities to the Repositories Configurable Dashboards reusing gadgets, with drilldown and cascade dependencies, URL exposition and CSS customization . Dashboards
  23. 23. 23 Energy Segmentation YouTube Sofia2 IoT Platform • Demos and viewers • Sofia2 presentations • Workshops and tutorials • Dashboards • Technical support • Solutions built on Sofia2
  24. 24. 24 Traceability in distribution: Executive Dashboards Context / complexity • Solution for automatic verification of delivery punctuality, next point arrival forecast update and temperature ranges verification during stores supply. • Monitoring and registration of Arrival time to store • Early warning + Expected arrival time notification to the next store • Traceability of goods delivered and collected at each store • Online monitoring of delivery operations Technological solution • Real Time Management & Reporting system. • Black-box (GPS & Mobile Comms.) + Beacons integrated with The Route Planning system. • Sofia2 as core for device integration, data analytics, calculations, events detection and notifications. Results • Greater efficiency in distribution, store unloading and platform loading • Avoid management conflicts with transportation providers and internal customers or franchisees • Compliance with temperature transport regulations • Reduction in the loss of means and its control, as well as of the goods in transit • Improvements in route planning and contracting suppliers • Improve efficiency in all internal processes of the value chain: • Verification of load units sent in the correct vehicle • Control of in route stops + openings control • Verification of load units left and loaded at each stop • Selective inventories of accumulated media in stores and warehouses • Truthful and timely information for the management of incidences of distribution or lack of means a a ab b b c c c (tª) (tª)
  25. 25. 25 Traceability in distribution: Executive Dashboards Apps & VerticalIoT Flow Social flow General SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW SystemsSDKs Java Javascript Python Android iOS .Net R C(C++ … Devices … RRSSAPIs G.Analytics PIWIK Y.Finance … IoT Gateway IoT Broker Ontology Stream Process Sofia2 Storage Real Time Database Notebooks API Manager HTML5 & API MANAGEMENT CONFIG DB Sofia2 Control Panel Security Historical Database Dashboards Ontology APIs VerticalSystemsApps Plugins REST Ext APIs CEP Ontology Flows MQTT … APIs Rules SPACES DEVICES Devices SEMANTIC 1 2 3 4
  26. 26. 26 Traceability in distribution: Executive Dashboards INGEST / PROCESS & STORE ANALYZE PUBLISH & VIEW Capabilities Communication connectors for different client’s programming language. Development APIs for the client side. Security capable in terms of communication, authentication and authorization. Light communication protocols (REST, OPC, MODBUS, WebSockets, MQTT, WS, JMS, AMQP…) Real-time processing and validation of the interchanged information. Homogenization of the information received, according to the data model defined (protocol independent) Java Extensible (APIs, Protocols, Plugins) Auditing of integration activity Integrated in Sofia2 Control Panel (HTML5)
  27. 27. 27 Traceability in distribution: Executive Dashboards Capabilities Real time processing of the information received Different coding languages supported (Groovy, R, Python). Integrated with other modules of the platform such as repositories or notebooks. Capabilities for simple IFTTT, CEP or visual definition of flows Alerts, notifications, enrichment of information… Managed and monitoried from Sofia2 Control Panel (HTML5) INGEST / PROCESS & STORE ANALYZE PUBLISH & VIEW
  28. 28. 28 Traceability in distribution: Executive Dashboards INGEST / STORE ANALYZE PUBLISH & VIEW Capabilities Integrated Security (authenticación, autorization, encription…) Data can be published independently of the repository storing the information (real time or historical) Custom Query commands, APIs clone, HTML5 CRUDs generated automatically Integration with third Party APIs including Open Data or WebServices to be Apified . APIs Lifecycle managemen (Created,, in Process of development, published, dDeprecated, Removed) Throtling control and auditing of the activity
  29. 29. 29 Traceability in distribution: Executive Dashboards YouTube Sofia2 IoT Platform • Demos and viewers • Sofia2 presentations • Workshops and tutorials • Dashboards • Technical support • Solutions built on Sofia2
  30. 30. 30
  31. 31. 31

×