Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sinfonier: How I turned my grandmother into a data analyst - Fran J. Gomez - Codemotion Amsterdam 2016

528 views

Published on

More than a technology, Sinfonier is the logic evolution of real time processing systems. It's the combination of a visual programming language (Yahoo Pipes), a collaborative philosophy (community) and a framework for real-time processing (Apache Storm) made available to users. During years solutions that exploit information using batch processing techs. have been growing. Now it is time to bring this philosophy to the world of real-time processing. Sinfonier was born with a clear focus on solving problems related to cybersecurity but technology supports other aspects we'll show this.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Sinfonier: How I turned my grandmother into a data analyst - Fran J. Gomez - Codemotion Amsterdam 2016

  1. 1. Sinfonier: How I turned my grandmother into a data analyst. @ffranz AMSTERDAM 11-12 MAY 2016
  2. 2. My Granma first time using Sinfonier
  3. 3. Why some technologies must be difficult to use?
  4. 4. Let's not repeat our mistakes
  5. 5. How many of you knows Apache Storm?
  6. 6. “Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use! “ http://storm.apache.org/
  7. 7. Hadoop Storm Large but finite jobs Infinite computations called Topologies Processes a lot of data at once Process Infinite streams of data one tuple at a time High Latency Low Latency
  8. 8. What do you need to use Storm?
  9. 9. Apache Storm Cluster
  10. 10. What do you need to use Sinfonier?
  11. 11. + + Drag & Drop Interface Automatic Deploy API Storm Cluster
  12. 12. Apache Storm Cluster Sinfonier Drawer
  13. 13. Apache Storm Sinfonier Project Topologies Programmatically DAG Visual DAG Components Spouts and Bolts Spouts, Bolts and Drains Grouping Shuffle, Field, All (+4) Shuffle Data Model Tuples JSON Tuple K,V
  14. 14. DRAIN BOLT SPOUT BOLT DRAIN DRAIN SPOUT
  15. 15. TITLE
  16. 16.  * RandomSentenceSpout: this is the spout which will be responsible for generating the phrases with which the word count will be performed.  * SplitSentence: this is the bolt responsible for identifying the words which appear in each of the sentences, generating a list whose elements are the different words in each of the sentences.  * WordCount: this is the bolt responsible for counting the number of words which make up the list generated by the previous module. TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("spout", new RandomSentenceSpout(), 5); builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")); Sinfonier vs. Storm
  17. 17. Sinfonier vs. Storm
  18. 18. RSS Demo
  19. 19. Developers, developers, developers…
  20. 20. • Name: Your module name. Must be UpperCamelCase • Icon: Add an image. • Entity: In order to catalog. • Type: Choose your type of module. Spout, Bolt and Drain. Won’t be change. • Language: Java or Python • Code: Url point to Gist.github.com • Description: Describe what you module do. • Fields: Declare your parameters.
  21. 21. States of the module Developing Pending Private Published Deleted Cannot use in Topologies Just declared Cannot use in Topologies Validation pending Owner can use in Topologies Code changes must be validated Module only available on existing topologies Everybody can use in Topologies
  22. 22. http://virtualmachine.sinfonier-project.net
  23. 23. Let’s Play
  24. 24. { "_id":{ "$oid":"5731928be4b0be5c2b777447" }, "guid":{ "content":"http://mqtt.org/?p=522", "isPermaLink":false }, "pubDate":"Fri, 07 Nov 2014 13:11:28 +0000", "category":"news", "title":"MQTT v3.1.1 now an OASIS Standard", "slash:comments":5 } { "_id-$oid":"5731928be4b0be5c2b777447”, "guid-content":"http://mqtt.org/?p=522", "guid-isPermaLink":false, "pubDate":"Fri, 07 Nov 2014 13:11:28 +0000", "category":"news", "title":"MQTT v3.1.1 now an OASIS Standard", "slash:comments":5 }
  25. 25. https://public.ducksboard.com/XBc7Sa8nZmspsh_Oc6Gr/
  26. 26. https://public.ducksboard.com/4xvXKxfjjSDkWBdN26Fk/
  27. 27. Today's world is no longer driven by data – it's driven by the connections between them.
  28. 28. http://fiware-cosmos.readthedocs.io/en/latest/ https://github.com/sinfonier-project/
  29. 29. Join us! http://sinfonier-project.net/ http://blog.sinfonier-project.net/ @e_sinfonier
  30. 30. All pictures belong to their respective authors AMSTERDAM 9-12 MAY 2016

×