Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SMACK : Une pile logicielle open source pour le traitement big data

361 views

Published on

Présentation de l'utilisation de Spark, Mesos, Akka, Cassandra et Kafka, dans le cadre d'un traitement de données avec des outils open source.

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

SMACK : Une pile logicielle open source pour le traitement big data

  1. 1. www.cetic.be SMACK: une pile logicielle open source pour le traitement big data Mathieu Goeminne 1
  2. 2. www.cetic.be Un mot sur le CETIC 2 Recherche appliquée Études de faisabilité Preuves de concept Transferts technologiques
  3. 3. www.cetic.be Secteurs d’activité • Ingénierie logicielle / qualité logicielle • Électronique / Embarqué • Traitement des données / Infrastructure 3
  4. 4. www.cetic.be Secteurs d’activité • Ingénierie logicielle / qualité logicielle • Électronique / Embarqué • Traitement des données / Infrastructure 3
  5. 5. www.cetic.be Un mot sur l’orateur • PhD (software engineering) sur les écosystèmes open source • Cetic depuis 2016 > Équipe Data • Projets Big Data et Data Science • Manufacturing et transport 4
  6. 6. www.cetic.be Le big data au sein des entreprises Des données toujours plus nombreuses, toujours variées, toujours rapides 5
  7. 7. www.cetic.be Manufacturing • (1-100) x 1000 capteurs @ (1 min - 1 sec)-1 • Mesures labo, ERP, etc. • Multiples sites (AGC Europe = 28 implantations) 6
  8. 8. www.cetic.be Data Center • Des milliers serveurs (OVH: 260 000) • Des dizaines de paramètres / serveur • Des millions de sites hébergés (OVH: 18 000 000) • (10-100-1000) messages de log applicatifs / seconde / site • Multiples implantations (OVH: 20) 7
  9. 9. www.cetic.be 8
  10. 10. www.cetic.be SMACK Un ensemble d’outils logiciels pour le traitement de données à large échelle 9
  11. 11. www.cetic.be Quelques propriétés • Projets Apache (à l’exception de Akka) • Répondre aux besoins suivants: • Passage à l’échelle: de plusieurs processeurs à plusieurs data centers • Résilience: N’importe quel élément peut tomber • Réplicabilité: On doit pouvoir refaire un travail à n’importe quel moment, et obtenir le même résultat • Efficacité: La donnée perd rapidement de sa valeur, donc un traitement doit être réalisé dans les délais. • Gestion intégrée de plusieurs workflows et plusieurs sources de données: typiquement, traitement de données historiques et de données « fraiches » 10
  12. 12. www.cetic.be Vue globale de l’architecture 11
  13. 13. www.cetic.be 1. Akka Pour la collecte de données 12
  14. 14. www.cetic.be Akka pour la collecte de données • Système de traitement distribué • Par échange de message, plutôt que par état partagé • Asynchrone, rellocalisable, passe à l’échelle. • Dans SMACK, chaque acteur représente un petit service de collecte • Reçoit une donnée (ex: via un service REST) • Traitement minimal (ex: validation du format) • Transmission à un service de persistance 13
  15. 15. www.cetic.be 2. Kafka Pour le transport des messages 14
  16. 16. www.cetic.be Kafka pour le transport des messages • Producteurs — Consommateurs • Optimisé pour écrire à grande vitesse (2 millions d’écritures / seconde sur 3 machines standard). • Organisation des messages en topics, chacun d’eux étant géré par un cluster de machines (réplication, résilience, etc.) • Persistance pendant x heures 15
  17. 17. www.cetic.be Kafka en tant que point central de communication 16
  18. 18. www.cetic.be Kafka en tant que point central de communication 16
  19. 19. www.cetic.be 3. Cassandra Pour le stockage 17
  20. 20. www.cetic.be Cassandra pour le stockage • Base de données NoSQL: pas votre SGBD habituel • Optimisée pour l’écriture à grande vitesse • Dictionnaire de dictionnaires de valeurs • Accès obligatoire par la clef de ligne, puis optionnellement par clef de colonne • Chaque ligne peut avoir des colonnes différentes • BDD distribuée, masterless • Chaque instance possède un sous-ensemble des lignes • Les lignes sont dupliquées (typiquement: 3x): vitesse et résilience • Alimentée par un agent (acteur Akka), qui consomme un topic Kafka. 18
  21. 21. www.cetic.be Cassandra: stockage de séries chronologiques 19
  22. 22. www.cetic.be Cassandra: stockage de séries chronologiques 19 Clef primaire composée. tag=clef de partition (de ligne) timestamp=clef de clustering Pour chaque tag, les timestamps sont ordonnés.
  23. 23. www.cetic.be Cassandra: stockage de séries chronologiques 19 Clef primaire composée. tag=clef de partition (de ligne) timestamp=clef de clustering Pour chaque tag, les timestamps sont ordonnés. tag time value quality sensor1 t1 v1 q1 sensor1 t2 v2 q2 sensor2 t3 v3 q3 sensor2 t4 v4 q4 sensor1 t1 t2 value v1 value v2 quality q1 quality q2 sensor2 t3 t4 value v3 value v3 quality q3 quality q4 Ce qu’on voit Ce qui est stocké
  24. 24. www.cetic.be 4. Spark Pour le traitement des données 20
  25. 25. www.cetic.be Spark pour le traitement des données • Resilient Distributed Dataset • Collections immuables • Partitions distribuées sur le cluster • Transformations • Appliquées sur chaque partition • Programmation fonctionnelle • filter, map, reduce, … • Actions • Rapatriement sur le driver 21
  26. 26. www.cetic.be Spark pour le traitement des données • Resilient Distributed Dataset • Collections immuables • Partitions distribuées sur le cluster • Transformations • Appliquées sur chaque partition • Programmation fonctionnelle • filter, map, reduce, … • Actions • Rapatriement sur le driver 21 TwitterRDD ImdbRDD Création
  27. 27. www.cetic.be Spark pour le traitement des données • Resilient Distributed Dataset • Collections immuables • Partitions distribuées sur le cluster • Transformations • Appliquées sur chaque partition • Programmation fonctionnelle • filter, map, reduce, … • Actions • Rapatriement sur le driver 21 TwitterRDD ImdbRDD Création map filter Transformation
  28. 28. www.cetic.be Spark pour le traitement des données • Resilient Distributed Dataset • Collections immuables • Partitions distribuées sur le cluster • Transformations • Appliquées sur chaque partition • Programmation fonctionnelle • filter, map, reduce, … • Actions • Rapatriement sur le driver 21 TwitterRDD ImdbRDD Création map filter Transformation join Transformation
  29. 29. www.cetic.be Spark pour le traitement des données • Resilient Distributed Dataset • Collections immuables • Partitions distribuées sur le cluster • Transformations • Appliquées sur chaque partition • Programmation fonctionnelle • filter, map, reduce, … • Actions • Rapatriement sur le driver 21 TwitterRDD ImdbRDD Création map filter Transformation join Transformation count Action
  30. 30. www.cetic.be Spark : une plateforme pour les gouverner tous 22
  31. 31. www.cetic.be Spark ❤ Cassandra 23
  32. 32. www.cetic.be Spark ❤ Cassandra 23
  33. 33. www.cetic.be Spark ❤ Cassandra 23
  34. 34. www.cetic.be 5. Mesos Pour gérer les ressources du cluster 24
  35. 35. www.cetic.be Mesos pour gérer les ressources du cluster • Efficace • Partage efficace des ressources entre les applications • Flexible • Supporte de nombreux frameworks actuels et à venir • Supporte des demandes de ressources de différentes natures • Passe à l’échelle • Des clusters de quelques nœuds à des milliers de nœuds • Robuste • Tolérance aux erreurs: si ça plante, on recommence sur le nœud d’à côté • Le scheduler lui-même est redondant 25
  36. 36. www.cetic.be Mesos pour gérer les ressources du cluster 26
  37. 37. www.cetic.be Démonstration! 27 https://youtu.be/QzJI4FeXns4
  38. 38. www.cetic.be linkedin.com/company/cetic info@cetic.be +32 71 490 700 twitter.com/@CETIC twitter.com/@CETIC_be www.cetic.be Aéropole de Charleroi-Gosselies Avenue Jean Mermoz 28 B-6041 Charleroi - Belgique Mathieu Goeminne mathieu.goeminne@cetic.be Merci

×