Start SmallBigger BiggestThe Event Driven Organisation                     Based on the Lambda Architecture
Who Is That?               Daan Gerits                       BigData.be co-organizer                       daan.gerits@gma...
Who Is That?               BigData.be                           Belgian BigData Group                              Founded...
AgendaTransCoThe Event Driven Organisation  Events  Views  Integration
TransCoMeet TransCo  International Transport  Track & Trace  Freelance Drivers  Custom Software  Enterprise Service Bus
TransCoDepartments  IT  Finance  Legal  Logistics  Sales  Communications  Marketing  ...
TransCoA customer requests a quoteA server goes downA website visitor clicks on a linkA financial transaction is bookedA d...
TransCoAll these have a similar thing:          They Generate Events
Events   IT   Finance   Legal   Logistics        Events   Sales   Communications   ...
Events    Events used to manipulate their             master data
Events  Today, events ARE their master data
TransCo   Isnt this a big data presentation?       Hang on, Well get to that
EventsAn event is  Time based  Record linked  Field oriented  Incremental  Immutable
EventsA new customer is added  ADD [ts=1,id=123,field="name",value="Daan Gerits"]  ADD [ts=1,id=123,field="address",value=...
EventsNotice I only created new records  No Updates  No Deletes  CRUD becomes CR
Events   Oh no, I shouldnt have done that!         DELETE [ts=43,id=123,field="address"]
TransCoDiscover market trendsApplication and server statisticsDirect MarketingFinancial validationTracking Dashboard
ViewsView = Fx   All Data
ViewsView =
ViewsDifferent View Types   Real-time   Relational   Reporting   Application   API   ...
ViewsViews are  Full data models  Based on all data  Focussed
Integration  How do I integrate with my existing            infrastructure?
IntegrationViews can also be stored in  - RDBMS  - Indexing Engine  - Files  - DataWarehouse  - ...But these views are rea...
Big DataSo what has all this to do with Big Data?
Big DataOnly appending events  need to store a lot of data    => HDFS, HBase, ...  need batch processing    => MapReduce, ...
Event Driven OrganisationAllows trending  driver statistics for his whole careerAllows state regeneration  the state of al...
Event Driven OrganisationBy storing eventsBy generating focussed views
Thank you!      Daan Gerits         @daangerits    daan.gerits@gmail.com
Backup Slides    Generating Views
Views        Realtime         Views
Views        Latency
Views         Start                  View   +   Generating   +   +   +   Ready   +Generating takes time  => View is always...
ViewsHow to solve this?  Store events in 2 locations    Master Data Set    Buffer DB  Buffer DB has    - data of last x ho...
Views              Speed               View        API              Batch              View
ViewsBatch Views  Use MapReduceSpeed Views  Use Storm or something similar
Upcoming SlideShare
Loading in …5
×

Start small bigger biggest

610 views

Published on

The Event Driven Company

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
610
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Start small bigger biggest

  1. 1. Start SmallBigger BiggestThe Event Driven Organisation Based on the Lambda Architecture
  2. 2. Who Is That? Daan Gerits BigData.be co-organizer daan.gerits@gmail.com @daangerits
  3. 3. Who Is That? BigData.be Belgian BigData Group Founded in juli 2011 +200 members http://www.bigdata.be http://www.meetup.com/bigdatabe @bigdatabe
  4. 4. AgendaTransCoThe Event Driven Organisation Events Views Integration
  5. 5. TransCoMeet TransCo International Transport Track & Trace Freelance Drivers Custom Software Enterprise Service Bus
  6. 6. TransCoDepartments IT Finance Legal Logistics Sales Communications Marketing ...
  7. 7. TransCoA customer requests a quoteA server goes downA website visitor clicks on a linkA financial transaction is bookedA delivery truck pings its GPS coördinates
  8. 8. TransCoAll these have a similar thing: They Generate Events
  9. 9. Events IT Finance Legal Logistics Events Sales Communications ...
  10. 10. Events Events used to manipulate their master data
  11. 11. Events Today, events ARE their master data
  12. 12. TransCo Isnt this a big data presentation? Hang on, Well get to that
  13. 13. EventsAn event is Time based Record linked Field oriented Incremental Immutable
  14. 14. EventsA new customer is added ADD [ts=1,id=123,field="name",value="Daan Gerits"] ADD [ts=1,id=123,field="address",value="MyStreet 40"]A customer is updated ADD [ts=13,id=123,field="address",value="AnotherStreet 1"]Eventually a customer is removed ADD [ts=43,id=123,field="address",value=]
  15. 15. EventsNotice I only created new records No Updates No Deletes CRUD becomes CR
  16. 16. Events Oh no, I shouldnt have done that! DELETE [ts=43,id=123,field="address"]
  17. 17. TransCoDiscover market trendsApplication and server statisticsDirect MarketingFinancial validationTracking Dashboard
  18. 18. ViewsView = Fx All Data
  19. 19. ViewsView =
  20. 20. ViewsDifferent View Types Real-time Relational Reporting Application API ...
  21. 21. ViewsViews are Full data models Based on all data Focussed
  22. 22. Integration How do I integrate with my existing infrastructure?
  23. 23. IntegrationViews can also be stored in - RDBMS - Indexing Engine - Files - DataWarehouse - ...But these views are read-only!If you want read-write use the API
  24. 24. Big DataSo what has all this to do with Big Data?
  25. 25. Big DataOnly appending events need to store a lot of data => HDFS, HBase, ... need batch processing => MapReduce, Pig, Hive, ... need real-time processing => Storm, S4, ...
  26. 26. Event Driven OrganisationAllows trending driver statistics for his whole careerAllows state regeneration the state of all master data on februari 12, 2005Is human-error-proof remove the last event on this recordScales very well
  27. 27. Event Driven OrganisationBy storing eventsBy generating focussed views
  28. 28. Thank you! Daan Gerits @daangerits daan.gerits@gmail.com
  29. 29. Backup Slides Generating Views
  30. 30. Views Realtime Views
  31. 31. Views Latency
  32. 32. Views Start View + Generating + + + Ready +Generating takes time => View is always x hours old
  33. 33. ViewsHow to solve this? Store events in 2 locations Master Data Set Buffer DB Buffer DB has - data of last x hours - data, not events
  34. 34. Views Speed View API Batch View
  35. 35. ViewsBatch Views Use MapReduceSpeed Views Use Storm or something similar

×