Advertisement

Event Pipe - Lambda Architecture

Clojure - Distributed Systems Developer a.i at MPARE
Sep. 18, 2015
Advertisement

More Related Content

Slideshows for you(20)

Advertisement

Event Pipe - Lambda Architecture

  1. EventPipe-λArchitecture w/NotsoBigData Bahadir Cambel - Senior Data Engineer @bahadircambel bahadir.io TravelBird
  2. There are 10^11 stars in the galaxy. That used to be a huge number. But it's only a hundred billion. It's less than the national deficit! We used to call them astronomical numbers. Now we should call them economical numbers. -RF We only have half a billion events, yet.
  3. What’sanEvent? a thing that happens (takes place), especially one of importance - Page view - Link Click - Order Save - Sales Flow - Mobile App Launch.. - A/B Visit - Newsletter View - Scroll - Ad View - and many more...
  4. What’saPipe?theBADBADONE - Events flowing in through. Once upon a time in LinkedIn
  5. Alsoaeventpipe(UNIFIED)
  6. logslogslogs->Kafka-Kinesis- Jay Kreps ( Ex-LinkedIn, co-creator Apache Kafka, Confluent.io founder) http://bit.ly/realtime-data-logs ( MUST READ - LONG READ - READ TWICE I’m aiming to read 3rd time) http://bit.ly/i-love-logs-video
  7. What’s λ Nathan Marz(Storm,Cascalog) come up with the term.
  8. EventPIPE-Batchpart
  9. EVentPipe-FULLFORceλ
  10. Howtocount? - Counter +1 - Aggregate in batch - What happens if you need to reprocess data ? - Page Views, Users, Sessions - How about Counter’s for Segments - per page/device/domain/ - How to handle Big DATA? Is it going to perform with - Million
  11. Hyperloglog-HLL - DVc ( almost accurate ) - Redis has it since 2.8.9 ( PFCOUNT - PFADD ) - 12 KBytes per key up to 2^64 elements per key ( 0.81% standard-error)
  12. CLOJURE - Lazy - Concise - Fast - Simple - Perfect for Data processing, distributed computing - the power of JVM
  13. stats
  14. OPensource - https://github.com/travelbird/star-track - https://github.com/travelbird/kinesis3 - https://github.com/travelbird/modular
  15. A/BTests-ExperimentsExperimentsExperiments It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong. - RF
  16. We’reallfoolsofourown The first principle is that you must not fool yourself and you are the easiest person to fool. - RF
  17. ORDERinCHaos
  18. Thanks - Andrii Mishkovski (Intro to Clojure) - Philipp Wassibauer (CTO of TravelBird) - Egle, Brandon, Niels, Brandon, Rob (Teammates) - Rich Hickey for Clojure - Nathan Marz for λArchitecture - Opensourcedevelopers-You’reamazing
Advertisement