Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Asynchronous micro-services and the unified log

1,401 views

Published on

On Friday October 7th 2016 at Crunch Conference in Budapest I gave a talk entitled "Asynchronous micro-services and the unified log".

The unified log enabled by Apache Kafka and Amazon Kinesis has been mostly understood as a better data processing architecture, replacing traditional data warehousing techniques. But the unified log also enables a new way of building transactional software, by enabling asynchronous micro-services. In this talk, I showed how event-driven micro-services designed around Kafka or Kinesis resolve many of the issues associated with traditional monolithic and synchronous micro-service based architectures.

Published in: Software
  • Be the first to comment

Asynchronous micro-services and the unified log

  1. 1. Asynchronous micro-services and the unified log Crunch Conference, Budapest, 7th October 2016
  2. 2. Introducing myself • Alexander Dean • Co-founder and technical lead at Snowplow, the open-source event data pipeline • Weekend writer of Unified Log Processing, available on the Manning Early Access Program • Co-author at Snowplow of Iglu, our open-source schema registry system, and Sauna, our open- source decisioning and response platform
  3. 3. We are witnessing the convergence of two separate technology tracks towards asynchronous or event-driven micro-services Transactional workloads Analytical workloads Asynchronous micro-services Software monoliths Synchronous micro-services Classic data warehousing Hybrid data pipelines Unified log architectures
  4. 4. Analytical workloads, and the rise of the unified log
  5. 5. A quick history lesson: the three eras of business data processing [1] 1. Classic data warehousing, 1996+ 2. Hybrid data pipelines, 2005+ 3. Unified log architectures, 2013+ [1] http://snowplowanalytics.com/blog/ 2014/01/20/the-three-eras-of-business-data-processing/
  6. 6. The era of classic data warehousing, 1996+ OWN DATA CENTER Data warehouse HIGH LATENCY Point-to-point connections WIDE DATA COVERAGE CMS Silo CRM Local loop Local loop NARROW DATA SILOES LOW LATENCY LOCAL LOOPS E-comm Silo Local loop Management reporting ERP Silo Local loop Silo Nightly batch ETL process FULL DATA HISTORY
  7. 7. The era of hybrid data pipelines, 2005+ CLOUD VENDOR / OWN DATA CENTER Search Silo Local loop LOW LATENCY LOCAL LOOPS E-comm Silo Local loop CRM Local loop SAAS VENDOR #2 Email marketing Local loop ERP Silo Local loop CMS Silo Local loop SAAS VENDOR #1 NARROW DATA SILOES Stream processing Product rec’s Micro-batch processing Systems monitoring Batch processing Data warehouse Management reporting Batch processing Ad hoc analytics Hadoop SAAS VENDOR #3 Web analytics Local loop Local loop Local loop LOW LATENCY LOW LATENCY HIGH LATENCY HIGH LATENCY APIs Bulk exports
  8. 8. The hybrid era: a surfeit of software vendors CLOUD VENDOR / OWN DATA CENTER Search Silo Local loop LOW LATENCY LOCAL LOOPS E-comm Silo Local loop CRM Local loop SAAS VENDOR #2 Email marketing Local loop ERP Silo Local loop CMS Silo Local loop SAAS VENDOR #1 NARROW DATA SILOES Stream processing Product rec’s Micro-batch processing Systems monitoring Batch processing Data warehouse Management reporting Batch processing Ad hoc analytics Hadoop SAAS VENDOR #3 Web analytics Local loop Local loop Local loop LOW LATENCY LOW LATENCY HIGH LATENCY HIGH LATENCY APIs Bulk exports
  9. 9. The hybrid era: company-wide reporting and analytics ends up like Rashomon The bandit’s story vs. The wife’s story vs. The samurai’s story vs. The woodcutter’s story
  10. 10. The hybrid era: the number of data integrations is unsustainable
  11. 11. So how do we unravel the hairball?
  12. 12. The advent of the unified log, 2013+ CLOUD VENDOR / OWN DATA CENTER Search Silo SOME LOW LATENCY LOCAL LOOPS E-comm Silo CRM SAAS VENDOR #2 Email marketing ERP Silo CMS Silo SAAS VENDOR #1 NARROW DATA SILOES Streaming APIs / web hooks Unified log LOW LATENCY WIDE DATA COVERAGE Archiving Hadoop < WIDE DATA COVERAGE > < FULL DATA HISTORY > FEW DAYS’ DATA HISTORY Systems monitoring Eventstream HIGH LATENCY LOW LATENCY Product rec’s Ad hoc analytics Management reporting Fraud detection Churn prevention APIs
  13. 13. CLOUD VENDOR / OWN DATA CENTER Search Silo SOME LOW LATENCY LOCAL LOOPS E-comm Silo CRM SAAS VENDOR #2 Email marketing ERP Silo CMS Silo SAAS VENDOR #1 NARROW DATA SILOES Streaming APIs / web hooks Unified log Archiving Hadoop < WIDE DATA COVERAGE > < FULL DATA HISTORY > Systems monitoring Eventstream HIGH LATENCY LOW LATENCY Product rec’s Ad hoc analytics Management reporting Fraud detection Churn prevention APIs The unified log is Amazon Kinesis, or Apache Kafka • Amazon Kinesis, a hosted AWS service • Extremely similar semantics to Kafka • Apache Kafka, an append- only, distributed, ordered commit log • Developed at LinkedIn to serve as their organization’s unified log
  14. 14. “Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization” [1] [1] http://kafka.apache.org/
  15. 15. So what does a unified log give us? A single version of the truth Our truth is now upstream from the data warehouse The hairball of point-to-point connections has been unravelled Local loops have been unbundled 1 2 3 4
  16. 16. Coming up to the end of 2016, and unified log architectures have seen extremely rapid and widespread adoption
  17. 17. Transactional data processing and the move to micro-services
  18. 18. In parallel, we have seen a steady (if spotty) rejection of software monoliths for transactional workloads
  19. 19. In a micro-services architecture, the individual capabilities of the system are split out into separate services Synchronous communication using request and response (often using RESTful HTTP or RPC)
  20. 20. What do synchronous micro-services give us? Strong module boundaries • Network boundaries between modules can be helpful for larger teams Independent deployment • Deploy individual micro-services independently • Simpler to deploy and less likely to cause whole system failures Support diversity • Can use the best language/framework/database for the capability • Reduces monoculture risk (anti-fragile) 1 2 3
  21. 21. Convergence on asynchronous or event- driven micro-services
  22. 22. When we re-architected Snowplow around the unified log 2.5 years ago, we designed it around small, composable workers Diagram from our February 2014 Snowplow v0.9.0 release post
  23. 23. This was based on the insight that real-time pipelines can be composed a little like Unix pipes
  24. 24. We avoided monolithic Spark Streaming or Storm jobs, based on our experiences with “heavy” Hadoop jobs in our batch pipeline Kinesis stream Kinesis stream Kinesis stream Kinesis stream Kinesis stream What we didn’t do: an inner Storm topology
  25. 25. We wanted to avoid the “inner topology” effect, with effectively two tiers of topology to reason about Difficult to unit test the inner topologies – complex behaviours inside each unit Difficult to operationalize the inner topologies – how do they handle backpressure, how do they scale, how do we upgrade them? Difficult to monitor the inner topologies Fundamental problem: the event streams in an inner topology are not first class entities 1 2 3
  26. 26. It worked: today the Snowplow real-time pipeline is a collection of individual event-driven micro-services Stream Collector Stream Enrich Kinesis S3 Kinesis Elasticsearch Kinesis Tee Kinesis Redshift (design stage) User’s AWS Lambda function User’s KCL worker app User’s Spark Streaming job
  27. 27. “the most compelling applications for stream processing are actually pretty different from what you would typically do with a Hive or Spark job—they are closer to being a kind of asynchronous microservice rather than being a faster version of a batch analytics job. …” Meanwhile, the Kafka team (now at Confluent) were seeing something interesting in the adoption of Kafka…
  28. 28. “…What I mean is that these stream processing apps were most often software that implemented core functions in the business rather than computing analytics about the business.” [1] [1] http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/ And these micro-services were substituting not just for batch analytical workloads, but also for transactional workloads
  29. 29. Why are async micro- services replacing “classic” request/response micro- services (at least amongst Kafka users)?
  30. 30. To start with, asynchronous micro-services have the same benefits as synchronous micro-services… Strong module boundaries • Network boundaries between modules can be helpful for larger teams Independent deployment • Deploy individual micro-services independently • Easier to deploy and less likely to cause whole system failures Support diversity • Can use the best language/framework/database for the capability • Reduces monoculture risk (anti-fragile) 1 2 3
  31. 31. … but in addition, event-driven asynchronous micro- services are extremely loosely-coupled, because they are intermediated by first class streams
  32. 32. Compare this to request/response synchronous micro-services, which have dependencies between upstream and downstream Downstream Upstream Single-page web app Login service Notifications service Content service Customer profile service Personalization service Ad service
  33. 33. If you can afford the (small) latency tax, there are some clear advantages in going asynchronous Much better toolkit for upgrades – schema evolution, running old and new service versions in parallel etc Adding new downstream services doesn’t increase the load on upstream services Failure of individual services introduces lag into the overall system, rather than overall system failure Easier to debug, because service inputs and outputs are directly inspectable in the event streams 1 2 3 4
  34. 34. So what’s next in this convergence? ?
  35. 35. More and more tooling for building asynchronous micro- services, including the unstoppable rise of “serverless”  Kinesis Client Library AWS Lambda IBM OpenWhisk Event-driven cloud functionsStream processing as a library Azure Functions
  36. 36. Greater adoption of open source schema registries as the canonical source of truth for the events in our topologies Confluent or Iglu schema registries
  37. 37. Request/response micro-services aren’t going away – they are just too useful But expect a move from slower HTTP-based to faster RPC- based options • e.g. Google’s gRPC • These are easier to read from and write from in high- volume event-driven architectures Expect wider adoption of API definition languages • OpenAPI (Swagger), RAML, API Blueprint Eventual harmonization of types? • Using JSON for RESTful APIs, Protocol Buffers for RPC and Avro for stream processing is crazy • Needs company-wide standardization, or dynamic translation (but this is lossy) 1 2 3
  38. 38. We also need new fabrics – or extensions of existing ones like Kubernetes – to address the challenges of running our topologies “How do we monitor this topology, and alert if something (data loss; event lag) is going wrong?” “How do we scale our streams and micro-services to handle event peaks and troughs smoothly?” “How do we re-configure or upgrade our micro-services without breaking things?”
  39. 39. At Snowplow we are working on a unified log fabric, called Tupilak, to solve this problem
  40. 40. Questions? http://snowplowanalytics.com https://github.com/snowplow/snowplow @snowplowdata To meet up or chat, @alexcrdean on Twitter or alex@snowplowanalytics.com

×