Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative
Commons Attr...
Upcoming SlideShare
Loading in …5
×

Building Resilient and Evolutionary Data Microservices

1,348 views

Published on

SpringOne Platform 2016
Speaker: Vinicius Carvalho; Advisory Platform Architect, Pivotal

How can we build data pipelines that are resilient to change? Data usually outlives application code, and we have to be prepared to deploy streams that can cope with the evolution of that data that is in motion. This talk will discuss the approach and supporting patterns to write resilient data microservices with Spring Cloud Stream and Spring Cloud Dataflow. We will discuss the role of a centralized Schema repository, and how can we work with different data models and protocols to achieve schema evolution. A centralized schema service will also help us understand what kind of data is currently being served and consumed by the different parts of our distributed system.

Samples for the presentation can be found here: https://github.com/viniciusccarvalho/schema-evolution-samples

Published in: Technology

Building Resilient and Evolutionary Data Microservices

  1. 1. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Building Resilient and Evolutionary Data Microservices By Vinicius Carvalho @vccarvalho viniciusccarvalho
  2. 2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Agenda • Data microservices • Why you should care about schema evolution • It’s all about the format • Introducing Schema evolution for Spring Cloud Stream • Demo 2
  3. 3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data microservices 3 Self contained Loose coupled Event Driven Data aware Independent
  4. 4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data microservices 4
  5. 5. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data pipelines 5
  6. 6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data microservices in action: Context maps 6 Sam Newman’s Building Microservices
  7. 7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data microservices in action: CQRS 7 MSDN CQRS MSDN CQRS Journey ebook
  8. 8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Learn more 8 Cloud Native Streaming and Event-Driven Microservices Wed 8/3 - 4:20 - 5:30 PM Starvine 2 Orchestrate All the Things! with Spring Cloud Data Flow Thu 8/4 - 11:10 - 12:15 Starvine 2 Architecting for Cloud Native Data: Data Microservices Done Right Using Spring Cloud Wed 8/3 2:00 - 3:10 PM Starvine 2
  9. 9. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Evolutionary? 9 What happens when data evolve?
  10. 10. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema Evolution 10 “The problem of evolving a data schema to adapt it to a change in modeled reality”
  11. 11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Backward compatibility 11 V1 V2 • A reader using a newer schema version can read from writers of an older version
  12. 12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Forward compatibility 12 V1 V2 • A reader using an older schema version can read from writers of a newer version
  13. 13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Request / Response paradigm 13 It’s all about a handshake client <V1> Recommendation Service GET /v1/… <V2> Recommendation Service client GET /v2/…
  14. 14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Messaging paradigm 14 <V1> Recommendation Service
  15. 15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Messaging paradigm 15 <V1> Recommendation Service <V2> Recommendation Service if msg.headers[version] == “v1” { } else if …
  16. 16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Microservices 16 But I thought microservices were all about smart endpoints and dumb pipes
  17. 17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Smart endpoints • Our goal is to make our endpoints capable of adapting to changes on the model without requiring changes on the system. 17
  18. 18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How can we achieve that? 18 We have the tools We just need to evolve on how to use it
  19. 19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ It all depends on the right format 19
  20. 20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Choosing the right format : Features 20 ✓Compact ✓Strongly typed ✓Adaptable ✓Versioned ✓Centralized
  21. 21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Choosing a format 21
  22. 22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Payload size 22 0 350 700 1050 1400 Payload Size (bytes) 1355 763 502 285243216 Avro Proto3 Kryo CSV JSON XML
  23. 23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Performance 23 Average marshaling time in nanos Read Write 3,4833,433 1,3001,333 Avro JSON
  24. 24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Benchmark : because we love one 24
  25. 25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Bottom line 25
  26. 26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why Avro • Binary and compact format • Mature and with active community. Heavily used on hadoop ecosystem • Schemas are a first class citizen on it • Out of the box support for Schema evolution • Support for GenericRecords : Easy integration with SpEL • Cross language implementations for clients 26
  27. 27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution support on SCSt 27 API Level 16 API Level 21 Registry •Stores schemas •Tests compatibility •Tracks utilization 1. Producers registers schemas 2. SCSt broadcast location in header
  28. 28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution support on SCSt 28 API Level 16 API Level 21 Registry 1. Consumers fetch message 2. Query registry for schema 3. If version is different apply resolution
  29. 29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Starting the Server 29 @SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }
  30. 30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema Registry Server 30 • POST / : Register a new schema, returning it’s version and a location header • GET /{subject}/{format}/{version} : Returns a schema definition • GET /{subject}/{format} : Returns all versions of a schema for a given format • POST /{subject}/{format}/test : Compatibility test against all registered versions • GET /{subject}/{format}/usage : List apps using all versions of a schema
  31. 31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Enabling Schema registration 31 @SpringBootApplication @EnableSchemaRegistryClient public class ProducerApplication { public static void main(String[] args) { SpringApplication .run(ProducerApplication.class,args); } }
  32. 32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Enabling Schema registration - Producer 32 spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: output: destination: sensor-topic contentType: "application/*+avro"
  33. 33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Enabling Schema registration - Consumer 33 spring: cloud: stream: schemaRegistryClient: endpoint: http://localhost:8990 bindings: input: destination: sensor-topic Content type is inferred from headers
  34. 34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution example 34 #v1 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"temperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]} ] } #v2 { "namespace" : "io.igx.android", "type" : "record", "name" : "Sensor", "fields" : [ {"name":"id","type":"string"}, {"name":"internalTemperature", "type":"float", "default":0.0, "aliases":["temperature"]} {"name":"externalTemperature", "type":"float", "default":0.0}, {"name":"acceleration", "type":"float","default":0.0}, {"name":"velocity","type":"float","default":0.0}, {"name":"accelerometer","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"magneticField","type":[ "null",{ "type":"array", "items":"float" } ]}, {"name":"orientation","type":[ "null",{ "type":"array", "items":"float" } ]} ] }
  35. 35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema Support on Spring Cloud Stream 35 - DEMO -
  36. 36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ References 36 • Martin Kleppmann Schema Evolution in avro, thrift and protobufers: https:// martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html • http://dataintensive.net/ - Martin Kleppmann • The CQRS Journey: https://msdn.microsoft.com/en-us/library/jj554200.aspx • Oracle Datastore schema evolution : https://docs.oracle.com/cd/NOSQL/html/ GettingStartedGuide/schemaevolution.html • Building Microservices by Sam Newman: http://samnewman.io/books/ building_microservices/ • Apache Avro: https://avro.apache.org/docs/1.7.7/gettingstartedjava.html • https://github.com/viniciusccarvalho/schema-evolution-samples

×