Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ThoughtWorks Technology Radar Roadshow - Melbourne

4,033 views

Published on

In the Melbourne edition of a 4-city Technology Radar roadshow, ThoughtWorks Australia's Head of Technology Scott Shaw and senior consultant Jen Smith cover topics from all 4 quadrants of the latest edition of the ThoughtWorks Technology Radar. This presentation covers Reactive Architectures, Hamms, Spring Boot vs. Nancy, and Impala.

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

ThoughtWorks Technology Radar Roadshow - Melbourne

  1. 1. TECHNOLOGY
 RADAR May 2015 — Our thoughts on the technology and trends that are shaping the future 1
  2. 2. 2
  3. 3. 3 TECHNOLOGYADVISORYBOARD
  4. 4. 4
  5. 5. 5
  6. 6. 6
  7. 7. THEMES FOR THIS ISSUE 7
  8. 8. TECHNIQUES 8
  9. 9. TECHNIQUES 8
  10. 10. 9 ADOPT 1. Consumer-driven contract testing NEW 2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team TECHNIQUES
  11. 11. 10 TECHNIQUES Architectures for the digital world (theme: innovations in architecture)
  12. 12. 10 TECHNIQUES 18 15 14 12 REACTIVE ARCHITECTURES FLUX ENTERPRISE DATA LAKE APPEND-ONLY DATA STORE Architectures for the digital world (theme: innovations in architecture)
  13. 13. Attrition Acquisition Retention Activation Referral Ads ARCHITECTURES FOR THE DIGITAL JOURNEY 11 Emails Google
  14. 14. Legacy Systems RIGHT DATA, RIGHT PLACE, RIGHT TIME 12 Legacy SystemsLegacy Systems Web Analytics Operational Metrics Insights
  15. 15. 13 Business Logic (functions) Legacy Systems Archivers File Store (S3) Insights Analytics Reports, Model Parameters Spark, Hadoop Microservices Append- only Database Flux-based Web Application User Actions View Rendering (react.js) Transactions, Web Analytics, Operational Logs Subscribed Events Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …)
  16. 16. 13 Business Logic (functions) Legacy Systems Archivers File Store (S3) Insights Analytics Reports, Model Parameters Spark, Hadoop Microservices Append- only Database Flux-based Web Application User Actions View Rendering (react.js) Transactions, Web Analytics, Operational Logs Subscribed Events Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …)
  17. 17. 14 Insights Analytics Reports, Model Parameters Spark, Hadoop Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) Transactions, Web Analytics, Operational Logs Subscribed Events Business Logic (functions) Legacy Systems Archivers File Store (S3)Microservices Append- only Database Flux-based Web Application User Actions View Rendering (react.js)
  18. 18. 14 Insights Analytics Reports, Model Parameters Spark, Hadoop Transactions, Web Analytics, Operational Logs Subscribed Events Business Logic (functions) Legacy Systems Archivers File Store (S3)Microservices Append- only Database Flux-based Web Application User Actions View Rendering (react.js)
  19. 19. 15 Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) Insights Analytics Reports, Model Parameters Spark, Hadoop Append- only Database Flux-based Web Application User Actions View Rendering (react.js) Transactions, Web Analytics, Operational Logs Subscribed Events Business Logic (functions) Legacy Systems Archivers File Store (S3)Microservices
  20. 20. 15 Transactions, Web Analytics, Operational Logs Subscribed Events Business Logic (functions) Legacy Systems Archivers File Store (S3)Microservices
  21. 21. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) Microservices Append- only Database Flux-based Web Application User Actions View Rendering (react.js) Transactions, Web Analytics, Operational Logs Subscribed Events Business Logic (functions) Legacy Systems Archivers
  22. 22. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …)
  23. 23. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) ALL DATA IN MOTION IS IMMUTABLE
  24. 24. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) ALL DATA IN MOTION IS IMMUTABLE FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
  25. 25. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …) ALL DATA IN MOTION IS IMMUTABLE FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
  26. 26. 16 Insights Analytics Reports, Model Parameters Spark, Hadoop File Store (S3) Event Queue (Time Series Database, Apache Kafka, AWS Kinesis, Eventstore, …)PUSHES RESPONSIBILITY FOR DATA QUALITY BACK ON THE SOURCE SYSTEMS ALL DATA IN MOTION IS IMMUTABLE FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
  27. 27. 17 ADOPT 1. Consumer-driven contract testing NEW 2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team TECHNIQUES
  28. 28. TOOLS 18
  29. 29. TOOLS 18
  30. 30. 19 TOOLS ADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul 55. Cursive 56. Gitlab 57. Hamms NEW 58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW HOLD 76. Citrix for development
  31. 31. 20 TOOLS
  32. 32. 20 TOOLS 57 HAMMS
  33. 33. WE BUILD DISTRIBUTED SYSTEMS BY DEFAULT 21
  34. 34. 22
  35. 35. 22 FAILUREISTHENORM… but testing it is tough
  36. 36. HAMMS Simulating badly behaved servers https://github.com/kevinburke/hamms
  37. 37. STARTING 24
  38. 38. STARTING 24
  39. 39. IF AT FIRST YOU DON’T SUCCEED… 25 http://127.0.0.1:5512/?key=demo&tries=6
  40. 40. IF AT FIRST YOU DON’T SUCCEED… 25 http://127.0.0.1:5512/?key=demo&tries=6
  41. 41. NOBODY IS 100% PERFECT 26 http://127.0.0.1:5513/?failrate=0.25
  42. 42. NOBODY IS 100% PERFECT 26 http://127.0.0.1:5513/?failrate=0.25
  43. 43. OTHER FAILURE MODES 27 Invalid content lengths and types Non-http responses Slow responses Partial responses
  44. 44. WHEN WOULD YOU USE HAMMS? 28 Automated testing Usability testing
  45. 45. BENEFIT OF COMMON PROTOCOLS 29 HTTP, TCP/IP Well-known failure cases Build on other people’s hard work!
  46. 46. GOING BEYOND UNIT TESTING… 30 ZED ATTACK PROXY CONSUMER-DRIVEN CONTRACT TESTING REST-ASSURED GORHAMMSTEST.GENERATIVE QUICKCHECK SIMULANT
  47. 47. RESOURCES 31 https://github.com/kevinburke/hamms
  48. 48. 32 TOOLS ADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul 55. Cursive 56. Gitlab 57. HAMMS NEW 58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW HOLD 76. Citrix for development
  49. 49. LANGUAGES & FRAMEWORKS 33
  50. 50. LANGUAGES & FRAMEWORKS 33
  51. 51. 34 LANGUAGES & FRAMEWORKS ADOPT 77. Nancy TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift HOLD 92. JSF
  52. 52. 35 LANGUAGES & FRAMEWORKS
  53. 53. 35 LANGUAGES & FRAMEWORKS 85 SPRING BOOT NANCY77
  54. 54. A TALE OF TWO WEB FRAMEWORKS 36 Java/Spring C#/.NET Lightweight ✓ ✓ Low-ceremony ✓ ✓ Self-hosted ✓ ✓ Opinionated ✓ ✓ boot
  55. 55. ON THE SURFACE, VERY SIMILAR 37
  56. 56. ON THE SURFACE, VERY SIMILAR 37
  57. 57. BUT WHAT’S UNDER THE COVERS? 38 Spring Boot’s pom.xml 1847 lines in total!
  58. 58. BUT WHAT’S UNDER THE COVERS? 38 Nancy’s Nuget page
  59. 59. BUT WHAT’S UNDER THE COVERS? 38 Nancy’s Nuget page
  60. 60. FRAMEWORKS VS. COMPOSITION 39 Spring Framework Your Spring Boot App Jetty Your App Code Owin Nancy.Owin Nancy Composes Calls higher-order functions
  61. 61. 40 LANGUAGES & FRAMEWORKS ADOPT 77. Nancy TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift HOLD 92. JSF
  62. 62. PLATFORMS 41
  63. 63. PLATFORMS 41
  64. 64. 42 PLATFORMS ADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
  65. 65. 43 PLATFORMS
  66. 66. 43 PLATFORMS 26CLOUDERA IMPALA
  67. 67. 44 IMPALA Elegant, nimble querying of big, lumbering data
  68. 68. 45 HADOOP Fast processing Scalable Commodity hardware Elegant failure handling HDFS
  69. 69. MAPREDUCE 46 • Powerful … but complicated • Not always the right model Input Splitting Mapping Sorting Reducing
  70. 70. SQL 47 SELECT … FROM … WHERE … GROUP BY … Declarative Simple Well known
  71. 71. RELATIONAL DATABASES 48 Not always good choice for adhoc analysis:
  72. 72. RELATIONAL DATABASES 48 Not always good choice for adhoc analysis: • Data needs to be optimised for consumption ETL ? ?
  73. 73. RELATIONAL DATABASES 48 Not always good choice for adhoc analysis: • Data needs to be optimised for consumption • Scaling is tough ? ? ? ? ETL ? ?
  74. 74. IMPALA 49 An SQL engine that runs on Hadoop
  75. 75. IMPALA IN ACTION 50 External tables create EXTERNAL TABLE books( id BIGINT, isbn STRING, category STRING, publish_date TIMESTAMP, publisher STRING, price FLOAT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/books/';
  76. 76. IMPALA IN ACTION 51 SELECT category, count(*) FROM books GROUP BY category Real-time querying of large datasets
  77. 77. IMPALA IN ACTION 51 SELECT category, count(*) FROM books GROUP BY category +---------------------------+----------+ | category | count(*) | +---------------------------+----------+ | COMPUTERS | 312604 | | SPORTS-RECREATION | 312766 | | PETS | 313039 | | CRAFTS-HOBBIES | 312438 | | FAMILY-RELATIONSHIPS | 312435 | | HOUSE-HOME | 313365 | Runtime : 5 seconds (~15 million rows) Real-time querying of large datasets
  78. 78. IMPALA IN ACTION 52 Arbitrary joins SELECT category, sum(quantity) as total_quantity FROM books JOIN [SHUFFLE] transactions ON books.id = transactions.book_id GROUP BY category;
  79. 79. IMPALA IN ACTION 52 Arbitrary joins SELECT category, sum(quantity) as total_quantity FROM books JOIN [SHUFFLE] transactions ON books.id = transactions.book_id GROUP BY category; +---------------------------+----------------+ | category | total_quantity | +---------------------------+----------------+ | STUDY-AIDS | 10171749 | | SELF-HELP | 10238182 | | TECHNOLOGY-ENGINEERING | 10207003 | | SOCIAL-SCIENCE | 10177075 | | HISTORY | 10238243 | | LAW | 10229825 | | PERFORMING-ARTS | 10231744 | Runtime : 19 seconds
  80. 80. WHEN WOULD YOU USE IMPALA? 53 Adhoc/exploratory data analysis Existing Business Intelligence tools via ODBC/JDBC
  81. 81. DIVERSIFY CONSUMERS OF YOUR DATA 54 ? ? ? ? Our Data Curious Person Financial Analyst Customer Experience Guru Dashboard Whizz
  82. 82. DIVERSIFY CONSUMERS OF YOUR DATA 54 ? BI Tools ? ? ? ? Our Data Curious Person Financial Analyst Customer Experience Guru Dashboard Whizz
  83. 83. DIVERSIFY CONSUMERS OF YOUR DATA 54 ? BI Tools Data Scientists Graph Fans Map Reducers ? ? ? ? Our Data Curious Person Financial Analyst Customer Experience Guru Dashboard Whizz
  84. 84. MADE POSSIBLE BY V2.0 DESIGN CHOICES 55 Diagram taken from: http://hortonworks.com/blog/apache-hadoop-2-is-ga/
  85. 85. FURTHER RESOURCES 56 http://impala.io/ Tutorials from Cloudera: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/ topics/impala_tutorial.html Impala and Amazon EMR: http://docs.aws.amazon.com/ElasticMapReduce/latest/ DeveloperGuide/emr-impala.html Hadoop: http://hadoop.apache.org/ Yarn + hadoop v2: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ YARN.html
  86. 86. 57 PLATFORMS ADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
  87. 87. 58 Scott Shaw @scottwshaw Jen Smith @JenniferSmithCo thoughtworks.com/radar

×