Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure
•
1 like•169 views
Life doesn't happen in batches. You have to process data on time as it happens to make use of the time-value of information. Apart from a general introduction to stream processing and Apache Flink, this presentation shows how we as a successful data-driven gaming company apply this concept with our data architecture and how we use Apache Flink to run several streaming applications.
Recommended
More Related Content
Similar to Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure
Similar to Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure (20)
Recently uploaded
Recently uploaded (20)
Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure
- 1. SQUIRRELS AND ELEPHANTS Big Data and Streaming at InnoGames
- 2. LET‘S GO ON A ROADTRIP
- 3. IMAGINE You are driving with your family on the backseat
- 4. IMAGINE There is a lot of traffic, you have to concentrate
- 5. IMAGINE And now, while driving, you are closing your eyes
- 7. HOW DO YOU FEEL?
- 8. WOULD YOU EVER DO THAT? NOPE
- 9. METAPHOR The car is your company, team or project The passengers are your colleagues
- 10. WOULD YOU EVER DO THAT? NOPE IN A METAPHORICAL SENSE
- 11. YOU HAVE TO PROCESS DATA ON TIME AS IT HAPPENS
- 12. BATCH PROCESSING… …might cause accidents Because…
- 13. LIFE DOESN’T HAPPEN IN BATCHES https://mapr.com/ebooks/streaming-architecture/chapter-01-why-event-streaming.html © Ellen Friedman, Ted Dunning
- 18. SIMILARITIES THE FIRST IMPRESSION COUNTS The moment the customer enters the shop or the player plays his first session is crucial HALO EFFECT When one trait of a person or thing is used to make an overall judgment of that person or thing
- 19. IN ORDER TO MAKE A POSITIVE IMPACT A RESPONSE NEEDS TO HAPPEN QUICKLY
- 21. REAL-TIME USER REPORTS TRAFFIC GAS PRICE SPEED TRAPS
- 23. STREAM PROCESSING STREAMS OF DATA GPS DATA WEB INTERACTION SENSOR DATA
- 24. STREAM PROCESSING PROCESSING DATA IN MOTION
- 28. APACHE FLINK
- 29. APACHE FLINK Framework and distributed process engine for stateful computations on unbounded and bounded data streams
- 31. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS
- 32. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS AKA BATCH PROCESSING
- 33. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? ORDERED BY EVENT TIME PROCESSING TIME
- 34. TIME IN STREAMING EPISODE I EPISODE II EPISODE IIIEPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 20051977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? EVENT TIME ORDERED BY PROCESSING TIME
- 35. TUMBLING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 SENSOR SUM
- 36. SLIDING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 19 21 SENSOR SUM
- 37. BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
- 38. API BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
- 39. BUILDING BLOCKS SQL / TABLE API DataStream API ProcessFunction APIs (dynamic tables) (streams, windows) (events, state, time) HIGH LEVEL ANALYTICS API STREAM AND BATCH DATA PROCESSING STATEFUL EVENT- DRIVEN APPLICATIONS CONCISENESS EXPRESSIVENESS
- 40. LET‘S HAVE A CLOSER LOOK
- 41. LET‘S HAVE A CLOSER LOOK final StreamExecutionEnvironment env = getExecutionEnvironment(); final DataStreamSource<Integer> stream = env.fromElements(1, 2, 3, 4); stream .map((MapFunction<Integer, Integer>) i -> i + 2) .filter((FilterFunction<Integer>) i -> i % 2 == 0) .print(); env.execute(); DATA SOURCE TRANSFORMATION DATA SINK
- 42. RUNTIME YOUR FLINK APP FLINK RUNTIMEDEPLOY
- 43. RUNTIME
- 44. BIG DATA AND STREAMING AT INNOGAMES
- 45. TEAM
- 46. TEAM BUSINESS INTELLIGENCE DATA ENGINEERING DATA SCIENCE OPERATIONS
- 49. DATA ARCHITECTURE DATA PIPELINE DATA PLATFORM milliseconds, seconds, minutes hours, days, years
- 51. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
- 52. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
- 54. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE STREAM PROCESSING
- 55. USE CASE EVENT METRICS
- 56. Metrics.java stream .map(streamEvent -> new Tuple2<>(streamEvent.getEventName(), 1)) .keyBy(0) .timeWindow(Time.minutes(1)) .sum(1) .addSink(graphiteSink).setParallelism(1).name("event_counts");
- 57. USE CASE EVENT METRICS
- 58. USE CASE LOG00 MONITOR
- 59. KeyedStream<StreamEvent, Integer> stream = events .filter(event -> Arrays.asList("reg", "login").contains(event.getEventName())) .keyBy((KeySelector<StreamEvent, Integer>) StreamEvent::getPlayerId); Log00.java
- 60. Pattern<StreamEvent, StreamEvent> pattern = Pattern.<StreamEvent>begin("reg").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("reg"); } }).followedBy("login").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("login"); } }).within(Time.seconds(60)); Log00.java
- 61. PatternStream<StreamEvent> patternStream = CEP.pattern(stream, pattern); DataStream<Either<PatternResult, PatternResult>> patternResultStream = patternStream.select( (p, ts) -> sendTimeoutToGraphite(p, ts), p -> sendSuccessToGraphite(p) ); Log00.java
- 62. USE CASE LOG00 MONITOR
- 63. USE CASE NEAR TIME CRM(NTCRM)
- 64. USE CASE NTCRM EVENT BUS EVENT CLIENT EVENTGATEWAY PLAYER DATANTCRM React to events with interstitials in less than 10 seconds
- 65. USE CASE NTCRM Elvenar has a trading feature that sometimes causes confusion. With NTCRM we can react to this and show more details within interstitials exactly when the player needs it.
- 66. JUST DO IT DEMO TIME Check it out on Github: https://github.com/prenomenon/codetalks-flinkdemo
- 67. GET IN TOUCH InnoGames GmbH Friesenstrasse 13 20097 Hamburg https://www.innogames.com Volker Janz Senior Software Developer Corporate Systems - Analytics
- 68. GET IN TOUCH @prenomenon feedback appreciated
- 70. EAT ICE CREAM AND STREAM ON Great Flink training: http://training.data-artisans.com
- 71. NEXT UP EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI BUSINESS INTELLIGENCE
- 72. THAT’S IT FOR NOW…
- 73. BACKUP / DETAILS The following slides are not part of my talk but might give the reader more insights later
- 74. COMPANY SNAPSHOT More than 400 employees Founded 2007 in Germany Headquarter in Hamburg +160m EUR revenue made in 2017 7 live games >30 language versions
- 75. I AM LEGEND OUR PORTFOLIO Simulation Strategy RPG Browser Multi-device Mobile
- 77. WINDOWING KEYED NON-KEYED TASK 1 TASK N SOURCE TASK 1SOURCE KEY 1 KEY N ALL DATA
- 78. STATE SOURCE MAP DATA SINK SOURCE MAP SUM(C,D) OFFSET OFFSET SUM(A,B) AB CD
- 79. RABBIT HOLE
- 80. RUNTIME SOURCE MAP PRINTFILTER STREAMING DATAFLOW (CONDENSED VIEW) OPERATOR CHAIN OPERATOR OPERATOR TASK TASK TASK SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
- 81. RUNTIME SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW) A Flink cluster has a JOB MANAGER and multiple TASK MANAGERS. Each of those is a JVM.
- 82. RUNTIME Each Task Manager can manage MULTIPLE THREADS executing TASKS / SUBTASKS. SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR THREAD THREAD THREADSUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR THREAD THREAD SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
- 83. CHECKPOINTING checkpoint barrier n checkpoint barrier n-1 checkpoint n+1 checkpoint n checkpoint n-1 Consistent, incremental snapshots of distributed data stream and operator state Based on a paper from 1985, inspired by the Chandy-Lamport-Algorithm
- 84. STATE OPERATOR STATE KEYED STATE Bound only to an operator Bound to an operator and key PLUGGABLE BACKEND MULTIPLE PRIMITIVES SUPPORTED GUARANTEED CONSISTENCY IN CASE OF A FAILURE