Recommended
Recommended
More Related Content
What's hot
What's hot (13)
Similar to Big Data London Meetup on Customer Experience
Similar to Big Data London Meetup on Customer Experience (20)
Recently uploaded
Recently uploaded (20)
Big Data London Meetup on Customer Experience
- 1. How real-time data processing is used for application in customer experience? by Christos Hadjinikolis | SeniorML Engineer| Data Reply UK
- 2. Data ReplyUK Data Reply is the Reply group company offering a broad range of analytics and data processing services. • Data Science • Data Engineering • Visit our website:https://www.reply.com/data-reply/en/ 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 2
- 3. Agenda • An Introduction to Streaming • Customer Experience & Real-time Analytics • Intro • Approaches • Use Cases • Pain points • Why Flink? • Capabilities • TheOpen SourceCommunity 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 3
- 4. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 4 Two maindrivespush us towardsevolution! 1. TheMarket incentives 2. Technologyrequirements
- 5. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 5 Two maindrivespush us towardsevolution! 1. TheMarket incentives Incentives aim to provide value for money and contribute towards some sort of success with regards to certain goals.
- 6. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 6 Two maindrivespush us towardsevolution! 1. TheMarket incentives 2. Technologyrequirements The second is really a direct consequence of the first. In our effort to increase our efficiency and the effectiveness of our solutions we constantly improve the tools we use.
- 7. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 7 …we come up withsolutions thatlead to the emergence of new paradigms which oftenend up defininghow the market evolves! “ “
- 8. …let’s take a step back
- 9. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 9 Traditional Data Infrastructures 1. TransactionalProcessing *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 10. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 10 Traditional Data Infrastructures 1. TransactionalProcessing • Microservices *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 11. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 11 Traditional Data Infrastructures 1. TransactionalProcessing • Microservices 2. Analytical Processing *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 12. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 12 Statefull Processing • Virtually all data is createdas continuous streams of events. (Event logs) • Not just record-at-a-time transformations (local state) • An Example *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 13. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 13 Statefull Processing • Event-Driven Applications • Real-timerecommendations(e.g., forrecommending productswhile customersbrowsearetailer’swebsite) • Patterndetection or complexevent processing (e.g., forfrauddetectionin credit cardtransactions) • Anomalydetection(e.g., todetect attemptstointrudea computernetwork) *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 14. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 14 Statefull Processing • Streaming analytics • Monitoringthequalityof customer communications • Analyzinguserbehavior in mobileapplications • Ad-hocanalysisoflive datain consumertechnology *from Stream Processing with Apache Flink, by F. Huaske & V. Kalavri
- 15. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 15 Customer Experience • Findand applythe best possibleway to buildloyalty, reduce churn and generate incrementalrevenue.
- 16. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 16 Timely Visibility is key • Reactive/ad-hoc marketing • Improved awareness • Immediately and automatically collect, correlate, analyzeand act on customerdata in diverse formats across systems within secondsofoccurrence. • Addressproblems before theycyclethrough and negatively affect the customerexperience
- 17. So, how can stream processing technologies help with that?
- 18. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 18 Real-time situational awareness • Problems may occurat anytime: • customerservice satisfaction; • networkproblems; • fraudulent activityandmore. • Being able to detect these issuesas they’rehappening protects the customer—and the broaderorganization— from the fall-out.
- 19. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 19 Real-time 1:1 marketing • Offer micro-targeted, personalized offers that both enhance the customer experiencethrough providing highly relevant, timely value and drive incremental revenuethrough upsell and cross-sell opportunities.
- 20. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 20 Real-time fraud detection and prevention • Speed is key • It’s critically important that companies are ableto notonly to detect and stop fraud as it’s happening—down to the secondof occurrence—but also to leverage predictive analytics to preventit from happening in the first place..
- 21. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 21 Real-time Recommendations Gather customer insights can be leveragedto: • Predict customerneeds • Predict customerchurn • Adjust product strategy • Adjust marketing strategy
- 22. …let’s look at some use-cases
- 23. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 23 Streaming Media Industry • FragmentedIndustry • RabbitHoleproblem! • Users get lost • Difficult to choose from the available content! • How can companies make a difference? • AdvancedUser Search • RecommenderSystems
- 24. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 24 NetFlix & YouTube • Monetization strategy: • Premium Memberships & Advertising Challenge: • Makerecommendationsfromavery large corpusofvideos,while being certainthatthesmall numberof videosthatwill appearonone’s device are personalised andengagingforthe user.
- 25. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 25 Solution • Collaborative Filtering • Technology:2 NNLayers Video Corpus Candidate Generation Ranking x106 x102 Video Features x10 User history
- 26. …why Flink then?
- 27. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 27 Scalability& Durability • Flink is a distributed processing framework • Millisecond latencies while processing millions of events persecond. • Exactly-once state consistency guarantees. • Ability to run streaming applications 24/7 • Fault Tolerant
- 28. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 28 IoTExplosion
- 29. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 29 5G estimated to reach 1.5 billion subscriptions in 2024 1.5 billion
- 30. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 30 Some very interesting libs • CEP:Complex Event Processing Combines datafrommultiple sourcestoinfer eventsorpatterns thatsuggest morecomplicatedcircumstances. • SQL on Streams
- 31. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 31 Smooth Model Serving • Stream serialised models into your processing engine just like any other message • Assemble the model andstart using it without disruptions!
- 32. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 32 Trusted bymajor companies • Alibaba • Ebay • Uber • AWS • Huawei
- 33. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 33 Open Source Community • Supported by a vast and active community • Fast and agile evolution • Transparency • Reliability • Cost effective
- 34. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 34 Flink Meetup in London Community • Around since 2016 • 600+ members • O’Reily Partners • Wehost events regularly (once a month)
- 35. Data ReplyUK Data Reply is the Reply group company offering a broad range of analytics and data processing services. • Data Science • Data Engineering • Visit our website:https://www.reply.com/data-reply/en/ 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 35
- 36. 18/07/2019 Dr C. Hadjinikolis|SeniorML Engineer| Data ReplyUK 36 Thanks
Editor's Notes
- Incentives aim to provide value for money and contribute towards some sort of success with regards to certain goals. The second is really a direct consequent of the first. In our effort to increase our efficiency and the effectiveness of our solutions we constantly improve the tools we use. However, sometimes as we address technological problems and challenges, we come up with solutions that lead to the emergence of new paradigms which often end up defining how the market evolves.
- Incentives aim to provide value for money and contribute towards some sort of success with regards to certain goals. The second is really a direct consequent of the first. In our effort to increase our efficiency and the effectiveness of our solutions we constantly improve the tools we use. However, sometimes as we address technological problems and challenges, we come up with solutions that lead to the emergence of new paradigms which often end up defining how the market evolves.
- Incentives aim to provide value for money and contribute towards some sort of success with regards to certain goals. The second is really a direct consequent of the first. In our effort to increase our efficiency and the effectiveness of our solutions we constantly improve the tools we use. However, sometimes as we address technological problems and challenges, we come up with solutions that lead to the emergence of new paradigms which often end up defining how the market evolves.
- However, sometimes as we address technological problems and challenges, we come up with solutions that lead to the emergence of new paradigms which often end up defining how the market evolves. Streaming is one such example and in particular statefull stream processing
- Applications are usually connected to external services or face human users and continuously process incoming events such as orders, email, or clicks on a website. When an event is processed, an application reads its state or updates it by running transactions against the remote database system. Often, a database system serves multiple applications that sometimes access the same databases or tables.
- This application design can cause problems when applications need to evolve or scale. Since multiple applications might work on the same data representation or share the same infrastructure, changing the schema of a table or scaling a database system requires careful planning and a lot of effort. A recent approach to overcoming the tight bundling of applications is the microservices design pattern. Microservices are designed as small, self-contained, and independent applications. They follow the UNIX philosophy of doing a single thing and doing it well.
- Instead of running analytical queries directly on the transactional databases, the data is typically replicated to a data warehouse, a dedicated datastore for analytical query workloads. In order to populate a data warehouse, the data managed by the transactional database systems needs to be copied to it. The process of copying data to the data warehouse is called extract–transform–load (ETL). An ETL process extracts data from a transactional database, transforms it into a common representation that might include validation, value normalization, encoding, deduplication, and schema transformation, and finally loads it into the analytical database.
- Local state access provides very good performance compared to reading and writing queries against remote datastores.
- Event-driven applications are stateful streaming applications that ingest event streams and process the events with application-specific business logic. Depending on the business logic, an event-driven application can trigger actions such as sending an alert or an email or write events to an outgoing event stream to be consumed by another event-driven application. Event-driven applications are an evolution of microservices. They communicate via event logs instead of REST calls and hold application data as local state instead of writing it to and reading it from an external datastore, such as a relational database or key-value store.
- How do batch analytics work? Depending on the scheduling intervals it may take hours or days until a data point is included in a report. To some extent, the latency can be reduced by importing data into the datastore with a data pipeline application. However, even with continuous ETL there will always be a delay until an event is processed by a query.
- When problems around quality service provision surface, they cycle through the customer experience at a large scale quickly and decisively. Dropped calls, network problems, fraudulent activity and more – the ability to detect these issues as they’re happening protects the customer – and the broader organization – from the fall-out. The problem for CSPs that aren’t leveraging streaming analytics is that they can’t detect the disruptions fast enough – within seconds – to deflect the impact on the customer base.
- If CSPs are able to track the activity and movements of their subscribers at an individual level in real-time, they are in a great position to offer micro-targeted, personalized offers that both enhance the customer experience through providing highly relevant, timely value and drive incremental revenue through upsell and cross-sell opportunities. For instance, if a CSP can track the movements of an individual subscriber through from cell to cell, and, based on the trajectory, predict that the subscriber is about to leave the country and turn off data service to avoid international roaming charges, the CSP can issue a timely offer for a short-term international roaming plan. This not only provides the subscriber with a valuable service during their travels when they otherwise would have probably not used the service at all but also drives incremental revenue through upsell.
- A fragmented market A marketplace where there is no one company that can exert enough influence to move the industry in a particular direction. AI Applications in the Streaming Media industry: It’s a fragmented market Rabbit hole problem: too much content, users get lost To differentiate, one needs to leverage AI in two ways: Recommender Systems and advanced User search
- Monetization strategy: Premium memberships and advertising The YouTube Recommender System has to make recommendations from a very large corpus of videos (with order of magnitude being much higher that Netflix’s) while being certain that the small number of videos that will appear on one’s device are personalised and engaging for the user.
- Collaborative Filtering Deep Learning: 2 steps: Candidate Generation, then Ranking (A/B testing to improve)
- Add these numbers with the emergence of 5G
- The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.
- The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.
- The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.
- Open source is becoming the norm!
- Open source is becoming the norm!
- Open source is becoming the norm!