![from boto import kinesis
import testdata
import datetime
import json
import time
class Users(testdata.DictFactory):
custid = testdata.RandomInteger(1, 10)
amount = testdata.RandomInteger(1, 100)
gateway = testdata.RandomSelection(['visa', 'paypal', 'master', 'stripe', 'wallet'])
if __name__ == '__main__':
kinesis = kinesis.connect_to_region("ap-southeast-1")
print kinesis.describe_stream("payments")
print kinesis.list_streams()
for user in Users().generate(10):
print(user)
print kinesis.put_record("payments", json.dumps(user), "partitionkey")
time.sleep(1)](https://image.slidesharecdn.com/breakingdownsilosunifyingdatastreams-200603210920/75/Breaking-Down-Silos-Unifying-Data-Streams-15-2048.jpg)
Uploaded byconfluent
Breaking Down Silos, Unifying Data Streams
The document outlines a virtual meetup focused on unifying data streams and addresses how organizations can break down data silos. It includes event details, interactive features, and information on the Confluent community and Apache Kafka resources. Additionally, it covers the use of a streaming platform for real-time data processing and provides a demo setup for data handling using Kinesis.
More Related Content
Similar to Breaking Down Silos, Unifying Data Streams
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Breaking Down Silos, Unifying Data Streams
- 1. You are beingrecorded and this footage may be added to public channels WELCOME! Thank you for joining us in these unique circumstances. We hope you’re safe and well. Get $50 of free usage every month for your first 3 months! cnfl.io/MU-Try-Cloud Breaking down Silos, Unifying Data Streams Naveen Nandan, Confluent Zoom-chat Any time during the virtual meetup! Unmute and ask! (only before and after talks) EVENT INFORMATION 5:10pm-5:50pm Jakarta Time How to ask questions and interact during and afterwards Community slack channel: #events Throughout and after the event ALL UPCOMING MEETUPS NEW MEETUP EMAIL ALERTS THE MEETUP HUB EVENTS.CONFLUENT.IO/MEETUPS VIDEOS OF PAST MEETUPS SLIDES FROM THE TALKS cnfl.io/slack Confluent Community Catalyst program nominations are open for the class of 2020-2021. Find out more and nominate your community heroes! Confluent.io/Nominate All the Apache Kafka® learning resources you’ll need developer.confluent.io MORE LINKS
- 2. Breaking Down Silos, UnifyingData Streams
- 3. Modern Architectures areAdapting to New Data Requirements NoSQL DBs Big Data Analytics App App DWH Transactional Databases Analytics Databases Data Flow DB DB App App MOM MOM ETL ETL ESB App App
- 4. The Solution isa Streaming Platform for Real-Time Data Processing NoSQL DBs Big Data Analytics App App DWH Transactional Databases Analytics Databases Data Flow DB DB App AppApp App Streaming Platform
- 5.
- 6. Silos can existwithin organisations
- 7. ride-requests-stream location-stream bookings-stream payments-stream ride-requests-stream location-stream bookings-stream payments-stream Organisation A Organisation B Averagebookings per second Other apps... Average bookings per second Other apps...
- 8. ride-requests-stream location-stream bookings-stream payments-stream ride-requests-stream location-stream bookings-stream payments-stream Organisation A Organisation B Averagebookings per second Other apps... Average bookings per second Other apps... ride-requests-stream location-stream bookings-stream payments-stream Unified Data Streams (post merger) Merger
- 9. clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation A Active usersper game Ad revenue Other apps... clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation B Active users per game Ad revenue Other apps... clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation C Active users per game Other apps...
- 10. clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation A clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation BActive users per game Ad revenue Other apps... clickstream game-activity-stream player-activity-stream in-game-purchase-stream Organisation C Merger clickstream game-activity-stream player-activity-stream in-game-purchase-stream Active users per game Ad revenue Other apps... Unified Data Streams (post merger)
- 11.
- 12. Let’s get setup first
- 13. Let’s get setup first
- 14. Let’s get setup first
- 15. from boto importkinesis import testdata import datetime import json import time class Users(testdata.DictFactory): custid = testdata.RandomInteger(1, 10) amount = testdata.RandomInteger(1, 100) gateway = testdata.RandomSelection(['visa', 'paypal', 'master', 'stripe', 'wallet']) if __name__ == '__main__': kinesis = kinesis.connect_to_region("ap-southeast-1") print kinesis.describe_stream("payments") print kinesis.list_streams() for user in Users().generate(10): print(user) print kinesis.put_record("payments", json.dumps(user), "partitionkey") time.sleep(1)
- 16.