Evolving s3 story
•Download as PPTX, PDF•
0 likes•144 views
This document summarizes the evolution of AppsFlyer's raw data product from a simple Spark script to a premium data service over 3 months. It began as a prototype to address large file sizes and numbers for BI clients. Challenges included scaling, monitoring, security and schema. Improvements such as Parquet format and stateful S3 reduced costs and improved performance. The service was abstracted into microservices with automated tasks, search, and notifications. Monitoring, cost optimization, and prioritizing jobs further refined the product. It concluded having transitioned to a premium, self-serve offering with onboarding and defined schemas.
![AppsFlyer
• 28M raised top VC
• 200M To 13B Daily Events [3 Years]
• 40GB To 5TB [gz] daily text data
• 25 → 60ppl R&D during 2016
• Top 15 Israeli startups by inc.com](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Evolving s3 story
Similar to Evolving s3 story (20)
Recently uploaded
Recently uploaded (20)
Evolving s3 story
- 1. Evolving a premium raw data product from simple spark script in 3 month Avi Perez, Big Data Team Leader @AppsFlyer
- 2. AppsFlyer • 28M raised top VC • 200M To 13B Daily Events [3 Years] • 40GB To 5TB [gz] daily text data • 25 → 60ppl R&D during 2016 • Top 15 Israeli startups by inc.com
- 3. What We Do Media SourcesApp Developers App Users X10 9 4B$ in media payments annually measured
- 4. AppsFlyer Raw Data Channels Raw vs Aggregated • Real Time Stream From kafka • Online Query Data API (csv) HTTP Columnar DB S3
- 5. New Use Case • Big Clients with BI Systems • Very large files / large number of files • Tackling current limitations
- 6. secor Amazon S3 Rapid Prototype ... write notify read
- 8. Challenges ... •Scale in #clients •Client monitoring •Security •Schema •Flow & Control
- 9. Requests keep coming... • More Clients • More Events Types • Customizable Columns
- 10. What are we facing here...
- 12. Improving Data Format • Scanning a lot of data is easy...but not that fast • Being a big data company is not necessarily saying you need to read all your data fast
- 13. Moving to Parquet . . . Twitter & Cloudera • Columnar storage (load only what you need) • Space efficient (50% improvement) • Read Time efficient (98% improvement )
- 14. Stateful S3 Bucket Structure For automatic bots parsing
- 15. View Layer • Flatten fields mapping • Versions
- 16. From script to Micro Service • Tasks creation (Buckets, IAM, Credentials etc) • Search on Task Executions • Access to the report files • Get statuses from the Job HTTP • Highly available
- 17. Abstraction . . .
- 18. Moving Toward A Product . . . • Clients want SLA . . .
- 19. Service transparency Push notification to slack once there is an issue
- 20. Data Segregation
- 21. Results Loading data for specific clients Load specific clients raw data from 2.5TB compressed topic Same load with partitioning 1.5 min 30sec
- 22. From hard coded List to RDS Client A Client B ... ...
- 23. Secured Email Notifications click Get link
- 24. Vault • Secure Secret Storage • Dynamics Secrets • Data Encryption • Leasing and renewal • Revocation
- 25. Cost Optimization Helping our clients with download Daily sessions output file for one of the clients The same report compressed (.gz) 60G B 2.1 GB
- 26. Moving to YARN
- 28. Support keep asking the same questions….
- 29. Monitoring . . . Monitor, monitor, monitor…. • Metrics • Re-tries • PDs
- 30. Going premium . . • On boarding • Well defined schema fields • Self Serve and pricing
- 31. What we learn . . .
Editor's Notes
- sales come to r&d and asked a way to get organic data
- Big data analytics נותנים כלים למשתמשים שלנו למדוד כמה איכותי הטרפיק שהם מביאים מערוצי פירסום שונים מאיפה מגיע אותו טראפיק איכותי וכלים לעזור להם לקבל החלטות מכמויות אדירות של מידע שמפפיעות באופן ישיר על הככנסות שלהם
- Raw vs aggregate
- Not always using out dashboard We asked them what we do with our API’s Jobs ETL to run on s3 High load on AF systems How we can solve Many queries per day We have inherint limit of 200k rows CMS big clients, remove limitations. Very large companies want all their data Script “issue” that cost us 50k
- פתרון: נדרשנו לקבל החלטות קשות ב r&d בידיעה שאנחנו נצטרך לשלם בתחזוקה ידנית, אבל לא היתה ממש ברירה ורצינו שהלקוח האסטרטגי הזה יהיה לנו. וזה הפתרון שהצגנו 13B events → kafka → secor (service for persisting kafka log to S3) As sequence files SparkSQL on top on that Creating manually a bucket on our production S3 for that account with only List \ READ permissions.creating IAM specifc user manually and Providing him the credentails And running the process with chrons \ mesos each morning עלינו לפרודקשיין בתוך כמה ימים, ואפשרנו גישה רק לטופיק הקטן ביותר של התקנות. הלקוח חתם.
- Mobile App Letgo Raises $100 Million From Naspers To Take Over Classifieds In The U.S.
- reports/<Home Folder>/account /<event-type>-<date YYYY-MM-dd> reports/<Home Folder>/apps/app-id /<event-type>-<date YYYY-MM-dd>
- Flatten the schema Schema on write Schema on read Code reuse Versioning Readability \ Simplification
- Tell the story of lets go Which build the entire marketier team work flow base on the dasgbaord they are creating
- Analytics process which calculate each day to X app (partiton keys) And saved that as meta-data on the files bucket
- על מנת להגן על הלקוחות שלנו וגם להגן עלינו מעצמנו מטעויות. הטמענו שירות שנתן לנו דרכים להפיק keys \ secret באופן שרירותי
- Helping our clients to improve their download time from our S3
- Scheduled tasks were not executed Same job executed twice Not trivial to maintance DAG Dynamic allocation