Brian Bulkowski : what startups can learn from real-time bidding
•Download as PPTX, PDF•
2 likes•2,105 views
Presentation about the technical issues of scaling out that apply to startup CTOs. https://ti.to/startup-cto-summit/sf
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Brian Bulkowski : what startups can learn from real-time bidding
Similar to Brian Bulkowski : what startups can learn from real-time bidding (20)
Recently uploaded
Recently uploaded (20)
Brian Bulkowski : what startups can learn from real-time bidding
- 1. What Starups Can Learn from Real-time Bidding Or “10 times faster, really?” Brian Bulkowski CTO and co-founder Aerospike © 2014 Aerospike. All rights reserved. Confidential 1
- 2. © 2014 Aerospike. All rights reserved. Confidential 2 Who am I ? ■ TRS-80, PC, Apple II, Vax 11/70, Wang ■ First product: lightpen university teaching kiosk ■ Networks: computers without people are boring ■ Liberate / NetComputer through the boom ■ 10B market cap in 1999, employee 32 ■ 2003-2007 “time off” ( startups ) ■ Citrusleaf / Aerospike history ■ 42 year old first-time CEO (me) ■ 2008 Prototype ■ 2010 First sale, get the band back together ■ 2011+ 3 rounds of funding (Draper, ALP, NEA, CNTP) ■ 70 employees, 2 offices brian@bulkowski.org brian@aerospike.com @bbulkow
- 3. © 2014 Aerospike. All rights reserved. Confidential 3 MILLIONS OF CONSUMERS BILLIONS OF DEVICES APP SERVERS DATA INSIGHTS WAREHOUSE Advertising Technology Stack WRITE CONTEXT In-memory NoSQL WRITE REAL-TIME CONTEXT READ RECENT CONTENT PROFILE STORE Cookies, email, deviceID, IP address, location, segments, clicks, likes, tweets, search terms... REAL-TIME ANALYTICS Best sellers, top scores, trending tweets BATCH ANALYTICS Discover patterns, segment data: location patterns, audience affinity
- 4. Introduction to Advertising: Real-time Bidding © 2014 Aerospike. All rights reserved. Confidential 4
- 5. North American RTB speeds & feeds ■ 1 to 6 billion cookies tracked ■ Some companies track 200M, some track 20B ■ Each bidder has their own data pool ■ Data is your weapon ■ Recent searches, behavior, IP addresses ■ Audience clusters (K-cluster, K-means) from offline Hadoop ■ “Remnant” from Google, Yahoo is about 0.6 million / sec ■ Facebook exchange: about 0.6 million / sec ■ “other” is 0.5 million / sec Currently about 3.0M / sec in North American © 2014 Aerospike. All rights reserved. Confidential 5
- 6. Financial Services – Intraday Positions ACCOUNT POSITIONS Read/Write Query Start of Day Data Loading End of Day Reconciliation LEGACY DATABASE (MAINFRAME) REAL-TIME DATA FEED © 2014 Aerospike. All rights reserved. Confidential 6 XDR 10M+ user records Primary key access 1M+ TPS planned Finance App Records App RT Reporting App
- 7. © 2014 Aerospike. All rights reserved. Confidential 7 Social Media MYSQL or POSTGRES (ROTATIONAL DISK) Java application tier Data abstraction and sharding Recent user generated content MODIFIED REDIS (SSD ENABLED) Content and Historical data
- 8. PRICING DATA Poll for Pricing Changes Store Latest Price © 2014 Aerospike. All rights reserved. Confidential 8 Travel Portal PRICING DATABASE (RATE LIMITED) SESSION MANAGEMENT Session Data Read Price XDR Airlines forced interstate banking Legacy mainframe technology Multi-company reservation and pricing Requirement: 1M TPS allowing overhead Travel App
- 9. QOS & Real-Time Billing for Telcos ■ In-switch Per HTTP request Billing ■US Telcos: 200M subscribers, 50 metros Execute Request © 2014 Aerospike. All rights reserved. Confidential 9 ■ In-memory use case SOURCE DEVICE/ USER Hot Standby Real-time Checks DESTINATION Request Update Device User Settings XDR Real-time Auth. QoS Billing Config Module App
- 10. Old Architecture ( scale out in 2000 ) CONTENT DELIVERY NETWORK Request routing and sharding LOAD BALANCER APP SERVERS CACHE DATABASE STORAGE © 2014 Aerospike. All rights reserved. Confidential 10
- 11. Modern Scale Out Architecture LOAD BALANCER Load balancer Simple stateless APP SERVERS CONTENT DELIVERY NETWORK Fast stateless IN-MEMORY NoSQL RESEARCH WAREHOUSE Long term cold storage HDFS BASED © 2014 Aerospike. All rights reserved. Confidential 11
- 12. How Fast You Can Go ( a few graphs ) © 2014 Aerospike. All rights reserved. Confidential 12
- 13. YCSB Performance Comparison 2014 © 2014 Aerospike. All rights reserved. Confidential 13
- 14. © 2014 Aerospike. All rights reserved. Confidential 14 Hot Analytics ■ High throughput Queries ■2 node cluster, 10 Indexes ■Query returns 100 of 50M records ■ Predictable low latency UN-PREDICTABLE LATENCY 128 – 300 ms 70 – 760 ms 7 – 10 ms QPS
- 15. © 2014 Aerospike. All rights reserved. Confidential 15 Amazon EC2 results
- 16. Mo’ speed, mo’ problems I don’t need that much speed ( you will ! ) © 2014 Aerospike. All rights reserved. Confidential 16 “ferrari speed” is bad ( but with camry reliability? ) I don’t believe you ( simple benchmark tooling ) Amazon will save me ( multicloud ) ( sell to API, platform companies )
- 17. Lessons Learned © 2014 Aerospike. All rights reserved. Confidential 17
- 18. Coding standards ( hiring is the obvious problem ) © 2014 Aerospike. All rights reserved. Confidential 18
- 19. Memory matters – the new coding style © 2014 Aerospike. All rights reserved. Confidential 19 CPU is free Memory is expensive Malloc is the ultimate enemy
- 20. Multithreading and reference counting “we multithread so you don’t have to” Hire old embedded guys Build reference counted libraries Memory access is the enemy © 2014 Aerospike. All rights reserved. Confidential 20
- 21. © 2014 Aerospike. All rights reserved. Confidential 21 Clients are hard
- 22. Creative corner cutting (opinionated) Server restart time doesn’t matter if the code is reliable Hash collisions don’t matter if the hash function hasn’t had a collision (RIPE-160) Rotational disk is dead ( correct for analytics ) Data commit doesn’t matter if the app server crashed © 2014 Aerospike. All rights reserved. Confidential 22
- 23. Aerospike’s Flash Experience © 2014 Aerospike. All rights reserved. Confidential 23 ■ Know your Flash ■ACT benchmark http://github.com/aerospike/act ■Read-write benchmark results back to 2011 ■ All clouds support flash now ■New EC2 instances ■Google Compute ■Internap, Softlayer, GoGrid… ■ Write durability usually not a problem with modern flash ■Durability is high (5 “drive writes per day” for 5 years, etc) ■Read performance suffers under write load anyway
- 24. Aerospike’s Flash Experience © 2014 Aerospike. All rights reserved. Confidential 24 ■ Densities increasing ■ 100G 2 years ago 800G today ■SATA vs PCI-E ■Appliances: 50T per 1U this year ■ Prices still dropping: perhaps $1/G next year ■ Intel P3700 results ■250K per device @ $2.5 / G ■ Old standard: Micron P320h 500K @ $8 / G ■ “Wide SATA” ■ 20 SATA drives ■ LSI “pass through mode” ■250K+ per server
- 25. © 2014 Aerospike. All rights reserved. Confidential 25 Use Open Source
- 26. © 2014 Aerospike. All rights reserved. Confidential 26
Editor's Notes
- Hello, my name is brian bulkowski My credentials are that I have spent over 25 years in silicon valley as an engineer, architect, technical lead, and CTO, working for companies like Novell, a streaming video company in the 90’s, a Netscape spinoff called Navio that went public under the name Liberate and reached a $10B valuation. As the initial Founder of Aerospike, I’ve had the privilege to work with companies reaching the highest levels of data scale, and help them achieve their business as well as technical goals. Companies such as BlueKai (now Oracle), Neustar, AppNexus, Yahoo, AOL, Ebay and a wide variety of others. Today my goal is to discuss the next generation data and scale architectures used inside the most leading-edge consumers of data – the advertising industry – which is now being used to power more responsive and personalized online experiences across many industries.
- This is the technology stack that major advertising technology companies built To sustain the crushing load of aggregating the clicks and views from so many websites Individual retailers are now using this same tech stack, for the same reason They wish to present an experience, and include Analytics-based results
- Let’s start with what happened in internet advertising that kicked off a scale revolution. In 2000, Google launched AdWords. This was the ability to buy advertising on a search keyword, instead of a web page. Display advertising was still static. Prior to 2005, internet advertising was traded statically. A person bought a certain number of impressions on a website – like buying 1M impressions on the Yahoo home page. These were “rotated” using a variety of technologies, but the model fit the existing model of media buying. Advertising companies would say “you want an article in Car and Driver? And on Car and Driver’s website?” In April 2007, Yahoo bought RightMedia. March 2008, Google acquired DoubleClick. Both of these systems matching display advertisements with consumers, based on “cost per click”, and revolutionized the industry. Google’s position in the center of advertising, as a black box, was challenged by open bidding exchanges. Founders of both RightMedia and DoubleClick created several companies, and an open “auction system” to democratize the flow of impressions (from publishers) and ads (from advertisers). These companies realized that real time pricing – individual auctions – were the only fair system for determining price. The RTB system has been used to monitize “long tail” (remenant) advertising, which catches users wherever they might go. “Premium” advertising is still in high demand, and may or may not enter the real-time bidding system. At the time of Facebook’s public launch, they used the same closed system that Google Search uses. They eventually found they didn’t have enough advertising content, income was down, thus they opened facebook exchange. At the time, many technologists said about advertising: the algorithms are simple, it’s only scaling that’s hard. Exchanges that slowed down publisher websites were quickly avoided. The 150 millisecond rule was established, with advertising platforms needing to deliver ads in 150ms to an end user. Platform companies realized the critical nature of keeping that contract – if they failed, there would be fewer ads per page, and less revenue to be had. Although some might argue that there is too much display advertising today, this exchange capacity has become necessary for satisfying mobile.
- Advertising was one of the pioneers, now other enterprises are understanding the need For this architecture. The traditional software & hardware providers scoff at these requirements and speed, but Cutting edge companies now understand that the technology is available, and are finding uses In their interprises. We are working with several financial services companies on providing an Intraday Positions Database. Instead of a cache / relational architecture, the requirement is around 1M TPS Velocity driven by: Faster trading Mobile customers Recommendations
- In China, Weibo, Alibaba, and TenCent are masters of agility, jumping on new trends in application design, at scale. A recent discussion with Pinterest showed a similar design. These companies see the benefits of the flexibility of in-memory NoSQL on the front application tier, but also abstract the application logic from database choice and scale using a separate layer. They also use this layer to separate users into “high traffic” and “low traffic”, and to allow different optimization patterns. An engineer at Weibo told me this was the most important optimization. Instead of allowing developers to directly access a database – making assumptions about that database’s performance and indexes – it is better to create an APPLICATION SPECIFIC – with DOMAIN KNOWLEDGE – layer.
- ( also will discuss retail here! The switch from CATALOG to INTERACTIVE and DEALS – same load! Must touch and track ALL USERS, those are the ones you need to influence, not those with existing affinity and logins ) Travel has long been an innovator in technology. Long ago, Airlines created some of the first credit card systems and intrastate banking Later, they pioneered real-time pricing for global seat reservation, but got hung up in technology They applied massive cache layers, but have consistancy problems. How often do you try to reserve and find the seat is no longer available, or at a different price? These companies are reaching out for the same internet technologies – with thousands of flights, removing caches, and allowing 100K queries per second, from partners and through open APIs to encourage rich apps.
- A new use case is happening with telecom and network providers. In order to provide rich network routing decisions, network operators are experimenting with a new form of Quality of service. These quality of service levels are driven by the CONTENT of HTTP interactions, using deep packet inspection. Use is also over 100K TPS, and requires ALWAYS UP database availability, just like the initial advertising users
- This is the old scale out architecture In the old system, you used cache and storage tiers, and traditional databases. This architecture does work. System like Facebook reached massive scale Few internet companies used storage vendor applications. Amazon and Google didn’t. Tell Srini’s story about scaling Yahoo Mobile with Netapp and Oracle.
- Here are the technologies and technology providers to watch in each area. Go through the App Layer in particular Research warehouse --- includes new systems like Spark --- easy to have multiple analytics systems, common in large deployments --- HDFS based systems
- Let’s get to some specifics
- Let’s get to some specifics
- Use enterprise offerings from major vendors like Intel, Micron, Samsung, …
- Use enterprise offerings from major vendors like Intel, Micron, Samsung, …
- Use OPEN SOURCE Engineers will resolve issues quickly if they can read the source Open source is the new escrow With a vibrant community, you can pay for extensions Pay those who wrote it Look for established projects with reference implementations Active releases Roadmap delivery