SlideShare a Scribd company logo
1 of 33
moshe . [email_address] . com http :// top - performance . blogspot . com 1 Billion Events Per Day The Internet Building Blocks Moshe Kaplan, RockeTier The Performance Experts
1 Billion Events Per Day The Internet Building Blocks [email_address] http://top-performance.blogspot.com
RockeTier ,[object Object],[object Object],[object Object],[object Object]
Assumptions ,[object Object],[object Object],[object Object],[object Object],[object Object]
Assumptions…
Major Options
Presentation Objectives ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Who is Using MySQL?
MySQL Limitations ,[object Object],[object Object]
Why Do I Care? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
100K New Users/Week
The Network Effect
What Should I Do? ,[object Object],[object Object],[object Object]
The 3 Stages System
Ad Network Reference Architecture
Step I – Load Balancing ,[object Object],[object Object]
Step II – Web Server
Step III – In Memory Database UPDATE …  SET HIT=HIT+1 UPDATE …  SET HIT=HIT+1 Validate Validate Banner 1 Hit Banner 2 Hit
Step III – In Memory Database IMDB UPDATE …  SET HIT=HIT+1 UPDATE …  SET HIT=HIT+1 Validate Validate UPDATE SET HIT=HIT+41 UPDATE SET HIT=HIT+22 UPDATE SET HIT=HIT+87 Banner 1 Hit Banner 2 Hit
And Finally… Sharding
Vertical Sharding
Horizontal Sharding ,[object Object],[object Object],[object Object],Mod 10 = 0 Mod 10 = 1 Mod 10 = 2 Mod 10 = 3 Mod 10 = 4 Mod 10 = 5 Mod 10 = 6 Mod 10 = 7 Mod 10 = 8 Mod 10 = 9
[object Object],[object Object],[object Object],Horizontal Sharding
Horizontal Sharding ,[object Object],[object Object],[object Object],Mod 1000 = 4
Horizontal Sharding ,[object Object],[object Object],[object Object]
Sharding Management ,[object Object],[object Object],[object Object]
Reporting
Best Practices ,[object Object],[object Object],[object Object]
Lessons ,[object Object],[object Object],[object Object],[object Object],[object Object]
Lessons ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Lessons ,[object Object],[object Object],[object Object]
The Bottom Line: Grow ∞
Startup your Engines  Thank you [email_address] http://top-performance.blogspot.com Our Methodology Performance  problems  are  extremely  complex  and  due  to  the  diferent  technologies  deployed,  each  case  is  unique.  A  “typical”  performance  problem  requires  delving  into  databases,  application  servers,  client  technology,  code  in  difering  programming  languages  and  system  and  software  architectures. RockeTier implements a unique methodology in  order to simplify the problem and evaluate each performance  bottleneck, providing both an  immediate efective relief and  when necessary, design a gradual roadmap to speed up your  software system and make it scalable and robust. Our 5 steps methodology : 1. Detect: Pinpoint  your  performance  bottlenecks  using  various  tools  including  load  and  stress  tools,  code  profiling,  database profiling, network sniffing and code  review  to  detect performance bottlenecks in specific components. 2. Rate: Grade  each  bottleneck  by  importance  and  provide  immediate practical recommendations and performance  boost estimations. 3. Immediate effective relief: Provide immediate fixes and workarounds in a short time  frame helping you meet your urgent business needs. 4. Roadmap Planning: When  necessary,  redesign  next  generation  Solutions,  using proven robust and scalable solutions such as grid  and in memory databases. 5. Scale up and Scale out: In  cases  where  redesign  is  necessary  -  RockeTier  provides implementation or software design description  (SDD),  and guidance  for  in house programmers  for  the  implementation of the next generation scalable system,  which will meet your growing business needs. Your Value Business:  Achieve your business performance requirements. GreenIT:  Protect the environment and reduce CO2 emissions. Bottom Line:  Reducing hardware and 3rd party software cost. The Performance Experts Success Stories The Finance Sector: An  international  insurance  company managing over  20  Billion US dollars  in assets was facing poor performance  in  its  core  life  insurance  policy  software  system.  The  RockeTier  team  detected  bottlenecks  originating  from  several  software  infrastructure  modules.  A  practical  solution was implemented. The customer’s success criteria  was a 20% decrease in insurance policy creation run time,  Our solution provided a 40% decrease in run time! Telecom: A VC backed start-up company was facing critical installation  problems  in  the  leading  Israeli  cellular  operator.  Knowing  that  existing  system  performance  would  not  meet  client  requirements,  the  company  asked  RockeTier  to  help  it  boost  its  performance.  RockeTier  evaluated  the  system  and  implemented  a  workaround  to  the  system  database  architecture, boosting the overall system performance by 30%.  Following  that.  the RockeTier  team designed  the  company’s  next  generation  architecture,  meeting  a  throughput  of  3000Mbps by design. RockeTier at a Glance RockeTier  is  a  software  solutions  company,  which  utilizes  its  knowledge  and  skills  to  help  companies  from  both  the  enterprise  sector  and  the  start-up  industry.  RockeTier  has  numerous  success  stories  in  solving  customers’  system  performance bottlenecks and scale out limitations, providing  immediate  improvements  and workarounds  in  a  short  time  frame  and,  when  necessary,  redesign  and  implementation  of  the  next  generation  solutions  employing  grid  and/or  in- memory  databases  in  the  Web  2.0,  Telecom  and  finance  markets. Web 2.0: a  start-up  company  providing  an  innovative  electronic  advertising and billing system was facing its technological  limits. The RockeTier  team evaluated and  redesigned  its  system architecture and is currently implementing a scale  out grid mechanism and caching algorithms. The solution  supports 20  times  the original  capacity using  the  same  hardware. Moreover  it  supports  semi-linear  growth  (by  simple scale out) and high availability requirements. “ 20% reduction in transaction time  within 3 months” “ Boost Performance by a factor of 200” “ 200 million events per day”

More Related Content

Viewers also liked

Q1 2009 Earning Report of Cooper Industries, Inc.
Q1 2009 Earning Report of Cooper Industries, Inc.Q1 2009 Earning Report of Cooper Industries, Inc.
Q1 2009 Earning Report of Cooper Industries, Inc.earningreport earningreport
 
Air Jordan XII Product Launch
Air Jordan XII Product LaunchAir Jordan XII Product Launch
Air Jordan XII Product LaunchAdrian Parker
 
Q1 2009 Earning Report of Selective Insurance Group, Inc.
Q1 2009 Earning Report of Selective Insurance Group, Inc.Q1 2009 Earning Report of Selective Insurance Group, Inc.
Q1 2009 Earning Report of Selective Insurance Group, Inc.earningreport earningreport
 
Q1 2009 Earning Report of Travelers Companies Inc.
Q1 2009 Earning Report of Travelers Companies Inc.Q1 2009 Earning Report of Travelers Companies Inc.
Q1 2009 Earning Report of Travelers Companies Inc.earningreport earningreport
 
Q1 2009 Earning Report of Independent Bank Corp.
Q1 2009 Earning Report of Independent Bank Corp.Q1 2009 Earning Report of Independent Bank Corp.
Q1 2009 Earning Report of Independent Bank Corp.earningreport earningreport
 

Viewers also liked (8)

Q1 2009 Earning Report of Cooper Industries, Inc.
Q1 2009 Earning Report of Cooper Industries, Inc.Q1 2009 Earning Report of Cooper Industries, Inc.
Q1 2009 Earning Report of Cooper Industries, Inc.
 
Air Jordan XII Product Launch
Air Jordan XII Product LaunchAir Jordan XII Product Launch
Air Jordan XII Product Launch
 
Q1 2009 Earning Report of Selective Insurance Group, Inc.
Q1 2009 Earning Report of Selective Insurance Group, Inc.Q1 2009 Earning Report of Selective Insurance Group, Inc.
Q1 2009 Earning Report of Selective Insurance Group, Inc.
 
Q1 2009 Earning Report of Comcast Corp.
Q1 2009 Earning Report of Comcast Corp.Q1 2009 Earning Report of Comcast Corp.
Q1 2009 Earning Report of Comcast Corp.
 
Q1 2009 Earning Report of Travelers Companies Inc.
Q1 2009 Earning Report of Travelers Companies Inc.Q1 2009 Earning Report of Travelers Companies Inc.
Q1 2009 Earning Report of Travelers Companies Inc.
 
Q1 2009 Earning Report of Gardner Denver, Inc.
Q1 2009 Earning Report of Gardner Denver, Inc.Q1 2009 Earning Report of Gardner Denver, Inc.
Q1 2009 Earning Report of Gardner Denver, Inc.
 
Q1 2009 Earning Report of Independent Bank Corp.
Q1 2009 Earning Report of Independent Bank Corp.Q1 2009 Earning Report of Independent Bank Corp.
Q1 2009 Earning Report of Independent Bank Corp.
 
Q3 2009 Earning Report of Banco Santander S.A.
Q3 2009 Earning Report of Banco Santander S.A.Q3 2009 Earning Report of Banco Santander S.A.
Q3 2009 Earning Report of Banco Santander S.A.
 

Similar to 1 Billion Events per Day, Israel 3rd Java Technology Day, June 22, 2009

Organization Wide Performance Methodology (ITIL)
Organization Wide Performance Methodology (ITIL)Organization Wide Performance Methodology (ITIL)
Organization Wide Performance Methodology (ITIL)Moshe Kaplan
 
Cloud Computing Design Best Practices
Cloud Computing Design Best PracticesCloud Computing Design Best Practices
Cloud Computing Design Best PracticesMoshe Kaplan
 
Delia_J_Micu_resume_arch_Jan_2015
Delia_J_Micu_resume_arch_Jan_2015Delia_J_Micu_resume_arch_Jan_2015
Delia_J_Micu_resume_arch_Jan_2015Delia J. Micu
 
70% Improvement in Service and Product Delivery on Implementing DevOps
70% Improvement in Service and Product Delivery on Implementing DevOps70% Improvement in Service and Product Delivery on Implementing DevOps
70% Improvement in Service and Product Delivery on Implementing DevOpsCygnet Infotech
 
Contino Webinar - Migrating your Trading Workloads to the Cloud
Contino Webinar -  Migrating your Trading Workloads to the CloudContino Webinar -  Migrating your Trading Workloads to the Cloud
Contino Webinar - Migrating your Trading Workloads to the CloudBen Saunders
 
Secure-Net Technologies
Secure-Net TechnologiesSecure-Net Technologies
Secure-Net TechnologiesNamita Mahajan
 
How to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightHow to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightRackspace
 
Coghead Overview 21 Aug08
Coghead Overview 21 Aug08Coghead Overview 21 Aug08
Coghead Overview 21 Aug08Tomoaki Sawada
 
How to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeHow to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeVMware Tanzu
 
Modernizing Traditional Applications
Modernizing Traditional ApplicationsModernizing Traditional Applications
Modernizing Traditional ApplicationsDocker, Inc.
 
Performance Optimization: Incorporating Database and Code Optimzitation Into ...
Performance Optimization: Incorporating Database and Code Optimzitation Into ...Performance Optimization: Incorporating Database and Code Optimzitation Into ...
Performance Optimization: Incorporating Database and Code Optimzitation Into ...Michael Findling
 
IEEE-SCCPresentation.290214544
IEEE-SCCPresentation.290214544IEEE-SCCPresentation.290214544
IEEE-SCCPresentation.290214544ypai
 
Preston Peterson Resume January 2017
Preston Peterson Resume January 2017Preston Peterson Resume January 2017
Preston Peterson Resume January 2017Preston Peterson
 
An Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyAn Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyRoger Giuffre
 
Automation, Audits, and Apps Tour
Automation, Audits, and Apps TourAutomation, Audits, and Apps Tour
Automation, Audits, and Apps TourChef
 
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...Mydbops
 
An Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyAn Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyRoger Giuffre
 

Similar to 1 Billion Events per Day, Israel 3rd Java Technology Day, June 22, 2009 (20)

Organization Wide Performance Methodology (ITIL)
Organization Wide Performance Methodology (ITIL)Organization Wide Performance Methodology (ITIL)
Organization Wide Performance Methodology (ITIL)
 
Cloud Computing Design Best Practices
Cloud Computing Design Best PracticesCloud Computing Design Best Practices
Cloud Computing Design Best Practices
 
Delia_J_Micu_resume_arch_Jan_2015
Delia_J_Micu_resume_arch_Jan_2015Delia_J_Micu_resume_arch_Jan_2015
Delia_J_Micu_resume_arch_Jan_2015
 
70% Improvement in Service and Product Delivery on Implementing DevOps
70% Improvement in Service and Product Delivery on Implementing DevOps70% Improvement in Service and Product Delivery on Implementing DevOps
70% Improvement in Service and Product Delivery on Implementing DevOps
 
Contino Webinar - Migrating your Trading Workloads to the Cloud
Contino Webinar -  Migrating your Trading Workloads to the CloudContino Webinar -  Migrating your Trading Workloads to the Cloud
Contino Webinar - Migrating your Trading Workloads to the Cloud
 
Secure-Net Technologies
Secure-Net TechnologiesSecure-Net Technologies
Secure-Net Technologies
 
How to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightHow to Bring Shadow IT to the Light
How to Bring Shadow IT to the Light
 
Coghead Overview 21 Aug08
Coghead Overview 21 Aug08Coghead Overview 21 Aug08
Coghead Overview 21 Aug08
 
How to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeHow to Migrate Applications Off a Mainframe
How to Migrate Applications Off a Mainframe
 
Modernizing Traditional Applications
Modernizing Traditional ApplicationsModernizing Traditional Applications
Modernizing Traditional Applications
 
Performance Optimization: Incorporating Database and Code Optimzitation Into ...
Performance Optimization: Incorporating Database and Code Optimzitation Into ...Performance Optimization: Incorporating Database and Code Optimzitation Into ...
Performance Optimization: Incorporating Database and Code Optimzitation Into ...
 
IEEE-SCCPresentation.290214544
IEEE-SCCPresentation.290214544IEEE-SCCPresentation.290214544
IEEE-SCCPresentation.290214544
 
Preston Peterson Resume January 2017
Preston Peterson Resume January 2017Preston Peterson Resume January 2017
Preston Peterson Resume January 2017
 
An Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyAn Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech Company
 
Automation, Audits, and Apps Tour
Automation, Audits, and Apps TourAutomation, Audits, and Apps Tour
Automation, Audits, and Apps Tour
 
TimZbydniewski.Resume
TimZbydniewski.ResumeTimZbydniewski.Resume
TimZbydniewski.Resume
 
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...
Scaling managed MySQL Platform in Flipkart - (Sachin Japate - Flipkart) - Myd...
 
Senate Technologies
Senate TechnologiesSenate Technologies
Senate Technologies
 
An Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech CompanyAn Innovative Big-Data Web Scraping Tech Company
An Innovative Big-Data Web Scraping Tech Company
 
Arunprakash Alagesan
Arunprakash AlagesanArunprakash Alagesan
Arunprakash Alagesan
 

More from Moshe Kaplan

Spark and C Integration
Spark and C IntegrationSpark and C Integration
Spark and C IntegrationMoshe Kaplan
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big DataMoshe Kaplan
 
Introduciton to Python
Introduciton to PythonIntroduciton to Python
Introduciton to PythonMoshe Kaplan
 
Creating Big Data: Methodology
Creating Big Data: MethodologyCreating Big Data: Methodology
Creating Big Data: MethodologyMoshe Kaplan
 
Redis training for java software engineers
Redis training for java software engineersRedis training for java software engineers
Redis training for java software engineersMoshe Kaplan
 
MongoDB training for java software engineers
MongoDB training for java software engineersMongoDB training for java software engineers
MongoDB training for java software engineersMoshe Kaplan
 
MongoDB from Basics to Scale
MongoDB from Basics to ScaleMongoDB from Basics to Scale
MongoDB from Basics to ScaleMoshe Kaplan
 
MongoDB Best Practices for Developers
MongoDB Best Practices for DevelopersMongoDB Best Practices for Developers
MongoDB Best Practices for DevelopersMoshe Kaplan
 
Scale and Cloud Design Patterns
Scale and Cloud Design PatternsScale and Cloud Design Patterns
Scale and Cloud Design PatternsMoshe Kaplan
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBMoshe Kaplan
 
Web systems architecture, Performance and More
Web systems architecture, Performance and MoreWeb systems architecture, Performance and More
Web systems architecture, Performance and MoreMoshe Kaplan
 
Do Big Data and NoSQL Fit Your Needs?
Do Big Data and NoSQL Fit Your Needs?Do Big Data and NoSQL Fit Your Needs?
Do Big Data and NoSQL Fit Your Needs?Moshe Kaplan
 
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...Moshe Kaplan
 
MySQL Multi Master Replication
MySQL Multi Master ReplicationMySQL Multi Master Replication
MySQL Multi Master ReplicationMoshe Kaplan
 
mongoDB Performance
mongoDB PerformancemongoDB Performance
mongoDB PerformanceMoshe Kaplan
 
Web Systems Architecture by Moshe Kaplan
Web Systems Architecture by Moshe KaplanWeb Systems Architecture by Moshe Kaplan
Web Systems Architecture by Moshe KaplanMoshe Kaplan
 
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuff
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuffBig Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuff
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuffMoshe Kaplan
 

More from Moshe Kaplan (20)

Spark and C Integration
Spark and C IntegrationSpark and C Integration
Spark and C Integration
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big Data
 
Introduciton to Python
Introduciton to PythonIntroduciton to Python
Introduciton to Python
 
Creating Big Data: Methodology
Creating Big Data: MethodologyCreating Big Data: Methodology
Creating Big Data: Methodology
 
Git Tutorial
Git TutorialGit Tutorial
Git Tutorial
 
Redis training for java software engineers
Redis training for java software engineersRedis training for java software engineers
Redis training for java software engineers
 
MongoDB training for java software engineers
MongoDB training for java software engineersMongoDB training for java software engineers
MongoDB training for java software engineers
 
MongoDB from Basics to Scale
MongoDB from Basics to ScaleMongoDB from Basics to Scale
MongoDB from Basics to Scale
 
MongoDB Best Practices for Developers
MongoDB Best Practices for DevelopersMongoDB Best Practices for Developers
MongoDB Best Practices for Developers
 
The api economy
The api economyThe api economy
The api economy
 
Big Data Workshop
Big Data WorkshopBig Data Workshop
Big Data Workshop
 
Scale and Cloud Design Patterns
Scale and Cloud Design PatternsScale and Cloud Design Patterns
Scale and Cloud Design Patterns
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 
Web systems architecture, Performance and More
Web systems architecture, Performance and MoreWeb systems architecture, Performance and More
Web systems architecture, Performance and More
 
Do Big Data and NoSQL Fit Your Needs?
Do Big Data and NoSQL Fit Your Needs?Do Big Data and NoSQL Fit Your Needs?
Do Big Data and NoSQL Fit Your Needs?
 
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...
The VP R&D Open Seminar on Project Management, SCRUM, Agile and Continuous De...
 
MySQL Multi Master Replication
MySQL Multi Master ReplicationMySQL Multi Master Replication
MySQL Multi Master Replication
 
mongoDB Performance
mongoDB PerformancemongoDB Performance
mongoDB Performance
 
Web Systems Architecture by Moshe Kaplan
Web Systems Architecture by Moshe KaplanWeb Systems Architecture by Moshe Kaplan
Web Systems Architecture by Moshe Kaplan
 
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuff
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuffBig Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuff
Big Data Seminar: Analytics, Hadoop, Map Reduce, Mongo and other great stuff
 

Recently uploaded

My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 

1 Billion Events per Day, Israel 3rd Java Technology Day, June 22, 2009

  • 1. moshe . [email_address] . com http :// top - performance . blogspot . com 1 Billion Events Per Day The Internet Building Blocks Moshe Kaplan, RockeTier The Performance Experts
  • 2. 1 Billion Events Per Day The Internet Building Blocks [email_address] http://top-performance.blogspot.com
  • 3.
  • 4.
  • 7.
  • 8. Who is Using MySQL?
  • 9.
  • 10.
  • 13.
  • 14. The 3 Stages System
  • 15. Ad Network Reference Architecture
  • 16.
  • 17. Step II – Web Server
  • 18. Step III – In Memory Database UPDATE … SET HIT=HIT+1 UPDATE … SET HIT=HIT+1 Validate Validate Banner 1 Hit Banner 2 Hit
  • 19. Step III – In Memory Database IMDB UPDATE … SET HIT=HIT+1 UPDATE … SET HIT=HIT+1 Validate Validate UPDATE SET HIT=HIT+41 UPDATE SET HIT=HIT+22 UPDATE SET HIT=HIT+87 Banner 1 Hit Banner 2 Hit
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32. The Bottom Line: Grow ∞
  • 33. Startup your Engines Thank you [email_address] http://top-performance.blogspot.com Our Methodology Performance problems are extremely complex and due to the diferent technologies deployed, each case is unique. A “typical” performance problem requires delving into databases, application servers, client technology, code in difering programming languages and system and software architectures. RockeTier implements a unique methodology in order to simplify the problem and evaluate each performance bottleneck, providing both an immediate efective relief and when necessary, design a gradual roadmap to speed up your software system and make it scalable and robust. Our 5 steps methodology : 1. Detect: Pinpoint your performance bottlenecks using various tools including load and stress tools, code profiling, database profiling, network sniffing and code review to detect performance bottlenecks in specific components. 2. Rate: Grade each bottleneck by importance and provide immediate practical recommendations and performance boost estimations. 3. Immediate effective relief: Provide immediate fixes and workarounds in a short time frame helping you meet your urgent business needs. 4. Roadmap Planning: When necessary, redesign next generation Solutions, using proven robust and scalable solutions such as grid and in memory databases. 5. Scale up and Scale out: In cases where redesign is necessary - RockeTier provides implementation or software design description (SDD), and guidance for in house programmers for the implementation of the next generation scalable system, which will meet your growing business needs. Your Value Business: Achieve your business performance requirements. GreenIT: Protect the environment and reduce CO2 emissions. Bottom Line: Reducing hardware and 3rd party software cost. The Performance Experts Success Stories The Finance Sector: An international insurance company managing over 20 Billion US dollars in assets was facing poor performance in its core life insurance policy software system. The RockeTier team detected bottlenecks originating from several software infrastructure modules. A practical solution was implemented. The customer’s success criteria was a 20% decrease in insurance policy creation run time, Our solution provided a 40% decrease in run time! Telecom: A VC backed start-up company was facing critical installation problems in the leading Israeli cellular operator. Knowing that existing system performance would not meet client requirements, the company asked RockeTier to help it boost its performance. RockeTier evaluated the system and implemented a workaround to the system database architecture, boosting the overall system performance by 30%. Following that. the RockeTier team designed the company’s next generation architecture, meeting a throughput of 3000Mbps by design. RockeTier at a Glance RockeTier is a software solutions company, which utilizes its knowledge and skills to help companies from both the enterprise sector and the start-up industry. RockeTier has numerous success stories in solving customers’ system performance bottlenecks and scale out limitations, providing immediate improvements and workarounds in a short time frame and, when necessary, redesign and implementation of the next generation solutions employing grid and/or in- memory databases in the Web 2.0, Telecom and finance markets. Web 2.0: a start-up company providing an innovative electronic advertising and billing system was facing its technological limits. The RockeTier team evaluated and redesigned its system architecture and is currently implementing a scale out grid mechanism and caching algorithms. The solution supports 20 times the original capacity using the same hardware. Moreover it supports semi-linear growth (by simple scale out) and high availability requirements. “ 20% reduction in transaction time within 3 months” “ Boost Performance by a factor of 200” “ 200 million events per day”

Editor's Notes

  1. Initial setup of a u-Page from toolbar 14M users today 20M users 2009Q1 600K downloads per day Moving from 1:N templates to 10-15% users that updates the page (from 1:1000 templates/users ratio to 1:7 templates/users) Statistics: user changes: 1 per week, upload: 2 times a day Wants to use the default if the user did not modified the template. However, wants to support pushed changes from templates to users pages, even if those were changed Have the same situation now, but now it’s saved on the users desktop Current applicative cache: - int: toolbar_id, list of int: widget_id[] - int: widget_id, object: widget - int: user_id, object: user Table design: - User id, tab id and user_tab_id are GUID – meaning that they can be distributed between databases - Other are not (may be needed to support?) - Expected XML scheme size: 45KB (up to 500KB) Why migration to MySQL will be problematic (they are already using the current SQL Server SP features): - Maximal row size * http://dev.mysql.com/doc/refman/5.0/en/innodb-restrictions.html * The maximum row length, except for VARBINARY , VARCHAR , BLOB and TEXT columns, is slightly less than half of a database page. That is, the maximum row length is about 8000 bytes - Scope identity - For XML - Bulk Insert from XML - Rollback/Transaction - BEGIN TRY Applicative changes by the user: - Move - Delete - Add - Minimize - Change internal parameters Options to be considered: - SQL Server based large machine - Sharding based on MySQL, SQL Server - Gigaspaces solution: felt that it’s a large machine, and they prefer the database way Other options - Saving only XML in the current way - Save the summed page configuration in an XML, so little read should be done from the DB (Tab based) - Write 20K files of 40KB each on the my laptop HD: 149s - Read XML: 109.5s - Write to DB: 1789s Use serialization of .Net - Save the XML on the disk in order to avoid variable length fields - Use memcached to hold the hash of users? Things to be considered: - What horizontal sharding algorithm should be selected - Hibernate Shards – provided by Google. Still beta-testing phase - What vertical sharding tables should be spitted to different databases - How do you manage so many databases (distribute data and so on) - There is not really an option to do that - Defining optimal table sizes - Retrieval of data from the disk vs. Getting data from the tables: Tab (1 per displayed page), Zone (3 per displayed page), Widget instance (10-300 per displayed page + should be extracted with/out zones) - OLAP solution to merge data OLAP solution Toolbar design: - Saving
  2. Our Methodology Performance problems are extremely complex and due to the diferent technologies deployed, each case is unique. A “typical” performance problem requires delving into databases, application servers, client technology, code in difering programming languages and system and software architectures. RockeTier implements a unique methodology in order to simplify the problem and evaluate each performance bottleneck, providing both an immediate efective relief and when necessary, design a gradual roadmap to speed up your software system and make it scalable and robust. Our 5 steps methodology : 1. Detect: Pinpoint your performance bottlenecks using various tools including load and stress tools, code profiling, database profiling, network sniffing and code review to detect performance bottlenecks in specific components. 2. Rate: Grade each bottleneck by importance and provide immediate practical recommendations and performance boost estimations. 3. Immediate effective relief: Provide immediate fixes and workarounds in a short time frame helping you meet your urgent business needs. 4. Roadmap Planning: When necessary, redesign next generation Solutions, using proven robust and scalable solutions such as grid and in memory databases. 5. Scale up and Scale out: In cases where redesign is necessary - RockeTier provides implementation or software design description (SDD), and guidance for in house programmers for the implementation of the next generation scalable system, which will meet your growing business needs. Your Value Business: Achieve your business performance requirements. GreenIT: Protect the environment and reduce CO2 emissions. Bottom Line: Reducing hardware and 3rd party software cost. The Performance Experts Success Stories The Finance Sector: An international insurance company managing over 20 Billion US dollars in assets was facing poor performance in its core life insurance policy software system. The RockeTier team detected bottlenecks originating from several software infrastructure modules. A practical solution was implemented. The customer’s success criteria was a 20% decrease in insurance policy creation run time, Our solution provided a 40% decrease in run time! Telecom: A VC backed start-up company was facing critical installation problems in the leading Israeli cellular operator. Knowing that existing system performance would not meet client requirements, the company asked RockeTier to help it boost its performance. RockeTier evaluated the system and implemented a workaround to the system database architecture, boosting the overall system performance by 30%. Following that. the RockeTier team designed the company’s next generation architecture, meeting a throughput of 3000Mbps by design. RockeTier at a Glance RockeTier is a software solutions company, which utilizes its knowledge and skills to help companies from both the enterprise sector and the start-up industry. RockeTier has numerous success stories in solving customers’ system performance bottlenecks and scale out limitations, providing immediate improvements and workarounds in a short time frame and, when necessary, redesign and implementation of the next generation solutions employing grid and/or in- memory databases in the Web 2.0, Telecom and finance markets. Web 2.0: a start-up company providing an innovative electronic advertising and billing system was facing its technological limits. The RockeTier team evaluated and redesigned its system architecture and is currently implementing a scale out grid mechanism and caching algorithms. The solution supports 20 times the original capacity using the same hardware. Moreover it supports semi-linear growth (by simple scale out) and high availability requirements. “ 20% reduction in transaction time within 3 months” “ Boost Performance by a factor of 200” “ 200 million events per day”
  3. The world is changing more and more fast You have to minimize NRE You must support unexpected demand You must provide top service (people now leaves cell operator after single incident, rather then 5, 10 years ago Firms a vulnerable – Citi worth 20Bill $ instead of 200 Bill a year ago Break and Mortar bookstores 15 years ago and Amazon…. Will it happen again to banks, insurance, real estate agencies… IS YOUR MARKET THE NEXT FOR PENETRATION – Finance? Real Estate? How to win a rival that is not existing yet? http://www.johnmwillis.com/ibm/cloud-computing-and-the-enterprise/ -
  4. Sharded database – OLTP Little/No reporting OLAP – must be implemented for reporting Loads data from sharded DBs Custom mechanism Any commercial
  5. Start with nothing: storage, FW, LB, Server and grow… Can buy servers for more than hour
  6. [email_address] http://top-performance.blogspot.com http://www.rocketier.com