Analysis of user experience is typically done by taking a random sample of users, measuring their experiences and extracting a single number from that sample. In terms of web performance, the experience we need to measure is user perceived page load time, and the single number we need to extract depends on the distribution of measurements across the sample.
There are a few contenders for what the magic number should be. Do you use the mean, median, mode, or something else? How do you determine the correctness of this number or whether your sample size is large enough? Is one number sufficient?
This talk covers some of the statistics behind figuring out which numbers one should be looking at and how to go about extracting it from the sample.
The Statistics of Web Performance AnalysisPhilip Tellis
If you're interested in measuring real user web performance, you'll find tools like boomerang or episodes quite handy. Some popular web frameworks even have modules that make it easy to add them to your site. However, what does one do once one has collected the data? How do you filter out the noise and get meaningful insights from the data?
In this talk, I'll go over the techniques we've picked up by analyzing millions of datapoints daily. I'll cover some simple rules to filter out invalid data, and the statistics to analyze and make sense of what's left. Do you use the mean, median or mode? What about the geometric mean and standard deviation? How confident are we in the results? And finally, why should we care?
This talk should help you gain useful insights from a histogram, or at the very least point you in the right direction for further analysis.
Data Science Case Studies: The Internet of Things: Implications for the Enter...VMware Tanzu
The Internet of Things: Implications for the Enterprise
The Internet Of Things (IoT) is already a reality but getting value out of that is still in its infancy. This session analyzes the implications of IoT for the enterprise with examples from the work we have done.
Rashmi Raghu is a Principal Data Scientist at Pivotal with a focus on the Internet-of-Things and applications in the Energy sector. Her work has spanned diverse industry problems including uncovering patterns & anomalies in massive datasets to predictive maintenance. She holds a Ph.D. in Mechanical Engineering with a minor in Management Science & Engineering from Stanford University. Her doctoral work focused on the development of novel computational models of the cardiovascular system to aid disease research. Prior to that she obtained Master’s and Bachelor’s degrees in Engineering Science from the University of Auckland, New Zealand.
The advance of solid state disk as a replacement for mechanical disk is driving gains in application performance and computing efficiency. However, the same advancements are also driving even more revolutionary changes in system memory. A new and even higher performance persistent data storage layer is emerging that will enable broad adoption of the as yet unrealized power real time computing.
This report analyzes the worldwide markets for Internet TV in US$ Million. The report provides separate comprehensive analytics for the US, Canada, Europe, Asia-Pacific, and Rest of World. Annual estimates and forecasts are provided for the period 2009 through 2017. Also, a six-year historic analysis is provided for these markets. The report profiles 98 companies including many key and niche players such as AOL, Inc., BT Group Plc, BBC, Boxee, Inc., Channel 5, China Telecom Corporation Limited, Comcast Corporation, NBC Universal, Cable News Network [CNN], Channel 4, Fox News Channel, Google TV, Hulu, Microsoft Corporation, Raidi
The Statistics of Web Performance AnalysisPhilip Tellis
If you're interested in measuring real user web performance, you'll find tools like boomerang or episodes quite handy. Some popular web frameworks even have modules that make it easy to add them to your site. However, what does one do once one has collected the data? How do you filter out the noise and get meaningful insights from the data?
In this talk, I'll go over the techniques we've picked up by analyzing millions of datapoints daily. I'll cover some simple rules to filter out invalid data, and the statistics to analyze and make sense of what's left. Do you use the mean, median or mode? What about the geometric mean and standard deviation? How confident are we in the results? And finally, why should we care?
This talk should help you gain useful insights from a histogram, or at the very least point you in the right direction for further analysis.
Data Science Case Studies: The Internet of Things: Implications for the Enter...VMware Tanzu
The Internet of Things: Implications for the Enterprise
The Internet Of Things (IoT) is already a reality but getting value out of that is still in its infancy. This session analyzes the implications of IoT for the enterprise with examples from the work we have done.
Rashmi Raghu is a Principal Data Scientist at Pivotal with a focus on the Internet-of-Things and applications in the Energy sector. Her work has spanned diverse industry problems including uncovering patterns & anomalies in massive datasets to predictive maintenance. She holds a Ph.D. in Mechanical Engineering with a minor in Management Science & Engineering from Stanford University. Her doctoral work focused on the development of novel computational models of the cardiovascular system to aid disease research. Prior to that she obtained Master’s and Bachelor’s degrees in Engineering Science from the University of Auckland, New Zealand.
The advance of solid state disk as a replacement for mechanical disk is driving gains in application performance and computing efficiency. However, the same advancements are also driving even more revolutionary changes in system memory. A new and even higher performance persistent data storage layer is emerging that will enable broad adoption of the as yet unrealized power real time computing.
This report analyzes the worldwide markets for Internet TV in US$ Million. The report provides separate comprehensive analytics for the US, Canada, Europe, Asia-Pacific, and Rest of World. Annual estimates and forecasts are provided for the period 2009 through 2017. Also, a six-year historic analysis is provided for these markets. The report profiles 98 companies including many key and niche players such as AOL, Inc., BT Group Plc, BBC, Boxee, Inc., Channel 5, China Telecom Corporation Limited, Comcast Corporation, NBC Universal, Cable News Network [CNN], Channel 4, Fox News Channel, Google TV, Hulu, Microsoft Corporation, Raidi
MeasureWorks eFinancials - Best practices for a successfull mobile experienc...MeasureWorks
Gebruikers van mobiel internet verwachten snelle transacties en betrouwbare sites en/of applicaties. Volgens recent onderzoek haakt meer dan 52% van de klanten af bij een slechte ervaring en overweegt daardoor geen gebruik meer te maken van een mobiele applicatie.
Nu mobiel internet een integraal onderdeel wordt van uw dienstverlening, en de verwachtingen van klanten toenemen, wordt het managen en monitoren van uw mobiele sites en applicaties een voorwaarde voor succes. Het niet tijdig identificeren van langzame, of erger, niet functionerende mobiele diensten zal onherroepelijk resulteren in verlies van klanten, omzet en uiteindelijk reputatie schade.
Aan de hand van praktijkvoorbeelden zullen we u laten zien:
* Wat de impact is van de adoptie van mobiel internet en groeiende klantverwachtingen op uw online dienstverlening
* Op welke wijze Mobiele Web Experience problemen kunnen worden herkend voordat klanten uw website verlaten
* Best practices voor het leveren van een kwalitatief uitstekende Mobile Web Experience
Sandvine Webinar – Making Cents of Internet Phenomena Through Network Busines...Computaris
To view the recorded version of the Sandvine webinar held at Computaris' 2 decade anniversary virtual event, please register here: http://webexpo.computaris.com/webinars/sandvine
The Best of Both Worlds - Combining Performance and Functional Mobile App Tes...Bitbar
We co-hosted a webinar with Neotys to shed some lights on
- How to overcome the challenges in mobile app performance and functional testing
- How to gain granular and actionable insights to measure and improve your app user experience
- Best practices to get the mobile readiness for 2017 Holiday Shopping Season
- A brief demo of the integration between Neoload and Bitbar Testing
Data Pioneers - Roland Haeve (Atos Nederland) - Big data in organisatiesMultiscope
Roland Haeve is cross competence manager Big Data voor Atos Nederland. Roland heeft ruim 18 jaar ICT-ervaring in het aanbieden van complete oplossingen binnen onder andere Business Intelligence (BI) en Big Data (Analytics). Big Data is voor veel bedrijven nog pionieren en uitzoeken wat de mogelijkheden zijn. In zijn presentatie zal Roland ingaan op succesvolle Big Data cases. Hij zal hierbij niet enkel inzoomen op Nederland, maar ook bredere, Europese voorbeelden meenemen.
The IoT Food Chain – Picking the Right Dining Partner is Important with Dean ...gogo6
Download our special report, IoT Tech for the Manager: http://bit.ly/report1-slideshare
The IoT Food Chain – Picking the Right Dining Partner is Important as presented at the IoT Inc Business' fourteenth Meetup. See: http://www.iot-inc.com/internet-of-things-value-chain-meetup/
In our fourteenth Meetup we have Dean Freeman, Research VP at Gartner presenting “The IoT Food Chain – Picking the Right Dining Partner is Important”.
Presentation Abstract
The Internet of Things means many different things to different people. What is key about the IoT is there is a distinct food chain that runs from the silicon devices to the services and then back. The level of success you will have in the IoT is heavily dependent upon where you fit in the food chain, and if you have the capability to move up the chain or across the chain into different verticals. In this presentation we will explore the food chain, what is important and what steps need to be taken to succeed in the world of the IoT.
Frontend Performance: Beginner to Expert to Crazy PersonPhilip Tellis
There’s no such thing as fast enough. You can always make your website faster. This talk will show you how. The very first requirement of a great user experience is actually getting the bytes of that experience to the user before they they get tired and leave.In this talk we’ll start with the basics and get progressively insane. We’ll go over several frontend performance best practices, a few anti-patterns, the reasoning behind the rules, and how they’ve changed over the years. We’ll also look at some great tools to help you.
MeasureWorks eFinancials - Best practices for a successfull mobile experienc...MeasureWorks
Gebruikers van mobiel internet verwachten snelle transacties en betrouwbare sites en/of applicaties. Volgens recent onderzoek haakt meer dan 52% van de klanten af bij een slechte ervaring en overweegt daardoor geen gebruik meer te maken van een mobiele applicatie.
Nu mobiel internet een integraal onderdeel wordt van uw dienstverlening, en de verwachtingen van klanten toenemen, wordt het managen en monitoren van uw mobiele sites en applicaties een voorwaarde voor succes. Het niet tijdig identificeren van langzame, of erger, niet functionerende mobiele diensten zal onherroepelijk resulteren in verlies van klanten, omzet en uiteindelijk reputatie schade.
Aan de hand van praktijkvoorbeelden zullen we u laten zien:
* Wat de impact is van de adoptie van mobiel internet en groeiende klantverwachtingen op uw online dienstverlening
* Op welke wijze Mobiele Web Experience problemen kunnen worden herkend voordat klanten uw website verlaten
* Best practices voor het leveren van een kwalitatief uitstekende Mobile Web Experience
Sandvine Webinar – Making Cents of Internet Phenomena Through Network Busines...Computaris
To view the recorded version of the Sandvine webinar held at Computaris' 2 decade anniversary virtual event, please register here: http://webexpo.computaris.com/webinars/sandvine
The Best of Both Worlds - Combining Performance and Functional Mobile App Tes...Bitbar
We co-hosted a webinar with Neotys to shed some lights on
- How to overcome the challenges in mobile app performance and functional testing
- How to gain granular and actionable insights to measure and improve your app user experience
- Best practices to get the mobile readiness for 2017 Holiday Shopping Season
- A brief demo of the integration between Neoload and Bitbar Testing
Data Pioneers - Roland Haeve (Atos Nederland) - Big data in organisatiesMultiscope
Roland Haeve is cross competence manager Big Data voor Atos Nederland. Roland heeft ruim 18 jaar ICT-ervaring in het aanbieden van complete oplossingen binnen onder andere Business Intelligence (BI) en Big Data (Analytics). Big Data is voor veel bedrijven nog pionieren en uitzoeken wat de mogelijkheden zijn. In zijn presentatie zal Roland ingaan op succesvolle Big Data cases. Hij zal hierbij niet enkel inzoomen op Nederland, maar ook bredere, Europese voorbeelden meenemen.
The IoT Food Chain – Picking the Right Dining Partner is Important with Dean ...gogo6
Download our special report, IoT Tech for the Manager: http://bit.ly/report1-slideshare
The IoT Food Chain – Picking the Right Dining Partner is Important as presented at the IoT Inc Business' fourteenth Meetup. See: http://www.iot-inc.com/internet-of-things-value-chain-meetup/
In our fourteenth Meetup we have Dean Freeman, Research VP at Gartner presenting “The IoT Food Chain – Picking the Right Dining Partner is Important”.
Presentation Abstract
The Internet of Things means many different things to different people. What is key about the IoT is there is a distinct food chain that runs from the silicon devices to the services and then back. The level of success you will have in the IoT is heavily dependent upon where you fit in the food chain, and if you have the capability to move up the chain or across the chain into different verticals. In this presentation we will explore the food chain, what is important and what steps need to be taken to succeed in the world of the IoT.
Frontend Performance: Beginner to Expert to Crazy PersonPhilip Tellis
There’s no such thing as fast enough. You can always make your website faster. This talk will show you how. The very first requirement of a great user experience is actually getting the bytes of that experience to the user before they they get tired and leave.In this talk we’ll start with the basics and get progressively insane. We’ll go over several frontend performance best practices, a few anti-patterns, the reasoning behind the rules, and how they’ve changed over the years. We’ll also look at some great tools to help you.
Frontend Performance: De débutant à Expert à Fou FurieuxPhilip Tellis
Frontend Performance Beginner to Expert to Crazy Person
The very first requirement of a great user experience is actually getting the bytes of that experience to the user before they they get tired and leave.
In this talk we'll start with the basics and get progressively insane. We'll go over several frontend performance best practices, a few anti-patterns, the reasoning behind the rules, and how they've changed over the years. We'll also look at some great tools to help you.
La performance front-end de débutant, à expert, à fou furieux !
La toute première condition nécessaire à une bonne expérience utilisateur est de pouvoir obtenir les octets de cette expérience avant que l'utilisateur ne se lasse et parte.
Nous débuterons cette conférence avec les bases pour progressivement devenir démentiel. Nous aborderons plusieurs des meilleurs pratiques de la performance front-end, quelques anti-patterns à éviter, le raisonnement derrière les règles, et comment ces dernières ont changé au fil des ans. Nous regarderons d'un peu plus près quelques très bon outils qui peuvent vous aider.
RUM isn’t just for page level metrics anymore. Thanks to modern browser updates and new techniques we can collect real user data at the object level, finding slow page components and keeping third parties honest.
In this talk we will show you how to use Resource Timing, User Timing, and other browser tricks to time the most important components in your page. We’ll also share recipes for several of the web’s most popular third parties. This will give you a head start on measuring object level performance on your own site.
Frontend Performance: Beginner to Expert to Crazy PersonPhilip Tellis
Boston Web Performance Meetup, April 22, 2014
The very first requirement of a great user experience is actually getting the bytes of that experience to the user before they they get fed up and leave. In this talk we'll start with the basics and get progressively insane. We'll go over several front-end performance best practices, a few anti-patterns, the reasoning behind the rules, and how they've changed over the years. We'll also look at some great tools to help you.
Schedule: 6:30, pizza
7:15: talk
Frontend Performance: Beginner to Expert to Crazy PersonPhilip Tellis
The very first requirement of a great user experience is actually getting the bytes of that experience to the user before they they get fed up and leave.
In this talk we'll start with the basics and get progressively insane. We'll go over several frontend performance best practices, a few anti-patterns, the reasoning behind the rules, and how they've changed over the years. We'll also look at some great tools to help you.
When we built boomerang at Yahoo!, we planned on it being a generic beaconing system with different payloads attached by plugins. We published an API, and wrote plugins to measure page roundtrip time, network throughput and latency. We received other plugins from Yahoo! to measure IPv6 and DNS latency, and then nothing happened...
Until one day, a certain Mr. Brewer submitted a NavTiming plugin. As it turns out, people were using boomerang in-house, and creating their own plugins that were never published.
In this talk, we’ll go over the basics of writing a boomerang plugin to measure anything you need, some best practices involved with writing plugins, and examples of third party plugins that others have written.
Abusing JavaScript to measure Web Performance, or, "how does boomerang work?"Philip Tellis
While building boomerang, we developed many interesting methods to measure network performance characteristics using JavaScript running in the browser. While the W3C's NavigationTiming API provides access to many performance metrics, there's far more you can get at with some creative tweaking and analysis of how the browser reacts to certain requests.
In this talk, I'll go into the details of how boomerang works to measure network throughput, latency, TCP connect time, DNS time and IPv6 connectivity. I'll also touch upon some of the other performance related browser APIs we use to gather useful information.
http://www.nywebperformance.org/events/78566362/
Abusing JavaScript to Measure Web PerformancePhilip Tellis
While building boomerang, we developed many interesting methods to measure network performance characteristics using JavaScript running in the browser. While the W3C's NavigationTiming API provides access to many performance metrics, there's far more you can get at with some creative tweaking and analysis of how the browser reacts to certain requests.
In this talk, I'll go into the details of how boomerang works to measure network throughput, latency, TCP connect time, DNS time and IPv6 connectivity. I'll also touch upon some of the other performance related browser APIs we use to gather useful information. I will NOT be covering the W3C Navigation Timing API since that's been covered by Alois Reitbauer in a previous Boston Web Perf talk.
Real user monitoring is one of the best ways of learning “the truth” about what visitors experience on your web site, but it comes at a cost. The real world is messy and noisy making it hard to know exactly what’s going on. Filtering your data, splitting it along multiple dimensions, and determining what to discard are important second steps on the path to insightful RUM analysis, and in this session, we’ll go into some of the details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
The Statistics of Web Performance
1. Introduction
Statistics - I
Statistics - II
The Statistics of Web Performance
Philip Tellis / philip@bluesmoon.info
ConFoo / 2010-03-12
ConFoo / 2010-03-12 The Statistics of Web Performance
2. Introduction
Statistics - I
Statistics - II
$ finger philip
Philip Tellis
philip@bluesmoon.info
@bluesmoon
yahoo
geek
ConFoo / 2010-03-12 The Statistics of Web Performance
3. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Introduction
ConFoo / 2010-03-12 The Statistics of Web Performance
4. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Accurately measure page performance
At least, as accurately as possible
ConFoo / 2010-03-12 The Statistics of Web Performance
5. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Accurately measure page performance
At least, as accurately as possible
ConFoo / 2010-03-12 The Statistics of Web Performance
6. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Be unintrusive
If you try to measure something accurately, you will change
something related
– Heisenberg’s uncertainty principle
ConFoo / 2010-03-12 The Statistics of Web Performance
7. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
And one number to rule them all
ConFoo / 2010-03-12 The Statistics of Web Performance
8. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Bandwidth
Real bandwidth v/s advertised bandwidth
Bandwidth to your server, not to the ISP
Bandwidth during normal internet usage
If the user’s always watching movies, you’re not winning
ConFoo / 2010-03-12 The Statistics of Web Performance
9. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Bandwidth
Real bandwidth v/s advertised bandwidth
Bandwidth to your server, not to the ISP
Bandwidth during normal internet usage
If the user’s always watching movies, you’re not winning
ConFoo / 2010-03-12 The Statistics of Web Performance
10. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Latency
How long does it take a byte to get to the user?
Wired, wireless, mobile, satellite?
How many hops in between?
Speed of light is constant
This is not a battle we will soon win.
When was the last time you heard latency mentioned in a
TV ad?
http://www.stuartcheshire.org/rants/Latency.html
ConFoo / 2010-03-12 The Statistics of Web Performance
11. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Latency
How long does it take a byte to get to the user?
Wired, wireless, mobile, satellite?
How many hops in between?
Speed of light is constant
This is not a battle we will soon win.
When was the last time you heard latency mentioned in a
TV ad?
http://www.stuartcheshire.org/rants/Latency.html
ConFoo / 2010-03-12 The Statistics of Web Performance
12. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
Latency
How long does it take a byte to get to the user?
Wired, wireless, mobile, satellite?
How many hops in between?
Speed of light is constant
This is not a battle we will soon win.
When was the last time you heard latency mentioned in a
TV ad?
http://www.stuartcheshire.org/rants/Latency.html
ConFoo / 2010-03-12 The Statistics of Web Performance
13. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
User perceived page load time
Time from “click on a link” to “spinner stops spinning”
This is what users notice
Depends on how long your page takes to build
Depends on what’s in your page
Depends on how long components take to load
Depends on how long the browser takes to execute and
render
ConFoo / 2010-03-12 The Statistics of Web Performance
14. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
We need to measure real user data
ConFoo / 2010-03-12 The Statistics of Web Performance
15. Introduction
The goal
Statistics - I
Performance Measurement
Statistics - II
The statistics apply to any kind of performance data though
ConFoo / 2010-03-12 The Statistics of Web Performance
16. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Statistics - I
ConFoo / 2010-03-12 The Statistics of Web Performance
17. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Disclaimer
I am not a statistician
ConFoo / 2010-03-12 The Statistics of Web Performance
18. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Population
All possible users of your system
ConFoo / 2010-03-12 The Statistics of Web Performance
19. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Sample
Representative subset of the population
ConFoo / 2010-03-12 The Statistics of Web Performance
20. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Bad sample
Sometimes it’s not
ConFoo / 2010-03-12 The Statistics of Web Performance
21. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
How to randomize?
Pick 10% of users at random and always test them
OR
For each user, decide at random if they should be tested
http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html
ConFoo / 2010-03-12 The Statistics of Web Performance
22. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Select 10% of users - I
if($sessionid % 10 === 0) {
// instrument code for measurement
}
Once a user enters the measurement bucket, they stay
there until they log out
Fixed set of users, so tests may be more consistent
Error in the sample results in positive feedback
ConFoo / 2010-03-12 The Statistics of Web Performance
23. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Select 10% of users - II
if(rand() < 0.1 * getrandmax()) {
// instrument code for measurement
}
For every request, a user has a 10% chance of being
tested
Gets rid of positive feedback errors, but sample size !=
10% of population
ConFoo / 2010-03-12 The Statistics of Web Performance
24. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
How big a sample is representative?
Select n such that
σ
1.96 √n ≤ 5%µ
ConFoo / 2010-03-12 The Statistics of Web Performance
25. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Standard Deviation
Standard deviation tells you the spread of the curve
The narrower the curve, the more confident you can be
ConFoo / 2010-03-12 The Statistics of Web Performance
26. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
MoE at 95% confidence
σ
±1.96 √n
ConFoo / 2010-03-12 The Statistics of Web Performance
27. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
MoE & Sample size
There is an inverse square root correlation between sample
size and margin of error
ConFoo / 2010-03-12 The Statistics of Web Performance
28. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
But wait... it’s not complicated enough.
We have different types of margins of error
...more about that later
ConFoo / 2010-03-12 The Statistics of Web Performance
29. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
But wait... it’s not complicated enough.
We have different types of margins of error
...more about that later
ConFoo / 2010-03-12 The Statistics of Web Performance
30. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
But wait... it’s not complicated enough.
We have different types of margins of error
...more about that later
ConFoo / 2010-03-12 The Statistics of Web Performance
31. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Ding dong
ConFoo / 2010-03-12 The Statistics of Web Performance
32. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
One number
Mean (Arithmetic)
Good for symmetric curves
Affected by outliers
Mean(10, 11, 12, 11, 109) = 30
ConFoo / 2010-03-12 The Statistics of Web Performance
33. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
One number
Median
Middle value measures central tendency well
Not trivial to pull out of a DB
Median(10, 11, 12, 11, 109) = 11
ConFoo / 2010-03-12 The Statistics of Web Performance
34. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
One number
Mode
Not often used
Multi-modal distributions suggest problems
Mode(10, 11, 12, 11, 109) = 11
ConFoo / 2010-03-12 The Statistics of Web Performance
35. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Other numbers
A percentile point in the distribution: 95th , 98.5th or 99th
Used to find out the worst user experience
Makes more sense if you filter data first
P95th (10, 11, 12, 11, 109) = 12
ConFoo / 2010-03-12 The Statistics of Web Performance
36. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Other means
Geometric mean
Good if your data is exponential in nature
(with the tail on the right)
GMean(10, 11, 12, 11, 109) = 16.68
ConFoo / 2010-03-12 The Statistics of Web Performance
37. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Wait... how did I get that?
N
ΠN xi — could lead to overflow
i=1
ΣN loge (xi )
i=1
N
e — computationally simpler
ConFoo / 2010-03-12 The Statistics of Web Performance
38. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Wait... how did I get that?
N
ΠN xi — could lead to overflow
i=1
ΣN loge (xi )
i=1
N
e — computationally simpler
ConFoo / 2010-03-12 The Statistics of Web Performance
39. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Wait... how did I get that?
N
ΠN xi — could lead to overflow
i=1
ΣN loge (xi )
i=1
N
e — computationally simpler
ConFoo / 2010-03-12 The Statistics of Web Performance
40. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Wait... how did I get that?
N
ΠN xi — could lead to overflow
i=1
ΣN loge (xi )
i=1
N
e — computationally simpler
ConFoo / 2010-03-12 The Statistics of Web Performance
41. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
Other means
And there is also the Harmonic mean, but forget about that
ConFoo / 2010-03-12 The Statistics of Web Performance
42. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
...though consequently
We have other margins of error
Geometric margin of error
Uses geometric standard deviation
Median margin of error
Uses ranges of actual values from data set
Stick to the arithmetic MoE
– simpler to calculate, simpler to read and not incorrect
ConFoo / 2010-03-12 The Statistics of Web Performance
43. Introduction Random Sampling
Statistics - I Margin of Error
Statistics - II Central Tendency
...though consequently
We have other margins of error
Geometric margin of error
Uses geometric standard deviation
Median margin of error
Uses ranges of actual values from data set
Stick to the arithmetic MoE
– simpler to calculate, simpler to read and not incorrect
ConFoo / 2010-03-12 The Statistics of Web Performance
44. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Statistics - II
ConFoo / 2010-03-12 The Statistics of Web Performance
45. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Outliers
Out of range data points
Nothing you can fix here
There’s even a book about
them
ConFoo / 2010-03-12 The Statistics of Web Performance
46. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Outliers
Out of range data points
Nothing you can fix here
There’s even a book about
them
ConFoo / 2010-03-12 The Statistics of Web Performance
47. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Outliers
Out of range data points
Nothing you can fix here
There’s even a book about
them
ConFoo / 2010-03-12 The Statistics of Web Performance
48. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Outliers
Out of range data points
Nothing you can fix here
There’s even a book about
them
ConFoo / 2010-03-12 The Statistics of Web Performance
49. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
DNS problems can cause outliers
2 or 3 DNS servers for an ISP
30 second timeout if first fails
... 30 second increase in page load time
Maybe measure both and fix what you can
http://nms.lcs.mit.edu/papers/dns-ton2002.pdf
ConFoo / 2010-03-12 The Statistics of Web Performance
50. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Band-pass filtering
ConFoo / 2010-03-12 The Statistics of Web Performance
51. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Band-pass filtering
Strip everything outside a reasonable range
Bandwidth range: 4kbps - 4Gbps
Page load time: 50ms - 120s
You may need to relook at the ranges all the time
ConFoo / 2010-03-12 The Statistics of Web Performance
52. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
IQR filtering
ConFoo / 2010-03-12 The Statistics of Web Performance
53. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
IQR filtering
Here, we derive the range from the data
ConFoo / 2010-03-12 The Statistics of Web Performance
54. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Let’s look at some real charts
ConFoo / 2010-03-12 The Statistics of Web Performance
55. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Bandwidth distribution for web devs
x-axis is linear
ConFoo / 2010-03-12 The Statistics of Web Performance
56. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Now let’s use log(kbps) instead of kbps
x-axis is exponential
ConFoo / 2010-03-12 The Statistics of Web Performance
57. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Exponential == Geometric
Categories/Buckets grow exponentially
Data is related geometrically
Use the geometric mean and geometric margin of error
gmean
Error _range = /gmoe , gmean ∗ gmoe
Non-linear ranges are hard for humans to grok
ConFoo / 2010-03-12 The Statistics of Web Performance
58. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Exponential == Geometric
Categories/Buckets grow exponentially
Data is related geometrically
Use the geometric mean and geometric margin of error
gmean
Error _range = /gmoe , gmean ∗ gmoe
Non-linear ranges are hard for humans to grok
ConFoo / 2010-03-12 The Statistics of Web Performance
59. Introduction
Filtering
Statistics - I
The Log-Normal distribution
Statistics - II
Exponential == Geometric
Categories/Buckets grow exponentially
Data is related geometrically
Use the geometric mean and geometric margin of error
gmean
Error _range = /gmoe , gmean ∗ gmoe
Non-linear ranges are hard for humans to grok
ConFoo / 2010-03-12 The Statistics of Web Performance
60. Introduction
Statistics - I
Statistics - II
So...
ConFoo / 2010-03-12 The Statistics of Web Performance
61. Introduction
Statistics - I
Statistics - II
Further reading
Web Performance - Not a Simple Number
http://www.netforecast.com/Articles/BCR+C25+Web+Performance+-+Not+A+Simple+Number.pdf
Revisiting statistics for web performance (introduction to
Log-Normal)
http://home.pacbell.net/ciemo/statistics/WhatDoYouMean.pdf
Random Sampling
http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html
Khan Academy’s tutorials on statistics
http://khanacademy.com/
Learning about Statistical Learning
http://measuringmeasures.blogspot.com/2010/01/learning-about-statistical-learning.html
Wikipedia articles on Random Sampling, Central Tendency,
Standard Error, Confounding, Means and IQR
ConFoo / 2010-03-12 The Statistics of Web Performance
62. Introduction
Statistics - I
Statistics - II
Summary
Choose a reasonable sample size and sampling factor
Tune sample size for minimal margin of error
Decide based on your data whether to use mode, median
or one of the means
Figure out whether your data is Normal, Log-Normal or
something else
Filter out anomalous outliers
ConFoo / 2010-03-12 The Statistics of Web Performance
63. Introduction
Statistics - I
Statistics - II
contact me
Philip Tellis
philip@bluesmoon.info
bluesmoon.info
@bluesmoon
ConFoo / 2010-03-12 The Statistics of Web Performance
64. Introduction
Statistics - I
Statistics - II
Photo credits
http://www.flickr.com/photos/leoffreitas/332360959/ by leoffreitas
http://www.flickr.com/photos/cobalt/56500295/ by cobalt123
http://www.flickr.com/photos/sophistechate/4264466015/ by Lisa
Brewster
http://www.flickr.com/photos/nchoz/243216008/ by nchoz
ConFoo / 2010-03-12 The Statistics of Web Performance
65. Introduction
Statistics - I
Statistics - II
List of figures
http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg
http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg
http://en.wikipedia.org/wiki/File:KilroySchematic.svg
http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png
ConFoo / 2010-03-12 The Statistics of Web Performance