This document discusses metrics and monitoring best practices based on lessons learned at Chrome. It covers the properties of good metrics, use cases for metrics in different contexts like labs, Web Performance APIs and Real User Monitoring (RUM). It provides the example metric of Largest Contentful Paint and discusses challenges in defining it accurately. The document also covers limitations and best practices for lab testing, A/B testing and understanding RUM data, emphasizing controlled experimentation and detecting changes and mix shifts. The overall message is that metrics require careful thought and validation but can provide valuable insights when done well.
The Early Stage Venture Series - part 1 - Paul Orlando This is part one of a talk series for early-stage startups. There are a lot of examples used in the talk that don't come out if you only view the slides, but here it is. Updates provided at: startupsunplugged.com
Always Be Testing - Learn from Every A/B Test (Hiten Shah)Future Insights
Session slides from Future Insights Live, Vegas 2015:
https://futureinsightslive.com/las-vegas-2015/
Hiten Shah, co-founder of KISSmetrics and Crazy Egg, talks you through his 4-step data-driven process for successful A/B testing.
How to be Successful with Responsive Sites (Koombea & NGINX) - EnglishKoombea
Can't decide if your organization should build a mobile app or responsive website? Do you interact with consumer-facing products or large scale developments?
This guide gives you an idea of what Responsive is, why you should use it, and then DIGS deep into the technical aspect and how to optimize for performance.
By: David Bohorquez & Rick Nelson
An overview on how to pick the right development agency, build your product, and reach a MVP or Minimum Viable Product. We specialize in bringing companies and ideas to fruition. For more information visit www.koombea.com/mvp/ or send us a note mvp@koombea.com and we will be more than happy to help.
By Kate Swanberg: www.twitter.com/keswanberg
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
Peep Laja, founder of ConversionXL, talks about some of the original UX research studies ConversionXL Institute has conducted over the last months. Inspiration for effective tests and website changes.
Website Redesigns: Why they Fail and How to Ensure SuccessOptimizely
Learn how to avoid website redesigns that don’t deliver results.
Website redesigns are a tremendous effort: they take a lot of planning, involve many team members and are very costly. The worst about them is that they often fail, meaning that the “new” website will perform worse than the old.
- How common redesign fails really are, and how costly they can be
- The key reasons most redesigns fail to increase sales/conversions
- A better approach to increase chances of redesign success
To learn more about Optimizely, find more info here: https://www.optimizely.com/
To get more inspiration for better website redesigns, visit our blog:
https://blog.optimizely.com/2014/08/14/2-alexa-500-site-redesigns-that-should-inspire-you-to-ab-test/
The Early Stage Venture Series - part 1 - Paul Orlando This is part one of a talk series for early-stage startups. There are a lot of examples used in the talk that don't come out if you only view the slides, but here it is. Updates provided at: startupsunplugged.com
Always Be Testing - Learn from Every A/B Test (Hiten Shah)Future Insights
Session slides from Future Insights Live, Vegas 2015:
https://futureinsightslive.com/las-vegas-2015/
Hiten Shah, co-founder of KISSmetrics and Crazy Egg, talks you through his 4-step data-driven process for successful A/B testing.
How to be Successful with Responsive Sites (Koombea & NGINX) - EnglishKoombea
Can't decide if your organization should build a mobile app or responsive website? Do you interact with consumer-facing products or large scale developments?
This guide gives you an idea of what Responsive is, why you should use it, and then DIGS deep into the technical aspect and how to optimize for performance.
By: David Bohorquez & Rick Nelson
An overview on how to pick the right development agency, build your product, and reach a MVP or Minimum Viable Product. We specialize in bringing companies and ideas to fruition. For more information visit www.koombea.com/mvp/ or send us a note mvp@koombea.com and we will be more than happy to help.
By Kate Swanberg: www.twitter.com/keswanberg
[Elite Camp 2016] Peep Laja - Fresh Out Of the OvenCXL
Peep Laja, founder of ConversionXL, talks about some of the original UX research studies ConversionXL Institute has conducted over the last months. Inspiration for effective tests and website changes.
Website Redesigns: Why they Fail and How to Ensure SuccessOptimizely
Learn how to avoid website redesigns that don’t deliver results.
Website redesigns are a tremendous effort: they take a lot of planning, involve many team members and are very costly. The worst about them is that they often fail, meaning that the “new” website will perform worse than the old.
- How common redesign fails really are, and how costly they can be
- The key reasons most redesigns fail to increase sales/conversions
- A better approach to increase chances of redesign success
To learn more about Optimizely, find more info here: https://www.optimizely.com/
To get more inspiration for better website redesigns, visit our blog:
https://blog.optimizely.com/2014/08/14/2-alexa-500-site-redesigns-that-should-inspire-you-to-ab-test/
Director of Product at Glassdoor Talks: How to Transition to Product ManagementProduct School
How to transition into Product Management with Phillip, who shared his experiences transitioning from Engineering into Product Management and discuss the following topics:
How to transition from an Engineer role to a PM role.
How to overcome the challenges that arise while transitioning from Engineering to Product Management
What you can do now to get a job in Product Management
Takeaways:
What is expected of a product manager
What tech startups look for on a resume for a product manager candidate
How to ace a product management interview
How to keep up with the product management discipline
Hoe kan je data gebruiken om beter te beslissen? Booking.com werkt samen met Netwerven en vertelt hoe je met data je beslissingsproces kan verbeteren.Think big, act small.
A/B Mythbusters: Common Optimization Objections DebunkedOptimizely
For every $92 marketers spend to drive traffic to their website, only $1 is spent on optimizing the experiences visitors encounter when they get there. But with the proven benefits of testing, why aren’t more companies spending on optimization? We’ve compiled a list of common objections to testing and asked digital marketers what they have to say about them.
Some teams think they can be agile by using a defined process or set of practices as defined by one of the agile approaches. This is just “doing Agile.” Other teams are agile in name only – the team says it’s “doing Agile” but ends up using the same old practices and achieving the same results. Teams adopt agile for a variety of reasons, but it’s not the process or set of practices they select that produces the results they seek. Teams are most successful when they adopt a particular mindset in order to “be agile”. Join Kent McDonald as he describes this mindset through 7 key ideas based on how people and organizations work best. We’ll discuss some specific techniques you can use to adopt the mindset on your project, how the project manager role changes along with the mindset, and how to help your team move from “doing Agile” to actually “being agile”.
7 Key Questions to Ask Your Prospective Tech AgencyKoombea
How’s your tech/web/mobile agency search going? You decided you need a dev shop, but what are the right questions to ask them?
At Koombea we want you to be prepared. And we are happy to help you figure out which agency is best for you—even if it’s not us.
When talking to prospective tech agencies, use these helpful 7 questions to filter your options.
*What is CRO? *CRO gains *When to do it? *CRO stack / tools
Learn how to make your website visitors and mobile app users do what you want them to do using Conversion Rate Optimization (CRO), or growth hacking..
This slide deck is part of the "Inbound Marketing & Growth" seminar held on Jan 20 2016. The seminar covered the topics Attract, Convert and Satisfy and this was the presentation for the Convert stage.
How great is your product idea? Every idea is great, but not every product idea can monetize itself. The product development process is crucial in deciding whether or not your product can be pushed to production. This guide will teach you how to validate your product ideas so that you can launch your next product successfully.
An overview of how to reach Minimum Sellable Product (MSP) for early stage startups and Labs. Class offered via NY Fashion Technology Accelerator at AlleyNYC, course contributed by Koombea.
Koombea is a product development agency specializing in mobile apps and technology.
More information at koombea.com/MVP
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...Optimizely
In this the keynote of the Optimizely Experience London, Matt Althauser (GM Optimizely Europe) shows where Optimizely has started in 2010 and how the product has evolved since.
During his talk fellow team members explained these additional features in more detail. Features include:
- Drag & Drop WYSIWYG editor
- Mobile
- API
- Audiences
- Balanced Content Delivery Network (CDN)
Why Things Go Off the Rails and How to Prevent Product-Engineering AngstOptimizely
Join us to hear from Claire Vo, Chief Product Officer, and Bill Press, SVP of Engineering, discuss how to better align and collaborate with their teams.
“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeOptimizely
Courtney Ball and Antonio Contreras from 23andMe’s Web Marketing and Engineering teams will share their experience using Optimizely Web’s latest development, Performance Edge. The new technology makes experiments run faster by processing in the edge (CDN) rather than in the browser. Optimizely Product Manager, Whelan Boyd, will join the discussion about why speed matters and how to maintain high performance when you are experimenting at scale.
Improve your content: The What, Why, Where and How about A/B Testingintrotodigital
A/B testing, also known as split testing, is a user experience research methodology where users are randomly split into two or more groups to see different versions of the same element. This presentation explains what is A/B testing, why you need it, where you can apply it and how to conduct an A/B test.
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal ReadinessSOASTA
‘Tis the Season – Holiday 2014 eCommerce Quality Checklist
Past Webinar
Archived (originally presented June 26th, 2014)
This year, your holiday traffic will increase 15% or more, and 50% of the users will be mobile. Recent research shows 71% of your revenue comes from multi-channel users, so if you haven’t started planning, you’re already behind. Leading retailers are preparing for Holiday “14 and testing their production sites for multi-channel access to 115% capacity, or beyond! If you’re not one of them, your plans are incomplete.
Cover your risks. Join Tenzing and SOASTA experts as they discuss the must-do checklist for peak performance.
In this webinar you’ll learn:
Align your Marketing and Quality plans
Cover the multichannel user experience
Test early in the lab and fully in production
Optimize end-to-end site speed and performance
When to freeze for the winter
Don’t miss this opportunity to “shop early” and see how the leading retailers are already beating the odds with cloud testing.
Director of Product at Glassdoor Talks: How to Transition to Product ManagementProduct School
How to transition into Product Management with Phillip, who shared his experiences transitioning from Engineering into Product Management and discuss the following topics:
How to transition from an Engineer role to a PM role.
How to overcome the challenges that arise while transitioning from Engineering to Product Management
What you can do now to get a job in Product Management
Takeaways:
What is expected of a product manager
What tech startups look for on a resume for a product manager candidate
How to ace a product management interview
How to keep up with the product management discipline
Hoe kan je data gebruiken om beter te beslissen? Booking.com werkt samen met Netwerven en vertelt hoe je met data je beslissingsproces kan verbeteren.Think big, act small.
A/B Mythbusters: Common Optimization Objections DebunkedOptimizely
For every $92 marketers spend to drive traffic to their website, only $1 is spent on optimizing the experiences visitors encounter when they get there. But with the proven benefits of testing, why aren’t more companies spending on optimization? We’ve compiled a list of common objections to testing and asked digital marketers what they have to say about them.
Some teams think they can be agile by using a defined process or set of practices as defined by one of the agile approaches. This is just “doing Agile.” Other teams are agile in name only – the team says it’s “doing Agile” but ends up using the same old practices and achieving the same results. Teams adopt agile for a variety of reasons, but it’s not the process or set of practices they select that produces the results they seek. Teams are most successful when they adopt a particular mindset in order to “be agile”. Join Kent McDonald as he describes this mindset through 7 key ideas based on how people and organizations work best. We’ll discuss some specific techniques you can use to adopt the mindset on your project, how the project manager role changes along with the mindset, and how to help your team move from “doing Agile” to actually “being agile”.
7 Key Questions to Ask Your Prospective Tech AgencyKoombea
How’s your tech/web/mobile agency search going? You decided you need a dev shop, but what are the right questions to ask them?
At Koombea we want you to be prepared. And we are happy to help you figure out which agency is best for you—even if it’s not us.
When talking to prospective tech agencies, use these helpful 7 questions to filter your options.
*What is CRO? *CRO gains *When to do it? *CRO stack / tools
Learn how to make your website visitors and mobile app users do what you want them to do using Conversion Rate Optimization (CRO), or growth hacking..
This slide deck is part of the "Inbound Marketing & Growth" seminar held on Jan 20 2016. The seminar covered the topics Attract, Convert and Satisfy and this was the presentation for the Convert stage.
How great is your product idea? Every idea is great, but not every product idea can monetize itself. The product development process is crucial in deciding whether or not your product can be pushed to production. This guide will teach you how to validate your product ideas so that you can launch your next product successfully.
An overview of how to reach Minimum Sellable Product (MSP) for early stage startups and Labs. Class offered via NY Fashion Technology Accelerator at AlleyNYC, course contributed by Koombea.
Koombea is a product development agency specializing in mobile apps and technology.
More information at koombea.com/MVP
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...Optimizely
In this the keynote of the Optimizely Experience London, Matt Althauser (GM Optimizely Europe) shows where Optimizely has started in 2010 and how the product has evolved since.
During his talk fellow team members explained these additional features in more detail. Features include:
- Drag & Drop WYSIWYG editor
- Mobile
- API
- Audiences
- Balanced Content Delivery Network (CDN)
Why Things Go Off the Rails and How to Prevent Product-Engineering AngstOptimizely
Join us to hear from Claire Vo, Chief Product Officer, and Bill Press, SVP of Engineering, discuss how to better align and collaborate with their teams.
“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeOptimizely
Courtney Ball and Antonio Contreras from 23andMe’s Web Marketing and Engineering teams will share their experience using Optimizely Web’s latest development, Performance Edge. The new technology makes experiments run faster by processing in the edge (CDN) rather than in the browser. Optimizely Product Manager, Whelan Boyd, will join the discussion about why speed matters and how to maintain high performance when you are experimenting at scale.
Improve your content: The What, Why, Where and How about A/B Testingintrotodigital
A/B testing, also known as split testing, is a user experience research methodology where users are randomly split into two or more groups to see different versions of the same element. This presentation explains what is A/B testing, why you need it, where you can apply it and how to conduct an A/B test.
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal ReadinessSOASTA
‘Tis the Season – Holiday 2014 eCommerce Quality Checklist
Past Webinar
Archived (originally presented June 26th, 2014)
This year, your holiday traffic will increase 15% or more, and 50% of the users will be mobile. Recent research shows 71% of your revenue comes from multi-channel users, so if you haven’t started planning, you’re already behind. Leading retailers are preparing for Holiday “14 and testing their production sites for multi-channel access to 115% capacity, or beyond! If you’re not one of them, your plans are incomplete.
Cover your risks. Join Tenzing and SOASTA experts as they discuss the must-do checklist for peak performance.
In this webinar you’ll learn:
Align your Marketing and Quality plans
Cover the multichannel user experience
Test early in the lab and fully in production
Optimize end-to-end site speed and performance
When to freeze for the winter
Don’t miss this opportunity to “shop early” and see how the leading retailers are already beating the odds with cloud testing.
Test Everything: TrustRadius Delivers Customer Value with ExperimentationOptimizely
When done right, experimentation can help you validate the product you’re building and create winning customer experiences. And it doesn’t take a big engineering team to make this happen.
TrustRadius, the most trusted review site for business technology, uses experimentation to build an online community through website and server-side experimentation. The small but mighty TrustRadius team runs experiments throughout the buyer’s journey to engage different user personas and understand outcomes in real-time.
Watch the webinar recording featuring Rilo Stark, product manager at TrustRadius, and Jack Peden, senior software engineer, to understand their data-driven experimentation strategy and how TrustRadius uses Optimizely Web and Full Stack products to tailor experiences to different customer segments and mitigate risk through A/B/N and painted door tests.
A primer on how ab testing can be set-up for success in an e-commerce environment. Includes guidelines of how to set-up ab tests including hypotheses definition, sample size determination, statistical testing and avoiding bias that can come in any experiment's set-up
Watch here: https://alexwilson.tech/content/770c26b5-f551-4795-90fa-0f24627ec510
Software Quality is really tricky: We all want to build the best thing for our users, but we want to ship good software, fast, without process overhead holding us back.
Join Alex in discussing what Software Quality actually is, how we can define and measure it, and how we can work towards it. And yes, there is room for a QA on your team.
A primer on AB testing and it's application in ecommerce. A necessary tool in every product manager's arsenal. Covers the principles behind setting up a good test and the statistical tools required to analyze results.
Supercharge your AB testing with automated causal inference - Community Works...Egor Kraev
An A/B test consists of splitting the customers into a test and a control group, and choosing a large enough sample size to observe the average treatment effect (ATE) we are interested in, in spite of all the other factors driving outcome variance. With causal inference models, we can do better than that, by estimating the effect conditional on customer features (CATE), thus turning customer variability from noise to be averaged over to a valuable source of segmentation, and potentially requiring smaller sample sizes as a result. Unfortunately, there are many different models available for estimating CATE, with many parameters to tune and very different performance. In this talk, we will present our auto-causality library, which combines the three marvelous packages from Microsoft – DoWhy, EconML, and FLAML – to do fully automated selection and tuning of causal models based on out-of-sample performance, just like any other AutoML package does. We will describe the projects inside Wise currently starting to apply it, and present results on comparative model performance and out-of-sample segmentation on Wise CRM data.
Webinar: Estrategias para optimizar los costos de testingFederico Toledo
Webinar en colaboración de QAminds y Abstracta Tech Talks.
Abstract: En estos días de lockdown y recesión económica muchas empresas están buscando formas de recortar costos por acá y por allá, y por supuesto, el testing es de las cosas que se suele recortar primero. El problema de recortar presupuesto y recursos para testing queda reflejado en el viejo dicho "pan para hoy hambre para mañana". En esta charla quiero compartir algunas estrategias con las que puedes optimizar costos de testing sin comprometer la calidad del sistema o producto que estás desarrollando.
Detailed presentation on performance testing and Loadrunner.
Complete course is available on udemy.
Use below link to get the course for just 20 USD
https://www.udemy.com/performance-testing-using-microfocus-loadrunner-basics-advanced/?couponCode=PTLR20D
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Launch Your Streaming Platforms in MinutesRoshan Dwivedi
The claim of launching a streaming platform in minutes might be a bit of an exaggeration, but there are services that can significantly streamline the process. Here's a breakdown:
Pros of Speedy Streaming Platform Launch Services:
No coding required: These services often use drag-and-drop interfaces or pre-built templates, eliminating the need for programming knowledge.
Faster setup: Compared to building from scratch, these platforms can get you up and running much quicker.
All-in-one solutions: Many services offer features like content management systems (CMS), video players, and monetization tools, reducing the need for multiple integrations.
Things to Consider:
Limited customization: These platforms may offer less flexibility in design and functionality compared to custom-built solutions.
Scalability: As your audience grows, you might need to upgrade to a more robust platform or encounter limitations with the "quick launch" option.
Features: Carefully evaluate which features are included and if they meet your specific needs (e.g., live streaming, subscription options).
Examples of Services for Launching Streaming Platforms:
Muvi [muvi com]
Uscreen [usencreen tv]
Alternatives to Consider:
Existing Streaming platforms: Platforms like YouTube or Twitch might be suitable for basic streaming needs, though monetization options might be limited.
Custom Development: While more time-consuming, custom development offers the most control and flexibility for your platform.
Overall, launching a streaming platform in minutes might not be entirely realistic, but these services can significantly speed up the process compared to building from scratch. Carefully consider your needs and budget when choosing the best option for you.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
7. Properties of a Good Top Level Metric
Representative
Accurate
Interpretable
Simple
Stable
Elastic
Realtime
Orthogonal
8. Use Cases of Top Level Metrics
Lab Web Performance API Chrome RUM
9. Lab
Fewer data points than RUM
Lab metrics need to be Stable and
Elastic.
This can put them at odds with
being Simple and Interpretable.
They don’t require Realtime.
Care should be put into handling
user input.
10. Web Perf API
Broad range of use cases
It is critical that metrics exposed to
Web Perf APIs are Representative
and Accurate.
They must be Realtime.
It’s more important to be Simple
and Interpretable than Stable or
Elastic.
Clear definitions that can be
implemented across engines are
important.
11. Chrome RUM
Understanding Chrome
performance in the wild.
We care a lot that our internal
metrics are Representative and
Accurate, but we can iterate on
them frequently.
We get a very large volume of data,
so it’s less important that they are
Stable or Elastic.
13. Key Moments in Page Load
Is it happening? Is it useful? Is it usable?
14. Goal: When is Main Content Loaded?
Is it happening? Is it useful? Is it usable?
15. Prior work:
Speed Index
Average time at which visible
parts of the page are displayed
Speed index is Representative,
Accurate, Interpretable, Stable, and
Elastic.
But it’s not Realtime, which is a
requirement for web perf APIs and
RUM.
And it’s not Simple.
16. Prior work: First
Meaningful
Paint
(deprecated)
Heuristic: first paint after
biggest layout change
First Meaningful Paint is
Representative, Interpretable, and
Realtime.
But it produces strange, outlier
results in 20% of cases, making it
not Accurate. It’s not Simple, or
Stable, or Elastic.
17. Main Content Painted Metric Priorities
1. Representative
2. Accurate
3. Realtime
4. Interpretable
5. Simple
18. We can get paint
events in
Realtime. Can we
make a metric
that is Simple and
Accurate?
19. Brainstorm Simple Metrics
● Largest image paint in viewport
● Largest text paint in viewport
● Largest image or text paint in viewport
● Last image paint in viewport
● Last text paint in viewport
● Last image or text paint in viewport...
20. And Test Them for Accuracy
● Generate filmstrips for 1000 sites.
● Look at the filmstrips and the layouts, and what each metric reports for
them.
● Pick the best.
21. And the Winner Is...
Largest image or text paint in viewport: Largest Contentful Paint.
23. What About Splash Screens?
Elements that are removed from the DOM are invalidated as candidates for
Largest Contentful Paint.
First contentful
paint (removed)
Largest
contentful paint
24. What About Background Images?
Background
image painted
Real largest
contentful paint
Page
background
image
25. What is a “Text Paint”?
Aggregate to block-level
elements containing text nodes
or other inline-level text elements
children.
26. What About User Actions During Load?
Largest Contentful Paint cannot be measured if there is user input.
Example: infinite scroll
33. Limitations
Lab Testing Pros and Cons
Good For
Fast iteration
Repeatable results
Debugging
Trying things you can’t launch
Can’t model every user
Can’t predict magnitude of
changes
36. Better Reproducibility: Test Environment
● Consider real hardware instead of VMs
● Use exact same model + configuration, or even same device
● On mobile, ensure devices are cool and charged
● Turn off all background tasks
● Reduce randomization
○ Simulate network
○ Mock Math.random(), Date.now(), etc
○ Freeze 3rd party scripts
42. Each Group of Runs is a Set of Samples
Use a hypothesis test to see if the sets
are from different.
Note that performance test results are
usually not normally distributed.
Recommend looking into Mann-Whitney
U, Kolmogorov-Smirnov, and Anderson-
Darling.
44. Limitations
A/B Testing Pros and Cons
Good For
Predicting real-world effect of
performance optimizations
Understanding potential
regressions from new features
Can’t A/B test every change
Many concurrent A/B tests can be
hard to manage
47. Best practices for experiment groups
● Use equal-sized control and experiment groups
● Compare groups on same version before experiment
● Options for getting more data
○ Run over a longer time period
○ Increase group size
49. Limitations
RUM Pros and Cons
Good For
Ground truth about user
experience
Hard to reproduce and debug
By the time a regression is
detected, real users are already
experiencing it
51. Reasons your RUM metric is doing
something you don’t understand
● Diversity of user base
● Mix shifts
● Things out of your control--patch Tuesday type updates
● Changes in metric definition
56. Make RUM more like A/B Testing
Launch new features via A/B test
Canary launches
Use a holdback
57. Takeaways
● Metrics are hard, but you can help! speed-metrics-dev@chromium.org
● Focus on user experience metrics
● Use lab for quick iteration, A/B testing for deeper understanding
Hi, I’m Annie. I’m a software engineer on Chrome, and I’ve been working on performance for the last several years. I led performance testing for 3 years, and now I work on performance metrics. I’ve also spent a lot of time helping track down performance regressions in the wild, and looking at unexpected A/B testing results. I wanted to talk about my experience in performance metrics and monitoring in Chrome, as I really hope it can be useful to web development as well.
I’m going to start off by talking about what we’ve learned over the years about developing performance metrics. I’ll go over monitoring in the second half of the talk.
There are three main use cases for top level metrics, and they each have different considerations, which means we weight the importance of properties for them differently.
First, in the lab, where we do performance testing before shipping to customers.
There’s far less data in the lab than in the real world, so it’s much more important that lab metrics be stable and elastic than field metrics.
Let’s take the example of first contentful paint. It’s very simple (first text or image painted on the screen) but it’s not very elastic, since things like shifts in viewport can change the value a lot, and it’s not very stable, since race conditions in which element is painted first can randomize the value. In the field, the large volume of data makes up for this. But in the lab, we can consider alternatives. One alternative Chrome uses is to monitor CPU time to first contentful paint instead of wall time, since it’s more stable and elastic, but it’s clearly less representative of user experience.
Another issue to be aware of in the lab is that timing for tests which require user input is quite difficult. Let’s say we wanted to write a performance test for first input delay. A simple approach would be to wait say, 3 seconds and click and element and measure the queueing time. But what happens in practice with a test like that is that work ends up being shifted after our arbitrary 3 second mark so that the benchmark does not regress, because developers assume we have a good reason for wanting an input in the first 3 seconds to be processed quickly. We could randomize the input time, but then the metric will be much less stable. So instead we wrote a metric that computes when the page will likely be interactable. This is Time to Interactive, and it’s what we use in the lab. But FID is much simpler and more accurate for RUM, which is why we have the two metrics.
Web Perf APIs are another use case for performance metrics. Ideally web perf APIs will be standardized across browsers, and used by both web developers and analytics providers to give site owners insight into the performance of their pages. This means that they need to be Representative, Accurate, and Interpretable. They also need to be computed in realtime, so that pages can access the data during the current session. To standardize, we need clear definitions, which means that being Simple is very important.
Because there is a higher volume of traffic in the field than is possible in the lab, we can sacrifice some stability and elasticity to meet these goals.
Chrome also has internal real user monitoring. We use this to generate the Chrome User Experience Report, and also to understand Chrome’s performance in the field. While we do want our toplevel metrics to be representative and accurate, we can iterate on them quickly so it’s less important from the start than it is for web perf APIs. The large volume of data we gets makes stability and elasticity less necessary. But of course, metrics need to be computed in real time.
That’s a lot of properties and use cases! How do we create metrics in practice? Here’s an example of the recent work the team did on largest contentful paint.
Of course, there’s been work in this area already. Speed index is the best known metric for main content painting. And it’s great! It hits almost all the properties, with one very important exception: it’s not realtime. We’ve tried to add it into Chrome multiple times, but we’ve never been able to get it run correctly and consistently.
Another metric in this space is the now-deprecated first meaningful paint. First meaningful paint is not a simple metric. It’s a heuristic; it records the first paint after the biggest layout change on the page. It’s often hard for developers to understand what might be the biggest layout change on the page, and there isn’t a clear explanation for why the first paint after the biggest layout change so often corresponds to the main content being shown. So it’s not a huge surprise that the metric has accuracy problems. In about 20% of cases, it produces outlier results. This is really hard to address.
So we started work on a new metric. And we had a set of goals. Speed index is great in the lab, but we wanted our metric to be available in web perf APIs and RUM contexts. So we wanted to make it realtime, simple, and interpretable, much more than stable and elastic.
We had a building block we thought we’d be able to use--we can get paint events in realtime. Can we use them to make a Simple, Accurate main content metric?
We started off by brainstorming ideas for metrics we could build with paint event. Maybe just when the last text is painted into the viewport? Or when the biggest image is painted?
We implemented all the variations, and used an internal tool to generate filmstrips and wireframes of paint events for 1000 sites.
And we got great results with the largest image OR text paint in the viewport. Hence, Largest Contentful Paint.
But there were some gotchas.
Pages with splash screens had very early largest contentful paint time. Invalidating images that are removed from the DOM improved the metric’s accuracy.
Body background images also caused metric values that were too early. Excluding them improved accuracy.
It also turned out that the “largest text paint” was not as simple to define as just a paint of a text element. In the image below, most people would say the paragraph is the largest text. But it is made up of multiple elements due to links, which you can see in the wireframe at the right. So we aggregate to a block-level element containing text to further improve accuracy.
One big problem that was left was pages with infinite scroll. Scroll a little, and more images and text are painted. The value of largest contentful paint just keeps going up as you scroll. We didn’t want to penalize these pages, but there wasn’t a great solution. So largest contentful paint isn’t recorded if there is user input.
After testing accuracy on over a thousand filmstrips, we further validated by checking how well largest contentful paint correlates with speed index on about 4 million HttpArchive sites. It’s about a 0.83 correlation, which we are really happy with.
It correlates much less well with First Contentful Paint, which we had also hoped for. Sometimes the largest and first contentful paint for a page are the same, which generally means that it’s painting most of its content at once. But sometimes the largest contentful paint comes a lot later, which means that we’re able to capture more “is the main content visible” in addition to “is the content loading”.
That’s the story of largest contentful paint so far. Looking back, one of the things we think we could have done better is collaborate with the broader web community to design the metric. So if you’re interested in this space, please come talk to me!
I’m also really excited to talk through what I’ve learned about monitoring performance over the years.
I think of performance monitoring in three phases:
First, we test in the lab to catch regressions and understand the basics of how the product is performing.
Next, when we release a new feature or make a big performance related change, we use A/B testing in the wild to get an understanding of real end-user performance.
Lastly, we monitor the performance that end users are seeing in the wild to make sure regressions aren’t slipping through the cracks.
Let’s start with lab testing.
Lab testing is a great place to start monitoring performance. You can catch regressions early, reproduce them locally, and narrow them down to a specific change.
But the downside is we can’t model every user in the lab. There are billions of people using the web on all sorts of different devices and networks. So no matter how good our lab testing is, there will always be regressions that slip through the cracks. And we also won’t be able to predict the magnitude of performance changes in the lab alone.
Lab testing usually comes down to two competing goals. First, reproducibility is critical. If the lab test says that there is a performance regression, we need to be able to repeat that test until we can narrow down which change caused the regression and debug the problem.
But we also want our tests to be as realistic as possible. If we’re monitoring scenarios that real users will never see, we’re going to waste a lot of time tracking down problems that don’t actually affect users. But realism means noise! Real web pages might do different things on successive loads. Real networks don’t necessarily have consistent bandwidth and ping times. Real devices often run differently when they’re hot.
Here’s a visual example of that shift to more realistic user stories from the v8 team (this was part of their Google I/O talk a few years back). The colors on the chart are a top level metric, time spent in different components of the v8 engine. At the top of the chart are more traditional synthetic benchmarks: Octane and Speedometer. At the bottom of the are real-world user stories, in this case some of the Alexa top sites. You can see that the metric breakdown looks very different on more realistic user stories. And this has a big impact on real-world performance. If we’re only looking at the synthetic benchmarks, it makes a lot of sense to optimize that pink JavaScript bar at the expense of the yellow Compliation bar. But this would have a negative impact on most of the real sites. We’ve found that measuring a metric on a variety of real user stories is an important source of realism for Chrome lab testing.
But what about reproducibility? We’ve done a lot of work there, too.
First, take a look at the test environment. We find that tests are more reproducible if run on real hardware than VMs. So that’s a good choice if you have the option. But if you run on multiple devices, you need to take care to make each of them as similar as possible to the others, even buying them in the same lot if you can. And for mobile devices in particular, it’s important to keep the temperature consistent. You may need to allow the device time to cool between runs.
Background tasks are a big source of noise, so try to turn as many of them off as possible.
And think about what else you can make consistent between test runs. Does it make sense to simulate the network? To mock out JavaScript APIs that introduce randomness? To freeze 3rd party scripts?
Diagnostic metrics can also be really useful for reproducibility. For instance, I mentioned earlier that we track CPU time to first contentful paint in Chrome in addition to just wall time. We also track breakdowns of top level metrics, so we can see which part changed the most when they regress.
And back to the fact that you should disable background processes. You can also track a diagnostic metric that just captures what processes were running on the machine when the test ran. When you get a noisy result, check the processes that were running to see if maybe there’s something you should kill.
We also graph what we call reference builds, which are pinned builds of Chrome. In this chart, the yellow line is tip of tree and the green line is a pinned reference version. When they both regress at the same time, you can tell that it’s likely a change in the metric or test environment, and not the performance of tip of tree.
How you detect a change in performance can also have a big impact on your ability to reproduce performance regressions.
It’s also important to be able to compare two versions. We can do this to check fixes for regressions, try out ideas that can’t be checked in, or narrow down a regression range on a timeline. You could even imagine throwing the timeline out altogether and blocking changes from being submitted if they regress performance. But to do that, you need to know if the difference between A and B here is a performance regression, or just noise.
The first thing to do is to add more data points. But then how do you compare them? Taking the averages has a smoothing effect, so you might miss regressions. You could try taking the medians, and I’ve even heard people say that the lowest values might be the best to compare, since those are probably the runs that had the least noise.
But really each group of runs is a set of samples. What we want to know is whether those samples are from the same distribution. We can use a hypothesis test to see if they’re from different distributions. One thing to note with this approach is that performance data is almost never normally distributed. It often has a long tail, or is bimodal. So the t test isn’t a good hypothesis test to use. There are lots of alternatives though.
After lab testing, the next phase is A/B testing.
A/B testing is great for understanding real world performance impacts, both of optimizations, and of regressions caused by new features. The big downside is that it’s hard to A/B test every single change.
And really, an A/B test should be called a controlled experiment. You really want to compare a control group to potential variations.
You DON’T want to do an experiment without a control group. And that’s why user opt-in isn’t a good replacement for A/B testing--the group that opted in is often different from the control group somehow.
An example that always sticks out to me is from about 10 years ago. I used to work on Google Docs, and before SSL was everywhere we had an option to turn it on by default. My director wanted to know the performance cost of turning it on everywhere, so he asked me how much slower performance was for users who opted in. And… it was 250ms FASTER, despite having to do the extra handshake. We thought it was because the group of users that opted in were some of our most tech savvy users, who also opted in to expensive hardware and faster network connections.
So what *is* a controlled experiment?
You should use equal-size, randomly chosen control and experiment groups.
Choose the groups before running the experiment, and allow what we call a preperiod, where you see if there are diffrences between the groups before the experiment starts.
If you don’t have enough data, you have a few options. Either increase the size of experiment groups, or run it for longer.
Lastly, there is real user monitoring.
Real user monitoring is the only ground truth about end user experience. You need RUM to know for sure if performance regressions are getting to end users, and if performance improvements are having an impact. But the downside to RUM data is that it’s really, really hard to understand.
First, a recommendation on what to monitor. It’s simplest to use percentiles instead of averages or composites. The median user experience should be great. And you should also monitor a high percentile, to understand the worst user experiences as well.
The first thing to do when digging into a performance change is to look at the volume. If there is a big change in the number of users at the same time as a change in performance, it’s often mix shift of some sort, and not a change to the product’s performance.
Another thing to do when looking at RUM data is to split it--by country, by operating system, by device type--you’ll often get a clearer picture of the conditions a regression happened under, and whether it was due to a mix shift of one of those things.
But that’s still really hard! Let’s say you do all that and you narrow down a date range at which a regression occurred. You still don’t have a lot to go on. Ideally you could have found the regression earlier. You can do that by A/B testing more aggressively. You can also check performance numbers as a release is canaried to users. Another thing to try after the fact is a holdback, where users are randomly served the old version. That can help narrow whether a version change was responsible for a regression.