This document discusses experimental design for measuring the effectiveness of advertising. It presents several ideas for experiments, including measuring total store sales after advertising campaigns, comparing online purchases of those who saw an ad to those who did not, and using randomized controlled experiments. It notes challenges like defining good controls and issues of external validity. Specific case studies discussed include measuring ad wear-out using a natural experiment with Yahoo ads and using Yelp rating rounding as a natural experiment to measure ratings' impact on restaurant revenue.
Dit is de presentatie die ik heb gehouden bij de schaduwverkiezingen voor de nieuwe burgemeester in de gemeente Heumen. Voor Paul Mengde was dat zijn eerste publieke optreden en ik heb hem daar namens
As multimedia are being used to facilitate and enhance teaching and creative learning, making rich media including images and videos searchable are critical in today and future elearning platform and management system. Traditionally, it's a difficult challenge. Some of the team members have invented an interdisciplinary framework which integrates the state-of-the-art image recognition technology and brain science research from MIT, intelligent cluster algorithm and patentable digital “tagging” technology
Dit is de presentatie die ik heb gehouden bij de schaduwverkiezingen voor de nieuwe burgemeester in de gemeente Heumen. Voor Paul Mengde was dat zijn eerste publieke optreden en ik heb hem daar namens
As multimedia are being used to facilitate and enhance teaching and creative learning, making rich media including images and videos searchable are critical in today and future elearning platform and management system. Traditionally, it's a difficult challenge. Some of the team members have invented an interdisciplinary framework which integrates the state-of-the-art image recognition technology and brain science research from MIT, intelligent cluster algorithm and patentable digital “tagging” technology
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
6. Sergei Vassilvitskii
Idea 1:
Idea 1: Measure the final effect:
– Track total store sales, compare to advertising budget
Findings:
– Total sales typically higher after intense advertising
6
Thursday, April 25, 13
7. Sergei Vassilvitskii
Idea 1:
Idea 1: Measure the final effect:
– Track total store sales, compare to advertising budget
Findings:
– Total sales typically higher after intense advertising
Problems:
– Stores advertise when people tend to spend
– Christmas shopping periods
– Travel during the summer
– Ski gear in winter, etc.
7
Thursday, April 25, 13
10. Sergei Vassilvitskii
Idea 2
“Measuring the online sales impact of an online ad or a
paid-search campaign -- in which a company pays to have
its link appear at the top of a page of search results -- is
straightforward:
We determine who has viewed the ad, then compare online
purchases made by those who have and those who have
not seen it."
10
Thursday, April 25, 13
11. Sergei Vassilvitskii
Idea 2
“Measuring the online sales impact of an online ad or a
paid-search campaign -- in which a company pays to have
its link appear at the top of a page of search results -- is
straightforward:
We determine who has viewed the ad, then compare online
purchases made by those who have and those who have
not seen it."
– Magid Abraham, CEO, President & Co-Founder of ComScore, in HBR
article (2008)
11
Thursday, April 25, 13
13. Sergei Vassilvitskii
Idea 2
Measure the difference between people who see ads and
who don’t.
Findings:
– People who see the ads are more likely to react to them
13
Thursday, April 25, 13
14. Sergei Vassilvitskii
Idea 2
Measure the difference between people who see ads and
who don’t.
Findings:
– People who see the ads are more likely to react to them
Problems:
– Ads are finely targeted. These are exactly the people who are likely to
click!
– Don’t advertise cars in fashion magazines.
– Even more extreme online -- which ads are shown depends on the
propensity of the user to click on the ad.
14
Thursday, April 25, 13
15. Sergei Vassilvitskii
Idea 3
Matching:
– Compare people in a group who saw an ad with people who are
similar, but didn’t see an ad, but are otherwise “the same.”
15
Thursday, April 25, 13
16. Sergei Vassilvitskii
Idea 3
Matching:
– Compare people in a group who saw an ad with people who are
similar, but didn’t see an ad, but are otherwise “the same.”
Problems:
– Hard to define “the same.” Beware of lurking variables.
16
Thursday, April 25, 13
18. Sergei Vassilvitskii
Case Study: Ad Wear-out
Few:
– Don’t want user to be annoyed
– No need to waste money if ad
is ineffective
Many:
– Make sure the user sees it
– Reinforce the message
18
What is the optimal number of times to show an ad?
Thursday, April 25, 13
20. Sergei Vassilvitskii
Observational Study
Look through the data:
– Find the users who saw the ad once
– Find the users who saw the ad many times
Measure Revenue for the two sets of users:
–
Conclusion: Limit the number of impressions
20
Thursday, April 25, 13
21. Sergei Vassilvitskii
Correlations
Why did some users only see the ad once?
– They must use the web differently
– : Sign on once a week to check email
– : Are always online
21
Thursday, April 25, 13
22. Sergei Vassilvitskii
Correlations
Why did some users only see the ad once?
– They must use the web differently
– : Sign on once a week to check email
– : Are always online
Correct conclusion:
– People who visit the homepage often are unlikely to click on ads
– Have not measured the effect of wear-out
22
Thursday, April 25, 13
23. Sergei Vassilvitskii
Idea 3
Matching:
– Compare people in a group who saw an ad with people who are
similar, but didn’t see an ad, but are otherwise “the same.”
Problems:
– Hard to define “the same.” Beware of lurking variables.
23
Thursday, April 25, 13
24. Sergei Vassilvitskii
Simpson’s Paradox
Kidney Stones [Real Data].
You have Kidney stones. There are two treatments A & B.
– Empirically, treatment A is effective 78% of time
– Empirically, treatment B is effective 83% of time
– Which one do you chose?
24
Thursday, April 25, 13
25. Sergei Vassilvitskii
Simpson’s Paradox
Kidney Stones [Real Data].
You have Kidney stones. There are two treatments A & B.
Digging into the data you see:
If they are large:
– Treatment A is effective 73% of the time
– Treatment B is effective 69% of the time
If they are small:
– Treatment A is effective 93% of the time
– Treatment B is effective 87% of the time
25
Thursday, April 25, 13
26. Sergei Vassilvitskii
Simpson’s Paradox
If they are large:
– Treatment A is effective 73% of the time
– Treatment B is effective 69% of the time
If they are small:
– Treatment A is effective 93% of the time
– Treatment B is effective 87% of the time
Overall:
– Treatment A is effective 78% of the time
– Treatment B is effective 83% of the time
26
Thursday, April 25, 13
27. Sergei Vassilvitskii
Simpson’s Paradox Summary Stats
27
A B
Small 81/87 (93%) 234/270 (87%)
Large 192/263 (73%) 55/80 (69%)
Combined 273/350 (78%) 289/350 (83%)
Thursday, April 25, 13
28. Sergei Vassilvitskii
Idea 3
Matching:
– Compare people in a group who saw an ad with people who are
similar, but didn’t see an ad, but are otherwise “the same.”
Problems:
– Hard to define “the same.” Beware of lurking variables.
– Simpson’s Paradox
28
Thursday, April 25, 13
29. Sergei Vassilvitskii
Getting at Causation
Randomized, Controlled Experiments.
– Select a target population
– Randomly decide whom to show the ad
– Subjects cannot influence whether they are in the treatment or control
groups
29
Thursday, April 25, 13
33. Sergei Vassilvitskii
Creating Parallel Universes
When user first arrives:
– Check browser cookie, assign to control or treatment group
– Control group: shown PSA
– Treatment group: shown ad
– Treatment the same on repeated visits
33
Thursday, April 25, 13
34. Sergei Vassilvitskii
Creating Parallel Universes
When user first arrives:
– Check browser cookie, assign to control or treatment group
– Control group: shown PSA
– Treatment group: shown ad
– Treatment the same on repeated visits
Advertising Effects:
– Positive !
– But smaller than reported through observational studies
34
Thursday, April 25, 13
36. Sergei Vassilvitskii
Online Experiments
Advantages:
– Can reach tens of millions of people!
• Can estimate very small effects. Lewis et al., "Here, There, and Everywhere:
Correlated Online Behaviors Can Lead to Overestimates of the Effects of
Advertising." (WWW 2011). Estimate effects of 0.01%!
36
Thursday, April 25, 13
37. Sergei Vassilvitskii
Online Experiments
Advantages:
– Can reach tens of millions of people!
• Can estimate very small effects. Lewis et al., "Here, There, and Everywhere:
Correlated Online Behaviors Can Lead to Overestimates of the Effects of
Advertising." (WWW 2011). Estimate effects of 0.01%!
– Can be relatively cheap (Mechanical Turk)
37
Thursday, April 25, 13
38. Sergei Vassilvitskii
Online Experiments
Advantages:
– Can reach tens of millions of people!
• Can estimate very small effects. Lewis et al., "Here, There, and Everywhere:
Correlated Online Behaviors Can Lead to Overestimates of the Effects of
Advertising." (WWW 2011). Estimate effects of 0.01%!
– Can be relatively cheap
– Can be recruit diverse subjects
• “20 students in a large Midwestern university.” Try to avoid subjects from WEIRD
societies (Western, Educated, Industrialized, Rich, and Democratic).
38
Thursday, April 25, 13
39. Sergei Vassilvitskii
WEIRD People
Which line is longer?
– Henrich, Joseph; Heine, Steven J.; Norenzayan, Ara (2010) : The
weirdest people in the world?, Working Paper Series des Rates für
Sozialund Wirtschaftsdaten
39
Thursday, April 25, 13
41. Sergei Vassilvitskii
Online Experiments
Advantages:
– Can reach tens of millions of people!
• Can estimate very small effects.
– Can be relatively cheap
– Can be recruit diverse subjects
• “20 students in a large Midwestern university.” Try to avoid subjects from WEIRD
societies (Western, Educated, Industrialized, Rich, and Democratic).
– Access: subjects in other countries, geographically diverse
– Can be quick
41
Thursday, April 25, 13
42. Sergei Vassilvitskii
Online Experiments
Advantages:
– Can reach tens of millions of people!
• Can estimate very small effects.
– Can be relatively cheap
– Can be recruit diverse subjects
• “20 students in a large Midwestern university.” Try to avoid subjects from WEIRD
societies (Western, Educated, Industrialized, Rich, and Democratic).
– Access: subjects in other countries, geographically diverse
– Can be quick
Challenges:
– Limited choice in range of treatments (no MRI studies)
– Do people behave differently offline?
42
Thursday, April 25, 13
43. Sergei Vassilvitskii
External Validity
Major Challenge in all lab experiments:
– Virtual and physical labs
– Do findings hold outside the lab?
Enter:
– Natural Experiments
43
Thursday, April 25, 13
44. Sergei Vassilvitskii
Natural Experiments
The experimental condition:
– Is not decided by the experimenter
– But is exogenous (subjects have no effect on the results)
44
Thursday, April 25, 13
45. Sergei Vassilvitskii
Case Study: Ad-wear out
Back to Ad-wear out.
Natural Experiment:
– When there were two competing campaigns, the Yahoo! ad server
decided which campaign to show at random!
– This was by engineering design -- both campaigns got an equal share
of pageviews. (Less complex, easy to distribute than a round robin
system)
45
Few:
– Don’t want user to be annoyed
– No need to waste money if ad is
ineffective
Many:
– Make sure the user sees it
– Reinforce the message
Thursday, April 25, 13
46. Sergei Vassilvitskii
Case Study: Ad-wear out
Natural Experiment:
– When there were two competing campaigns, the Yahoo! ad server
decided which campaign to show at random!
– This was by engineering design -- both campaigns got an equal share
of pageviews. (Less complex, easy to distribute than a round robin
system)
Experiments:
– Compare behavior of people who saw the same total number of ads,
but different number of each campaign.
46
Thursday, April 25, 13
47. Sergei Vassilvitskii
Case Study: Ad-wear out
47
Yes:
– Some advertisements see a 5x drop in click-through rate after the
first exposure
– These typically have very high click-through rates
No:
– Others see no decrease in click-through rate even after ten exposures
– Have lower, but steady click-through rates
Thursday, April 25, 13
48. Sergei Vassilvitskii
Case Study 2: Yelp
Does a higher Yelp Rating lead to higher revenue?
How to do the experiment?
48
Thursday, April 25, 13
49. Sergei Vassilvitskii
Case Study 2: Yelp
Does a higher Yelp Rating lead to higher revenue?
How to do the experiment?
– Observational -- no causality.
– Control -- deception.
– Natural?
49
Thursday, April 25, 13
50. Sergei Vassilvitskii
Case Study 2: Yelp
Does a higher Yelp Rating lead to higher revenue?
Natural Experiment:
– Yelp rounds ratings to the nearest half star.
– 4.24 becomes 4 stars, 4.26 is 4.5 stars
50
Thursday, April 25, 13
51. Sergei Vassilvitskii
Case Study 2: Yelp
Natural Experiment:
– Yelp rounds ratings to the nearest half star.
– 4.24 becomes 4 stars, 4.26 is 4.5 stars
Data:
– Raw ratings from Yelp
– Restaurant revenue (from tax records)
51
Thursday, April 25, 13
52. Sergei Vassilvitskii
Case Study 2: Yelp
Natural Experiment:
– Yelp rounds ratings to the nearest half star.
– 4.24 becomes 4 stars, 4.26 is 4.5 stars
Data:
– Raw ratings from Yelp
– Restaurant revenue (from tax records)
– Finding: a one star increase leads to a 5-9% increase in revenue.
52
Thursday, April 25, 13
53. Sergei Vassilvitskii
Case Study 3: Badges
How do Badges influence user behavior?
Specifically:
– The “epic” badge on stackoverflow.
– Awarded after hitting the maximum number of points (through posts,
responses, etc.) on 50 distinct days.
53
Thursday, April 25, 13
54. Sergei Vassilvitskii
Case Study 3: Badges
How do Badges influence user behavior?
Specifically:
– The “epic” badge on stackoverflow.
– Awarded after hitting the maximum number of points (through posts,
responses, etc.) on 50 distinct days.
Experimental Design:
– Within subject pre-post test (again)
– Look at user behavior before/after receiving badge
– Averaged over different user, different timings, (hopefully) all other
factors.
54
Thursday, April 25, 13
56. Sergei Vassilvitskii
Overall
Experimental Design is hard!
– Be extra skeptical in your analyses. Lots of spurious correlations
Experiments:
– Natural and Controlled are best way to measure effects
Observational Data:
– Sometimes best you can do
– Can lead interesting descriptive insights
– But beware of correlations!
56
Thursday, April 25, 13