1. The document discusses A/B testing approaches for game design, noting key areas that can be tested like onboarding experiences, monetization strategies, and retention mechanics.
2. It introduces Bayesian approaches to A/B testing, noting that observing results allows updating beliefs about hypotheses rather than relying on passing a threshold for significance.
3. Key challenges with frequentist approaches are discussed like multiple comparisons inflating false positive rates, and "peeking" at intermediate results invalidating conclusions. Bayesian methods account for uncertainty and can incorporate prior information and new evidence iteratively.
우리는 지금 무엇을 하고있는지를 고민하나요? 아니면 무엇이 되어가고 있는지를 고민하나요? 네 맞습니다. 우리는 매년 무엇을 할지 고민합니다. 그런데 중요한것은 방향 즉 어디를 가고 있는지 입니다.
그래서 넷플릭스의 추천 시스템이 어디를 향해 가고 있는지를 살펴보고 추천시스템의 향해 가야할 Goal에 대하여 같이 이야기를 해보고자 합니다
3 lessons from 9 years of locomotive offers: Data based user segmentation and...GameCamp
How did TrainStation evolve from simple offers to automated personalized monetisation systems? How to combine using data, design insights and community outreach to create the most compelling offers for players? How do we integrate that information with content creation and automation? Presentation based on practical examples.
Test & Learn: How to Find Your Product's North Star Metric Optimizely
Product-led companies are winning in this digital era by rallying their teams around a product North Star: the metric that connects the business outcomes they are trying to deliver to the work they do each day.
In this talk, you’ll learn a practical framework for how to determine your product’s North Star and how it can shift your strategy.
By attending this session, you will:
Learn how to define a North Star and why it is so important for product development teams
Learn how the best product teams use a narrow set of meaningful metrics to align product strategy with actual team initiatives
See examples of how North Stars change your product roadmap and drive your business forward
Walk through a specific example of setting and evolving a North Star
Feel ready to take learnings back to your teams
우리는 지금 무엇을 하고있는지를 고민하나요? 아니면 무엇이 되어가고 있는지를 고민하나요? 네 맞습니다. 우리는 매년 무엇을 할지 고민합니다. 그런데 중요한것은 방향 즉 어디를 가고 있는지 입니다.
그래서 넷플릭스의 추천 시스템이 어디를 향해 가고 있는지를 살펴보고 추천시스템의 향해 가야할 Goal에 대하여 같이 이야기를 해보고자 합니다
3 lessons from 9 years of locomotive offers: Data based user segmentation and...GameCamp
How did TrainStation evolve from simple offers to automated personalized monetisation systems? How to combine using data, design insights and community outreach to create the most compelling offers for players? How do we integrate that information with content creation and automation? Presentation based on practical examples.
Test & Learn: How to Find Your Product's North Star Metric Optimizely
Product-led companies are winning in this digital era by rallying their teams around a product North Star: the metric that connects the business outcomes they are trying to deliver to the work they do each day.
In this talk, you’ll learn a practical framework for how to determine your product’s North Star and how it can shift your strategy.
By attending this session, you will:
Learn how to define a North Star and why it is so important for product development teams
Learn how the best product teams use a narrow set of meaningful metrics to align product strategy with actual team initiatives
See examples of how North Stars change your product roadmap and drive your business forward
Walk through a specific example of setting and evolving a North Star
Feel ready to take learnings back to your teams
Pristones가 실제 서비스에 Growth Haking을 적용한 경험을 '삼성전자 미디어 솔루션 센터'와 상암 DMC 스타트업 모임 Spark@DMC에서 발표한 자료입니다.
주요 내용
- Growth Hacking의 개념 정의와 대표적 사례
- Growth Hacking의 기본 분석 방법론
- Growth Hacking 기법의 로켓펀치(http://rocketpun.ch/), 클럽믹스(http://clubmix.co.kr/) 실제 적용 사례
어떻게 하면 게임 유저의 행동을 분석하여 개인 맞춤형 관리를 할 수 있을까요? 텐투플레이의 개인화 라이브옵스(LiveOps)로 게임 수익을 향상시킬 수 있는 노하우를 공개합니다.
1. 게임 로그 데이터 수집, 2. 가설 세우기, 3. 가설 검정, 4. 추천 아이템 및 전략 가설 검정
영상: https://www.youtube.com/watch?v=foqt4Y21uUA
NDC 2016 이은석 - 돌죽을 끓입시다: 창의적 게임개발팀을 위한 왓 스튜디오의 업무 문화Eunseok Yi
창의적이고 고품질의 게임 개발 결과물을 낼 수 있게 돕는 조직 내부의 개방적 업무 문화에 대한 강연입니다. 강연자가 책임자로 몸담고 있는 왓 스튜디오가 <야생의>를 만들면서 겪는 예시들을 들어서 설명합니다.
꿈과 열정에 기반한 자발적 업무 문화, 개인이 아닌 집단이 창의적인 결과물을 내게 일하는 방법, 지향점의 공유와 정렬, 효율적이고 개방적인 조직 구조, 의사소통에 쓰이는 수단들과 특성, 왓 스튜디오라는 집단이 굴러가며 <야생의> 같은 독특한 게임을 만들어가는 시스템 등에 대한 소개가 있을 예정입니다.
오늘 밤부터 쓰는 google analytics (구글 애널리틱스, GA) Yongho Ha
http://ga.yonghosee.com 에서 진행하는 구글 어날리틱스(google analytics) 에 대한 강의 슬라이드 입니다. 이 슬라이드는 샘플이지만, 초반부는 실재 강의 교재 그대로 입니다. 이것 자체로도 여러분이 GA를 이해하는데 좀 도움이 된다면 기쁘겠습니다^^ 감사합니다.
[IGC 2017] 라이엇게임즈 유석문 - 게임 개발의 Agile Best Practices강 민우
플레이어들이 원하는 기능을 적시에 개발하여 배포하기 위해 필요한 Agile의 Practice를 소개 합니다. 기획 단계에서부터 개발, 운영까지 필요한 기술 Practice와 함께 효과적인 조직 문화 구축에 필요한 Practice를 Riot Games의 사례를 기반으로 설명합니다.
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들Minho Lee
2021-09-04 프롬 특강 발표자료입니다.
---
많은 사람들이 A/B 테스트가 중요하다고 말합니다.
그런데 우리는 뭘 믿고 A/B 테스트에 의사결정을 맡기는 걸까요?
A/B 테스트는 그냥 돌리면 성과를 만들어주는 마법의 도구가 아닙니다.
신뢰할 수 있는 실험 결과를 위해 어떤 고민이 더 필요한지 살펴보려고 합니다.
[우리가 데이터를 쓰는 법] 모바일 게임 로그 데이터 분석 이야기 - 엔터메이트 공신배 팀장Dylan Ko
Gonnector(고넥터) 고영혁 대표가 주최한 스타트업 데이터 활용 세미나 '우리가 데이터를 쓰는 법' 의 세 번째 발표 자료
세미나 : 우리가 데이터를 쓰는 법 (How We Use Data)
일시 : 2016년 4월 12일 화요일 10:00 ~ 18:00
장소 : 마루180 (Maru180) B1 Think 홀
제목 : 모바일 게임 로그 데이터 분석 이야기
연사 : 엔터메이트 공신배 팀장
The benefits of operating a free-to-play "game-as-a-service" are well known: elastic pricing, a direct relationship with your players, longer lifespan, and an opportunity to fine-tune after launch. But to fully realize these benefits, you need to plan your live operations strategy as carefully as you plan your game. This talk will show how you can build an effective LiveOps strategy using PlayFab.
Mobile Analytics (Mobile BI) - A Game Changer Jitender Aswani
Big Data - According to IDC, data is doubling every two years and is expected to reach 1.8 ZB (a trillion GB) in 2011.
Eight mobility related mega trends are locked in a virtuous cycle and will be the bedrock for growth and adoption of Mobile BI solutions and for the overall Enterprise Mobility.
According to Gartner, more than 33% of Analytics will be consumed using mobile devices, a prediction well supported by the 8 mobility related mega trends discussed here.
Mobile BI market could grow at 20% plus CAGR over the next 5 years and could likely become over a $2 billion market by 2015.
Therefore, by 2015 more than 15% of Analytics revenues could come from Mobile Analytics solutions. This should be a serious strategic priority for every Analytics vendor if not already.
Pristones가 실제 서비스에 Growth Haking을 적용한 경험을 '삼성전자 미디어 솔루션 센터'와 상암 DMC 스타트업 모임 Spark@DMC에서 발표한 자료입니다.
주요 내용
- Growth Hacking의 개념 정의와 대표적 사례
- Growth Hacking의 기본 분석 방법론
- Growth Hacking 기법의 로켓펀치(http://rocketpun.ch/), 클럽믹스(http://clubmix.co.kr/) 실제 적용 사례
어떻게 하면 게임 유저의 행동을 분석하여 개인 맞춤형 관리를 할 수 있을까요? 텐투플레이의 개인화 라이브옵스(LiveOps)로 게임 수익을 향상시킬 수 있는 노하우를 공개합니다.
1. 게임 로그 데이터 수집, 2. 가설 세우기, 3. 가설 검정, 4. 추천 아이템 및 전략 가설 검정
영상: https://www.youtube.com/watch?v=foqt4Y21uUA
NDC 2016 이은석 - 돌죽을 끓입시다: 창의적 게임개발팀을 위한 왓 스튜디오의 업무 문화Eunseok Yi
창의적이고 고품질의 게임 개발 결과물을 낼 수 있게 돕는 조직 내부의 개방적 업무 문화에 대한 강연입니다. 강연자가 책임자로 몸담고 있는 왓 스튜디오가 <야생의>를 만들면서 겪는 예시들을 들어서 설명합니다.
꿈과 열정에 기반한 자발적 업무 문화, 개인이 아닌 집단이 창의적인 결과물을 내게 일하는 방법, 지향점의 공유와 정렬, 효율적이고 개방적인 조직 구조, 의사소통에 쓰이는 수단들과 특성, 왓 스튜디오라는 집단이 굴러가며 <야생의> 같은 독특한 게임을 만들어가는 시스템 등에 대한 소개가 있을 예정입니다.
오늘 밤부터 쓰는 google analytics (구글 애널리틱스, GA) Yongho Ha
http://ga.yonghosee.com 에서 진행하는 구글 어날리틱스(google analytics) 에 대한 강의 슬라이드 입니다. 이 슬라이드는 샘플이지만, 초반부는 실재 강의 교재 그대로 입니다. 이것 자체로도 여러분이 GA를 이해하는데 좀 도움이 된다면 기쁘겠습니다^^ 감사합니다.
[IGC 2017] 라이엇게임즈 유석문 - 게임 개발의 Agile Best Practices강 민우
플레이어들이 원하는 기능을 적시에 개발하여 배포하기 위해 필요한 Agile의 Practice를 소개 합니다. 기획 단계에서부터 개발, 운영까지 필요한 기술 Practice와 함께 효과적인 조직 문화 구축에 필요한 Practice를 Riot Games의 사례를 기반으로 설명합니다.
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들Minho Lee
2021-09-04 프롬 특강 발표자료입니다.
---
많은 사람들이 A/B 테스트가 중요하다고 말합니다.
그런데 우리는 뭘 믿고 A/B 테스트에 의사결정을 맡기는 걸까요?
A/B 테스트는 그냥 돌리면 성과를 만들어주는 마법의 도구가 아닙니다.
신뢰할 수 있는 실험 결과를 위해 어떤 고민이 더 필요한지 살펴보려고 합니다.
[우리가 데이터를 쓰는 법] 모바일 게임 로그 데이터 분석 이야기 - 엔터메이트 공신배 팀장Dylan Ko
Gonnector(고넥터) 고영혁 대표가 주최한 스타트업 데이터 활용 세미나 '우리가 데이터를 쓰는 법' 의 세 번째 발표 자료
세미나 : 우리가 데이터를 쓰는 법 (How We Use Data)
일시 : 2016년 4월 12일 화요일 10:00 ~ 18:00
장소 : 마루180 (Maru180) B1 Think 홀
제목 : 모바일 게임 로그 데이터 분석 이야기
연사 : 엔터메이트 공신배 팀장
The benefits of operating a free-to-play "game-as-a-service" are well known: elastic pricing, a direct relationship with your players, longer lifespan, and an opportunity to fine-tune after launch. But to fully realize these benefits, you need to plan your live operations strategy as carefully as you plan your game. This talk will show how you can build an effective LiveOps strategy using PlayFab.
Mobile Analytics (Mobile BI) - A Game Changer Jitender Aswani
Big Data - According to IDC, data is doubling every two years and is expected to reach 1.8 ZB (a trillion GB) in 2011.
Eight mobility related mega trends are locked in a virtuous cycle and will be the bedrock for growth and adoption of Mobile BI solutions and for the overall Enterprise Mobility.
According to Gartner, more than 33% of Analytics will be consumed using mobile devices, a prediction well supported by the 8 mobility related mega trends discussed here.
Mobile BI market could grow at 20% plus CAGR over the next 5 years and could likely become over a $2 billion market by 2015.
Therefore, by 2015 more than 15% of Analytics revenues could come from Mobile Analytics solutions. This should be a serious strategic priority for every Analytics vendor if not already.
Milion Dollar Impact Through Metrics, Analytics & A/B TestingAzhar Bandeali
This is an updated version of the talk I gave at Product Camp ATL 2017.
Atlanta doesn't talk enough about data analytics and A/B testing. I was able to follow my product's performance on a product scorecard built on (free) analytics tools to identify key areas of improvement. Once identified, I conducted A/B testing on potential fixes to find the best performing change that resulted in real revenue impact. This session is a case study on what I did and how you can do the same for your product.
Note: I presented this session last year and it helped someone get a promotion at work! So, I'll be presenting an updated version of it in 2017.
Mark's HoneyTracks presentation at Casual Connect 2012 in Hamburg on Feb 9th: What metrics / KPIs should you focus on along the different life cylce stages of your game.
Introduction to Neural Networks and Deep Learning from ScratchAhmed BESBES
If you're willing to understand how neural networks work behind the scene and debug the back-propagation algorithm step by step by yourself, this presentation should be a good starting point.
We'll cover elements on:
- the popularity of neural networks and their applications
- the artificial neuron and the analogy with the biological one
- the perceptron
- the architecture of multi-layer perceptrons
- loss functions
- activation functions
- the gradient descent algorithm
At the end, there will be an implementation FROM SCRATCH of a fully functioning neural net.
code: https://github.com/ahmedbesbes/Neural-Network-from-scratch
Theory to consider an inaccurate testing and how to determine the prior proba...Toshiyuki Shimono
I presented a mathematical theory on a medical testing method. This fundamental theory can be taken account of both cases when the resource of the testing is limited or not. One implication is that "negative proof" may not function well, and another implication is that excessively high specificity and accuracy are required for meaningful diagnosis unless the careful usage of the diagnosis is considered.
Runtime Analysis of Population-based Evolutionary AlgorithmsPer Kristian Lehre
Populations are at the heart of evolutionary algorithms (EAs). They provide the genetic variation which selection acts upon. A complete picture of EAs can
only be obtained if we understand their population dynamics. A rich theory on runtime analysis (also called time-complexity analysis) of EAs has been
developed over the last 20 years. The goal of this theory is to show, via rigorous mathematical means, how the performance of EAs depends on their
parameter settings and the characteristics of the underlying fitness landscapes. Initially, runtime analysis of EAs was mostly restricted to
simplified EAs that do not employ large populations, such as the (1+1) EA. This tutorial introduces more recent techniques that enable runtime
analysis of EAs with realistic population sizes.
The tutorial begins with a brief overview of the population‐based EAs that are covered by the techniques. We recall the common stochastic selection
mechanisms and how to measure the selection pressure they induce. The main part of the tutorial covers in detail widely applicable techniques tailored to
the analysis of populations.
To illustrate how these techniques can be applied, we consider several fundamental questions: When are populations necessary for efficient
optimisation with EAs? What is the appropriate balance between exploration and exploitation and how does this depend on relationships between mutation and
selection rates? What determines an EA's tolerance for uncertainty, e.g. in form of noisy or partially available fitness?
Runtime Analysis of Population-based Evolutionary AlgorithmsPK Lehre
Populations are at the heart of evolutionary algorithms (EAs). They provide the genetic variation which selection acts upon. A complete picture of EAs can only be obtained if we understand their population dynamics. A rich theory on runtime analysis (also called time-complexity analysis) of EAs has been developed over the last 20 years. The goal of this theory is to show, via rigorous mathematical means, how the performance of EAs depends on their parameter settings and the characteristics of the underlying fitness landscapes. Initially, runtime analysis of EAs was mostly restricted to simplified EAs that do not employ large populations, such as the (1+1) EA. This tutorial introduces more recent techniques that enable runtime analysis of EAs with realistic population sizes.
The tutorial begins with a brief overview of the population‐based EAs that are covered by the techniques. We recall the common stochastic selection mechanisms and how to measure the selection pressure they induce. The main part of the tutorial covers in detail widely applicable techniques tailored to the analysis of populations. We discuss random family trees and branching processes, drift and concentration of measure in populations, and level‐based analyses.
To illustrate how these techniques can be applied, we consider several fundamental questions: When are populations necessary for efficient optimisation with EAs? What is the appropriate balance between exploration and exploitation and how does this depend on relationships between mutation and selection rates? What determines an EA's tolerance for uncertainty, e.g. in form of noisy or partially available fitness?
This tutorial was presented at the 2015 IEEE Congress on Evolutionary Computation at Sendai, Japan, May 25th 2015.
How to track and improve Customer Experience with LEO CDPTrieu Nguyen
1) Why CX measurement is so important
2) Introduction to key metrics of CX
2.1 Customer Feedback Score (CFS)
2.2 Customer Effort Score (CES)
2.3 Customer Satisfaction Score (CSAT)
2.4 Net Promoter Score (NPS)
3) Using Journey Map to CX Data Management
4) Introduction to LEO CDP and demo
[Notes] Customer 360 Analytics with LEO CDPTrieu Nguyen
Part 1: Why should every business need to deploy a CDP ?
1. Big data is the reality of business today
2. What are technologies to manage customer data ?
3. The rise of first-party data and new technologies for Digital Marketing
4. How to apply USPA mindset to build your CDP for data-driven business
Part 2: How to use LEO CDP for your business
1. Core functions of LEO CDP for marketers and IT managers
2. Data Unification for Customer 360 Analytics
3. Data Segmentation
4. Customer Personalization
5. Customer Data Activation
Part 3: Case study in O2O Retail and Ecommerce
1. How to build customer journey map for ecommerce and retail
2. How to do customer analytics to find ideal customer profiles
The ideal customer profile in a B2B context
The ideal customer profile in a B2C context
3. Manage product catalog for customer personalization
4. Monitoring Data of Customer Experience (CX Analytics)
CX Data Flow
CX Rating plugin is embedded in the website, to collect feedback data
An overview of CX Report
A CX Report in a customer profile
5. Monitoring data with real-time event tracking reports
Event Data Flow
Summary Event Data Report
Event Data Report in a Customer Profile
Part 4: How to setup an instance of LEO CDP for free
1. Technical architecture
2. Server infrastructure
3. Setup middlewares: Nginx, ArangoDB, Redis, Java and Python
Network requirements
Software requirements for new server
ArangoDB
Nginx Proxy
SSL for Nginx Server
Java 8 JVM
Redis
Install Notes for Linux Server
Clone binary code for new server
Set DNS hosts for LEO CDP workers
4. Setup data for testing and system verification
Part 5: Summary all key ideas
Why should you invest in LEO CDP ?
Purpose: Big data and AI democracy for SMEs companies
Problem: Customer Analytics and Customer Personalization
Solutions: CDP + CX + Personalization Engine
Product demo: LEO CDP for Ecommerce and Fintech
Business model: Freemium → Ecosystem → Subscription
Market size: 20 billion USD in 2026 and CAGR 34.6%
Differentiation: cloud-native software
Go-to-market approach: Community → Free → Paid
Team: 1 full-stack dev, 1 data scientist and 12,000 fans of BigDataVietnam.org Community
Need 150,000 USD for scaling business (you get 20% share)
Lộ trình triển khai LEO CDP cho ngành bất động sảnTrieu Nguyen
1) Hiểu bài toán số hoá trải nghiệm khách hàng
2) Nghiên cứu giải pháp LEO CDP
3) Lộ trình triển khai
Phát triển / số hoá điểm chạm khách hàng
Xây dựng bản đồ hành trình khách hàng
Định nghĩa các metrics và KPI quan trọng
Xây dựng web portal và mobile data hub
Xây dựng kế hoạch Digital Marketing
Triển khai CDP và Marketing Automation
Xây dựng đội Analytics để phân tích dữ liệu
From Dataism to Customer Data PlatformTrieu Nguyen
1) How to think in the age of Dataism with LEO CDP ?
2) Why is Dataism for human, business and society ?
3) How should LEO Customer Data Platform (LEO CDP) work ?
4) How to use LEO CDP for your business ?
Data collection, processing & organization with USPA frameworkTrieu Nguyen
1) How to think in the age of Dataism with USPA framework ?
2) How to collect customer data
3) Data Segmentation Processing for flexibility and scalability
4) Data Organization for personalization and business activation
Part 1: Introduction to digital marketing technologyTrieu Nguyen
Outline of this course
1. Digital Media Models in the age of marketing 4.0
2. Strategic Thought as It Relates to Digital Marketing
3. Web: The Center of Digital Marketing Delivery Mix
4. Content Management System (CMS) and headless CMS
5. Search Engine Marketing
6. Email Marketing
7. Social Media and Mobile Marketing
8. Introduction to Advertising Technology (Ad Tech)
9. Introduction to Customer Database and Customer Data Platform (CDP)
10. Legal Issues: Data privacy, Security, and Intellectual Property
11. Case study: IKEA - from business strategy to digital marketing strategy
12. Recommended books for self-study
Transform your marketing and sales capabilities with Big Data and A.I
1) Why is Customer Data Platform (CDP) ?
Case study: Enhancing the revenue of your restaurant with CDP and mobile app marketing
Question: Why can CDP disrupt business model for restaurant industry (B2C) ?
2) How would CDP work in practice ?
Introducing USPA.tech as logical framework for implementing CDP in practice
How Can a Customer Data Platform Enhance Your Account-Based Marketing Strategy (B2B) ?
3) How can we implement CDP for business?
Introducing the CDP as customer-first marketing platform for all industries (my key idea in this slide)
Video Ecosystem and some ideas about video big dataTrieu Nguyen
Introduction to Video Ecosystem Mind Map
Video Streaming Platform
Video Ad Tech Platform
Video Player Platform
Video Content Distribution Platform
Video Analytics Platform
Summary of key ideas
Q & A
Concepts, use cases and principles to build big data systems (1)Trieu Nguyen
1) Introduction to the key Big Data concepts
1.1 The Origins of Big Data
1.2 What is Big Data ?
1.3 Why is Big Data So Important ?
1.4 How Is Big Data Used In Practice ?
2) Introduction to the key principles of Big Data Systems
2.1 How to design Data Pipeline in 6 steps
2.2 Using Lambda Architecture for big data processing
3) Practical case study : Chat bot with Video Recommendation Engine
4) FAQ for student
Apache Hadoop and Spark: Introduction and Use Cases for Data AnalysisTrieu Nguyen
Growth of big datasets
Introduction to Apache Hadoop and Spark for developing applications
Components of Hadoop, HDFS, MapReduce and HBase
Capabilities of Spark and the differences from a typical MapReduce solution
Some Spark use cases for data analysis
Introduction to Recommendation Systems (Vietnam Web Submit)Trieu Nguyen
1) Why do we need recommendation systems ?
2) How can we think with recommendation systems ?
3) How can we implement a recommendation system with open source technologies ?
RFX framework https://github.com/rfxlab
Apache Kafka: https://kafka.apache.org
Apache Spark: https://spark.apache.org
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
11. 1. Split population 2. Show variations 4. Choose winner 4. Deploy winner3. Measure response
12. State
Sync State
Sync
Game
State
DB
Data
Warehouse
Analyse
Events
(tagged
with
A
or
B)
Game
Server
Content
Management
System
A
or
B
variant
A/B
Test
Server
19. (f) Zoomed in version of the previous scatterplot. Note that jitter and transparency
Value of first purchase
Totalspend
$70$30$15
A/B Test
VIPs
Repeat
Purchases
$5$2
31. p-Value
95%
2.5% 2.5%
=
1
p
2
e
(x µ)2)
2 2
(n − r)!r!
µ = np σ = np(1 − p)
z =
¯x − µ
σ
z =
¯x − µ
ˆσ√
n
SE(¯x) =
ˆσ
√
n
n =
(zα + zβ)2
σ2
δ2
µ + 1.96σ
µ − 1.96σ
z =
¯x − µ
σ
z =
¯x − µ
ˆσ√
n
SE(¯x) =
ˆσ
√
n
n =
(zα + zβ)2
σ2
δ2
µ + 1.96σ
µ − 1.96σ
32. p-Value
The probability of observing as extreme a result
assuming the null hypothesis is true
The probability of the data given the model
OR
33. Null Hypothesis: H0
H H
Accept H Type-II Error
Reject H Type-I Error
Truth
Observation
False Positive
34. All we can ever say is either
not enough evidence that retention rates are the same
the retention rates are different, 95% of the time
p < 0.05
p-Value
35. actually...
The evidence supports a rejection of the null hypothesis, i.e.
the probability of seeing a result as extreme as this, assuming
the retention rate is actually 30%, is less than 5%.
p < 0.05
40. Family-wise Error
p = P( Type-I Error) = 0.05
5% of the time we will get a false positive - for one treatment
P( no Type-I Error) = 0.95
P( at least 1 Type-I Error for 2 treatments) = (1 - 0.9025) = 0.0975
P( no Type-I Error for 2 treatments) = (0.95)(0.95) = 0.9025
46. Tossing the Coin
0 10 20 30 40 50
0.00.20.40.60.81.0
Number of flips
ProportionofHeads
Heads
Tails
THTHTTTHHTHH…
47. 1 5 10 50 500
0.00.20.40.60.81.0
Number of flips
ProportionofHeads Tossing the Coin
Long run average = 0.5
48. p(y|x) =
p(y, x)
p(x)
p(x|y) =
p(x, y)
p(y)
p(y, x) = p(x, y) = p(y|x)p(x) = p(x|y)p(y)
p(y|x) =
p(x|y)p(y)
p(x)
p(y|x) =
p(x|y)p(y)
P
Probability of x
p(y|x) =
p(y, x)
p(x)
p(x|y) =
p(x, y)
p(y)
p(y, x) = p(x, y) = p(y|x)p(x) = p(x|y)p(y)
p(y|x) =
p(x|y)p(y)
p(x)
Probability of x and y
(conjoint)
p(y|x) =
p(y, x)
p(x)
p(x|y) =
p(x, y)
p(y)
p(y, x) = p(x, y) = p(y|x)p(x) = p(x|y)p
p(x|y)p(y)
Probability of x given y
(conditional)
Terminology
49. The Bernoulli distribution
For a “fair” coin, θ = 0.5
Head (H) = 1, Tails (T) = 0
p(x|θ) = θx(1 − θ)(1−x)
A single toss:
p(x|θ) = θx(1 − θ)(1−x)
p(heads = 1|0.5) = 0.51(1−0.5)(1−1) = 0.5
0 (1−0)
p(x|θ) = θx(1 − θ)(1−x)
p(heads = 1|0.5) = 0.51(1−0.5)(1−1) = 0.5
p(tails = 0|0.5) = 0.50(1−0.5)(1−0) = 0.5
50. The Binomial
p(x|θ) = θx(1 − θ)(1−x)
Probability of heads in a single throw:
Probability of x heads in n throws:
p(heads = 1|0.5) = 0.51
(1 − 0.5)(1−1)
= 0.5
p(tails = 0|0.5) = 0.50
(1 − 0.5)(1−0)
= 0.5
p(h|n, θ) =
n
h
θh
(1 − θ)(n−h)
p(x|θ, n) =
n
x
θx
(1 − θ)(n−x)
51. 20 tosses of a fair coin
=
1
p
2⇡ 2
e
(x µ)2)
2 2
1 − θ)(n−x)p(θ)dθ
pc − pv
1−pc)+pv(1−pv)
n
p(x|0.5, 20)
p(x|θ)p(θ)
x
(1 − θ)(1−x)
0.51
(1 − 0.5)(1−1)
= 0.5
52. 20 tosses of an “un-fair” coin
1
p
2⇡ 2
e
(x µ)2)
2 2
θx(1 − θ)(n−x)p(θ)dθ
ztest =
pc − pv
pc(1−pc)+pv(1−pv)
n
p(x|0.2, 20) p(x|0.5, 20)
p(¯θ) = p(x|θ)p(θ)
p(x|θ) = θx
(1 − θ)(1−x)
s = 1|0.5) = 0.51
(1 − 0.5)(1−1)
= 0.5
53. Likelihood of θ given observation i of x heads in n throws:
“Binomial Likelihood”
h
p(x|θ, n) =
n
x
θx
(1 − θ)(n−x)
1
p(h|n, θ) =
n
h
θh
(1 − θ)(n−h)
p(x|θ, n) =
n
x
θx
(1 − θ)(n−x)
L(θ|xi, ni) =
ni
xi
θxi
(1 − θ)(ni−xi)
54. The Likelihood
Increasing likelihood of θ with more observations…
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.4
1 heads in 2 throws.
θ
Likelihood
0.0 0.2 0.4 0.6 0.8 1.0
0.000.100.20
5 heads in 10 throws.
θ
Likelihood
0.0 0.2 0.4 0.6 0.8 1.0
0.0000.0100.020
500 heads in 1000 throws.
θ
Likelihood
55. p(¯θ) = p(x|θ)p(θ)
p(x|θ) = θx
(1 − θ)(1−x)
p(heads = 1|0.5) = 0.51
(1 − 0.5)(1−1)
= 0.5
p(tails = 0|0.5) = 0.50
(1 − 0.5)(1−0)
= 0.5
The observations (#heads)
p(x|0.2, 20) p(x|0.5, 20)
p(¯θ) = p(x|θ)p(θ)
p(x|θ) = θx
(1 − θ)(1−x)
p(heads = 1|0.5) = 0.51
(1 − 0.5)(1−1)
= 0.5
p(tails = 0|0.5) = 0.50
(1 − 0.5)(1−0)
= 0.5
The model parameter (e.g. fair coin)
p(x|θ, n) = θx
(1 − θ)(n−x)
B(a, b) =
Γ(a)Γ(b)
Γ(a + b)
=
(a − 1)!(b − 1)!
(a + b − 1)!
beta(θ|a, b) = θ(a−1)
(1 − θ)(b−1)
/ B(a, b)
p(θ|x) = θx
(1 − θ)(n−x)
p(θ) / p(x)We want to know this
ztest =
c v
pc(1−pc)+pv(1−pv)
n
p(x|0.2, 20) p(x|0.5, 20)
p(¯θ) = p(x|θ)p(θ)
p(x|θ) = θx
(1 − θ)(1−x)
p(heads = 1|0.5) = 0.51
(1 − 0.5)(1−1)
= 0.5
Probability of data given model
A recap…
56. Note that
p(x, y) = p(x|y)p(y) = p(y|x)p(x)
p(x|θ) ̸= p(θ|x)
p(cloudy|raining) ̸= p(raining|cloudy)
p(θ|x, n) = θx
(1 − θ)(n−x)
θ(a−1)
(1 − θ)(b−1)
/ B(a, b) p(
p(θ|x, n) = θx+a−1
(1 − θ)(n−x+b−1)
/ B(x + a, n − x + b
61. p(✓|x)
| {z }
= p(x|✓)
| {z }
p(✓)
|{z}
/ p(x)
|{z}
p(x) =
Z
p(x|✓)p(✓)d✓
The prior
Captures our “belief” in the model
based on prior experience, observations or knowledge
63. Selecting a prior
We’d like the product of prior and likelihood
to be “like” the likelihood
We’d like the integral to be easily evaluated
p(x, n|θ) = θx
(1 − θ)(n−x)
p(θ|x) = θx
(1 − θ)(n−x)
p(θ) / p(x)
p(θ|x) =
θx
(1 − θ)(n−x)
p(θ)
θx(1 − θ)(n−x)p(θ)dθ
ztest =
pc − pv
pc(1−pc)+pv(1−pv)
n
p(h|n, θ) =
n
h
θh
(1 − θ)(n−h)
p(x|θ, n) =
n
x
θx
(1 − θ)(n−x)
1
67. beta priorbinomial likelihood
Putting it together…
p(θ|x, n) = θx
(1 − θ)(n−x)
θ(a−1)
(1 − θ)(b−1)
/ B(a, b) p(x, n)
p(θ|x, n) = θx+a−1
(1 − θ)(n−x+b−1)
/ B(x + a, n − x + b)
p(x, n|θ) = θx
(1 − θ)(n−x)
number of heads x number of tails (n-x)
p(x|y) ̸= p(y|x)
p(cloudy|raining) ̸= p(raining|cloudy)
p(θ|x, n) = θx
(1 − θ)(n−x)
θ(a−1)
(1 − θ)(b−1)
/ B(a, b) p(x)
p(θ|x, n) = θx+a−1
(1 − θ)(n−x+b−1)
/ B(x + a, n − x + b)
p(θa > θb) =
θa>θb
p(θa|xa)p(θb|xb) dθadθb
68. 1. Decide on a prior, which captures your belief
2. Run experiment and observe data for heads, tails
3. Determine your posterior based on the data
4. Use posterior as your new belief and re-run
experiment
5. Rinse, repeat until you hit an actionable certainty
Putting it together…
77. Captures our prior belief, expertise, opinion
Strong prior belief means:
‣ we need lots of evidence to contradict
‣ results converge more quickly (if prior is “relevant”)
Provides inertia
With enough samples, prior’s impact diminishes,
rapidly
The Prior
79. With 1 or more variants we have a multi-dimensional
problem
Need to evaluate volumes under the posterior
In general requires numerical quadrature = Markov
Chain Monte-Carlo (MCMC)
Multiple variant tests
95. Benefits / Features
Continuously observable
No need to fix population size in advance
Incorporate prior knowledge / expertise
Result is a “true” probability
A measure of the difference magnitude is given
Consistent framework for lots of different scenarios
99. Multi-arm bandits
Draw sample from
each "arm" prior
distribution
Evaluate Success
Update posterior for
chosen "arm"
Pick largest & deploy
Thompson Sampling
Thompson, William R. "On the likelihood that one unknown probability exceeds another in view of the evidence of two samples". Biometrika, 25(3–4):285–294