Do you want to know how real users are interacting with your product? Do you want to know which features they don’t use? Would you like to understand how your product works internally under real operational conditions? Then you need telemetry—the instrumentation of your product to record this information and transmit it back to you for analysis. Windows 10 implemented this capability. Today, there are more than 450 million devices running Windows 10 providing constant feedback on its operation. Ken Johnston says Microsoft learned a lot about what they did right for that launch—and what they have to do better going forward. Ken shares the internal architecture of Microsoft Windows Telemetry and how its data cloud is constructed. He describes how Windows implemented privacy controls for users, how they gather and categorize customer feedback, and what they learned by sharing features early with users. All this infrastructure is now an integral part of how Microsoft prioritizes bugs and feature requests. Learn how to add telemetry to your applications to answer your use and quality questions.

1. BW9
Test & QA
6/7/2017 2:45:00 PM
BW9 Drive Product Improvements with
Telemetry
Presented by:
Ken Johnston
Microsoft
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - https://www.techwell.com/

2. Ken Johnston
Microsoft
Ken Johnston is a principal data science manager on the Microsoft core data
science team where he and his team focus their research on Windows post sales
monetization and device usage in the commercial and education segments. Since
joining Microsoft in 1998 Ken's roles have included GPM for Bing data quality and
measurements; group manager for Bing shopping and data operations; test lead
and test manager on MSN, hosted exchange, subscription and billing platform,
and office products. He previously served as the Microsoft director of test
excellence. Ken is a frequent presenter, a regular blogger, coauthor of How We
Test Software at Microsoft, and contributor to Experiences of Test Automation:
Case Studies of Software Test Automation. Contact Ken on Twitter @rkjohnston.

3. 6/14/2017
1
Drive Product Improvements
with Telemetry
Th D S i B hi d Th
#WindowsBistro @RKJohnston
The Data Science Behind The
Windows 10 Bistro
3 Great Loves
#WindowsBistro @RKJohnston

4. 6/14/2017
2
#WindowsBistro @RKJohnston
#WindowsBistro @RKJohnston

5. 6/14/2017
3
Las Vegas has Great Restaurants
#WindowsBistro @RKJohnston
You
sho ldshould
open a
Restaurant
#WindowsBistro @RKJohnston
Restaurant

6. 6/14/2017
4
Open a Restaurant Together
#WindowsBistro @RKJohnston
6 Keys to Success
Safe & Private Location Listen to our past
Customers
Taste Test the New
Menu
#WindowsBistro @RKJohnston
Big Grand Opening Listen to Feedback Grow the Business

7. 6/14/2017
5
#WindowsBistro @RKJohnston
Privacy=Personal Security
#WindowsBistro @RKJohnston

8. 6/14/2017
6
Privacy is in the Users Hands
Windows as a Service
Most people just use services and
trus them to maintain privacy
Windows is evolving to be aWindows is evolving to be a
service.
Official Position:
1. Windows 10 collects
information so the product will
work better for you.
#WindowsBistro @RKJohnston
http://www.winbeta.org/news/how-check-
and-adjust-your-privacy-settings-windows-10
f y
2. You are in control with the
ability to determine what
information is collected.
A Look Inside the Kitchen
•Opt Out features
Random Sampling Rates•Random Sampling Rates
•Data Scrubbing at
Ingestion
•PII Purging
N T t d M k ti
#WindowsBistro @RKJohnston
•No Targeted Marketing

9. 6/14/2017
7
Three Levels of Telemetry
Using the Data
Level 1 is heartbeat.
• Is the Device Alive
Level 1: PC Census Data (Very Limited Data Points)
Auto-populated by the platform
DeviceID, DateTime, OS SKU, Make & Model of PC, etc
• How many win 10 PCs do we have
in the world.
Level 2 is system configuration and usage
• Trend in Hardware
• Driver Stability
• App Popularity for App compat
validation
Level 2: Software & Hardware (Dozens of Data
Points)
Network connection Type, Disk Configuration, Activity start/stop,
system Carsh, etc.
Level 3: App Level (Hundreds of Data Points)
#WindowsBistro @RKJohnston
Level 3 is App Level
• Are the apps actually used
• What features of the apps are
popular
• App Hangs, Crashes, and
Abandonment traced to OS features
Level 3: App Level (Hundreds of Data Points)
Used for component specific data
Feature Specific Error Codes, etc.
Sometimes structured for eventing, sometimes free-form for
tracing
App Crash or Hang
High Level Data Platform Diagram
Alterting
DB
Common
encrypts
ntationpoints
Map Reduce Job Manager
Super Fast Data
AlertingCrash
Data
Big Data
Aggregat
e
Streams Quality
Business
Insights
nPowerBIDashboard
owsCommonTelemetryClient-
ybeforesendingitup,instrumen
scannedforPIIviolations
Big Data Cloud
(Cosmos)
Secured to
Data Scientists
Map Reduce Job Manager
(USQL)
Level2&3Usage
Data
A&Big
ata
TelemetryWebFrontEnd
AlgosdroppingPII
#WindowsBistro @RKJohnston
Apps
Insights
Analyze
(SQL)
App Team
Power BI
Dashboard
s
Report,
Visualize &
Collaborate
Instrument
Windo
telemetry
Crash Data has PII
risk so locked down
Store, Transform & CorrelateCollect
Debug&
LogsDa

10. 6/14/2017
8
Kitchen Safety First
#WindowsBistro @RKJohnston
#WindowsBistro @RKJohnston

11. 6/14/2017
9
Must Listen to all your Customers
#WindowsBistro @RKJohnston
Three Sources of Text Data
• Social media carried a
strong signal different than
Social (Twitter +
strong signal different than
SIUF or UIF
• SIUF and UIF were our
engaged users. Very
different population
Forums)
SIUF (System Initiated
User Feedback)
#WindowsBistro @RKJohnston
different population
UIF (User Initiated
Feedback)

12. 6/14/2017
10
Developed the Windows Insider Program
[Jan 21 2015]Since we
launched the Windows Insider
Program in September [2014],
we’ve been joined by 1.7
million Windows Insiders,
who have delivered over
800,000 pieces of feedback.
#WindowsBistro @RKJohnston
Insider are over 4 million now
Insiders Feedback
Tool
#WindowsBistro @RKJohnston
Feedback on
Feedback

13. 6/14/2017
11
SysSieve Architecture
KNOWLEDGE BUILDING PHASE
Repeated
Knowledge Ontology1 2 3
TROUBLE TICKET
REPOSITORY
Phrase
Extraction
Knowledge
Discovery
Ontology
Modeling
Goal: Find frequently
occurring phrases
power supply unit is faulty <power supply unit is faulty>
Goal: Find phrases
important in the
“networking” domain
• ENTITY: power supply unit
Goal: Semantic
interpretation of the
domain-specific phrases
1 2 3
#WindowsBistro @RKJohnston
… power supply unit is faulty…
… access router inoperative…
… run config script …
… is to inform you that there …
<power supply unit is faulty>
<access router inoperative>
<run config script>
• ENTITY: power supply unit
-> STATE: faulty
• ENTITY: access router ->
CONDITION: inoperative
• ENTITY: config script ->
ACTION: run
MS Research ‐ http://research.microsoft.com/en‐us/um/people/navendu/syssieve/
Step – II: Knowledge Discovery
• Goal: Find phrases important in the current
domain to do problem inference
Phrase Important?
• Challenges:
– Filter meaningful phrases from noisy ones
– Expert‐labeling is time‐consuming
• Approach (19M phrases  5 6K phrases)
power disruption on
access router
key corruption due to
expired certificate
bad memory on server
prior communication
#WindowsBistro @RKJohnston22
• Approach: (19M phrases  5.6K phrases)
1. Apply a pipeline of linguistic filters
2. Rank phrases by importance using
information theoretic measures
prior communication
best regards
informing you that

14. 6/14/2017
12
#WindowsBistro @RKJohnston
Taste Test = Flighting
#WindowsBistro @RKJohnston

15. 6/14/2017
13
What does Device OS Flighting mean?
Deploying the latest code, scripts and diagnostics to
a range of audiences. From internal self-hosters to
2. Staged Deployment. Verify the
quality of a new component or change
by sending to a specific sample
di ( i l l
g
external “test pilots”.
#WindowsBistro @RKJohnston
audience (or progressively larger
audiences).
Who do you want to flight to?
Everyone
Selfhost is here
OSG
Microsoft
External
Test Pilots
y
Engineer
Team
#WindowsBistro @RKJohnston

16. 6/14/2017
14
#WindowsBistro @RKJohnston
Reservation System
#WindowsBistro @RKJohnston

17. 6/14/2017
15
The Funnel Patch Ready for Win 10
Update GDR
Reservation Notification
Targeted Devices with High Likelihood of success
>1 billion Eligible Windows 7‐8.1 Devices *
• Exceeded Target of 60 million
i
User
Initiated
Upgrade Starts
Bandwidth
Background Download
reservations
• Flooded Internet on Day 1
• > 80,000,000 upgrades within
weeks of launch
#WindowsBistro @RKJohnston
Initiated
110M
Upgrade Success
• Watch the Funnel every day
• Enterprise Upgrade Blockers
#WindowsBistro @RKJohnston

18. 6/14/2017
16
Net Promotor SATs
Word Cloud Example
The upgrade went quickly and smoothly, the UI is clean and intuitive,
Edge and Continuum are a joy to use, and Cortana and the stock apps
Easy upgrade
are a great! It's a great experience.
Very impressed. Smooth, clean interface. The Nervous Nellies clinging
to Win7 should finally be able to put on big boy pants :)
Fast and clean
The UI is clean and simple to use And it's fast
It installed flawlessly, works out of the box, moves fast and it is a great
experience overall. Great job!
#WindowsBistro @RKJohnston
Great desktop OS again: start menu is back, apps don't fill screen,
virtual desktops. Even better touch‐device OS than Windows 8.1: date,
time, notifications, wifi always visible on start screen. Notifications are
great! Transition between tablet and desktop mode is great!
Start menu
The UI is clean and simple to use. And it s fast.
Net Promotor DSATs
Word Cloud Example
“You made too many changes. I can't find anything. I
am so frustrated and angry that I accepted your
offer. I wish I could go back to the way I had it.”
“I find that it can be too confusing to those who just
want simple when it comes to their electronics.”
#WindowsBistro @RKJohnston
“Have not had enough time with the product yet to
form an opinion.”

19. 6/14/2017
17
#WindowsBistro @RKJohnston
Free Samples as a part of CAC
#WindowsBistro @RKJohnston

20. 6/14/2017
18
ARPU vs. ARPD
Average Revenue Per User
• Revenue Per Month (RPM)
Average Revenue Per Device
• Revenue Per Month (RPM)( )
• Active Users (AU)
• User Churn Rate in Months (Uchurn)
• Customer Acquisition Costs
RPM/AU=ARPU
ARPU*Uchurn=Lifetime Value
( )
• Active Devices (AD )
• User Churn Rate in Months (Uchurn)
• Customer Acquisition Costs
RPM/AD=ARPD
ARPD*Uchurn=Lifetime Value
#WindowsBistro @RKJohnston
ARPU Uchurn=Lifetime Value
LTV – CAC = Gross Profit per User
ARPD Uchurn=Lifetime Value
LTV – CAC = Gross Profit per User
Bing Revenue is >50% Anonymous
Same Store Sales
Without Total $$sWithout Total $$s
#WindowsBistro @RKJohnston

21. 6/14/2017
19
Boosted Trees Classification
• The matched data is ~40% of‐ ‐ ‐
‐ ‐ ‐
the total devices
• 45% of our matched data
have revenues > 0
• We trained a binary classifier
on the matched and predicted
on the unmatched
40%
+ + +
55%
‐ ‐ ‐
‐ ‐ ‐
‐ ‐ ‐
‐ ‐ ‐
~40
~60
%
+ + +
+ + +
+ + +
‐ ‐ ‐
#WindowsBistro @RKJohnston
on the unmatched.
• Cross Validation results: 73%
precision and 74% positive
recall.
60%
45%
+ + +
+ + +
+ + +
%
Big Data Cooking
#WindowsBistro @RKJohnston

22. 6/14/2017
20
#WindowsBistro @RKJohnston
6 Keys to Success
Safe & Private Location Listen to our past
Customers
Taste Test the New
Menu
#WindowsBistro @RKJohnston
Big Grand Opening Listen to Feedback Grow the Business

23. 6/14/2017
21
#WindowsBistro @RKJohnston
NEWS: Microsoft has more Data Scientists…
#WindowsBistro @RKJohnston