The document discusses the importance of malleable, observable, and engaged systems of experimentation in achieving effective learning and innovation in technology. It emphasizes the need for experiments to be cost-effective, well-instrumented, and conducted at significant scale to yield meaningful results. Additionally, it highlights the growing accessibility of data tools and the relevance of big data in various applications, including customer acquisition and product improvement.

1. Dynamics Day 2015
Systems of Intelligence in Action

2. Experimentation
!=
Learning From
Mistakes

3. 1
MALLEABLE: It must be cost effective to change the
way that the system or process behaves. Ideally for
just a subset of usage or users.
WHAT WE NEED
2
3
OBSERVABLE: The system or process must be
sufficiently well instrumented such that we have the
information available to test the hypothesis.
ENGAGED: The system or process must be being used
at sufficient scale and in a sufficiently ‘real’
environment for any results to be significant.
An experiment is a procedure carried out to verify, refute,
or establish the validity of a hypothesis… Experiments vary
greatly in goal and scale, but always rely on repeatable
procedure and logical analysis of the results.
SYSTEMS OF EXPERIMENTATION
MALLEABLE
OBSERVABLEENGAGED
Experiments

4. MALLEABLE
OBSERVABLEENGAGED

5. “arguments about whether or not a
feature idea is worth doing or not
generally get resolved by just spending a
week implementing it and then testing it
on a sample of users, e.g., 1% of Nevada
users.”

7. There are over 500 different K-Cups

8. There Are Cartridge Loaded Software Systems

9. Microsoft Azure Demand Center
Internal Systems & Product/Platforms
Dynamics CRM
Marketo
Web Email Call Center
Data
Warehouse
PowerBI
Systems
Of
Record
Systems
Of
Experimentation
Systems
Of
Engagement

10. MALLEABLE
OBSERVABLEENGAGED

12. Storage is Cheap…
…Data Is Not

13. Smart Appliances (Think Printers + Ink)
Existing ‘Dumb’ Product
SensorTag
IoT Suite
Smart Screen Mobile Apps(s)
Raw Data Store
???
Personalization
Platform
PowerBI
Systems
Of
Experimentation
Systems
Of
Engagement

14. MALLEABLE
OBSERVABLEENGAGED

15. “…Our tool works best when you’ve got at
least 1 million or so app installs to work
with…”

16. Towards Systems
of Intelligence

17. Flipping The Model
Inputs OutputsIT System
Expertise
InterpretationExperimentation

18. Flipping The Model
Inputs OutputsIT System
ExplanationExperimentation
Automation
Models
Machine Learning

19. Example:
Building Systems
Provider (~HVAC)

20. Much of the Tooling
is Free

21. Is Becoming More
Widely Available

22. Is Becoming Easier to Use

23. And Is Entering The Public Consciousness

24. And can be crowd-sourced

25. In Boots & All
Use Cases We Are Seeing

26. Big Data Use Cases, BARC Research 2015

27. STORE
OPTIMIZATION
TRACKING,
TARGETING
SEGMENT OF 1
SMART
PRODUCTS
SMART
ASSETS
PRODUCT
IMPROVEMENT
CUSTOMER
ACQUISITION &
RETENTION
USER ANALYTICS
&
OPTIMIZATiON
SERVICE/B2B RETAIL SUPPLY CHAIN
FRAUD
&
COMPLIANCE

28. • Embrace Experimentation
• Malleable, Agile, Cartridges
• Observable; Capture More Data
• Engaged; Ship Early & Learn
• Systems of Intelligence
• Doing Data Science <- Easier
• Knowing What We Can Do <- Hard
• Analytics Inside Your Organization
• Survey for Opportunities
• Most CRM Systems
• Customer Facing Apps/Sites
• IoT