Being woken up at 3 am by the pager is never fun but seeing an incident resolve before you’ve even left the bed is maddening. Sleepily the next day you tune the alert for a better night’s sleep yet more untuned alerts sing to you in your sleep. After a few rounds of alert-tuning whack-a-mole you wonder: Could I predict if an incident will resolve itself? This is the story of how a weary engineer used a Cloud ML model with Cloud Functions to reduce pager noise. Recounting some of the challenges faced, we’ll explore training a model with a limited data set & continual training in a serverless environment. We’ll also explore the implications of using a bot as a first responder to a pager.

1. October 1st 2019
@mlfowler_
@Claranet
Reducing Pager Fatigue using a Serverless ML Bot
Mike Fowler - Senior Site Reliability Engineer - Public Cloud Practice
PLACE CUSTOMER LOGO HERE

2. London PostgreSQL Meetup
January 24th 2019
About Me

3. October 1st 2019
@mlfowler_
@Claranet
I Like to Think I Know Data
Source: https://peakcare.wordpress.com/2011/10/05/heads-in-the-sand/https://i.pinimg.com/originals/cb/32/5f/cb325f9c268bf2135125f512d95

4. October 1st 2019
@mlfowler_
@Claranet
Scene: Our Engineer Rests Peacefully
Source: https://peakcare.wordpress.com/2011/10/05/heads-in-the-sand/https://i.pinimg.com/originals/cb/32/5f/cb325f9c268bf2135125f512d95

5. October 1st 2019
@mlfowler_
@Claranet
Scene: Red Alert!
Source: https://peakcare.wordpress.com/2011/10/05/heads-in-the-sand/https://vignette.wikia.nocookie.net/memoryalpha/images/6/6b/RedAlert.jpg/revision/latest?cb=20100117050244&path-prefix=en

6. October 1st 2019
@mlfowler_
@Claranet
Scene: Peace
Source: https://peakcare.wordpress.com/2011/10/05/heads-in-the-sand/
https://www.lakelouiseinn.com/wp-content/uploads/2019/01/LakeLouise2-1.jpg

7. October 1st 2019
@mlfowler_
@Claranet
Identify a Problem to Solve

8. October 1st 2019
@mlfowler_
@Claranet
The Problem
Many PagerDuty incidents resolve before I respond disrupting my
sleep needlessly

9. October 1st 2019
@mlfowler_
@Claranet
First Approach: Tuning
https://i.ytimg.com/vi/5ZfaNpKvg5w/maxresdefault.jpg

10. October 1st 2019
@mlfowler_
@Claranet
Second Approach: Machine Learning
https://irishtechnews.ie/wp-content/uploads/2017/07/chi-inc-robots-doing-more-office-work-bsi-hub-20150617.jpg

11. October 1st 2019
@mlfowler_
@Claranet
The Shape of Data
23
2 timestamps
6 numeric
15 text
features
20678samples

12. October 1st 2019
@mlfowler_
@Claranet
The Shape of Usable Data
5
1 timestamp
1 numeric
3 text
features
19354samples

13. October 1st 2019
@mlfowler_
@Claranet
Losing Hope
https://io9.gizmodo.com/the-exact-moment-when-battlestar-galactica-won-our-hear-1670313315

14. October 1st 2019
@mlfowler_
@Claranet
Feature Engineering
• Worked examples starting from simple
number manipulations to complex
processes such as principal component
analysis (PCA)
• Lots of Python code and decent
explanations
• Primarily scikit-learn
• Decent bibliography per chapter

15. October 1st 2019
@mlfowler_
@Claranet
The Shape of Usable Data
47
2185 positive class
2185 negative class
features
4370samples

16. October 1st 2019
@mlfowler_
@Claranet
Choosing a Model: Random Forest
https://miro.medium.com/max/2612/0*f_qQPFpdofWGLQqc.png

17. October 1st 2019
@mlfowler_
@Claranet
Validating the Model: Cross Validation
https://towardsdatascience.com/cross-validation-explained-evaluating-estimator-performance-e51e5430ff85

18. October 1st 2019
@mlfowler_
@Claranet
The Model
79%
Accuracy of

19. October 1st 2019
@mlfowler_
@Claranet
The Model is only a Keystone
https://io9.gizmodo.com/https://miro.medium.com/max/3036/1*SQg9Buf5w-rR2T8vCIVy3g.jpeg

20. October 1st 2019
@mlfowler_
@Claranet
Going Serverless
https://datacenterfrontier.com/wp-content/uploads/2017/11/equinix-dc12-data-hall.jpg

21. October 1st 2019
@mlfowler_
@Claranet
A Basic Serverless Architecture

22. October 1st 2019
@mlfowler_
@Claranet
Introducing Mr Data

23. October 1st 2019
@mlfowler_
@Claranet
Introducing Mr Data

24. October 1st 2019
@mlfowler_
@Claranet
Introducing Mr Data

25. October 1st 2019
@mlfowler_
@Claranet
Introducing Mr Data

26. October 1st 2019
@mlfowler_
@Claranet
AI Platform
• Hosted Jupyter notebooks
• Distributable training with automatic resource
provisioning
• Supports CPUs, GPUs and TPUs
• Run across many nodes and multiple
experiments
• Automated hyperparameter tuning with
HyperTune
• Exportable models
• Model hosting for online prediction

27. October 1st 2019
@mlfowler_
@Claranet
Exporting a scikit-learn Model
from sklearn.ensemble import RandomForestClassifier
from sklearn.externals import joblib
model = RandomForestClassifier(n_estimators=n)
...
model.predict = model.predict_proba
joblib.dump(model, 'model.joblib')

28. October 1st 2019
@mlfowler_
@Claranet
Deploying a Model: Upload
$ gsutil cp ./model.joblib gs://your-bucket/model.joblib
NB: The directory containing the model must be 250MB or less

29. October 1st 2019
@mlfowler_
@Claranet
Deploying a Model: Create a Model
$ gcloud ai-platform models create mrdata

30. October 1st 2019
@mlfowler_
@Claranet
Deploying a Model: Create a Version
$ gcloud ai-platform versions create v1
--model mrdata
--origin gs://your-bucket/model.joblib
--runtime-version=1.14
--framework SCIKIT_LEARN
--python-version=3.5

31. October 1st 2019
@mlfowler_
@Claranet
Cloud Functions
• Function-as-a-Service supporting:
- Node.js 6, 8 & 10
- Python 3.7.1
- Go 1.11.6
• Triggerable from:
- HTTP
- Cloud Storage
- Pub/Sub
- Cloud Scheduler

32. October 1st 2019
@mlfowler_
@Claranet
A Go Cloud Function
import (
"net/http"
)
func PagerDuty(w http.ResponseWriter, r *http.Request) {
//awesome code
}

33. October 1st 2019
@mlfowler_
@Claranet
Function Deployment Preliminaries
$ gcloud iam service-accounts create mrdata --display-name
"Mr Data's Service Account"
$ gcloud beta projects add-iam-policy-binding myproject
--member
serviceAccount:mrdata@myproject.iam.gserviceaccount.com
--role roles/ml.developer

34. October 1st 2019
@mlfowler_
@Claranet
Function Deployment
$ gcloud beta functions deploy mrdata --entry-point
PagerDuty --runtime go111 --trigger-http --service-account
mrdata@myproject.iam.gserviceaccount.com

35. October 1st 2019
@mlfowler_
@Claranet
Needs Improvement
http://www.bellabeachproperties.com/wp-content/uploads/2014/04/house-falling-apart-1.jpg

36. October 1st 2019
@mlfowler_
@Claranet
Needs Improvement

37. October 1st 2019
@mlfowler_
@Claranet
Improved Serverless Architecture

38. October 1st 2019
@mlfowler_
@Claranet
Cloud Pub/Sub
• Publish/Subscribe messaging service
• At-least-once delivery
• Seek & Replay
- A subscription only sees from after it
was created

39. October 1st 2019
@mlfowler_
@Claranet
Topic & Subscription Creation
$ gcloud pubsub topics create pd-notify
$ gcloud pubsub subscriptions create
--topic pd-notify pd-notify-model
$ gcloud pubsub subscriptions create
--topic pd-notify pd-notify-firestore

40. October 1st 2019
@mlfowler_
@Claranet
Cloud Firestore
• Serverless NoSQL document database
• ACID transactions
• Automatic scaling & indexing
• Multi-region replication
• Client libraries provide live and offline
synchronization

41. October 1st 2019
@mlfowler_
@Claranet
A Simple struct for Recording Inferences
type Prediction struct {
Incident string `firestore:”incident”`
Prediction bool `firestore:”prediction”`
Confidence float64 `filestore:”confidence”`
}

42. October 1st 2019
@mlfowler_
@Claranet
Adding a Document to a Collection
pred := Prediction{
Incident: “PRX7NJU”,
Prediction: true,
Confidence: 0.7677249,
}
_, err := client.Collection(“predictions”)
.Doc(“PRX7NJU”)
.Set(ctx, pred)

43. October 1st 2019
@mlfowler_
@Claranet
Reporting

44. October 1st 2019
@mlfowler_
@Claranet
Towards Continual Training & Deployment

45. October 1st 2019
@mlfowler_
@Claranet
Diagnostics

46. October 1st 2019
@mlfowler_
@Claranet
What Next?
https://nerdist.com/article/star-trek-picard-data-where-he-is-now/

47. October 1st 2019
@mlfowler_
@Claranet
Fin

48. October 1st 2019
@mlfowler_
@Claranet
Mike Fowler mlfowler
Questions ?
gh-mlfowler @mlfowler_
www.mlfowler.com
mike.fowler@claranet.uk