2. You can talk to me about:
● Making on-call better for humans
● High Availability and Load Balancing
● Mirroring free software
● Zero trust networking
● Home Automation and garage racks
● Pretty much anything
James Forman

3. I am a:
Linux Sysadmin, Network Engineer and People Manager
I work for:

4. https://en.wikipedia.org/wiki/File:Wellington_montage_2.jpghttps://en.wikipedia.org/wiki/File:New_Zealand_relief_map.jpg

5. Overlay by:
Wikipedia User Hazhk

6. Catalyst’s Wellington Team - December 2017

7. OSMC.de 2019

8. First things first

9. Content Warning
Photos of earthquake damage

10. Hot Potato is not a
monitoring system

11. It’s a message broker

12. Life before Hot Potato

13. One pager team
The “pager peeps”

14. Every person has a pager
One number, multiple pagers

15. A range of different
monitoring builds
Customer managed, remote, out of country, in
country with pager access, email to pager gateways

16. 18 Monitoring Servers
Nagios 3, Icinga 1.x and Icinga2

17. Support Hotline
Call number, leave voicemail, wake person up

18. IRC based handovers
<jforman> Pager on
<redacted> Going to sleep

19. Why did we build it?

20. We already wanted a
replacement system

21. Aging technology
The pager system was becoming unreliable

25. Image credit: Matthew Inman / theoatmeal.com

28. The top 3 questions

29. 1. Why not use a service?

30. SLAs
None of the options could meet our requirements

31. 2. Why not use SMS?

33. 3. Why not go staffed 24/7?

35. Our other motivations

36. Open Source

37. Customer data
Stop sending notifications in cleartext

38. https://techcrunch.com/2019/10/30/nhs-pagers-medical-health-data/

39. We thought we had time
to find a replacement

40. https://www.spark.co.nz/content/dam/kb/public/docs/media-release-paging-network-closure.pdf

41. We thought we had time
to find a replacement

42. At first it was good news

43. Aging technology

45. It was too good to be true

46. The pager network
became unreliable

48. “In response to Radio New Zealand
queries Spark said it had talked to many
of its customers before the
announcement was made and that
included the Fire Service.”

50. Time ran out
(in the middle of the night)

51. NO CARRIER
:(

53. “1st cab off the rank was those pager numbers
that had not signed up to the new pager
network were disconnected.”

54. “We have then worked with the customers who
have migrated across to replace their old
access points (ways they send a pager
message) to either Email or an API option.”

55. “This is because the old access points are
being turned off.”

56. Photo by:
BRENDON O'HAGAN/FAIRFAX NZ

58. Solving the
immediate problem
so people could sleep

59. eMail -> SMS
We sent all notifications via SMS
as an emergency measure

60. eMail == :(
Nameless project == :)

61. The first version of
Hot Potato

62. A really bad “API”
The worst thing I’ve put into production

63. A dodgy script
Rolled out to all the monitoring servers

64. Insert and Send
Add to database, send pager message

65. select * from notifications
A table of notifications

66. A handover button
sends a message saying you have the pager

67. v0.1 - Much more reliable than email

68. It worked (mostly)

69. It gave us the time and
opportunity to do better

70. We had some goals

71. Don’t get in the way
make it easy to be on-call

73. Enable alert reduction
let people sleep

75. Survive natural hazards
the reality of building systems in NZ

76. Volcanoes

77. https://www.nationalgeographic.org/news/plate-tectonics-ring-fire/

79. Earthquakes

80. Recent fatal earthquakes
22 February 2011 - Christchurch - 185 people
13 June 2011 - Christchurch - 1 person
14 November 2016 - Kaikoura - 2 people

81. Diagrams by:
Wikipedia User Mikenorton

82. Photo by:
New Zealand Defence Force

83. Photo by:
New Zealand Defence Force

84. Photo by:
RNZ / Rebekah Parsons-King

85. Photo of:
MP Stuart Smith

86. Photo by:
RNZ / Simon Morton

87. Photo by:
RNZ / Conan Young

88. Photo by:
RNZ / Aaron Smale

89. Photo by:
Phillip Pearson

90. Tsunamis

91. https://wremo.nz/hazards/tsunami-zones/

92. https://wremo.nz/hazards/tsunami-zones/

93. https://wremo.nz/hazards/tsunami-zones/

94. https://wremo.nz/hazards/tsunami-zones/

95. Survive any loss of
International Connectivity
we had 1 main undersea cable (2 landings)

96. Image credit: Tourism New Zealand

97. https://www.submarinecablemap.com/

98. https://www.submarinecablemap.com/

99. https://www.submarinecablemap.com/

100. Then we had some
requirements

101. Survive disasters
Earthquakes, tsunamis, volcanoes, team lunches..

102. Support existing
monitoring
Nagios3, Icinga 1.x, Icinga2

103. Get rid of email
No more using email to deliver messages

104. Confirm message delivery
Move from paging and SMS to Push Notifications

105. Improve handover
Is your pager on yet? I want to go to sleep

106. #deathTo Pagers
“I’d rather have a bee burrow into my
skull than carry a pager again”
- Me

108. What did we build?

109. A web app with an API
built with Python and Flask

110. With a funky database
and some queuing
CockroachDB and RabbitMQ

111. Our production
environment has 5 nodes
NZ: Porirua, Wellington and Hamilton
AU: Sydney
US: California

113. How does it work?

114. Sending notifications

118. Heartbeats

120. How does it look?

124. What else can it do?

125. Failure notifications
the pager network is down again!

126. Heartbeats
ensuring connectivity

127. Teams
put everyone on-call!

128. Team escalations
because sometimes bad things happen

129. Reports
A breakdown of the week that was

130. Promote alert reduction
With the help of some neopixels

132. What notification
providers does it support?

133. Twilio
For delivery of SMS messages

134. Modica
For delivery of SMS messages and pager messages

135. Pushover
For delivery of push notifications to Android and iOS

138. What’s planned?

139. Mobile app
for Android and iOS, no more pagers

140. Support hotline
direct calls to the on-call person or take messages

141. Planned work
stop forgetting to extend downtime on things

142. Language support
German and Italian coming soon

143. What do I need to try it?

144. What do I need to deploy
it to production?

145. One server
If you don’t want redundancy,
you don’t have to have it

146. Demo?

147. James Forman
Callum Dickinson
Filip Vujičić
Zac Pullar-Strecker
Opal Symes
Rhys Davies
Michael Fincham
Tim Bruce
Jamie McClymont
Toni Gardener
Manuela Spies
Sapir Ben-Shahar
Brynn Wilde
Hemanth Sonthi
Emanuel Evans
Hazel Meehan
Baxter Gray
Sam Banks
Thank you to our contributors

148. Open Source Academy

149. https://hotpotato.nz

151. Questions?

152. https://hotpotato.nz
@teamHotPotato
#hotpotato on freenode