This document discusses considerations for building an Internet of Things (IoT) backend infrastructure. It outlines lessons learned from building multiple versions of a DIY IoT platform project over 4 years. Key recommendations include using open source time series databases instead of building custom solutions; treating devices and browsers equally with API keys for authentication; planning for change by versioning APIs; adopting a schema-less data model; and using standard HTTP with websockets for data flows. While a DIY approach can work depending on needs and skills, the document also acknowledges that outsourcing to vendors may be cheaper.

1. DIY IoT Backend
Voon Siong WONG

2. greenfields

3. greenfields
4 years ago, wanted IoT platform
no clear requirements
grew organically
flaws in hindsight
never in production, never fixed
hobby project

4. revisited the project
made list of good, bad, and how it can be better
development ~2 weeks, ~800 LOC Ruby
runs on 1x t2.small instance
response time average 50ms, 95% < 80ms
learnt from a lot of bad experiences

5. disclaimer
not necessarily advocating DIY
outsourcing to 3rd party vendors can be cheaper
depends on in-house capabilities

6. Let's Get Started

7. item 1: time series
don't DIY time series
v1 had to do it, not many open source
v2 used open source database, threw out a ton of code

8. read/write optimisation is hard
use 1-D sort for index optimisation
usually time as primary index
overwrite or append for same timestamp?
aggregating across multiple tables?
geo-spatial capabilities
eg. measurements in this area at this time
4-D "sort"

9. we tried building one based on
careful planning of Row-key & Column-key
only supported overwrite, no appends
fixed time intervals
aggregation were scheduled tasks, rather than queries
poor performance
HBase

10. unless your job is to build databases, don't
outside of your control if you're using vendor

11. item 2: authentication
treat browsers and iot devices equally as first class citizens
accessible with fixed or variable API keys
optionally with username/password

12. avoid client SSL
cumbersome to generate/manage
impractical to install on browsers
thus requires proxy server
use API Keys
fixed API keys
variable API key, HMAC request and time
deploy with environment variables
interactive logins
username/password
oauth

13. item 3: expect change
plan for it
very relevant when DIY
but use as criteria for judging vendors
many vendors are still young

14. ensure APIs are versioned
using path, headers, request params, etc.
deprecations
devices don't / can't understand warning messages
warn developers directly
return success on dead APIs
don't care about their behaviour any more
avoid old devices's retry logic from hammering it

15. API ownership
APIs can make or break your application
one of strongest reasons to DIY
don't forget that it changes!

16. Item 4: model associations
don't enforce a schema for model relationships
embrace schema-less, because sensors are

17. schema / syntactic
fixed associations, inflexible
need to know "position" in hierarchy / grouping
conflicts when reusing names
schema-less / semantic
uniqueness by credentials and name
hierarchies or grouping stored in text, eg. "state/suburb/street",
arbitrarily delimited

18. item 5: data flows
trickles from device to server
streams from server to advanced clients

19. use regular session-less HTTP for small volumes
for greater volume, use a pub-sub mechanism
, subscribe to desired channels
modern clients use (uni-directional) websockets, resume on
connection failures
older clients fallback to long-polling
firehose server

20. item *: nuts & bolts
measurement units
overwrite vs append
don't require SNI
prefer time since epoch
text over binary

21. IoT Backend: Solved
Problem?

22. IoT for hire
authentication, data aggregation, remote control, etc. for hire
still maturing sector...

23. focus on value
this list apply whether you're DIY backend, or evaluating vendors
not suggesting one or the other, that's up to you
suggest you think about value, rather than technical capability
I can, should I?

24. worth it?
in our case, DIY was worth it
(re)development effort was small
lessons from previous generation
in-house knowledge on how to fix things

25. LiveVu
Realtime energy monitoring

26. Real-time Energy Monitor
Your Energy Use
CONSUMPTION
6.06
kW
COST
$1.51
per hour
HISTORY
kW
6
5.8
6.2
6.4
10:40 10:45
Last 10 mins 

27. SmokeAlert
Air Quality Monitoring

28. 22/09/2015 @ 10:00  
 22-September-2015
 
 
+
-
SmokeAlert Enter a location Legend About Privacy Terms

29. disclaimer (repeat)
not necessarily advocating DIY
outsourcing to 3rd party vendors can be cheaper
depends on in-house capabilities

30. Thanks!