Storing sensor data can be notoriously difficult, especially if you don't know what sort of sensors you will be working with before hand. A typical device can have a large number of sensors, with each a different type of data, integer float, string etc. This can lead to complicated relational solutions or even tables filled with nulls. This session will show how using Cassandra's User Defined Types along with Collections a simple table can probably handle most sensor data that you care to throw at it.

1. Storing Sensor data using User Defined Types

2. Andy Cobley
• Senior lecturer, (Professor), University of Dundee
• Program Director of MSc programs in
– Data Science (http://www.dundee.ac.uk/study/pg/data-science/ )
– Data Engineering (http://www.dundee.ac.uk/study/pg/data-engineering/ )
2© 2015. All Rights Reserved.

3. 1 Sensors, Our problem
2 User defined types
3 With collections
4 Reading and Writing Sensor data
3© 2015. All Rights Reserved.

4. Sensors
• We were interested in sampling soil moisture (and other values)
• At scale
• Potentially allows us to calibrate satellites.
– Sentinel-1, SMAP
• Many existing data sets are very old, especially in developing
countries
4© 2015. All Rights Reserved.
http://www.ukeof.org.uk/documents/dagsoiltestingkit.pdf

5. Sensors
• Soil sensors can be very expensive
– Data Loggers can be $100’s
• Can have problems with trailing wires
– Data muling can overcome that.
• We want lots of people to take part (20,000)
5© 2015. All Rights Reserved.

6. Sensors
• So:
– Lets let people make their own
– Raspberry pi and AirPi
– Smart Citizen
– MudPi
6© 2015. All Rights Reserved.

7. Open Source
• Each device sends different data
• Airpi:
– Temperature, humidity, air pressure, light levels, UV levels, carbon
monoxide, nitrogen dioxide and smoke level
• Smart Citizen
– Carbon Monoxide (CO), Nitrogen Oxide (NO2),
Temperature,Humidity,LightSound
7© 2015. All Rights Reserved.

8. Open Source
• Each has it’s own source code, data can be sent in different
data types
– Int, Float, String ?
• So we don’t know what each sensor will send or what type it will
be.
• This was experimental, a kind of what if ?
8© 2015. All Rights Reserved.

9. Relational solutions
• We can create relational tables, but we are likely to end up
dealing with a lot of nulls.
• Decisions we make might go out of date (sensors that send
much more data than expected)
• You can probably find a relational solution, but it won’t be pretty!
9© 2015. All Rights Reserved.

10. Lets use User Defined Types
• Why ?
– Allow us to ignore the nulls
© 2015. All Rights Reserved. 10

11. User Defined Types
11© 2015. All Rights Reserved.
CREATE TYPE if not exists SensorReading (
fValue float,
sValue text,
iValue int,
Accuracy float
);

12. If we have one sensor reading
12© 2015. All Rights Reserved.
Create table if not exists untypedSensor(
name uuid,
insertion_time timestamp,
reading frozen<SensorReading>,
Primary Key (name,insertion_time)
)WITH CLUSTERING ORDER BY (insertion_time DESC);

13. Frozen
• Frozen means that sensor reading can only be overwritten, not
changed a bit at a time
• None frozen UDT’s are coming in 3.0
• A frozen UDT is essentially a blob
13© 2015. All Rights Reserved.

14. With Multiple sensor readings
• Replace the sensor reading with a map :
14© 2015. All Rights Reserved.
Create table if not exists untypedSensor(
name uuid,
insertion_time timestamp,
readings map <text,frozen<SensorReading>>,
Primary Key (name,insertion_time)
)WITH CLUSTERING ORDER BY (insertion_time DESC);

15. Meta data
15© 2015. All Rights Reserved.
Create table if not exists untypedSensor(
name uuid,
insertion_time timestamp,
metadata map <text,text>,
readings map <text,frozen<SensorReading>>,
Primary Key (name,insertion_time)
)WITH CLUSTERING ORDER BY (insertion_time DESC);

16. Writing to our UDT
• Our data will be sent in
JSON format:
• Lets assume we
decode it
16© 2015. All Rights Reserved.
{ "SensorData":
{ "device":"5f388896-71c4-402c-b7c6-8610e305422f",
"insertion_time":"Wed Jan 28 15:54:29 GMT 2015" },
"meta": { "name":"Andy", "latitude":"56.4233156",
"longitude":"-2.9754031" },
"sensors": [
{ "name":"Sensor1", "fValue":"20.1”},
{ "name":"Sensor2", "fValue":"20.2" },
{ "name":"Sensor3", "iValue":"80" },
{ "name":"Sensor4", "sValue":"HIGH!" }
]
}

17. Quick Steps
17© 2015. All Rights Reserved.
UserType SensorReadingType = null;
SensorReadingType =
cluster.getMetadata().getKeyspace("sensorsync").getUserType("SensorReading");
UDTValue SensorReading = SensorReadingType.newValue();
//Get reading from JSON, Use correct statement depending on type
sensorReading.setFloat("fValue", (float) Value);
//or
sr.setInt("iValue", value);
//etc

18. 18© 2015. All Rights Reserved.
Putting the value into a map and saving
Map<String, UDTValue> mp = new HashMap<String, UDTValue>();
mp.put(Name, sr);
Statement statement = QueryBuilder.insertInto("sensorsync", "Sensors")
.value("name", dUuid)
.value("insertion_time", dd)
.value("reading", mp);
getSession().execute(statement);

19. Reading back the values
• Select is nothing special
– Select reading where Blah Blah Blah
• Then just get the map
– row.getMap("reading", String.class, UDTValue.class)
19© 2015. All Rights Reserved.

20. SensorSync Application
• https://github.com/acobley/Sensorsync
– Sensor simulator and Sensor Store
• https://github.com/acobley/SensorWeb
– Displays sensor data on a webpage with D3
• https://github.com/acobley/AirPi
– Python code to read the AirPi and send as Json string
20© 2015. All Rights Reserved.

21. Conclusion
• Storing Sensor data can be a problem for some Databases
• Cassandra makes it easy with User Defined Types
• Maybe even easier with 2.2 using native JSON support
• INSERT INTO users JSON '{"id": "user123", "age": 42, "state":
"TX"}';
– But that’s another story !
© 2015. All Rights Reserved. 21

22. Thank you