It is often unclear how to store data in Accumulo because of its focus on storing raw binary data. While it may seem trivial to store simple values, it can difficult to determine how to store structured data like nested records or values of different types (e.g. boolean, integer, date, string). One way to alleviate this is to formulate a consistent manner for storing typed data and organizing them into structured records. When data is stored in this manner, it becomes possible to extract the schema from the data rather than predefining a schema and attempting to make the data fit. The data contained in these typed records can then be efficiently analyzed to create a Multi-Typed Schema representing the data types they contain. Further, a Multi-Typed Schema can then be reduced to a Single-Typed schema and used in a more traditional manner, such as Data Frames in Spark SQL. – Speaker – Chad Hardin Dir. Engineering, Koverse Chad is a Computer Science Doctoral Candidate at Nova Southeastern University and is finishing up his dissertation on the multi-agent patrol problem. He has a master's in Computer Science and a B.S. in Information Technology. He is currently the Director of Engineering at Koverse, where he has been working for 5 years. — More Information — For more information see http://www.accumulosummit.com/

1. Chad Hardin
Data-Defined Typed Schema Generation in Accumulo
Accumulo Summit 2017

2. ● Store semi-structured data in Accumulo
● Not require any type of schema to be defined before storing it.
● Query or otherwise process that data with an existing schema-oriented
framework like Spark SQL.
● Discover the schema instead of create it.
©Koverse 2
Motivation
What do we want to do?

3. ©Koverse 3
The Problem
A lack of:
• Methodologies
• Algorithms
• Experience
What keeps us from doing this?

4. ©Koverse 4
Approach
● Represent all data as records.
● Store records in a data set.
● Determine the schema of each record.
● Combine the schema of all records into a data set schema.
● Apply the data set schema to all records.

5. ©Koverse 5
What is a Record?
Think of it like JSON. They have named fields with different types of values like
numbers, strings, and booleans. Consider these records:
{
“name” : “Samantha”,
“age” : 13,
“homeTown” : “Seattle”,
“favoriteColor” : “blue”
}
{
“name” : “Jackson”,
“age” : 2,
“favoriteColor” : false
}
R R

6. ©Koverse 6
What is a Data Set?
A Data Set is filled with such records and has a name:
R
R
R
RR
R
R
R
“Stuff”

7. ©Koverse 7
What is a Record Schema?
{
“name” : “string”,
“age” : “integer”,
“homeTown” : “string”,
“favoriteColor” : “string”
}
{
“name” : “Jackson”,
“age” : 2,
“favoriteColor” : false
}
{
“name” : “Samantha”,
“age” : 13,
“homeTown” : “Seattle”,
“favoriteColor” : “blue”
}
{
“name” : “string”,
“age” : “integer”,
“favoriteColor” : “boolean”
}
R
R
S
S

8. ©Koverse 8
The schemas?
A Data Set has as many schemas as it does records. So what is the schema of
the Data Set?
S
S
S
S
S
SS
S
S S ?R
R
R
RR
R
R
R

9. ©Koverse 9
Merging Record Schemas {
“name” : [
“string”
],
“age” : [
“integer”
],
“homeTown” : [
“string”
],
“favoriteColor” : [
“string”,
“boolean”
]
}
{
“name” : “string”,
“age” : “integer”,
“homeTown” : “string”,
“favoriteColor” : “string”
}
{
“name” : “string”,
“age” : “integer”,
“favoriteColor” : “boolean”
}
S
S
S

10. When records have different types for the same field, we have to decide what the
ultimate type will be. The string type is the most general.
©Koverse 10
Merging field types
How to handle conflicts of data types?
string boolean integer double
string string string string string
boolean string boolean string string
integer string string integer double
double string string double double

11. ©Koverse 11
Collapsing the Schema
{
“name” : [
“string”
],
“age” : [
“integer”
],
“homeTown” : [
“string”
],
“favoriteColor” : [
“string”,
“boolean”
]
}
{
“name” : “string”,
“age” : “integer”,
“homeTown” : “string”
“favoriteColor” : “string”
}
S
S

12. ©Koverse 12
Applying the Collapsed Schema
{
“name” : “Jackson”,
“age” : 2,
“homeTown” : null,
“favoriteColor” : “false”
}
{
“name” : “Jackson”,
“age” : 2,
“favoriteColor” : false
}
{
“name” : “string”,
“age” : “integer”,
“homeTown” : “string”
“favoriteColor” : “string”
}
S
R
R

13. ● Key
– Row ID: Data Set Identifier + Record Identifier
– Column Family: Field Name
– Column Qualifier: N/A
– Visibility: Whatever you need
– Timestamp: Write time
● Value (byte array)
– 1st byte: Field type
– Remaining bytes: Field value
©Koverse 13
Record Implementation in Accumulo
Many possibilities. A basic idea for storing records in a table...

14. ● Key
– Row ID: Data Set Id
– Column Family: Field Name
– Column Qualifier: N/A
– Visibility: Whatever you need
– Timestamp: Write time
● Value: Field Type
©Koverse 14
Schema Implementation in Accumulo
Write record schemas to a different table, use a Combiner Iterator (next slide)

15. Combine the Accumulo field value using this table.
©Koverse 15
Schema Combiner Iterator
Create a custom Combiner Iterator to reduce the schema types.
string boolean integer double
string string string string string
boolean string boolean string string
integer string string integer double
double string string double double

16. 1. Read the records of a data set into an RDD (from Accumulo)
2. Read the data set schema (from Accumulo)
3. Convert that schema into a Spark Data Frame Schema
4. Map the RDD so that every record conforms to the schema
5. Create a Spark Data Frame using the schema and the RDD
6. You can now use SQL queries for your schema-less records.
©Koverse 16
Spark SQL
How to make it work?

17. Use Spark SQL hooks:
● Column Filtering (Pruned Scans)
● Push-Down Predicates (Pruned Filtered Scans)
● Use a Spark SQL Data Set (same name but different)
● Use Spark SQL Catalogs (like a real SQL database!)
● Use Spark SQL Data Streams
©Koverse 17
Spark SQL
Future Improvements

18. Questions?
©Koverse 18
Thank You!
http://www.koverse.com/