Data Works MD July 2021 - https://www.meetup.com/DataWorks/events/278394107/ Video - https://youtu.be/WXA1yX8O3Lc ------------------------------------------------- Introducing Datawave: Scalable Data Ingest and Query on Apache Accumulo Out of the box, Accumulo's strengths are difficult to appreciate without first building an application that showcases its capabilities to handle massive amounts of data. Unfortunately, building such an application is non-trivial for many would-be users, which affects Accumulo's adoption. In this talk, we introduce Datawave, a complete ingest, query, and analytic framework for Accumulo. Datawave, recently open-sourced by the National Security Agency, capitalizes on Accumulo's capabilities, provides an API for working with structured and unstructured data, and boasts a robust, flexible, and scalable backend. We'll do a deep dive into Datawave's project layout, table structures, and APIs in addition to demonstrating the Datawave quickstart—a tool that makes it incredibly easy to hit the ground running with Accumulo and Datawave without having to develop a complete application. Datawave - https://code.nsa.gov/datawave/ ------------------------------------------------- Hannah Pellón received her B.S. in Mathematics from the University of Maryland while working as a software engineering intern at Northrop Grumman conducting RF signal analysis and spectrometry. She spent 11 years at Northrop Grumman thereafter contributing to IR&D efforts and programs centered around Accumulo and Hadoop. She is currently a software developer and lead at Tiber Technologies focusing on Datawave and distributed computing technologies

1. Introducing
DataWave
Hannah Pellon

2. Agenda
• Prerequisites
• High Level Architecture
• Key Concepts
• Ingest
• Query
• Applications
• FAQ

3. Foundations

4. Prerequisites
• Hadoop (HDFS, YARN, MapReduce)
• Accumulo (General Architecture, Iterators, Authorizations, Shell)
• Zookeeper
• Wildfly

5. What is Datawave?
Storage & Retrieval engine built on Apache Accumulo Providing:
• Tunable ingest workflow
• Table schema
• Query API
• Query parsing, planning, and execution
• Support for a variety of data types and formats

6. Datawave Overview
• Storage & Retrieval engine built on Apache Accumulo
Ingest Query
text
image
html
...
Various structured and unstructured data formats
csv Datawave Datawave

7. warehouse
ingest
Datawave
Ingest
Datawave Architecture
YARN
Wildfly
Datawave
Web
Accumulo
Datawave
Iterators
MapReduce
Zookeeper
Datawave
Flag Maker
Ingress
(HAProxy)
Dataflow
(NiFi)
Many Data Sources / Multiple Hadoop Clusters / Multiple HDFS Instances / Many Web Front-ends
Datawave
Bulk Loader
HDFS
Datawave
Query
Datawave
Ingest Job
Datawave
Tables
text
html
image
...
csv Datawave
REST API

8. Key Concepts

9. Data Model
• Records
• RawRecordContainer, Event, or Document
• A collection of fields and content data
• Fields
• Can be multi valued
• Can be indexed, reverse indexed, excluded, or simply stored
• Special fields: index-only, tokenized, and virtual fields
Record Field
Field
Type
Normalizer
Mytype-config.xml
<property>
<name>mytype.data.category.index</name>
<value>NAME,ID,ID,EXTERNALS_THETVDB,EXTERNALS_TVRAG
E,EXTERNALS_IMDB,EMBEDDED_CAST_CHARACTER_NAME,EMB
EDDED_CAST_PERSON_NAME,EMBEDDED_CAST_PERSON_ID,GE
NRES</value
</property>
<property>
<name>mytype.data.category.index.reverse</name>
<value>NAME,NETWORK_NAME,OFFICIALSITE,URL</value>
</property>
…

10. Normalizers
• Field Types & Normalizers
• Raw field data into index entries using a Normalizer.
• Normalized in index tables, but not in shard
Record Field
Field
Type
Normalizer
LcNoDiacritics
NumberType
…
BaNaNa banana
10 +bE1
27 +bE2.7
100 +cE1
Mytype-config.xml
<property>
<name>mytype.data.default.type.class</name>
<value>datawave.data.type.LcNoDiacriticsType</value>
</property>
<property>
<name>mytype.START.data.field.type.class</name>
<value>datawave.data.type.DateType</value>
</property>
<property>
<name>mytype.WEIGHT.data.field.type.class</name>
<value>datawave.data.type.NumberType</value>
</property>

11. Tables, Indexes & Shards
• Overall Structure
• Data is partitioned into date-based shards
• Each Shard is contained in a single tablet
• Each Shard has its own Field & Term Index, Data & Record Storage
• Certain Field values tracked in Edges
Shard Table
Global Index Table
Shard
Field Index
Record Storage
Data Storage
Term Index
Edge
Table
Shard Shard
Shard
Meta
Data
Table …
…

12. Shard Table
Dates
• LOAD_DATE
• ACTIVITY_DATE
• EVENT_DATE
• …
Mytype-config.xml
<property>
<name>mytype.data.category.date</name>
<value>EVENT_DATE</value>
</property>
YYYYMMDD_id

13. Table Structures

14. Shard
• Field Index – Index of fields within a shard
• Record – All fields and values for an Event
• Term Index- Index of terms within a record
• Document Data – All content for an object
RowId ColumnFamily ColumnQualifier Value
Shard id Datatype0DocUID FieldName0Value
RowId ColumnFamily ColumnQualifier Value
Shard id ‘fi’0FieldName Value0datatype0DocUID
RowId ColumnFamily ColumnQualifier Value
Shard id ‘tf’ Datatype0DocUId0Value0FieldName Protocol Buffer
(Offsets, stats)
RowId ColumnFamily ColumnQualifier Value
Shard id ‘d’ Datatype0DocUId0ContentName Document Contents
(GZIP and base64
encoded)

15. Shard Index and Shard Reverse Index
• shardIndex
• shardReverseIndex
RowId Column Family Column Qualifier Value
Normalized Field Value Normalized Field Name ShardID0DataType Doc Uid.List or
Uid.List.size
RowId Column Family Column Qualifier Value
eulaV dleiF dezilamroN Normalized Field Name ShardID0DataType Doc Uid.List or
Uid.List.size

16. FOOD == ‘Banana’ AND CAR_MAKE =~ ‘.*i’
shardIndex
…
banana FOOD : 20210724_0favorites doc1
banana FOOD : 20210724_1favorites doc2
…
shardReverseIndex
…
inihgrobmal CAR_MAKE : 20210724_0favorites doc1
irarref CAR_MAKE : 20210724_1favorites doc2
ittagub CAR_MAKE : 20210724_2favorites doc3,doc4,doc5
…
Shard Table
20210724_0 favoritesdoc1 : CAR_MAKE.0Lamborghini
favoritesdoc1 : COLOR.0Blue
favoritesdoc1 : FOOD.0Banana
favoritesdoc1 : NAME.0John Smith
20210724_1 favoritesdoc2 : CAR_MAKE.0Ferrari
favoritesdoc2 : COLOR.0Red
favoritesdoc2 : FOOD.0Banana
favoritesdoc2 : NAME.0Jane Doe

17. Edge
Use rowID Column Family Column Qualifier Value
Edge
Event
Count per
Day
EntityA0EntityB edgeType/relationshipA-
relationshipB
yyyyMMdd/edgeAttributes Protocol Buffer containing count
for day as Long and int32 bitmask
for hour of day.
Entity
Activity
per Hour
EntityA STATS/ACTIVITY/edgeType/edge
Relationship
yyyyMMdd/edgeAttributes Protocol Buffer containing count
per hour as Long[24]
• Relationship between field values in a record
• Allows all records with the same field values to be grouped together
• Supports Iterative graph building

19. Edges
…
kevin bacon0emily tremaine TV_COSTARS/PERSON-PERSON : 20210724/TVMAZE_METADATA-TVMAZE_METADATA/tremors/29754/A
kevin bacon0valentine mckee TV_CHARACTERS/PERSON-CHARACTER : 20210724/TVMAZE_METADATA-TVMAZE_METADATA/tremors/29754/A
…
Edge-definitions.xml
<bean id="myjson" class="datawave.ingest.mapreduce.handler.edge.define.EdgeDefinitionConfigurationHelper" scope="prototype">
…
<property name="edges">
<list> <!-- Create bidirectional edges for each pair of costars in a tv show --> <bean
class="datawave.ingest.mapreduce.handler.edge.define.EdgeDefinition">
<property name="edgeType" value="TV_COSTARS"/>
<property name="direction" value="bi"/>
<property name="AllPairs">
<list>
<bean class="datawave.ingest.mapreduce.handler.edge.define.EdgeNode">
<property name="selector" value="EMBEDDED_CAST_PERSON_NAME.EMBEDDED_0.CAST_0.PERSON_0.NAME_0"/>
<property name="relationship" value="PERSON"/>
<property name="collection" value="TVMAZE_METADATA"/>
</bean>
<bean class="datawave.ingest.mapreduce.handler.edge.define.EdgeNode">
<property name="selector" value="EMBEDDED_CAST_PERSON_NAME.EMBEDDED_0.CAST_1.PERSON_0.NAME_0"/>
<property name="relationship" value="PERSON"/>
<property name="collection" value="TVMAZE_METADATA"/>
</bean>
…

20. Datawave Metadata
Use rowID ColumnFamily Column Qualifier Value
Event Metadata
Datatypes this field has been
ingested for
Field name ‘e’ DataType NULL
Index Metadata
Datatypes this field is indexed for
Field name ‘i’ DataType NULL
Reverse Index Metadata
Datatypes this field is reverse
indexed for
Field name ‘ri’ DataType NULL
Type Metadata
class used to normalize field
Field name ‘t’ DataType0DatatypeClassName NULL
Event Column Frequency
Count per field per type per day
Field name ‘f’ DataType0YYYYMMDD (event date) Count
Description Field name ‘desc’ DataType Text description of field
• Tracks field characteristics and frequencies per datatype

21. Datawave Metadata Continued
Use rowID ColumnFamily Column Qualifier Value
Query Model
Mapping
Field name Model Name Field Name0Mapping Direction NULL
Edge metadata edgeType/
relationshipA-
relationshipB
‘edge’ infoA-infoB/edgeAttributes Protocol Buffer containing the list of field
names that contributed to this edge type with
counts

22. Query

23. Datawave Query
• JEXL or Lucene query syntax with functions
• Execution shaped by data characteristics
• Iterators to perform low level functions (e.g: Intersection)
• REST API
Query/Logic/create
Query/id/next
Query/id/close
POST
Parses, validates, optimizes query
Considers field cardinality, index, ...
GET
Returns pages of results
REST endpoints
POST
Frees up Resources

24. Grouping Contexts
{FIELD}.{GROUPING_CONTEXT}
• Example Query: ITEM == ‘banana’ AND COST == 500
Record 1
20210724_0 purchasedoc1 ITEM.0BANANA
20210724_0 purchasedoc1 ITEM.1MATRESS
20210724_0 purchasedoc1 ITEM.2MITERSAW
20210724_0 purchasedoc1 COST.00.69
20210724_0 purchasedoc1 COST.1500
20210724_0 purchasedoc1 COST.2249.99
20210724_0 purchasedoc1 STORE.0NILE
20210724_0 purchasedoc1 TRANSACTION_ID.01234

25. Grouping Contexts
{FIELD}.{GROUPING_CONTEXT}
• Example Query: ITEM == ‘banana’ AND COST == 500
• ITEM == ‘banana’ AND #MATCHES_IN_GROUP(ITEM, ‘banana’, COST, 500)
•
Record 1
20210724_0 purchasedoc1 ITEM.0BANANA
20210724_0 purchasedoc1 ITEM.1MATRESS
20210724_0 purchasedoc1 ITEM.2MITERSAW
20210724_0 purchasedoc1 COST.00.69
20210724_0 purchasedoc1 COST.1500
20210724_0 purchasedoc1 COST.2249.99
20210724_0 purchasedoc1 STORE.0NILE
20210724_0 purchasedoc1 TRANSACTION_ID.01234

26. Query Functions
• MATCHES_IN_GROUP
• INCLUDE(field, regex)
• EXCLUDE(field, regex)
• OCCURRENCE(field, operator, count)
• …
GeoWave
• INTERSECTS_BOUNDING_BOX(field, westLon, eastLon, southLat, southLon)
• INTERSECTS_RADIUS_KM(field, centerLon, centerLat, radiusKm)
• …

27. Query Logics
• EventQuery - record retrieval
• LookupUUID - find records given an ID
• EdgeQuery - find records given edge members and attributes
• DiscoveryQuery - record counts by attribute
• MetricsQuery - find query metrics
Query/Logic/create
Query/next
Query/close

28. Query Models
• Search many similar fields with a single term
FRUIT OR VEGETABLE OR MEAT OR GRAIN OR DAIRY FOOD
ACCOUNT_ID OR ACCOUNT_NUMBER OR TRANSACTION_ID OR …
… OR REPORT_ID OR STATEMENT_NUMBER IDS

29. Applications

30. Applications
 Healthcare
 Symptom correlation
 Outbreak tracking
 Purchase Trends
 Network Security
 …
 Any data set/feed you can translate into FIELD/VALUE pairs
myocarditis0bacterial infection SYMPTOMS / SYMPTOM-DIAGNOSIS : 20210724/UCI-HEART-DISEASE
myocarditis0covid-19 SYMPTOMS / SYMPTOM-DIAGNOSIS : 20210724/COVID-WHO
myocarditis0fever SYMPTOMS / SYMPTOM-SYMPTOM : 20210724/COVID-WHO

31. Thanks!

32. Backups

33. Query
Webserver
QueryExpiration
Bean
QueryExecutor
Bean
CachedResultsBean
QueryLogic
Pushdown
Scheduler
QueryPlanner
Ingress
(HAProxy)
Load
Balancer assigns
a query to a
webserver
Tserver
Query Execution
QueryTransformer
Results
Shard Table
Global Index Table Edge
Table

34. Ingest Classes
InputFormat
RecordReader
EventMapper
DataTypeHandler
IngestHelper
Raw file
Record1
Record2
…
split
{k, RawRecordContainer}
getHelper
getEventFields
process
Mutations/
Bulk Ingest Keys
RRC F/V Multimap

35. EventMapper Anatomy: Raw Records to K,V Pairs