Bubbles – Virtual Data Objects

Bubbles
Virtual Data Objects
June 2013Stefan Urbanek
data brewery

Contents
■ Data Objects
■ Operations
■ Context
■ Stores
■ Pipeline

Brewery 1 Issues
■ based on streaming data by records
buﬀering in python lists as python objects
■ stream networks were us...

About
Python framework for data
processing and quality probing
v3.3

Objective
focus on the process,
not data technology

Data
■ keep data in their original form
■ use native operations if possible
■ performance provided by technology
■ have ot...

for categorical data
* you can do numerical too, but there are
plenty of other, better tools for that
*

Data Objects

data object represents structured data
Data do not have to be in its final form,
neither they have to exist. Promise of
pr...

virtual data object
fields
virtual data
SQL statement
iterator
id
product
category
amount
unit price
representations

Data Object
■ is deﬁned by ﬁelds
■ has one or more representations
■ might be consumable
one-time use objects such as stre...

Fields
■ deﬁne structure of data object
■ storage metadata
generalized storage type, concrete storage type
■ usage metadat...

100
Atari 1040ST
computer
10
400.0
1985
no
integer
string
string
integer
ﬂoat
integer
string
typeless
nominal
nominal
disc...

SQL statement
iterator
SELECT *
FROM products
WHERE price < 100
engine.execute(statement)
Representations
SQL statement th...

Representations
■ represent actual data in some way
SQL statement, CSV ﬁle, API query, iterator, ...
■ decided on runtime
...

Representations
SQL statement
iterator
natural, most eﬃcient
for operations
default, all-purpose,
might be very expensive

Representations
>>> object.representations()
[“sql_table”, “postgres+sql”, “sql”, “rows”]
data might have been
cached in a...

Data Object Role
■ source: provides data
various source representations such as rows()
■ target: consumes data
append(row)...

Append From ...
Iterator SQL
target.append_from(source)
for row in source.rows():
INSERT INTO target (...)
SQLSQL
INSERT I...

Operations

Operation
✽… ? ...
… ? ...… ? ...
… ? ...
does something useful with data object and
produces another data object
or somet...

Signature
@operation(“sql”)
def sample(context, object, limit):
...
signature
accepted representation
SQL
✽ … ? ...
iterat...

@operation
@operation(“sql”)
def sample(context, object, limit):
...
@operation(“sql”, “sql”)
def new_rows(context, target...

List of Objects
@operation(“sql[]”)
def append(context, objects):
...
@operation(“rows[]”)
def append(context, objects):
....

Any / Default
@operation(“*”)
def do_something(context, object):
...
default operation – if no signature matches

Context

Context
SQL iterator
iterator
SQL iterator
✽
✂
⧗
✽
⧗
Mongo
✽
collection of operations

Operation Call
context = Context()
context.operation(“sample”)(source, 10)
sample sample
iterator ⇢SQL ⇢
iterator
SQL
call...

Simplified Call
context.operation(“sample”)(source, 10)
context.o.sample(source, 10)

Dispatch
SQL
✽iterator
SQL
iterator
✽iterator
MongoDB
operation is chosen based on signature
Example: we do not have this ...

Dispatch
dynamic dispatch of operations based on
representations of argument objects

Priority
SQL
✽iterator
SQL
iterator
✽SQL
iterator
order of representations matters
might be decided during runtime
same re...

Incapable?

SQL
SQL
join details
A
A

SQL
SQL
join details
A
B
SQL
join details
SQL

same connection diﬀerent connectio...

Retry!

SQL
SQL
A
B
iterator
iteratorSQL
join details join details

SQL

retry another
signature
raise RetryOperation(“...

Retry when...
■ not able to compose objects
because of diﬀerent connections or other reasons
■ not able to use representat...

Modules
*just an example
collection of operations
SQL Iterator MongoDB
SQL iterator
iterator
SQL iterator
✽
✂
⧗
✽
⧗
Mongo
✽

Extend Context
context.add_operations_from(obj)
any object that has operations as
attributes, such as module

Stores

Object Store
■ contains objects
tables, ﬁles, collections, ...
■ objects are named
get_object(name)
■ might create objects...

Object Store
store = open_store(“sql”, “postgres://localhost/data”)
store factory
Factories: sql, csv (directory), memory,...

Stores and Objects
source = open_store(“sql”, “postgres://localhost/data”)
target = open_store(“csv”, “./data/”)
source_o...

Pipeline

Pipeline
SQLSQL SQL SQL
Iterator
sequence of operations on “trunk”

$Pipeline Operations stores = { “source”: open_store(“sql”, “postgres://localhost/data”) ”target” = open_store(“csv”, “./da...$

Pipeline
p.source(store, object_name, ...)
store.get_object(...)
p.create(store, object_name, ...)
store.create(...)
store...

Operation Library

$Filtering ■ row filters filter_by_value, filter_by_set, filter_by_range ■ field_filter(ctx, obj, keep=[], drop=[], rename={}) ke...$

Uniqueness
■ distinct(ctx, obj, key)
distinct values for key
■ distinct_rows(ctx, obj, key)
distinct whole rows (ﬁrst occu...

Master-detail
■ join_detail(ctx, master, detail, master_key, detail_key)
Joins detail table, such as a dimension, on a spe...

Dimension Loading
■ added_keys(ctx, dim, source, dim_key, source_key)
which keys in the source are new?
■ added_rows(ctx, ...

more to come…

Conclusion

To Do
■ consolidate representations API
■ deﬁne basic set of operations
■ temporaries and garbage collection
■ sequence ob...

Version 0.2
■ processing graph
connected nodes, like in Brewery
■ more basic backends
at least Mongo
■ bubbles command lin...

Future
■ separate operation dispatcher
will allow custom dispatch policies

Contact:
@Stiivi
stefan.urbanek@gmail.com

databrewery.org