Download as PDF, PPTX
![Representations
>>> object.representations()
[“sql_table”, “postgres+sql”, “sql”, “rows”]
data might have been
cached in a table
we might use PostgreSQL
dialect specific features...
… or fall back to
generic SQL
for all other
operations](https://image.slidesharecdn.com/databrewery2-dataobjects-130605022102-phpapp01/85/Bubbles-Virtual-Data-Objects-17-320.jpg)
![List of Objects
@operation(“sql[]”)
def append(context, objects):
...
@operation(“rows[]”)
def append(context, objects):
...
matches one of common representations
of all objects in the list](https://image.slidesharecdn.com/databrewery2-dataobjects-130605022102-phpapp01/85/Bubbles-Virtual-Data-Objects-24-320.jpg)
![Filtering
■ row filters
filter_by_value, filter_by_set, filter_by_range
■ field_filter(ctx, obj, keep=[], drop=[], rename={})
keep, drop, rename fields
■ sample(ctx, obj, value, mode)
first N, every Nth, random, …](https://image.slidesharecdn.com/databrewery2-dataobjects-130605022102-phpapp01/85/Bubbles-Virtual-Data-Objects-47-320.jpg)
Uploaded byStefan Urbanek
Bubbles – Virtual Data Objects
This document describes Bubbles, a Python framework for data processing and quality probing. Bubbles focuses on representing data objects and defining operations that can be performed on those objects. Key aspects include: - Data objects define the structure and representations of data without enforcing a specific storage format. - Operations can be performed on data objects and are dispatched dynamically based on the objects' representations. - A context stores available operations and handles dispatching. - Stores provide interfaces to load and save objects from formats like SQL, CSV, etc. - Pipelines allow sequencing operations to transform and process objects from source to target stores. - The framework includes common operations for filtering, joining, aggreg
More Related Content
Similar to Bubbles – Virtual Data Objects
Bubbles – Virtual Data Objects
- 1. Bubbles Virtual Data Objects June2013Stefan Urbanek data brewery
- 2. Contents ■ Data Objects ■Operations ■ Context ■ Stores ■ Pipeline
- 3. Brewery 1 Issues ■based on streaming data by records buffering in python lists as python objects ■ stream networks were using threads hard to debug, performance penalty (GIL) ■ no use of native data operations ■ difficult to extend
- 4. About Python framework fordata processing and quality probing v3.3
- 5. Objective focus on theprocess, not data technology
- 6. Data ■ keep datain their original form ■ use native operations if possible ■ performance provided by technology ■ have other options
- 7. for categorical data *you can do numerical too, but there are plenty of other, better tools for that *
- 8.
- 9. data object representsstructured data Data do not have to be in its final form, neither they have to exist. Promise of providing data in the future is just fine. Data are virtual.
- 10. virtual data object fields virtualdata SQL statement iterator id product category amount unit price representations
- 11. Data Object ■ isdefined by fields ■ has one or more representations ■ might be consumable one-time use objects such as streamed data SQL statement iterator
- 12. Fields ■ define structureof data object ■ storage metadata generalized storage type, concrete storage type ■ usage metadata purpose – analytical point of view, missing values, ...
- 13. 100 Atari 1040ST computer 10 400.0 1985 no integer string string integer float integer string typeless nominal nominal discrete measure ordinal flag id product category amount unit price year shipped FieldList storage type name analytical type (purpose) sample metadata
- 14. SQL statement iterator SELECT * FROMproducts WHERE price < 100 engine.execute(statement) Representations SQL statement that can be composed actual rows fetched from database
- 15. Representations ■ represent actualdata in some way SQL statement, CSV file, API query, iterator, ... ■ decided on runtime list might be dynamic, based on metadata, availability, … ■ used for data object operations filtering, composition, transformation, …
- 16. Representations SQL statement iterator natural, mostefficient for operations default, all-purpose, might be very expensive
- 17. Representations >>> object.representations() [“sql_table”, “postgres+sql”,“sql”, “rows”] data might have been cached in a table we might use PostgreSQL dialect specific features... … or fall back to generic SQL for all other operations
- 18. Data Object Role ■source: provides data various source representations such as rows() ■ target: consumes data append(row), append_from(object), ... target.append_from(source) for row in source.rows(): print(row) implementation might depend on source
- 19. Append From ... IteratorSQL target.append_from(source) for row in source.rows(): INSERT INTO target (...) SQLSQL INSERT INTO target SELECT … FROM source same engine
- 20.
- 21. Operation ✽… ? ... …? ...… ? ... … ? ... does something useful with data object and produces another data object or something else, also useful
- 22. Signature @operation(“sql”) def sample(context, object,limit): ... signature accepted representation SQL ✽ … ? ... iterator SQL
- 23. @operation @operation(“sql”) def sample(context, object,limit): ... @operation(“sql”, “sql”) def new_rows(context, target, source): ... @operation(“sql”, “rows”, name=“new_rows”) def new_rows_iter(context, target, source): ... unary binary binary with same name but different signature:
- 24. List of Objects @operation(“sql[]”) defappend(context, objects): ... @operation(“rows[]”) def append(context, objects): ... matches one of common representations of all objects in the list
- 25. Any / Default @operation(“*”) defdo_something(context, object): ... default operation – if no signature matches
- 26.
- 27.
- 28. Operation Call context =Context() context.operation(“sample”)(source, 10) sample sample iterator ⇢SQL ⇢ iterator SQL callable reference runtime dispatch sample SQL ⇢
- 29.
- 30. Dispatch SQL ✽iterator SQL iterator ✽iterator MongoDB operation is chosenbased on signature Example: we do not have this kind of operation for MongoDB, so we use default iterator instead
- 31. Dispatch dynamic dispatch ofoperations based on representations of argument objects
- 32. Priority SQL ✽iterator SQL iterator ✽SQL iterator order of representationsmatters might be decided during runtime same representations, different order
- 33. Incapable? SQL SQL join details A A SQL SQL join details A B SQL joindetails SQL same connection different connection use this fails
- 34. Retry! SQL SQL A B iterator iteratorSQL join details joindetails SQL retry another signature raise RetryOperation(“rows”, “rows”) if objects are not compose-able as expected, operation might gently fail and request a retry with another signature:
- 35. Retry when... ■ notable to compose objects because of different connections or other reasons ■ not able to use representation as expected ■ any other reason
- 36. Modules *just an example collectionof operations SQL Iterator MongoDB SQL iterator iterator SQL iterator ✽ ✂ ⧗ ✽ ⧗ Mongo ✽
- 37. Extend Context context.add_operations_from(obj) any objectthat has operations as attributes, such as module
- 38.
- 39. Object Store ■ containsobjects tables, files, collections, ... ■ objects are named get_object(name) ■ might create objects create(name, replace, ...)
- 40. Object Store store =open_store(“sql”, “postgres://localhost/data”) store factory Factories: sql, csv (directory), memory, ...
- 41. Stores and Objects source= open_store(“sql”, “postgres://localhost/data”) target = open_store(“csv”, “./data/”) source_obj = source.get_object(“products”) target_obj = target.create(“products”, fields=source_obj.fields) for row in source_obj.rows(): target_obj.append(row) target_obj.flush() copy data from SQL table to CSV
- 42.
- 43. Pipeline SQLSQL SQL SQL Iterator sequenceof operations on “trunk”
- 44. Pipeline Operations stores ={ “source”: open_store(“sql”, “postgres://localhost/data”) ”target” = open_store(“csv”, “./data/”) } p = Pipeline(stores=stores) p.source(“source”, “products”) p.distinct(“color”) p.create(“target”, “product_colors”) operations – first argument is result from previous step extract product colors to CSV
- 45. Pipeline p.source(store, object_name, ...) store.get_object(...) p.create(store,object_name, ...) store.create(...) store.append_from(...)
- 46.
- 47. Filtering ■ row filters filter_by_value,filter_by_set, filter_by_range ■ field_filter(ctx, obj, keep=[], drop=[], rename={}) keep, drop, rename fields ■ sample(ctx, obj, value, mode) first N, every Nth, random, …
- 48. Uniqueness ■ distinct(ctx, obj,key) distinct values for key ■ distinct_rows(ctx, obj, key) distinct whole rows (first occurence of a row) for key ■ count_duplicates(ctx, obj, key) count number of duplicates for key
- 49. Master-detail ■ join_detail(ctx, master,detail, master_key, detail_key) Joins detail table, such as a dimension, on a specified key. Detail key field will be dropped from the result. Note: other join-based operations will be implemented later, as they need some usability decisions to be made
- 50. Dimension Loading ■ added_keys(ctx,dim, source, dim_key, source_key) which keys in the source are new? ■ added_rows(ctx, dim, source, dim_key, source_key) which rows in the source are new? ■ changed_rows(ctx, target, source, dim_key, source_key, fields, version_field) which rows in the source have changed?
- 51.
- 52.
- 53. To Do ■ consolidaterepresentations API ■ define basic set of operations ■ temporaries and garbage collection ■ sequence objects for surrogate keys
- 54. Version 0.2 ■ processinggraph connected nodes, like in Brewery ■ more basic backends at least Mongo ■ bubbles command line tool already in progress
- 55. Future ■ separate operationdispatcher will allow custom dispatch policies
- 56.
- 57.