Here are a few ways SciQL could help with this seismology use case: 1. The mseed array allows storing and querying the large seismic data in an efficient columnar format. 2. Window-based aggregation with dimensional grouping enables filtering signals by station/LTA ratios over time windows. 3. Views and queries on dimensional groups facilitate removing false positives by comparing signals across nearby stations over time. 4. Further window-based grouping and UDFs can extract signal windows for additional heuristic analysis. By integrating the array and relational models, SciQL provides a declarative way to analyze large multidimensional scientific datasets like seismic signals interactively.

1. Finding something different:
Arrays in database systems,
the next frontier ?
Martin Kersten
CWI
© M Kersten 2012

2. Science applications
© M Kersten 2012

3. Public database of 4-40 TB
Relational schema of around 200 pages SQL
Relational tables up to 20B elements
Finding closely related sky objects
446
columns
>585
million
rows
6
columns
>
20
Billion
© M Kersten 2012 rows

4. The LOFAR radio telescope
Complex image processing pipeline (Blue-gene )
Transient Sky Objects database (50TB/yr)
Finding transients within 4 seconds timeframe
© M Kersten 2012

5. Datawarehouse of seismic data
Highly compressed file repository
(>3.5M files and 15- 150 TB)
About to explode due to sensor network
Finding warning signals
© M Kersten 2012

6. Remote sensing
Processing pipeline to interpret images < 1TB/ yr
Finding and detecting forest fires
© M Kersten 2012

7. Matlab
RDBMS Python
SQL C R
*-API
SciQL
Interdependent
Software libaries
FITS, mSEED,
geoTIFF,…
HDF5,
NETCFD
Datavault
© M Kersten 2012

8. Agenda
Array support in database systems
SciQL array query language
A crash course on column-stores
SciQL implementation approach
© M Kersten 2012

9. What is an array?
An array is a systematic arrangement of objects
addressed by dimension values.
Get(A, X, Y,…) => Value
Set(A, X, Y,…) <= Value
There are many species:
vector, bit array, dynamic array, parallel array,
sparse array, variable length array, jagged array
© M Kersten 2012

10. Who needs them anyway ?
Seismology – partial time-series
Climate simulation – temporal ordered grid
Astronomy – temporal ordered images
Remote sensing – image processing
Social networks – graph algorithms
Genomics – ordered strings
Forensics – images, strings, graphs
Scientists ‘love them’ : MSEED, NETCDF, FITS,
CSV,..
© M Kersten 2012

11. Arrays in DBMS
Relational prototype built on arrays, Peterlee IS
Vehicle(1975)
Persistent programming languages, Astral (1980), Plain
(1980)
Object-orientation and persistent languages were the
make belief to handle them, O2(1992)
Several array algebras AML(2002), Aquery(2003), RAM
(2004), SRAM(2012)
© M Kersten 2012

12. Array declarations:
CREATE TABLE sal_emp ( name text, pay_by_quarter integer[], schedule text[][]);
CREATE TABLE tictactoe ( squares integer[3][3] );
Array operations: denotation ([]), contains (@>), is
contained in (<@), append, concat (||),
dimension, lower, upper, prepend, to-string, from-
string, …
Array constraints: none, no enforcement of
dimensions.
© M Kersten 2012

13. SQL 2003
Arrays are attribute type constructors
Arrays can be declared without a maximum cardinality
Array nesting is unrestricted.
Query results can be converted into arrays.
CREATE TABLE listbox( choices CHAR(3) ARRAY[1000] NOT NULL);
INSERT INTO listbox_choices
VALUES( 'Department Names',
ARRAY(SELECT name FROM sales.depts ORDER BY 1));
© M Kersten 2012

14. Breaks large C++ arrays (rasters) into disjoint chunks
Maps chunks into large binary objects (blob)
Provide function interface to access them
RASCAL, a SQL92 extension
Known to work up to 12 TBs.
© M Kersten 2012

15. Breaks large C++ arrays (rasters) into overlapping
chunks
Built storage manager from scratch
Map-reduce processing model
Provide function interface to access them
AQL, a crippled SQL92
© M Kersten 2012

16. What is the problem?
- Appropriate array denotations? Query language
- Functional complete operation set ?
- Mature implementations? Systems
- Size limitations due to (blob) representations ?
- Scale out?
- Community awareness? Education
© M Kersten 2012

17. Agenda
Array support in database systems
SciQL array query language
A crash course on column-stores
SciQL implementation approach
© M Kersten 2012

18. MonetDB SciQL
SciQL (pronounced ‘cycle’ )
• A backward compatible extension of SQL’03
• Symbiosis of relational and array paradigm
• Flexible structure-based grouping
• Capitalizes the MonetDB physical array storage
• Recycling, an adaptive ‘materialized view’
• Zero-cost attachment contract for cooperative clients
http://www.scilens.org/Resources/SciQL
© M Kersten 2012

19. Table vs Arrays
CREATE TABLE tmp
A collection of tuples
Indexed by a (primary) key
Default handling
Explicitly created using
INS/UPD/DEL
© M Kersten 2012

20. Table vs arrays
CREATE TABLE tmp CREATE ARRAY tmp
A collection of tuples A collection of a priori defined tuples
Indexed by a (primary) key Indexed by dimension expressions
Default handling Implicitly defined by default value,
Explicitly created using To be updated with INS/DEL/UPD
INS/UPD/DEL
© M Kersten 2012

21. SciQL examples
CREATE TABLE matrix (
x integer,
y integer,
value float
PRIMARY KEY (x,y) );
INSERT INTO matrix VALUES
(0,0,0),(0,1,0),(1,1,0)(1,0,0);
0 0 0
0 1 0
1 1 0
1 0 0
© M Kersten 2012

22. SciQL examples
CREATE TABLE matrix ( CREATE ARRAY matrix (
x integer, x integer DIMENSION[2],
y integer, y integer DIMENSION[2],
value float value float DEFAULT 0);
PRIMARY KEY (x,y) );
INSERT INTO matrix VALUES
(0,0,0),(0,1,0),(1,1,0)(1,0,0);
null … … …
0 0 0
null null null …
0 1 0
0 0
0 null …
1 1 0 1
0 0 0
0 null null
1 0 0
0 1
© M Kersten 2012

23. SciQL examples
CREATE TABLE matrix ( CREATE ARRAY matrix (
x integer, x integer DIMENSION[2],
y integer, y integer DIMENSION[2],
value float value float DEFAULT 0);
PRIMARY KEY (x,y) );
DELETE matrix WHERE y=1 DELETE matrix WHERE y=1
A hole in the array
0 0 0
null null
1 0 0 1
0 0 0
0 1
© M Kersten 2012

24. SciQL examples
CREATE TABLE matrix ( CREATE ARRAY matrix (
x integer, x integer DIMENSION[2],
y integer, y integer DIMENSION[2],
value float value float DEFAULT 0);
PRIMARY KEY (x,y) );
INSERT INTO matrix VALUES INSERT INTO matrix VALUES
(0,1,1), (1,1,2) (0,1,1), (1,1,2)
0 0 0
1 0 0
1 2
1
0 1 1
0 0 0
1 1 2
0 1
© M Kersten 2012

25. SciQL unbounded arrays
CREATE TABLE matrix ( CREATE ARRAY matrix (
x integer, x integer DIMENSION,
y integer, y integer DIMENSION,
value float value float DEFAULT 0);
PRIMARY KEY (x,y) );
INSERT INTO matrix VALUES INSERT INTO matrix VALUES
(0,2,1), (0,1,2) (0,2,1), (0,1,2)
0 2 1 2 1 0
0 1 2 1 0 0
0 0 2
© M Kersten 2012 0 1

26. SciQL Dimensions
Unbounded Dimensions
scalar-type DIMENSION
Bounded Dimensions
scalar-type DIMENSION[stop]
scalar-type DIMENSION[first: step: stop]
scalar-type DIMENSION[*: *: *]
timestamp DIMENSION [ ‘2010-01-19’ : ‘1’ minute : *]
© M Kersten 2012

27. SciQL table queries
-- Dimension names make query formulation easier
CREATE ARRAY matrix (
x integer DIMENSION,
y integer DIMENSION,
value float DEFAULT 0 );
-- simple checker boarding aggregation
SELECT sum(value) FROM matrix WHERE (x + y) % 2 = 0
© M Kersten 2012

28. SciQL array queries
CREATE ARRAY matrix ( CREATE ARRAY result(
x integer DIMENSION, x integer DIMENSION,
y integer DIMENSION, value float DEFAULT 0 );
value float DEFAULT 0 );
-- group based aggregation to construct an unbounded vector
SELECT [x], sum(value) FROM matrix
WHERE (x + y) % 2 = 0
GROUP BY x;
© M Kersten 2012

29. SciQL array views
CREATE ARRAY vmatrix (
x integer DIMENSION[-1:5],
y integer DIMENSION[-1:5],
value float DEFAULT -1 )
AS SELECT x, y, value FROM matrix;
-1 -1 -1 -1
-1 0 0 -1
-1 0 0 -1
-1 -1 -1 -1
© M Kersten 2012

30. SciQL tiling examples
V0,3 V1,3 V2,3 V3,3
V0,2 V1,2 V2,2 V3,2
V0,1 V1,1 V2,1 V3,1
Anchor
Point V0,0 V1,0 V2,0 V3,0
SELECT x, y, avg(value)
FROM matrix
GROUP BY matrix[x : 1 : x+2][y : 1 : y+2];
© M Kersten 2012

31. SciQL tiling examples
V0,3 V1,3 V2,3 V3,3
V0,2 V1,2 V2,2 V3,2
V0,1 V1,1 V2,1 V3,1
Anchor
Point V0,0 V1,0 V2,0 V3,0
SELECT x, y, avg(value)
FROM matrix
GROUP BY DISTINCT matrix[x:1:x+2][y:1:y+2];
© M Kersten 2012

32. SciQL tiling examples
V0,3 V1,3 V2,3 V3,3
Anchor
Point V0,2 V1,2 V2,2 V3,2
V0,1 V1,1 V2,1 V3,1
null
V0,0 V1,0 V2,0 V3,0
null null
SELECT x, y, avg(value)
FROM matrix
GROUP BY DISTINCT matrix[x-1:1:x+1][y:1:y+2];
© M Kersten 2012

33. SciQL tiling examples
V0,3 V1,3 V2,3 V3,3
Anchor
Point V0,2 V1,2 V2,2 V3,2
V0,1 V1,1 V2,1 V3,1
V0,0 V1,0 V2,0 V3,0
SELECT x, y, avg(value)
FROM matrix
GROUP BY matrix[x][y],
matrix[x-1][y], matrix[x+1][y],
matrix[x][y-1], matrix[x][y+1];
© M Kersten 2012

34. SciQL, A Query Language for Science Applications
• Seamless integration of array-, set-, and sequence-
semantics.
• Dimension constraints as a declarative means for
indexed access to array cells.
• Structural grouping to generalize the value-based
grouping towards selective access to groups of cells
based on positional relationships for aggregation.
© M Kersten 2012

35. Agenda
Array support in database systems
SciQL array query language
Use-case exercise
A crash course on column-stores
SciQL implementation approach
© M Kersten 2012

36. Seismology use case
Rietbrock: Chili earthquake
… 2TB of wave fronts
… filter by sta/lta
… remove false positives
… window-based 3 min cuts
… heuristic tests
… interactive response required …
How can a database system help?
Scanning 2TB on modern pc takes >3 hours
© M Kersten 2012

37. Use case, a SciQL dream
Rietbrock: Chili earthquake
create array mseed (
tick timestamp dimension[ ‘2010’:*],
data decimal(8,6),
station string );
© M Kersten 2012

38. Use case, a SciQL dream
Rietbrock: … filter by sta/lta
--- average by window of 5 seconds
select A.tick, avg(A.data)
from mseed A
group by A[tick:1:tick + 5 seconds]
© M Kersten 2012

39. Use case, a SciQL dream
Rietbrock: … filter by sta/lta
select A.tick
from mseed A, mseed B
where A.tick = B.tick
and avg(A.data) / avg(B.data) > delta
group by A[tick:tick + 5 seconds],
B[tick:tick + 15 seconds]
© M Kersten 2012

40. Use case, a SciQL dream
Rietbrock: … filter by sta/lta
create view candidates(
station string,
tick timestamp,
ratio float ) as
select A.station, A.tick, avg(A.data) / avg(B.data) as ratio
from mseed A, mseed B
where A.tick = B.tick
and avg(A.data) / avg(B.data) > delta
group by A[tick:tick + 5 seconds],
B[tick:tick + 15 seconds]
© M Kersten 2012

41. Use case, a SciQL dream
Rietbrock: … remove false positives
-- remove isolated errors by direct environment
-- using wave propagation statics
create table neighbors(
head string,
tail string,
delay timestamp,
weight float)
© M Kersten 2012

42. Use case, a SciQL dream
Rietbrock: … remove false positives
select A.tick, B.tick
from candidates A, candidates B, neighbors N
where A.station = N.head
and B.station = N.tail
and B.tick = A.tick + N.delay
and B.ratio * N.weight < A.ratio;
© M Kersten 2012

43. Use case, a SciQL dream
Rietbrock: … remove false positives
delete from candidates
select A.tick
from candidates A, candidates B, neighbors N
where A.station = N.head
and B.station = N.tail
and B.tick = A.tick + N.delay
and B.ratio * N.weight < A.ratio;
© M Kersten 2012

44. Use case, a SciQL dream
Rietbrock: … window-based 3 min cuts
… heuristic tests
select B.station, myfunction(B.data)
from candidates A, mseed B
where A.tick = B.tick
group by distinct B[tick:tick + 3 minutes];
-- using a User Defined Function written in C.
© M Kersten 2012

45. Agenda
Array support in database systems
SciQL array query language
A crash course on column-stores
SciQL implementation approach
© M Kersten 2012

46. Storing Relations in MonetDB
Void Void Void Void Void
1000 1000 1000 1000 1000
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Virtual OID: seqbase=1000 (increment=1)
© M Kersten 2012

47. BAT Data Structure
BAT:
binary association table
Head Tail
BUN:
binary unit
Hash tables, Head & Tail:
BUN heap:
T-trees, - consecutive memory
R-trees, blocks (arrays)‫‏‬
block (array)‫‏‬
... - memory-mapped file
files
Tail Heap:
- best-effort duplicate
elimination for strings
© M Kersten 2012 (~ dictionary encoding)

48. Processing Model (MonetDB Kernel)‫‏‬
l Bulk processing:
l full materialization of all intermediate results
l Binary (i.e., 2-column) algebra core:
l select, join, semijoin, outerjoin
l union, intersection, diff (BAT-wise & column-wise)‫‏‬
l group, count, max, min, sum, avg
l reverse, mirror, mark
l Runtime operational optimization:
l Choosing optimal algorithm & implementation according to
input properties and system status
© M Kersten 2012

49. The Software Stack
Strategic optimization
Front-ends SQL 03 MAL
Optimizers Tactical optimization:
MAL -> MAL rewrites
Back-end(s) MonetDB 5 MAL
Runtime
Kernel MonetDB kernel operational
optimization
© M Kersten 2012

50. MonetDB Front-end: SQL
EXPLAIN SELECT a, z FROM t, s WHERE t.c = s.x;
function user.s2_1():void;
barrier _73 := language.dataflow();
_2:bat[:oid,:int] := sql.bind("sys","t","c",0);
_7:bat[:oid,:int] := sql.bind("sys","s","x",0);
_10 := bat.reverse(_7);
_11 := algebra.join(_2,_10);
_13 := algebra.markT(_11,0@0);
_14 := bat.reverse(_13);
_15:bat[:oid,:int] := sql.bind("sys","t","a",0);
_17 := algebra.leftjoin(_14,_15);
_18 := bat.reverse(_11);
_19 := algebra.markT(_18,0@0);
_20 := bat.reverse(_19);
_21:bat[:oid,:int] := sql.bind("sys","s","z",0);
_23 := algebra.leftjoin(_20,_21);
exit _73;
_24 := sql.resultSet(2,1,_17);
sql.rsColumn(_24,"sys.t","a","int",32,0,_17);
sql.rsColumn(_24,"sys.s","z","int",32,0,_23);
_33 := io.stdout();
sql.exportResult(_33,_24);
end s2_1;
© M Kersten 2012

51. Agenda
Array support in database systems
SciQL array query language
A crash course on column-stores
SciQL implementation approach
© M Kersten 2012

52. SciQL implementation
• Use the complete MonetDB software stack
• Extend the SQL catalog to support SciQL
• Extend the Kernel to support array processing
• Extend the optimizer stack for performance
• Aim for a functional implementation first
• Use tabular representation of arrays
• Reuse the SQL code generator
© M Kersten 2012

53. © M Kersten 2012

54. © M Kersten 2012

55. © M Kersten 2012

56. © M Kersten 2012

57. © M Kersten 2012

58. © M Kersten 2012

59. Slicing a portion of an array is a ‘selection’
© M Kersten 2012

60. ˜
© M Kersten 2012

61. It works
© M Kersten 2012

62. Conclusions
• The language definition is ‘finished’
• Functional prototype is ‘around the corner’
• Exposure to real life cases and external libraries
• MonetDB’s core technology was essential
• Challenge:
ARRAYS
FILES
© M Kersten 2012

63. © M Kersten 2012

64. © M Kersten 2012

65. © M Kersten 2012

66. Science DBMS landscape
MonetDB 5.23 SciDB 0.5 Rasdaman
Architecture Server approach Server approach Plugin(Oracle, DB2, Informix,
Mysql, Postgresql)
Open source Mozilla License GPL 3.0 Commercial GPL 3.0 Dual license
Downloads >12.000 /month Tens up to now ??
SQL SQL 2003 ?? SQL92++
Interoperability {JO}DBC, C(++),Python, … C++ UDF C++, Java, OGC
Array language SciQL AQL RASQL
Array model Fixed+variable bounds Fixed arrays Fixed+variable bounds
Science Linked libraries Linked libraries Linked libraries
Foreign files Vaults of csv, FITS, ?? Tiff,png,jpg..,
NETCDF, MSEED csv,,NETCDF,HDF4,
Distribution 50-200 node cluster 4 node cluster 20-node
Distribution tech Dynamic partial replication Static fragmentation Static fragmentation
Executor Various schemes Map-reduce Tile streaming
Largest demo Skyserver SDSS 6 3TB --- 12TB, IGN –F (on Postgresql)
Storage tuning Query adaptive Schema definitions Workload driven
© M Kersten Heuristics + cost base
Optimization 2012 ?? Heuristics +cost based