Ugif 04 2011 france ug04042011-jroy_ts

364 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
364
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ugif 04 2011 france ug04042011-jroy_ts

  1. 1. TimeSeries Technical PresentationJacques RoyApril 6, 2011 © 2010 IBM Corporation
  2. 2. Agenda■ What is TimeSeries■ Why TimeSeries■ Components■ Usage■ 2 © 2010 IBM Corporation
  3. 3. “Give me the Jan 1st element from time series “X” Most useful when a range of data is normally read “Give me the Jan 1st thru Jan 10th elements from time series “X” Access to one time series is usually completed before moving to the next time series.3 © 2010 IBM Corporation
  4. 4. Challenges Managing Time Series Data■ Slow Performance – Extremely slow data access specially for ordered set of rows due to the data layout and disk I/O – Operations hard or impossible to do in standard SQL■ High Storage Requirements – Time series are usually stored as "tall – thin" tables with a very large number of rows – May need one index to enforce uniqueness and another for index only read, more space used for index than data – Huge space requirements in standard relational layout, due to the volume and data■ Complex Querying – Can be difficult to write SQL to work with the data4
  5. 5. Informix Solution● TimeSeries Data Type : Native time series support■ Store time series elements as an ordered set of elements – Uses less space because the "key" is factored out and the time field takes either 0 (for regular) or 11 ( for irregular) bytes – Access is faster than index-only-read – SQL can be made much simpler■ Freedom to manage time series data: – Freedom to choose what and how it is stored – Freedom to choose the time series interval – Freedom to choose where the time series is storedMeterId Reading No other RDBMS 1001 2010-01-01,daily,{(12.34,12567),(12.56,9000),(12.34,55567),..} has native time 2001 2010-05-05,daily,{(199.08,6780),(198.55,3400),(198.12,250),..} series support 2011 2010-09-01,daily,{(9.34,8067),(9.56,9000),(9.40,10780),..}5
  6. 6. Key Strengths of Informix TimeSeries Performance – Extremely fast data access: Data clustered on disk to reduce I/O – Provides very high degree of parallelism on reads and writes – Provides continuous loading of data with minimal impact on concurrent queries Space Savings – Provides high level of compression – Can be over 50% space savings over standard relational layout Usability – Time series tool kit allows custom analytics to be written – Handles operations hard or impossible to do in standard SQL – Conceptually closer to how users think of time series – No other RDBMS has native time series support6 © 2010 IBM Corporation
  7. 7. Smart Meters Data: Schema Example Primary Key mtr_id date Col1 Col2 ColN 1 Mon Value 1 Value 2 ……. Value N 1 Tue Value 1 Value 2 ……. Value N 1 Wed Value 1 Value 2 ……. Value N Relational Schema ... ... ... ... ……. ... 13 Mon Value 1 Value 2 ……. Value N 13 Tue Value 1 Value 2 ……. Value N 13 Wed Value 1 Value 2 ……. Value N ... ... ... ... ……. ... mtr_id Series (int) timeseries(mtr_data) 1 [(Mon, v1, ...)(Tue,v1…)] 2 [(Mon, v1, ...)(Tue,v1…)]Above schema using 3 [(Mon, v1, ...)(Tue,v1…)]Informix TimeSeries 4 [(Mon, v1, ...)(Tue,v1…)] … … Save space and increase performance with faster data access with Informix7 © 2010 IBM Corporation
  8. 8. TimeSeries Space Savings ExampleTimeSeries data type takes much less space than traditional relational storage● – Proof of concept example: • Regular TimeSeries, 15 minute interval • Relational database used ~ 1TB (1000GB) • Informix used ~340GB The reason for this is: – The TimeSeries does not repeat data •MeterID: 4 bytes per reading •TimeStamp: Could be 12 bytes per reading •Assuming a 8 byte reading, that ~66% savings •3X less storage! Data Storage Comparison for 1 million meters 8 © 2010 IBM Corporation
  9. 9. TimeSeries Performance Performance – Faster accessing sets of data • Ordered data – Much faster combining time series – For data loading into timeseries, Informix outperforms the nearest competition by more than 30x times – For report generation from Performance Comparison for Data timeseries, Informix outperforms Loads and Reports for 1 Million Meters the nearest competition by more than 90x times9 © 2010 IBM Corporation
  10. 10. Who’s Interested in TimeSeries Energy: smart meters Capital Markets – Arbitrage opportunities, breakout signals, risk/return optimization, portfolio management, VaR calculations, simulations, backtesting... Telecommunications: – Network monitoring, load prediction, blocked calls (lost revenue) from load, phone usage, fraud detection and analysis... Manufacturing: – Machinery going out of spec; process sampling and analysis Logistics: – Location of a fleet (e.g. GPS); route analysis Scientific research: – Temperature over time...10 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  11. 11. TimeSeries: Key Concepts ■ Containers – Specialized storage for TimeSeries EXECUTE PROCEDURE TSContainerCreate(raw_container, rootdbs, meter_data, 100, 50); ■ Timeseries data element: row type – Flexibility to define as many parts as needed CREATE ROW TYPE meter_data ( tstamp datetime year to fraction(5), value decimal(14,3) ); ■ Timeseries types: regular, irregular – Covers regular intervals and sparse data distribution ■ Calendar – Defines business patterns 11 © 2010 IBM Corporation
  12. 12. Features Unique to Regular TimeSeries Only one element per “on” interval Value "persists" to end of interval An element for an “on” interval may be missing, entire element will be NULL Calendar determines offset in TimeSeries of given time point Elements can be accessed by offset or time point Time point not stored; calculated from header + date/time arithmetic12 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  13. 13. Features Unique to Irregular TimeSeries Data can be entered at any time point within a valid "on" interval Element persist until next element No NULL elements Elements can only be accessed by time No duplicate time points allowed If element already exists at given time point either an error is raise or a unique time point is found: – round time point up to nearest second – search back for first element – add 10 microseconds, this is new time point13 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  14. 14. Accessing Timeseries Access through standard tabular view – Makes TimeSeries look like a standard relational table SQL Functions – 103 functions Customized functions – Written in Stored Procedure Language (SPL), “C”, Java – 65 “C” functions14 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  15. 15. TimeSeries Header A TimeSeries needs information that sets its context: – Calendar: Time period where data is found – Origin: Time origin of the TimeSeries – Threshold: in-row storage threshold – Container: where to store the out-of-row data – Metadata: optional data added by the TimeSeries creator15 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  16. 16. Calendar and Calendar Patterns A calendar pattern is needed before we can create a calendar: INSERT INTO CalendarPatterns VALUES(day, {1 on, 2 off, 4 on}, day ); A Calendar defines a set of valid times at which the TimeSeries can record data. (July 8, 2005 is a Friday) INSERT INTO CalendarTable(c_name, c_calendar) VALUES(calday, startdate(2005-07-08 00:00:00.00000), pattstart(2005-07-08 00:00:00.00000), pattname(day) ); You can provide a pattern explicitly: INSERT INTO CalendarTable(c_name, c_calendar) VALUES(weekcal, startdate(2005-07-08 00:00:00.00000), pattstart(2005-07-08 00:00:00.00000), pattern({1 on, 2 off, 4 on}, day) );16 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  17. 17. TimeSeries: Table A TimeSeries resides in a table: CREATE TABLE ts_data ( loc_esi_id char(20) NOT NULL, measure_unit varchar(10) NOT NULL, direction char(1) NOT NULL, multiplier TimeSeries(meter_data), raw_reads timeseries(meter_data), PRIMARY KEY(loc_esi_id, measure_unit, direction) ) LOCK MODE ROW;17 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  18. 18. Populating a TimeSeries A timeSeries must first be created: INSERT INTO taqtrade_day VALUES("IBM.N", TSCreate(calday, 2005-07-08 00:00:00.00000, 20, 0, 0, taqtrade_day) ); It can be created through the input function: INSERT INTO taqtrade VALUES("AA.N", irregular, container(taqtrade), origin(2007-04-03 06:30:00.00000), calendar(calsec), [(4.48, . . .)@2007-04-03 06:30:03.00003, (4.50,. . .)@2007-04-03 06:30:03.00119, . . .] );18 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  19. 19. The Virtual Table Interface Makes a TimeSeries look like a table: EXECUTE PROCEDURE TSCreateVirtualTab(ts_data_v, ts_data, origin(2010-11-10 00:00:00.00000), calendar(cal15min),container(raw_container), threshold(0), regular, 0, raw_reads); Virtual table created: CREATE TABLE ts_data_v ( loc_esi_id char(20), measure_unit varchar(10,0), direction char(1), tstamp datetime year to fraction(5), value decimal(14,3) );19 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  20. 20. Quick Review A TimeSeries resides in a container – The container resides in a dbspace – The container is for a specific element type (row type) – A container is for either a regular or irregular TimeSeries (not both) – A container can contain multiple TimeSeries A TimeSeries requires a calendar – Defines when the data starts, defines a pattern of valid values A TimeSeries data is defines as a row type – Defines the values tracked You can operate on TimeSeries through special SQL functions or use the virtual table interface and standard SQL20 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation
  21. 21. DEMO21 Informix Dynamic Server, TimeSeries DataBlade Module class © 2007 IBM Corporation

×