The document discusses the ECB's statistical data warehouse (SDW), which has been the primary channel for disseminating ECB statistics since 2005, designed according to the SDMX information model. It outlines key features such as data structure definitions, metadata storage, and data retrieval processes available through the SDW portal. Additionally, it highlights the capability for users to download data in various formats and subscribe for notifications on changes to time series observations.

1. Using SDMX data
model for data
dissemination and
data retrieval.
Introducing ECB’s
Statistical Data Warehouse
Gábor Horváth
SDMX Expert Meeting, October 2016

2. Rubric
www.ecb.europa.eu ©
Background
• SDW  key dissemination channel for ECB statistics since 2005!
• Primary source of data & metadata for end users, websites (ECB
website, Our statistics), visualization applications (ECBstatsApp);
• Fully designed based on the SDMX information model
• Access the public SDW via: http://sdw.ecb.europa.eu
SDW = Data & metadata + SDW Portal
repository
2
SDMX and ECB’s Statistical Data Warehouse
SDW
Databas
e
Data
Metadata
SDW
Databas
e
Data
Metadata
SDMX and ECB’s Statistical Data Warehouse

3. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 5

4. Rubric
www.ecb.europa.eu ©
• Data Structure Definitions
• Data flows: 280+, (18+ million time series)
ECB SDW’s Content – Current status
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 6
Maintained by ECB Other Maintenance Agencies
110 50+

5. Rubric
www.ecb.europa.eu ©
SDW Database – Structural Metadata Storage
SMD = DSD, Statistical concepts, code lists are stored with their
timestamps of validity.
From 2005 – until the introduction of separate, dedicated SDMX
registries, the SDW database fulfilled the role of SDMX registry.
The SDW database
stores for each
artifact also its
Maintenance agency….
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 7
SDMX and ECB’s Statistical Data
Warehouse

6. Rubric
www.ecb.europa.eu ©
Accessing the Data Structure Definitions 1.
1. Studying the Complete DSD of a Data Flow
(all dimensions, attributes, their codelists with complete code list and their usage on the
portal is displayed).
2. Dimensions are used in the ‘MS Excel Pivot Export’s
3. ECB SDMX WS 2.1 :
https://sdw-wsrest.ecb.europa.eu/service/datastructure/ECB/ECB_FMD2
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 8
Link to portal
1. 2.

7. Rubric
www.ecb.europa.eu ©
The SDW portal displays all attributes defined in the DSD.
They can be accessed on different pages of the portal.
Note:
The list of pages is available on the ‘DSD’ of the Data Flow
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 9
Link to portal
Level of attribute Page, where it is displayed
Observation Quickview + Data Table
Series Quickview + Metadata
Sibling Quickview + Metadata
Dataset Metadata

8. Rubric
www.ecb.europa.eu ©
The SDW portal uses attributes:
for the Data display (grid) and Data visualization (graphs).
•COLLECTION
(beginning of period, end of period, etc.)
•DECIMAL attributes
(default setting, can be changed under ‘My settings…’
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 10

9. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 11

10. Rubric
www.ecb.europa.eu ©
Concepts and Codelists in SDW
1. Displayed in the DSDs of a Data Flow
(all dimensions and attributes with their concepts with complete code list).
2. See Codelists of Dimension Filters in the ‘MS Excel Pivot Export’s
3. ECB SDMX WS 2.1 :
https://sdw-wsrest.ecb.europa.eu/service/datastructure/ECB/ECB_FMD2
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 12
Link to portal
1. 2.

11. Rubric
www.ecb.europa.eu ©
Concepts and Codelists in SDW 2.
3. Dimension Filters can be used to “drill down” in the data cubes.
3.1 Base Filters
(common in “all” data sets)
3.2. Data Set specific Dimension Filters.
(Constraints preset the codes in the filters)
Note: ‘Safe Drill Down’
After each step during the ‘drill down’ in the
data cube, the content of the filters is always
set to the set of available codes.
Users will always find series during the drill down.
4. ECB SDMX WS 2.1 :
https://sdw-wsrest.ecb.europa.eu/service/codelist/ECB/CL_CURRENCY
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 13
Link to portal

12. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 14

13. Rubric
www.ecb.europa.eu ©
SDW Database – Data Storage
• Observations are stored in the traditional data
warehousing design solutions: in start-schemas.
• The structure of a Star-schema is defined in a DSD.
• Data Set: Three Fact tables with the corresponding codelists.
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 15
DSDs can contain four levels of attributes:
1.Observation level
2.Time Series level
3.Sibling level
(Sibling = groups where Frequency dim is wildcarded)
4.Data Set level
Fact tables / data set Content
1 OBS table Dimensions, period, observations,
Observation level attributes
1 SERIES table Dimensions, Series-level attributes
1 SIBLING table Dimensions, Sibling-level attributes
Notes:
• No observation is ever deleted!
• Full life span of time series is
available.

14. Rubric
www.ecb.europa.eu ©
SDW Database – star schema of observationsobservations
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 16
Period
Dimension 1
Dimension 2
Dimension n
Observation value
Observation attribute 1
Observation attribute m
Valid from
Valid to
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Fact tables with dimension tables

15. Rubric
www.ecb.europa.eu ©
SDW Database – star schema of seriesseries attributes
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 17
Period
Dimension 1
Dimension 2
Dimension n
Series attribute 1
Series attribute l
Valid from
Valid to
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Fact tables with dimension tables

16. Rubric
www.ecb.europa.eu ©
SDW Database – star schema of siblingsibling attributes
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 18
Period
Dimension 1
Dimension 2
Dimension n-1
Sibling attribute 1
Sibling attribute k
Valid from
Valid to
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Code Description
… …
Fact tables with dimension tables

17. Rubric
www.ecb.europa.eu ©
SDW Database – Management of data set tables
1. Creation of new data sets
all data tables and views are automatically created before the first
dissemination of the data.
2. Updates/changes in the DSD
changes which does not affect the ‘backward compatibility’ of
database tables are handled automatically.
E.g: Introduction of new attributes in fact tables,
introduction of new code values in dimension tables,
increase of the length of codes in dimension tables.
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 19

18. Rubric
www.ecb.europa.eu ©
SDW Database – Data & Metadata loads
On average:
•daily 200 data loads (on working days)
•weekly 4 updates of SMD (including any change in codes,
descriptions, etc)
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 20
File type File format Source/agency
Data SDMX-ML Compact
format (UTF-8)
ECB + other external
sources (converted)
Structural Metadata SDMX-ML Structure
Message (UTF-8)
ECB, BIS + from other
agencies.

19. Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
Data Download options
The SDW portal offers SDMX-ML Data Download feature since 2006.
The format has been slightly changed, and it is SDMX-ML 2.0 at the
moment.
In case of interest in data files in the SDMX-ML 2.1 format, please,
use the SDMX RESTful Web-Services of the SDW.
It can return statistical information also in JSON and different CSV
formats!
Link ‘API’ leads you to the Documentation of the Web-Services.
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 21
Link to portal

20. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 22

21. Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
• Navigation hierarchy - Category schema
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 23
Data flows are categorized always
to the leaf nodes in the navigation tree.
E.g: 12 series of the FM Data Flow
to the ‘Official interest rates’ category.
Constraints of FM dataset:
FREQ in [B, D]
PROVIDER_FM = 4F
INSTRUMENT_FM = KR
Link to portal

22. Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
• Data Flows in the navigation hierarchy
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 24
All Data Sets without Constraints are
Categorized under a dedicated Category
‘Full Content’

23. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 25

24. Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
Reference Metadata is available on the ‘Metadata’ pages of SDW
A list of Concepts from the SDMX Metadata Common Vocabulary
(MCV) can be used
e.g: Reference MD
of the EXR Data Set
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 26
Link to portal

25. Rubric
www.ecb.europa.eu ©27
SDW portal – Reference Metadata storage in SDW
<<Contact>>
„Alberto González Pandiella”
<<Responsible business area>>
“External Statistics Division”
<<FAME shortcut >>
„$ECB”
Reference Metadata can be attached to
data at different levels:
- Data Sets
-Groups/Siblings
-Time Series
SDMX and ECB’s Statistical Data Warehouse
<<attach>>
<<attach>>
<<attach>>

26. Rubric
www.ecb.europa.eu ©
SDW Database – Reference metadata storage
SDMX and ECB’s Statistical Data Warehouse 28
SDMX and ECB’s Statistical Data Warehouse
Metadata item
Metadata id
Metadata value
Metadata type
…
….
Attach
Attachment
id
Metadata id
Description
Dataflow
…
Constraint
Attachment id
Concept
Code value
…
…

27. Rubric
www.ecb.europa.eu ©
1. Data Structure Definitions (DSDs)
2. Concepts and Code lists
3. Data flows
4. Category schema
5. Reference Metadata
6. Notification
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 29

28. Rubric
www.ecb.europa.eu ©
SDW portal – supported by the SDMX model
SDW offers a Notifications service
•Users can subscribe to email notifications in cases of changes in the
observations of one or multiple time series from one or more Data
Flows.
•Changes include: (i) new observation added, (ii) existing observation
revised (iii) observation deleted.
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 30
The SDW database and portal do not
server as Registries and they do not
provide Notification services for changes
in the Structural Metadata.

29. Rubric
www.ecb.europa.eu ©
Thank you for your attention!
GÁBOR HORVÁTH
European Central Bank
Statistical Information Services Division
SDMX and ECB’s Statistical Data Warehouse
SDMX and ECB’s Statistical Data Warehouse 31