Sdmx Tools


Published on

Presentation about the most interesting SDMX tools. How tio start working with SDMX using the ISTAT SDMX Framework, a set of building blocks to easily set up an SDMX architecture

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • <pagebreak> The SDMX standard specifies a single Information Model in order to describe what and how data and metadata can be exchanged in the context of SDMX. Based on this Information Model, SDMX defines two ways of representing its messages. Two different formats are available for expressing SDMX messages. The first format is SDMX-EDI and is equivalent to GESMES/TS. This format is based on an EDIFACT syntax and is Time-Series oriented. This means that only one type of observation throughout time can be carried in a single DataSet. Moreover, the DataSet messages have only one format. This format covers a subset of the SDMX-IM. For example, non time-series messages or reference metadata messages are not supported. The second format is SDMX-ML. It is an XML format that covers the whole SDMX-IM. The SDMX-ML format supports four different (although equivalent) formats, for data messages, in order to serve different purposes. It also supports reference metadata messages as well as messages for querying SDMX Web Services and Registry Interface messages. Two simple “rules” in order to go from the SDMX-IM to the SDMX-ML implementation are that: (concrete) classes become XML elements and their attributes become XML attributes. Of course, there are exceptions in these “rules”. In the SDMX-ML implementation there are XML elements that do not correspond directly to classes from the SDMX-IM and vice versa.
  • <pagebreak> The conversion process comprises two main activities; reading an input data message and writing out the converted data message. The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers. The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry.
  • <pagebreak> The SDMX standard, in its 5 th document of specifications (, gives details on the interfaces that an SDMX compliant Registry should implement. Based on these specifications, Eurostat’s SDMX Registry has been developed and is now deployed in the European Commission’s production environment ( ). Eurostat’s SDMX Registry has been developed in order to be used a central repository for: Structural metadata: Code Lists, Concept Schemes, Data Structure Definitions, Metadata Structure Definitions, Category Schemes, Organisation Schemes, Hierarchical Code Lists Provisioning metadata: Data flows, Metadata flows, Provision Agreements The major interface of Eurostat’s SDMX Registry is a Web Service implementing the SDMX Registry Interface messages. These are SDMX-ML messages and are specified within the standard and by the following XSD: A Graphical User Interface (GUI) has been also implemented in order to enable human interaction of the World Wide Web. The user-authentication is realized using CIRCA accounts. A standalone tool implementing almost all functionality of the SDMX registry is the DSW (already presented).
  • Sdmx Tools

    1. 1. SDMX TOOLS Workshop on Data Transmission Becici – Montenegro 17- 19 June 2009 Vincenzo PATRUNO ISTAT
    2. 2. Overview There are several tools that help us working with SDMX files Released under GPL or EUPL license
    3. 3.
    4. 4.
    5. 5. SDMX Converter
    6. 6. From SDMX-IM to messages SDMX-IM SDMX-EDI SDMX-ML
    7. 7. <ul><li>Based on a common Information Model </li></ul><ul><ul><li>SDMX-EDI (GESMES/TS) </li></ul></ul><ul><ul><ul><li>EDIFACT syntax </li></ul></ul></ul><ul><ul><ul><li>Time series oriented – One format for Data Sets </li></ul></ul></ul><ul><ul><li>SDMX-ML </li></ul></ul><ul><ul><ul><li>XML syntax </li></ul></ul></ul><ul><ul><ul><li>Four different formats for Data Sets </li></ul></ul></ul><ul><ul><ul><li>Easier validation (XML based) </li></ul></ul></ul>From SDMX-IM to messages
    8. 8. SDMX-ML XML format for the exchange of SDMX-structured data and metadata.
    9. 9. SDMX-ML: Six standard messages Name of message Short description Schema file 1 Structure Definition Message Contains a data structure definition Fixed 2 Generic Data Message Conveys data in a form independent of a data structure definition. It is designed for data provision on websites and in any scenario where applications receiving the data may not have detailed understanding of the data set's structure before they obtain the data set itself. Fixed 3 Compact Data Message Exchange of large data sets in a data structure definition-dependent form Derived from data structure definition message 4 Utility Data Message For schema-based functions, such as validation, in a data structure definition-dependent form Derived from data structure definition message 5 Cross-sectional Data Message Exchange of many observation types in a data structure definition-dependent form Derived from data structure definition message 6 Query message To query a database to obtain an SDMX-ML message as the result Fixed
    10. 10. Metadata <ul><li>Structural metadata must be associated with the data, otherwise it becomes impossible to identify, retrieve and browse the data. </li></ul>
    11. 11. <ul><li>The simplest explanation of a data structure definition is that it is a set of descriptor concepts, associated with a set of data, which allow us to understand what that data means. </li></ul>Data Structure Definition
    12. 12. SDMX-ML “Model-Driven” XML Approach
    13. 13. SDMX Converter <ul><li>Main Activities </li></ul><ul><ul><li>Reading the input message </li></ul></ul><ul><ul><ul><li>parsing of the message </li></ul></ul></ul><ul><ul><ul><li>populating the data model of the tool (based on the SDMX v2.0 information model ) </li></ul></ul></ul><ul><ul><li>Writing the converted message </li></ul></ul><ul><ul><ul><li>uses the data model to write the output message in the target format. </li></ul></ul></ul><ul><li>Supported formats </li></ul><ul><ul><li>SDMX-ML (Generic, Compact, Utility Compact) </li></ul></ul><ul><ul><li>Gesmes (TS, 2.1. DSIS) </li></ul></ul><ul><ul><li>Flat files (CSV, FLR) </li></ul></ul><ul><li>DSD retrieved from the Registry that is mandatory for completing a conversion </li></ul>
    14. 14. ISTAT SDMX Framework
    15. 15. SDMX Framework <ul><li>It is based on the SDMX Information Model </li></ul><ul><li>It covers all the dissemination business process </li></ul><ul><li>It can be used entirely </li></ul><ul><li>or </li></ul><ul><li>Only some building bocks can be used, in order to integrate them in an already existing information system </li></ul><ul><li>Available APIs </li></ul><ul><ul><li>SDMX Query Parser </li></ul></ul><ul><li>It allows to set up an SDMX architecture in a short time </li></ul><ul><li>It can be used for didactic purpose </li></ul><ul><li>It will be available as Open Source </li></ul>
    16. 16. Technical requirements <ul><li>windows applications: </li></ul><ul><ul><li>MS Windows XP or sup. </li></ul></ul><ul><ul><li>Microsoft .Net framework 2.0 </li></ul></ul><ul><li>web applications </li></ul><ul><ul><li>MS Windows 2003 or sup. </li></ul></ul><ul><ul><li>Internet Information Server 6.0 or sup. </li></ul></ul><ul><ul><li>Microsoft .Net framework 2.0 </li></ul></ul><ul><ul><li>Microsoft ASP.NET 2.0 </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Oracle 9 or sup </li></ul></ul><ul><ul><li>MS SQL Server 2005 or sup. </li></ul></ul><ul><ul><li>MS Access </li></ul></ul>
    17. 17. NSI 1 2 3 Data Sharing Model
    18. 18. Query SDMX Data SDMX-ML RSS WS NSI Data Sharing Model
    19. 19. Link to SDMX file Data SDMX-ML RSS WS NSI Data Sharing Model
    20. 20. Query Message <ul><li>This is another special-purpose message. It is used to convey a query to a database which then returns an SDMX-ML message </li></ul><ul><li>For web services and database-driven applications </li></ul>
    21. 21. Data Structure Wizard
    22. 22. SDMX Registry <ul><li>A repository for keeping </li></ul><ul><ul><li>Structural metadata (e.g. CodeLists, ConceptSchemes, DSDs, MSDs) </li></ul></ul><ul><ul><li>Provisioning information (e.g. Dataflows, Provision Agreements) </li></ul></ul><ul><li>Repository is accessible via a Web Service accepting SDMX-ML messages </li></ul><ul><li>GUI for user interaction over the Web </li></ul><ul><li>Authentication using CIRCA accounts </li></ul><ul><li>DSW – “standalone” Java GUI </li></ul>
    23. 23. <ul><li> </li></ul>SDMX Registry
    24. 24. Thank You for Your Attention [email_address] ....and Good SDMX-ing