Your SlideShare is downloading. ×
0
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Sdmx Tools
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Sdmx Tools

3,301

Published on

Presentation about the most interesting SDMX tools. How tio start working with SDMX using the ISTAT SDMX Framework, a set of building blocks to easily set up an SDMX architecture

Presentation about the most interesting SDMX tools. How tio start working with SDMX using the ISTAT SDMX Framework, a set of building blocks to easily set up an SDMX architecture

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,301
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
40
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • <pagebreak> The SDMX standard specifies a single Information Model in order to describe what and how data and metadata can be exchanged in the context of SDMX. Based on this Information Model, SDMX defines two ways of representing its messages. Two different formats are available for expressing SDMX messages. The first format is SDMX-EDI and is equivalent to GESMES/TS. This format is based on an EDIFACT syntax and is Time-Series oriented. This means that only one type of observation throughout time can be carried in a single DataSet. Moreover, the DataSet messages have only one format. This format covers a subset of the SDMX-IM. For example, non time-series messages or reference metadata messages are not supported. The second format is SDMX-ML. It is an XML format that covers the whole SDMX-IM. The SDMX-ML format supports four different (although equivalent) formats, for data messages, in order to serve different purposes. It also supports reference metadata messages as well as messages for querying SDMX Web Services and Registry Interface messages. Two simple “rules” in order to go from the SDMX-IM to the SDMX-ML implementation are that: (concrete) classes become XML elements and their attributes become XML attributes. Of course, there are exceptions in these “rules”. In the SDMX-ML implementation there are XML elements that do not correspond directly to classes from the SDMX-IM and vice versa.
  • <pagebreak> The conversion process comprises two main activities; reading an input data message and writing out the converted data message. The are specific modules that read and write datasets i.e SDMX-ML (Compact, Generic, Utility, Cross-Sectional) Gesmes (TS, 2.1, DSIS) Flat files (CSV FLR). The information of a dataset to converter is stored in classes that are based on the SDMX Information Model v2.0. These classes play the role of an intermediate format between readers and writers. The Data Structure Definition related to the converted datasets is needed for performing a conversion. SDMX Converter, if it’s is not provided manually, can retrieved the DSD from the Registry.
  • <pagebreak> The SDMX standard, in its 5 th document of specifications (http://www.sdmx.org/docs/2_0/SDMX_2_0%20SECTION_05_RegistrySpecification.pdf), gives details on the interfaces that an SDMX compliant Registry should implement. Based on these specifications, Eurostat’s SDMX Registry has been developed and is now deployed in the European Commission’s production environment ( https://webgate.ec.europa.eu/sdmxregistry/ ). Eurostat’s SDMX Registry has been developed in order to be used a central repository for: Structural metadata: Code Lists, Concept Schemes, Data Structure Definitions, Metadata Structure Definitions, Category Schemes, Organisation Schemes, Hierarchical Code Lists Provisioning metadata: Data flows, Metadata flows, Provision Agreements The major interface of Eurostat’s SDMX Registry is a Web Service implementing the SDMX Registry Interface messages. These are SDMX-ML messages and are specified within the standard and by the following XSD: http://www.sdmx.org/docs/2_0/SDMXRegistry.xsd A Graphical User Interface (GUI) has been also implemented in order to enable human interaction of the World Wide Web. The user-authentication is realized using CIRCA accounts. A standalone tool implementing almost all functionality of the SDMX registry is the DSW (already presented).
  • Transcript

    • 1. SDMX TOOLS Workshop on Data Transmission Becici – Montenegro 17- 19 June 2009 Vincenzo PATRUNO ISTAT
    • 2. Overview There are several tools that help us working with SDMX files Released under GPL or EUPL license
    • 3. http://sdmx.org/?page_id=13
    • 4. http://www.osor.eu/communities/oss-and-statistics/wiki/sdmx-tools
    • 5. SDMX Converter http://sdmx.org/?page_id=13
    • 6. From SDMX-IM to messages SDMX-IM SDMX-EDI SDMX-ML
    • 7. <ul><li>Based on a common Information Model </li></ul><ul><ul><li>SDMX-EDI (GESMES/TS) </li></ul></ul><ul><ul><ul><li>EDIFACT syntax </li></ul></ul></ul><ul><ul><ul><li>Time series oriented – One format for Data Sets </li></ul></ul></ul><ul><ul><li>SDMX-ML </li></ul></ul><ul><ul><ul><li>XML syntax </li></ul></ul></ul><ul><ul><ul><li>Four different formats for Data Sets </li></ul></ul></ul><ul><ul><ul><li>Easier validation (XML based) </li></ul></ul></ul>From SDMX-IM to messages
    • 8. SDMX-ML XML format for the exchange of SDMX-structured data and metadata.
    • 9. SDMX-ML: Six standard messages Name of message Short description Schema file 1 Structure Definition Message Contains a data structure definition Fixed 2 Generic Data Message Conveys data in a form independent of a data structure definition. It is designed for data provision on websites and in any scenario where applications receiving the data may not have detailed understanding of the data set's structure before they obtain the data set itself. Fixed 3 Compact Data Message Exchange of large data sets in a data structure definition-dependent form Derived from data structure definition message 4 Utility Data Message For schema-based functions, such as validation, in a data structure definition-dependent form Derived from data structure definition message 5 Cross-sectional Data Message Exchange of many observation types in a data structure definition-dependent form Derived from data structure definition message 6 Query message To query a database to obtain an SDMX-ML message as the result Fixed
    • 10. Metadata <ul><li>Structural metadata must be associated with the data, otherwise it becomes impossible to identify, retrieve and browse the data. </li></ul>
    • 11. <ul><li>The simplest explanation of a data structure definition is that it is a set of descriptor concepts, associated with a set of data, which allow us to understand what that data means. </li></ul>Data Structure Definition
    • 12. SDMX-ML “Model-Driven” XML Approach
    • 13. SDMX Converter <ul><li>Main Activities </li></ul><ul><ul><li>Reading the input message </li></ul></ul><ul><ul><ul><li>parsing of the message </li></ul></ul></ul><ul><ul><ul><li>populating the data model of the tool (based on the SDMX v2.0 information model ) </li></ul></ul></ul><ul><ul><li>Writing the converted message </li></ul></ul><ul><ul><ul><li>uses the data model to write the output message in the target format. </li></ul></ul></ul><ul><li>Supported formats </li></ul><ul><ul><li>SDMX-ML (Generic, Compact, Utility Compact) </li></ul></ul><ul><ul><li>Gesmes (TS, 2.1. DSIS) </li></ul></ul><ul><ul><li>Flat files (CSV, FLR) </li></ul></ul><ul><li>DSD retrieved from the Registry that is mandatory for completing a conversion </li></ul>
    • 14. http://bms.istat.it/sodidownload/download.aspx ISTAT SDMX Framework
    • 15. SDMX Framework <ul><li>It is based on the SDMX Information Model </li></ul><ul><li>It covers all the dissemination business process </li></ul><ul><li>It can be used entirely </li></ul><ul><li>or </li></ul><ul><li>Only some building bocks can be used, in order to integrate them in an already existing information system </li></ul><ul><li>Available APIs </li></ul><ul><ul><li>SDMX Query Parser </li></ul></ul><ul><li>It allows to set up an SDMX architecture in a short time </li></ul><ul><li>It can be used for didactic purpose </li></ul><ul><li>It will be available as Open Source </li></ul>
    • 16. Technical requirements <ul><li>windows applications: </li></ul><ul><ul><li>MS Windows XP or sup. </li></ul></ul><ul><ul><li>Microsoft .Net framework 2.0 </li></ul></ul><ul><li>web applications </li></ul><ul><ul><li>MS Windows 2003 or sup. </li></ul></ul><ul><ul><li>Internet Information Server 6.0 or sup. </li></ul></ul><ul><ul><li>Microsoft .Net framework 2.0 </li></ul></ul><ul><ul><li>Microsoft ASP.NET 2.0 </li></ul></ul><ul><li>Database </li></ul><ul><ul><li>Oracle 9 or sup </li></ul></ul><ul><ul><li>MS SQL Server 2005 or sup. </li></ul></ul><ul><ul><li>MS Access </li></ul></ul>
    • 17. NSI 1 2 3 Data Sharing Model
    • 18. Query SDMX Data SDMX-ML RSS WS NSI Data Sharing Model
    • 19. Link to SDMX file Data SDMX-ML RSS WS NSI Data Sharing Model
    • 20. Query Message <ul><li>This is another special-purpose message. It is used to convey a query to a database which then returns an SDMX-ML message </li></ul><ul><li>For web services and database-driven applications </li></ul>
    • 21. Data Structure Wizard http://sdmx.org/?page_id=13
    • 22. SDMX Registry <ul><li>A repository for keeping </li></ul><ul><ul><li>Structural metadata (e.g. CodeLists, ConceptSchemes, DSDs, MSDs) </li></ul></ul><ul><ul><li>Provisioning information (e.g. Dataflows, Provision Agreements) </li></ul></ul><ul><li>Repository is accessible via a Web Service accepting SDMX-ML messages </li></ul><ul><li>GUI for user interaction over the Web </li></ul><ul><li>Authentication using CIRCA accounts </li></ul><ul><li>DSW – “standalone” Java GUI </li></ul>
    • 23. <ul><li>https://webgate.ec.europa.eu/sdmxregistry/ </li></ul>SDMX Registry
    • 24. Thank You for Your Attention [email_address] ....and Good SDMX-ing

    ×