2016 SDMX Experts meeting, SDMX Starter Kit for National Statistical Agencies, Denis Ward
1. SDMX Starter Kit for
National Statistical Agencies
SDMX Expert Meeting - October 2016
Aguascalientes, Mexico
2. SDMX Starter Kit for National Statistical Agencies
Presentation overview
1. What is the Starter Kit?
2. Main elements of the Kit – also, what’s new
3. Importance of using standardised concepts and terminology
4. Overview of structured implementation steps
5. Issues to be considered by countries before embarking on SDMX
implementation
6. Most difficult issues to resolve for countries new to SDMX –
suggested solutions
7. Suggestions / observations for future
3. 1. What is the Starter Kit?
• Online document outlining a step-by-step process that could be used by agencies
considering SDMX implementation
• Intended for use by agencies who are new to SDMX
• Assumes little existing knowledge of SDMX concepts and uses
• Aimed at a range of players who need to be involved in SDMX implementation within a
national organisation – senior management, subject matter experts, dissemination and
coordination units, IT
• Complements the Checklist for SDMX Design Projects located on the SDMX.org website
Current version located on the SDMX.org website at
https://sdmx.org/wp-content/uploads/SDMX_Starter_Kit_Version_23-9-2015.pdf
4. 2. Main elements of the Starter Kit
Current version
A. Objectives of the Starter Kit
B. Business case for implementing SDMX standards and guidelines
C. Structured process for implementing SDMX by national agencies – Five steps
Annexes
• Document will never be “finalised”
• Constantly evolving as new material comes to light
5. Starter Kit sources
• Minimal duplication of existing documentation, etc
• Makes extensive use of links to existing material
• Provides context for a range of information sources
Starter
Kit
SDMX .org
website
• Standards
• New material
Presentations at SDMX
Global Conferences, Expert
Meetings
• Evolving standards
• National implementations
Other relevant
modernisation standards
DDI, GSBPM, GSIM,
GAMSO, CSPA, etc
Experiences with national
implementations
Especially in developing
countries
Websites of
Eurostat, IMF,
ECB and NSOs
6. Main elements of the Starter Kit
Current version New version
A. Starter Kit objectives A. Objectives
B. Business case for implementing SDMX B. Business case
C. Structured process for implementing SDMX by
national agencies – Five steps
C. Structured process – Five steps
Annexes D. DSDS and MSDs
E. SDMX Concept Scheme and Information Model
F. Data reporting, exchange and dissemination scenarios
G. Business process modernisation – other standards
H. SDMX implementation tools
I. Complex SDMX issues – where countries have most trouble
Annexes
Constantly on lookout for new material / better ways of explaining
concepts, etc
7. Starter Kit – What’s new
Topic / issue Observations
1. Further information on use of SDMX for data
reporting and dissemination using push / pull modes
Provision of links to relevant national agency / int. org.
presentations at SDMX meetings / conferences
2. Inclusion of additional “Available resources”
information boxes
Provide links to where more detailed information can
be accessed
3. Further information on SDMX Registries Links to Global SDMX Registry and registries
developed by Eurostat, IMF
4. Further information on international glossaries and
importance of using standard concepts, etc
Includes new SDMX Glossary and Metadata Glossary
currently being developed
5. Step-by-step approach from designing and creating
core SDMX artefacts
Based mostly on presentation by Pellegrino and
Suranyi at 2015 SDMX Global Conference
6. More information on data exchange, reporting and
dissemination scenarios
7. Further information on business processing
modernisation standards – GSBPM, GAMSO, etc
More information on their relationship to SDMX
standards and guidelines
8. More examples of statistical tables reformatted into
SDMX Concept Structures
Provided as an Annex
8. Starter Kit underlying premises
• Initial SDMX implementation and ongoing maintenance is a resource
consuming process
• Obtaining resources requires senior management commitment and
buy in
• SDMX is but one of a set of standards necessary for modernization of
statistical processes – others include DDI, GSBPM, CSPA, GSIM
9. 3. Use of common terminology is crucial!
MCV
SDMX Glossary
Metadata Glossary
CSPA
GSBPM
10. 4. Structured approach aims to:
• Link SDMX implementation to a range of broad strategic issues across
statistical agency and / or the NSS – these need to be considered prior to
implementation
• Emphasise the need to step back and consider SDMX within the context of
existing corporate goals and planning processes – e.g. modernisation,
statistical integration
• Highlight the importance of developing a proper business case for
implementing SDMX or other relevant modernisation standard
• Consider a range of implementation tools on offer (SDMX-RI, DevInfo,
Fusion, IMF SDDS+) side by side to compare relative strengths
11. Five SDMX implementation steps
Development of business case is a precondition for senior management buy-in
1. Acquire basic understanding of key SDMX
concepts / artefacts
Links to existing material / resources
2. Consider range of issues beforehand Cover: institutional; IT; statistical; skills /
resource-related issues
3. Identify skill development needs • Cover: SDMX information model; how to
access / use tools;
• How to acquire required skills
4. Identify which SDMX implementation
tools to use
• Not necessary to reinvent wheel
• Consider range of tools available
5. Link into SDMX regional / global networks • SDMX.org
• With implementation countries
• Expert meetings / Global conferences
12. Business case – elements / issues
• Analysis of current situation
• Analysis of potential benefits / objectives of SDMX
• Realistic identification of human and financial resources needed for SDMX
implementation
• Relationship to other elements of standards-based modernization efforts
• Specify collaboration required for successful implementation – internal,
with other agencies in national statistical system, international statistical
agencies
Comprise issues to consider well prior to SDMX implementation
13. 5. Issues to consider prior to SDMX implementation
Answers to these issues could form: content of business case and modalities of an SDMX pilot project
1. Institutional issues / objectives • Objectives
• Drivers within organisation
• Coordination issues
• Which statistical domains
2. IT-related issues • Where are data stored
• Current database environment
• Structural metadata
• Which implementation software to use
3. Statistical issues • Conformity to international standards
• Existence of reference metadata
4. Resource-related issues • Resources adequate
• Have necessary skills
• How to acquire skills
14. 6. Most difficult issues for countries new to SDMX
ISSUE SUGGESTIONS FOR IMPROVEMENT
a. Getting senior management buy-in - resources Develop Business Case for SDMX implementation. Covered in Starter
Kit.
b. Coming to grips with SDMX Information model Provide further examples of national datasets expressed in terms of
SDMX concepts, data structure and concept scheme
c. Which Data Structure Definition (DSD) to use Starter Kit provides some comparative information. Well covered in
SDMX Modelling a Statistical Domain resource.
d. Which implementation tools to use Starter Kit provides some comparative information. More needed
e. Placing data / metadata exchange scenarios in
context of current national IT platform(s)
Provide information on how to assess relationship between current /
future national IT platform(s) where data / metadata reside and various
exchange scenarios. LINK TO OBJECTIVES
f. Identifying pilot project modalities Starter Kit outlines issues to be considered beforehand. These are key
elements of implementation modalities. Emphasise benefits of a pilot.
LEARNING BY DOING.
g. Skill development Covered in Business Case - role of pilot project
Will now quickly discuss issues b. to g.
15. b. Coming to grips with SDMX Information Model
• Coming to grips with SDMX-IM can
be tricky
• Especially for non-IT experts
• Number of SDMX terms / concepts,
artefacts
• Need to relate national tables,
datasets, data flows to SDMX
concepts, etc
• Starter Kit provides lots of
examples
16. Need to provide even more examples of national datasets expressed in terms of the SDMX data structures,
concept schemes
Source: Eurostat
Such as ……
17. c. Which Data Structure Definition to use?
• Choice between global, local, shared DSDs &
between single and multi-domain DSDs
• Existing DSDs outlined in Starter Kit & on
SDMX.org. Not complete
• Global DSDs accessed via SDMX Global
Registry. Shared DSDs accessed via other
SDMX registries (e.g. Eurostat, IMF, etc). No
one-stop shop
• Criteria for selection of appropriate DSD –
refer resource Modelling a Statistical Domain
for Data Exchange
DSD Title Maintenan
ce agency
Nature of
DSD
Domain specific
/ multi-domain
Current
status
Data coverage No. of
dimensions
Balance of Payments (BOP) IMF global Domain specific Operational Balance of payments, external reserves,
international investment position (IIP),
co-ordinated portfolio investment
survey (CPIS), co-ordinated direct
investment survey (CDIS)
16
National Accounts (NA) [30
September 2013]
Eurostat global Domain specific Operational 26
Foreign Direct Investment
(FDI)
OECD global Domain specific Operational 18
Government Finance
Statistics (GFS)
IMF global Domain specific Operational 9
MDGs UNSD global Multi-domain Operational 12
Debt Reporting by
Developing Countries
World Bank global Domain specific Operational DSD developed jointly with COMSEC and
UNCTAD. External debt and selected
foreign assets from creditor, debtor and
market sources and institutions
17
R&D statistics UNESCO* global Domain specific Under
developmen
t
Concept scheme covers government
budget appropriations or outlays for
research and development (GBAORD)
and R&D statistics. DSD is being
developed in cooperation with Eurostat
and the OECD.
Education UNESCO* global Domain specific Under
developmen
t
Concept scheme created covers the
whole of ISCED 2011. DSD is being
developed in cooperation with Eurostat
and the OECD. The maintenance agency
is still to be identified.
International merchandise
trade
UNSD global Domain specific Under
developmen
t
Concept scheme covers the whole
domain, 42 concepts in total. DSD
developed in cooperation with the
OECD, United Nations and Eurostat. The
maintenance agency is still to be
identified. Expected to be finalised in
2015.
26
Eco-Fin IMF shared Multi-domain Operational Used for SDDS Plus 5
Fisheries statistics Eurostat local Multi-domain Operational Includes catch, landings and aquaculture
statistics. DSDs are available on the Euro
SDMX Registry.
UN CountryData UNSD local Multi-domain Operational 9
Culture statistics UNESCO local Multi-domain Under
developmen
t
ICT UNESCO local Domain specific Under
developmen
t
Communications UNESCO local Domain specific Under
developmen
t
National Statistics Data Page
for SDDS Plus
IMF local Multi-domain
Short-term Economic
Indicators
OECD local Multi-domain Comprises two DSDs for: short-term
indicators (prices, real indicators, etc);
infra-annual labour indicators
Labour force statistics ILO local Multi-domain Operational Comprises two separate DSDs
18. d. SDMX implementation tools
Range of implementation tools /
software have been developed by the
Sponsoring Agencies and private
companies:
• SDMX Reference Infrastructure (SDMX-RI)
• SDMX Converter
• IMF SDDS+ tools – used in the Open Data
Platform project with African Development
Bank
• DevInfo
• SDMX Global Registry
• Tools developed by Metadata Technology;
Space-Time Research
Which one(s) should a country use?
More comparative information needed
International perspective
SDMX-RI
IMFSDDS+
Converter
DevInfo
Others
National perspective
SDMX-RI
Converter
IMFSDDS+
DevInfo
Others
Need for information on: relationship,
differences between tools available
19. Data / Metadata exchange
scenarios
• Data exchange: Push mode
• Data Sharing: Pull mode –
using Web service
• Data Sharing: Pull – using
SDMX files
• Data Hub concept based
Data / Metadata exchange
scenarios
• Data exchange: Push mode
• Data Sharing: Pull mode –
using Web service
• Data Sharing: Pull – using
SDMX files
• Data Hub concept based
Implementation based
on current IT platform
Implementation based
on future IT platform
Efficiencygain
objectives
Efficiencygain
objectives
Current IT
Platform
Future IT
Platform
SDMX Technical
Standards
SDMX Content-
Oriented
guidelines
e. Current IT platform and exchange scenarios
SDMX implementation is national IT platform independent. Which data metadata exchange scenario to
use? How conforms with current / future national IT platform where data / metadata reside? Link to
SDMX objectives.
20. f. Pilot project – Tips and Tricks
Tips
• Don’t underestimate time and
resources required.
• Don’t be too ambitious re scale of
project.
• Formulate clear set of objectives
• Identify potential efficiency gains.
• Identify skills required
• Linkage of pilot outcomes to full-
scale implementation
• Analyse pilot outcomes
Issues to be considered when preparing
for a pilot project
Priority
• Domain(s) to be included
• How would SDMX be used for domain(s)
included
• Area(s) within NSO to be main drivers for pilot
project
Other issues [outlined previously]
• Institutional and pilot objectives
• Statistical issues
• IT-related issues
• Resource-related issues
21. g. Resource related issues – Current experience
• Initial SDMX implementation and ongoing maintenance, etc., beyond pilot stage is
resource consuming
• Requires senior management commitment in terms of resources / funding –
therefore, need for a Business Case
• There is a steep learning curve for all involved to come to grips with standards /
guidelines, implementation tools
• Careful consideration of skill needs required even before pilot. An outcome of
pilot project to firm up resource-related issues.
• Most countries in Europe and in Asia-Pacific, Africa that have used SDMX have
required technical support from one of the Sponsoring Agencies – WILL THIS
ALWAYS BE THE CASE?
22. 7. Suggestions – Observations for future
• Understanding of SDMX Information Model probably OK. There is also
sufficient information on SDMX.org and Eurostat websites
• Though perhaps need further examples of national datasets expressed in
terms of SDMX concepts, concept scheme, etc. Process outlined in
presentation by Pellegrino and Suranyi at 2015 Global Conference
• List of existing DSDs now on SDMX.org website. Also includes DSDs under
development. Needs to be kept up-to-date.
• Need for more comparative information on the SDMX Implementation tools
developed by Eurostat, IMF, UN, etc
23. 7. Suggestions – Observations for future (cont.)
• Perhaps further information needed on relationship between current
national IT platforms where data / metadata are currently stored and the
various SDMX data exchange scenarios – push / pull / data sharing / hubs –
link to SDMX implementation objectives
• More detailed step-by-step documentation needed on the use of SDMX
Implementation Tools that are available – e.g. the Eurostat webinars on use
of SDMX Converter
• Need to think through a documentation strategy that would enable
countries (especially developing countries) to implement SDMX and access
/ use implementation tools without need for support from I/Os
Use of common terms and concepts lies at the heart of DDI, SDMX implementation
Enhances interoperability of systems, processes, data / metadata exchange
Enabled through the use of common architcures – GSIM, GSBPM, CSPA
Standard glossaries being developed further