Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
_TS_SDMX_Thailand.pdf
1. 1
Innovations in Data Dissemination in ASIA-PACIFIC
Ms. Taweesap Srikwan
Statistician, Senior Professional Level
National Statistical Office, Thailand
SDMX data portal with .Stat Suite
2. 2
2
Thailand National Statistical Office
01
03
02
Why .Stat Suite ?
04
The process of implementation
The Challenges
Adopting to other tools related .Stat Suite
The Contents
05 The next step
3. 3
3
Thailand National Statistical Office
Why .Stat Suite ?
• The open-source tool and supporting GSBPM and SDMX standard
• Co-development by the global community
• Technology partners program facilitation for service development
• Enhancing functionalities by connecting to Power Bi, MS Excel, R, Python.
TNSO’s views
Before After
.Stat Suite is an SDMX native standard-based, componentized, open-source
platform for the efficient production and dissemination of high-quality
statistical data. (https://sdmx.org)
Based on the SDMX Reference Infrastructure provided by Eurostat
http://statv2.nic.go.th/
https://stathub.nso.go.th/
SAE
NSI Client .Stat Suite
2013 2019
4. 4
4
Thailand National Statistical Office
Stakeholders
Data Provider
Data Consumer
Agency
A Data Model is represented by a
Data Structure Definition(DSD)
A Data Flow represents
the Data provider and
Data Structure
Definition
Statistics Sharing Hub (.Stat Suite)
disseminates both Data and Metadata in
various format (A dataset and the
description of the dataset)
Dimension, Attribute, Measure
The process of implementation
SDMX Artefacts such as Concept Scheme,
Category Scheme, Categorization, etc.
Labor statistics Data Flow BPMN Matrix Sheet
Labor Category by NSS
Concept Scheme
TNSO will conduct a pilot project to exchange labor statistical data with Ministry of Labour
5. 5
Thailand National Statistical Office
Fusion Registry
SDMX Structure Generation
SDMX Data Preparation
Excel2CSV Excel Power Query
SDMX Artefact Repository
Thailand Fusion Registry
https://sdmx.nso.go.th/FusionRegistry
SDMX Matrix Generator
TNSO will conduct a pilot project to exchange labor statistical data with Ministry of Labour
https://stathub.nso.go.th
Automatically
Format Data Source
= manually
https://dlm-oshub.nso.go.th/
Thailand Statistics Sharing Hub
by .Stat Suite
by .Stat Suite
The process of implementation
5
6. 6
Thailand National Statistical Office
https://stathub.nso.go.th
Data Explorer on .Stat Suite
Labour
8 Datasets
ICT
5 Datasets
Population
2 Datasets
Natural resources
and environment
4 Datasets
Statistical sectors
(NSS)
SDGs
16
Datasets
Get Data from API Query with
SDMX link.
Microsoft Power BI
.Stat DLM Excel-Addin is a facilitator of
data editing and sharing
Microsoft Excel
Develop a generic SDMX
web-client
R Programming
There is a library can directly
connect to
SDMX Web Services import and
transform
SDMX data into native formats
Python
an Apache 2.0-licensed Python
library that implements SDMX
Panda SDMX
Get data from API
Query with SDMX link
From .Stat Suite
The process of implementation
6
Supporting Thailand National Statistical System
https://app.powerbi.com/links/_dY4sCqELf?ctid=f64d3183-7691-4fcb-
a5ef-39ebc082bff9&pbi_source=linkShare
https://app.powerbi.com/groups/me/reports/517a65bc-b9a8-4d0f-9309-
68b51703a746?ctid=f64d3183-7691-4fcb-a5ef-39ebc082bff9&pbi_source=linkShare
https://app.powerbi.com/groups/me/reports/1d42f1e9-e043-47ef-adb2-
2f5112e830ed?ctid=f64d3183-7691-4fcb-a5ef-39ebc082bff9&pbi_source=linkShare
https://app.powerbi.com/groups/me/reports/2fb7bdd2-a9ae-4610-b1c2-
a387e2e83c67?ctid=f64d3183-7691-4fcb-a5ef-39ebc082bff9&pbi_source=linkShare
7. 7
7
Thailand National Statistical Office
Dissemination of
Statistical data in non-
machine readable formats
No agreement of Common
Codelist Standards
Connectivity among
statistical units
Apply the SDMX standard to
government agencies
statistical data
Personnel skill gaps
Knowledge transfer
Critical
Points
The challenges
8. 8
Thailand National Statistical Office
Adopting to other tools related .Stat Suite
Cabinet Resolution
In 2021-2022 : TNSO has developed the
Government Data Catalog Platform for collecting
the list of data and metadata from all Agency
Data Catalogs.
In 2022 : Developing the central
platform for Government Data
Catalog and Directory services Data Catalog for All
Government Data Catalog Platform (GD Platform)
Agency Data Catalog (CKAN Open-D) GD Catalog Registration & Management GD Catalog Services
(CKAN Open-D)
Utilization
In 2020-2022 : TNSO has supported
government agencies In developing their
data catalog.
Agency Data
Catalog A
Agency Data
Catalog B
Agency Data
Catalog C
Statistics Sharing Hub
.Stat Suite
CKAN Open-D is a Thailand extension of
CKAN Software (https://ckan.org/),
which is an open-source Data Management
System software.
Government Data Service
Framework
Getting the readable machine data file
and upload to .Stat Suite
8
9. 9
9
Thailand National Statistical Office
Cabinet Resolution
In 2021-2022 : TNSO has developed the
Government Data Catalog Platform for collecting
the list of data and metadata from all Agency
Data Catalogs.
In 2022 : Developing the central
platform for Government Data
Catalog and Directory services Data Catalog for All
Government Data Catalog Platform (GD Platform)
Agency Data Catalog (CKAN Open-D) GD Catalog Registration & Management GD Catalog Services
(CKAN Open-D)
Utilization
In 2020-2022 : TNSO has supported
government agencies In developing their
data catalog.
Agency Data
Catalog A
Agency Data
Catalog B
Agency Data
Catalog C
Government Data Service
Framework
Data
Feeding
Staging
Dataset
Prepare
Dataset
Service
(SDMX-JSON,
SDMX-XML,
SDMX-CSV)
Primary Data
Secondary data Dashboard Template
Dataset
Loader
Secondary data
Power BI SDMX
R-SDMX
Python SDMX
Sharing
Hub
.Stat Suite
The next step
Project in 2023
The Data Integration
• Supporting the government
agencies to create the
machine readable data file
and disseminate in Agency
Data Catalog.
• promoting the exchange of
Official Statistics according
to the SDMX standards for
mutual benefits
MS Excel
10. 10
10
Thailand National Statistical Office
Statistics Sharing Hub
Government Data Catalog
https://gdcatalog.go.th https://stathub.nso.go.th/