Presentation of CORE Research Network (ESSnet) at MSIS (Management of Statistical Information Systems) 2011 - Luxembourg

1. CORE ESSnet
(COmmon Reference Environment)
progress report
Carlo Vaccari
Istat - Italy
MSIS Meeting - Luxembourg May 23-25 2011 1

2. Outline
Introduction and history
CORE objectives
CORE where we are
Architecture implementation
Information model
CORE and SDMX
CORE and GSIM
MSIS Meeting - Luxembourg May 23-25 2011 2

3. CORA ESSnet
Financed by Eurostat under 2009
Statistical Workprogramme
Countries involved: it (coordinator),
ch, dk, lv, nl, no, se
Duration: October 2009 - October
2010
MSIS Meeting - Luxembourg May 23-25 2011 3

4. CORA Technical
Architecture
CORA Model: two dimensions
Functional dimension
Construction dimension
Functional dimension
Adoption of GSBPM 4.0
9 subprocesses of level 2
1
2 3 4 5 6 7 8 9
Specify
Design Build Collect Process Analyse Disseminate Archive Evaluate
Needs
MSIS Meeting - Luxembourg May 23-25 2011 4

5. Construction Dimension:
Layers
A domain of interest documented by
Figures
statistical products
Time Series Statistical series over time
Integrated or simple statistical product for a
Statistic given time
Population A population at a given time
Unit A statistical unit at a given time
Variable A statistical variable at a given time
A logical representation of the value of a
Value
variable
MSIS Meeting - Luxembourg May 23-25 2011 5

6. CORA Model Grid
Statistical processes compliant to CORA model are
intended to be designed by statisticians
MSIS Meeting - Luxembourg May 23-25 2011 6

7. After CORA…CORE!
COmmon Reference Environment (CORE),
financed by Eurostat under 2010
Statistical Workprogramme
Countries involved: it (coordinator), fr, nl,
no, pt, se
Duration: December 2010 - January 2012
MSIS Meeting - Luxembourg May 23-25 2011 7

8. CORE Principal
Outcomes
Environment for the definition and
execution of statistical processes
Definition of a process in terms of
services selected from an available
repository
Execution of the composed workflow
MSIS Meeting - Luxembourg May 23-25 2011 8

9. CORE Outcomes:
Design
CORA model → CORA information
model
Design of CORE services and
processes
MSIS Meeting - Luxembourg May 23-25 2011 9

10. CORE Outcomes:
Implementation
Selection of available middleware solutions for
process execution
Realization of an environment able to permit the
execution of processes:
Interfaces (GUIs) for defining CORE processes for
statistical users
Integration APIs
Repository of integration layers
MSIS Meeting - Luxembourg May 23-25 2011 10

11. CORE Outcomes:
Testing
Realization of processes starting from
services implementing some GSBPM phase
Evaluation of costs related to integration
Prototype implementation (to be
engineered)
MSIS Meeting - Luxembourg May 23-25 2011 11

12. CORE Architecture (1)
GUIs to support modelling of CORE processes
according to the CORA grid
Modeling & control flow constructs
Drag & drop facilities for process design
Global schema
Implementation: we are evaluating the usage of
an open process editor tool (Oryx -
http://bpt.hpi.uni-potsdam.de/Oryx/WebHome )
MSIS Meeting - Luxembourg May 23-25 2011 12

13. CORE Architecture (2)
Process Runtime
Controlled execution of services
Implementation: integration of existing workflow
solutions, currently in evaluation phase
Service Runtime
Integration APIs (in-out data transformation)
Service execution
Implementation: CSV and SQL data
transformations are currently being implemented
Service Repository
Deployment of services
MSIS Meeting - Luxembourg May 23-25 2011 13

14. CORE Information Model (1)
First draft of CORE information model
Del. 2.1 released: requirements for the
model of the interface through which
statistical services will communicate
Information Model to be released,
currently in discussion phase
MSIS Meeting - Luxembourg May 23-25 2011 14

15. CORE Information Model (2)
Design Principles (in discussion):
Rectangular data sets (rows & columns)
Strong typing (data, rules, parameters)
Dataset kinds (eg micro/aggreg)
Free-style arguments (eg scripts tool
dependent)
Other (service arguments and infos)
MSIS Meeting - Luxembourg May 23-25 2011 15

16. CORE and SDMX
Both initiatives foster standardization
CORE
Focus on standardization of processes and data
exchanges (mainly) intra-NSI
SDMX
Focus on standardization of processes and data
exchanges (mainly) inter-NSIs (or between NSIs
and international organizations)
MSIS Meeting - Luxembourg May 23-25 2011 16

17. CORE and SDMX - 2
CORE
Focus on all phases of statistical processes
Both micro and macro data considered
SDMX
Focus (mainly) on dissemination phase
Mainly macro data considered
MSIS Meeting - Luxembourg May 23-25 2011 17

18. Information Model
Both propose an information model
CORE information model
Takes explicitly process dimension into account
through GSBPM
Data dimension
SDMX information model
Mainly focused on data dimension
MSIS Meeting - Luxembourg May 23-25 2011 18

19. CORA Information
Model
Service +belongs_to Layer
level
+contains
1
+implements +belongs_to
n
Constructor
prescript +has
+input Element
n
n
output.belongs_to.level =
input.belongs_to.level + 1 +output n
Construct Object
+represented_by
1
Figure Time series Statistic Population Unit Variable
MSIS Meeting - Luxembourg May 23-25 2011 19

20. SDMX Data & Metadata Information
Model
Data or Metadata
Structure
Definition
uses specific
data or
metadata can have child
structure categories
can be linked with
Data or Data or categories from multiple
category schemes
Metadata Metadata Category
Set conforms to business Flow
rules of the data or
metadata flow comprised
can get data of subject
from multiple or reporting
data providers categories
Data Provision Category
Provider Agreement Scheme
MSIS Meeting - Luxembourg May 23-25 2011 20

21. On Information Models
Different abstraction levels
CORE
“Higher” modelling level
E.g.: statistics as tabular data
SDMX
“Lower” modeling level
E.g.: aggregated data set with dimensions, attributes
and measures
MSIS Meeting - Luxembourg May 23-25 2011 21

22. Open Issues - 1
Can we use SDMX for micro and macro
data exchanges in a CORE process?
Need for mapping of information
models
MSIS Meeting - Luxembourg May 23-25 2011 22

23. On-Going Work - 1
CORE implementation scenario within
Istat
Main phases: Sample selection and
allocation
CORE wrapping of available SAS and
R procedures
MSIS Meeting - Luxembourg May 23-25 2011 23

24. On-Going Work - 2
Design and implementation of CORE
Integration APIs
Possible in/out SDMX translations
CORE TOOL
SDMX IAPI TOOL IAPI SDMX
MSIS Meeting - Luxembourg May 23-25 2011 24

25. Open Issues - 2
What about metadata?
CORE: Data and metadata managed in the
same way
SDMX:
Distinction between structural metadata and
reference metadata
Dedicated effort for metadata management
MSIS Meeting - Luxembourg May 23-25 2011 25

26. Collaboration between
CORE/SDMX ESSnets
CORE planned deliverable on “Feedbacks
on SDMX Usage in CORE”
Periodical meetings inside Istat between
coordinators of the two ESSnets
Exchanges of resources between the two
ESSnets
MSIS Meeting - Luxembourg May 23-25 2011 26

27. CORE and GSIM
GSIM: Generic Statistical Information Model
deliverable from OCMIMF Operationalising a
Common Metadata/Information Management
Framework activity inside Statistical Network
Ambiguity on the acronym: reference to
“generic statistical information model” in CORE
ESSnet proposal
In March started activity to clarify relationships
(thanks to J.P.Kent and A.Hamilton)
MSIS Meeting - Luxembourg May 23-25 2011 27

28. CORE and GSIM
First analysis and discussions: the deliverables
from the two initiatives are complementary in
intent and do not overlap in concept
Necessary to avoid gaps and/or duplications
and ensure the complementary relationship
MSIS Meeting - Luxembourg May 23-25 2011 28

29. CORE Information
Model
CORE will define a very generic information model
(CORE-IM) for the interface through which statistical
services will communicate with each other within
the framework of the CORA model
As a communication protocol, CORE-IM focuses on
the “postal envelope” used when passing
information between services, rather than focusing
in detail on the information being communicated (ie
what is inside)
MSIS Meeting - Luxembourg May 23-25 2011 29

30. CORE and GSIM
CORE-IM current hypothesis: to support a flag to
indicate if the information being communicated is
described within GSIM
→ without claiming to align the semantics of the
content (eg, “classification”), but only to alert a
consuming service which “understands” GSIM that
it can relate the content to GSIM
MSIS Meeting - Luxembourg May 23-25 2011 30

31. Complementary nature
CORE-IM supporting semantic interoperability at a
very high, abstract level (“here is an information
object, along with the ‘envelope’ information about
it”) where GSIM can provide greater semantic
precision to a subset of information objects
communicated using CORE
CORE supporting communication between services
→ substantial interoperability benefits
Information aligned with GSIM semantics
MSIS → further level of interoperability
Meeting - Luxembourg May 23-25 2011 31

32. Complementary nature
GSBPM reference model for statistical business
processes
GSIM reference model for information input to, used
by and produced by those processes
Models are independent → it's possible to use one
without the other
CORE-IM recognizes and uses GSBPM and (hopefully)
will do the same with regard to GSIM, giving them a
potential contact point
MSIS Meeting - Luxembourg May 23-25 2011 32

33. Coordination in
practice
Need to maximize the extent to which these
synergies are achieved in practice:
Members in common (no, se)
ABS leader for the OCMIMF and observer in CORE
CORE members external reviewers for GSIM
material
CORE WP2 “co-ordination input” from the OCMIMF
collaboration team in regard to deliverables
Half day session at METIS workshop (October)
presenting CORE and OCMIMF works to external
metadata specialists
Common documents in preparation
...
MSIS Meeting - Luxembourg May 23-25 2011 33