SDMX @ ABS:SDMX @ ABS:Why and How we use SDMXWhy and How we use SDMXGraeme OakleyGraeme OakleyAustralian Bureau of Statist...
Content of PresentationContent of PresentationIntroduction - objectives, overview of ABSIntroduction - objectives, overvi...
"Towards Implementation of SDMX - Jan 9-11, 2007"Objectives of PresentationObjectives of PresentationWhy SDMX fits with A...
Australian Bureau of StatisticsAustralian Bureau of StatisticsAustralias national statistical agency, statisticalAustrali...
ABS PrinciplesABS PrinciplesRelevanceRelevance: good planning with a good: good planning with a goodunderstanding of curr...
ABS Principles (ctd)ABS Principles (ctd)ProfessionalismProfessionalism: exercise highest professional: exercise highest p...
ABS - ClientsABS - ClientsAustralias national statistical agency serving:Australias national statistical agency serving:...
Principles and Standards ApproachPrinciples and Standards ApproachData ManagementData Management - data is visible, acces...
Standards (ctd)Standards (ctd)"Towards Implementation of SDMX - Jan 9-11, 2007"DisseminationDissemination --all publisha...
Case StudiesCase StudiesInternationalInternational - NAWWE with OECD (Note:- NAWWE with OECD (Note:ABS responds to many i...
International - OECD NAWWEInternational - OECD NAWWEInvolvement in pilot project since 2002/03Involvement in pilot projec...
OECDFAME Time SeriesDBABS InformationWarehouseABSStatisticalProductsEXCEL FileHand enteredCurrent Process"Towards Implemen...
OECDFAME TimeSeries DBABS InformationWarehouseABSStatisticalProductsTime Series XMLTransform toSDMXPublish towww.abs.gov.a...
ABS ExperienceABS ExperienceChanged business process requiredChanged business process requiredAlthough supporting the in...
National - RBANational - RBA"Towards Implementation of SDMX - Jan 9-11, 2007"Central Bank for Australia (RBA) - receivesC...
National - RBA (ctd)National - RBA (ctd)ABS provided detailed Labour Force data inABS provided detailed Labour Force data...
ABS RBALabour Forcedata in SDMXformatSQL Server DBholding timeseries dataXSLTSQL ServerIntegrationServices(SSIS)Email with...
What RBA see as benefits?What RBA see as benefits?Efficient data transfer - reduces need for clericalEfficient data trans...
What does RBA want ABS to change?What does RBA want ABS to change?"Towards Implementation of SDMX - Jan 9-11, 2007"Nothin...
VisionVision"Towards Implementation of SDMX - Jan 9-11, 2007"Vision - framework and standard that ABS can useVision - fra...
LessonsLessonsSDMX has the potential to provide integrated dataSDMX has the potential to provide integrated dataexchange ...
IssuesIssues"Towards Implementation of SDMX - Jan 9-11, 2007"Use SDMX for wide range of users, from technicallyUse SDMX f...
ConclusionConclusion"Towards Implementation of SDMX - Jan 9-11, 2007"Move to SDMX V2, recognising need to keep back-Move ...
QUESTIONSANDDISCUSSION"Towards Implementation of SDMX - Jan 9-11, 2007"
Upcoming SlideShare
Loading in …5
×

Sdmx at australian bureau of statistics

441 views
350 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
441
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • SITC - Standard International Trade Classification ICD - International Classification of Diseases and Related Health Problems
  • Reason that a new data load was required is that the Data Structure Definition (DSD) is different in some ways to the way the ABS otherwise disseminates NAB data currently. The ideal for the future will be that the data is loaded only once and downstream transformation processes "shape" it to meet the DSD requirement and separately to meet other requirements.
  • Sdmx at australian bureau of statistics

    1. 1. SDMX @ ABS:SDMX @ ABS:Why and How we use SDMXWhy and How we use SDMXGraeme OakleyGraeme OakleyAustralian Bureau of StatisticsAustralian Bureau of Statistics"Towards Implementation of SDMX - Jan 9-11, 2007"
    2. 2. Content of PresentationContent of PresentationIntroduction - objectives, overview of ABSIntroduction - objectives, overview of ABSPrinciples relevant to ABS data and metadataPrinciples relevant to ABS data and metadatadisseminationdisseminationCase studies - one national, oneCase studies - one national, oneinternationalinternationalVision, Lessons, IssuesVision, Lessons, Issues"Towards Implementation of SDMX - Jan 9-11, 2007"
    3. 3. "Towards Implementation of SDMX - Jan 9-11, 2007"Objectives of PresentationObjectives of PresentationWhy SDMX fits with ABS philosophyWhy SDMX fits with ABS philosophyconcerning principles and standardsconcerning principles and standardsTell our stories about a national andTell our stories about a national andinternational experience using SDMXinternational experience using SDMXDiscuss the future vision, concerns andDiscuss the future vision, concerns andlessonslessons
    4. 4. Australian Bureau of StatisticsAustralian Bureau of StatisticsAustralias national statistical agency, statisticalAustralias national statistical agency, statisticalleadership role in National Statistical Serviceleadership role in National Statistical ServiceABS MissionABS Mission".. assist and encourage informed decision-making, research".. assist and encourage informed decision-making, researchand discussion .."and discussion .."A broad range of statistics on Australian societyA broad range of statistics on Australian societyEconomic, Demographic, Social, Health, Tourism,Economic, Demographic, Social, Health, Tourism,Culture and Leisure, Environment, ...Culture and Leisure, Environment, ...Central and 8 Regional offices, about 2750 staff (FTE)Central and 8 Regional offices, about 2750 staff (FTE)plus 800 interviewersplus 800 interviewers"Towards Implementation of SDMX - Jan 9-11, 2007"
    5. 5. ABS PrinciplesABS PrinciplesRelevanceRelevance: good planning with a good: good planning with a goodunderstanding of current and future needs ofunderstanding of current and future needs ofusers;users; timely and relatable statisticstimely and relatable statisticsIntegrityIntegrity: analysis and interpretation is objective;: analysis and interpretation is objective;always publish our statistics;always publish our statistics; explain and informexplain and inform;;methodologies and approaches open to scrutinymethodologies and approaches open to scrutinyAccess for allAccess for all: strive for accurate and widespread: strive for accurate and widespreadmedia coverage;media coverage; access through range of productsaccess through range of productsand servicesand services"Towards Implementation of SDMX - Jan 9-11, 2007"
    6. 6. ABS Principles (ctd)ABS Principles (ctd)ProfessionalismProfessionalism: exercise highest professional: exercise highest professionalstandards; staff highly trained;standards; staff highly trained; development ofdevelopment ofstatistical standardsstatistical standards; open about the; open about the quality of ourquality of ourstatisticsstatisticsTrust of providersTrust of providers: encouraged to provide accurate: encouraged to provide accurateinformation;information; data is strictly protecteddata is strictly protected; explain why; explain whyinformation is collected; every ABS officer requiredinformation is collected; every ABS officer requiredto give an undertaking of secrecyto give an undertaking of secrecySource: ABS Corporate PlanSource: ABS Corporate Plan"Towards Implementation of SDMX - Jan 9-11, 2007"
    7. 7. ABS - ClientsABS - ClientsAustralias national statistical agency serving:Australias national statistical agency serving:Australian GovernmentAustralian GovernmentState/Territory GovernmentsState/Territory GovernmentsLocal GovernmentsLocal GovernmentsInternational statistical organisationsInternational statistical organisationsBusiness communityBusiness communityEducation, religious, welfare, etc organisationsEducation, religious, welfare, etc organisationsMediaMediaIndividual members of societyIndividual members of society"Towards Implementation of SDMX - Jan 9-11, 2007"
    8. 8. Principles and Standards ApproachPrinciples and Standards ApproachData ManagementData Management - data is visible, accessible,- data is visible, accessible,understandable, reliable, relatable, presentableunderstandable, reliable, relatable, presentableMetadataMetadata - definitions (understanding), discovery,- definitions (understanding), discovery,quality (fitness for purpose)quality (fitness for purpose)ABS Enterprise ArchitectureABS Enterprise Architecture - follow mainstream- follow mainstreamstandards, collaborate & partner where effective,standards, collaborate & partner where effective,promote development of low complexity &promote development of low complexity &maintainable solutions, remove dependence on vendormaintainable solutions, remove dependence on vendorformats, extensible and scalable systems, use emergingformats, extensible and scalable systems, use emergingtechnology as appropriatetechnology as appropriate"Towards Implementation of SDMX - Jan 9-11, 2007"
    9. 9. Standards (ctd)Standards (ctd)"Towards Implementation of SDMX - Jan 9-11, 2007"DisseminationDissemination --all publishable data in ABS Information Warehouseall publishable data in ABS Information Warehousemetadata associated with productsmetadata associated with productsweb site is prime dissemination channelweb site is prime dissemination channelclients can pull data for freeclients can pull data for freedata stored once but published in many commondata stored once but published in many commonformatsformatsStatisticalStatistical - use international standards eg ICD,- use international standards eg ICD,SITC; and frameworks eg SNASITC; and frameworks eg SNA
    10. 10. Case StudiesCase StudiesInternationalInternational - NAWWE with OECD (Note:- NAWWE with OECD (Note:ABS responds to many information requests fromABS responds to many information requests frominternational organisations, in variety of formats -international organisations, in variety of formats -we are seeking an efficient solution that aligns withwe are seeking an efficient solution that aligns withour principles and standards)our principles and standards)NationalNational - Reserve Bank of Australia (RBA)- Reserve Bank of Australia (RBA)[A large, sophisticated and important user of ABS[A large, sophisticated and important user of ABSdata. There are a number of other users of thisdata. There are a number of other users of thistype eg Treasury.]type eg Treasury.]"Towards Implementation of SDMX - Jan 9-11, 2007"
    11. 11. International - OECD NAWWEInternational - OECD NAWWEInvolvement in pilot project since 2002/03Involvement in pilot project since 2002/03Tasks include:Tasks include:Set up metadata and load National Accounts data toSet up metadata and load National Accounts data toABS Information Warehouse to suit agreedABS Information Warehouse to suit agreedNAWWE Data Structure DefinitionNAWWE Data Structure DefinitionExtract from ABSIW and transform to agreedExtract from ABSIW and transform to agreedSDMX message structure (XML format)SDMX message structure (XML format)Place on ABS website (recently decommissionedPlace on ABS website (recently decommissionedweb page for original case study and yet to put upweb page for original case study and yet to put upone for current study)one for current study)"Towards Implementation of SDMX - Jan 9-11, 2007"
    12. 12. OECDFAME Time SeriesDBABS InformationWarehouseABSStatisticalProductsEXCEL FileHand enteredCurrent Process"Towards Implementation of SDMX - Jan 9-11, 2007"
    13. 13. OECDFAME TimeSeries DBABS InformationWarehouseABSStatisticalProductsTime Series XMLTransform toSDMXPublish towww.abs.gov.auFuture Process"Towards Implementation of SDMX - Jan 9-11, 2007"
    14. 14. ABS ExperienceABS ExperienceChanged business process requiredChanged business process requiredAlthough supporting the initiative, ABS NationalAlthough supporting the initiative, ABS NationalAccounts group could not always provideAccounts group could not always provideresources to do necessary setup workresources to do necessary setup workLong periods on inactivity whilst OECD workedLong periods on inactivity whilst OECD workedon new Data Structure Definitionon new Data Structure DefinitionExperimental SDMX V1 files were deliveredExperimental SDMX V1 files were deliveredNow have SDMX V2 DSD to understand andNow have SDMX V2 DSD to understand andimplementimplement"Towards Implementation of SDMX - Jan 9-11, 2007"
    15. 15. National - RBANational - RBA"Towards Implementation of SDMX - Jan 9-11, 2007"Central Bank for Australia (RBA) - receivesCentral Bank for Australia (RBA) - receivesmainly time series data from many sourcesmainly time series data from many sourcesMany formats and different frequenciesMany formats and different frequenciesPreviously maintained their time series data inPreviously maintained their time series data inan EXCEL based spreadsheet systeman EXCEL based spreadsheet systemIn past year, developed new system toIn past year, developed new system tocentralise time series data storage andcentralise time series data storage andretrieval, based on SQL serverretrieval, based on SQL server
    16. 16. National - RBA (ctd)National - RBA (ctd)ABS provided detailed Labour Force data inABS provided detailed Labour Force data inSDMX V1 format as a trial (RBA currently takesSDMX V1 format as a trial (RBA currently takesLFS data in a large number of EXCEL files andLFS data in a large number of EXCEL files andsome multidimensional cubes)some multidimensional cubes)The RBAs new system has been demonstratedThe RBAs new system has been demonstratedto internal business areas, and acceptedto internal business areas, and acceptedRBA want to extend use of SDMX to other ABSRBA want to extend use of SDMX to other ABSseries and improve the timelinessseries and improve the timeliness"Towards Implementation of SDMX - Jan 9-11, 2007"
    17. 17. ABS RBALabour Forcedata in SDMXformatSQL Server DBholding timeseries dataXSLTSQL ServerIntegrationServices(SSIS)Email with file attachedWhat RBA do with SDMX files"Towards Implementation of SDMX - Jan 9-11, 2007"RBAeconomists
    18. 18. What RBA see as benefits?What RBA see as benefits?Efficient data transfer - reduces need for clericalEfficient data transfer - reduces need for clericalintervention when dealing with spreadsheets, canintervention when dealing with spreadsheets, caneasily manipulate with programseasily manipulate with programsMetadata rich - SDMX allows more metadata toMetadata rich - SDMX allows more metadata tobe conveyed compared to spreadsheetsbe conveyed compared to spreadsheetsAble to handle changes in time series metadataAble to handle changes in time series metadataand data more easily - can programmatically checkand data more easily - can programmatically checkmetadata for changesmetadata for changes"Towards Implementation of SDMX - Jan 9-11, 2007"
    19. 19. What does RBA want ABS to change?What does RBA want ABS to change?"Towards Implementation of SDMX - Jan 9-11, 2007"Nothing related to the transfer format - SDMXNothing related to the transfer format - SDMXWant the SDMX files on ABS web site so canWant the SDMX files on ABS web site so candownload at 11.30am (embargo time) ratherdownload at 11.30am (embargo time) ratherthan wait for emailthan wait for emailWant ABS to extend use of SDMX to otherWant ABS to extend use of SDMX to otherstatistical seriesstatistical seriesAdditional information related to identification ofAdditional information related to identification ofdiscontinued series, code setsdiscontinued series, code sets
    20. 20. VisionVision"Towards Implementation of SDMX - Jan 9-11, 2007"Vision - framework and standard that ABS can useVision - framework and standard that ABS can usefor all our international reporting; and also to usefor all our international reporting; and also to usewith large major national users of ABS data.with large major national users of ABS data.SDMX is suitable as the primary disseminationSDMX is suitable as the primary disseminationvehicle for machine readable aggregate and timevehicle for machine readable aggregate and timeseries datasets for both national and internationalseries datasets for both national and internationalclients. The SDMX format files will become ABSclients. The SDMX format files will become ABSdata products held on the web site for free access.data products held on the web site for free access.
    21. 21. LessonsLessonsSDMX has the potential to provide integrated dataSDMX has the potential to provide integrated dataexchange across organisational boundaries.exchange across organisational boundaries.Challenge is not the SDMX technology, butChallenge is not the SDMX technology, butdealing with meaning and mapping issuesdealing with meaning and mapping issuesRecognise costs involved in implementation,Recognise costs involved in implementation,especially mapping local metadata and dataespecially mapping local metadata and datastructures to international requirement in eachstructures to international requirement in eachsubject domain, and the importance of engagingsubject domain, and the importance of engaginglocal subject matter expertslocal subject matter experts"Towards Implementation of SDMX - Jan 9-11, 2007"
    22. 22. IssuesIssues"Towards Implementation of SDMX - Jan 9-11, 2007"Use SDMX for wide range of users, from technicallyUse SDMX for wide range of users, from technicallycompetent to small data integrator. A short shallowcompetent to small data integrator. A short shallowlearning curve with plenty of support, such as easylearning curve with plenty of support, such as easydocumentation, examples, tutorials, and toolsdocumentation, examples, tutorials, and toolsSimple time series identifier, in addition to the DataSimple time series identifier, in addition to the DataStructure Definition - eg local identifier as part ofStructure Definition - eg local identifier as part ofexchange - the key back to internal data holdings;exchange - the key back to internal data holdings;existing international code for each series used inexisting international code for each series used incurrent exchanges - doesnt vary from country tocurrent exchanges - doesnt vary from country tocountry.country.
    23. 23. ConclusionConclusion"Towards Implementation of SDMX - Jan 9-11, 2007"Move to SDMX V2, recognising need to keep back-Move to SDMX V2, recognising need to keep back-end systems independent of version changes.end systems independent of version changes.Importance of establishing business benefit ofImportance of establishing business benefit ofSDMX - it is not just a technology.SDMX - it is not just a technology.Ideal of a common framework that would guideIdeal of a common framework that would guideeach statistical domain to determine their exchangeeach statistical domain to determine their exchangerequirements and definitions.requirements and definitions.Need for good registration processes around allNeed for good registration processes around allaspects of SDMX, and involvement of NSOs inaspects of SDMX, and involvement of NSOs ingovernance.governance.
    24. 24. QUESTIONSANDDISCUSSION"Towards Implementation of SDMX - Jan 9-11, 2007"

    ×