History In year 1949, the Bureau of Statistics was formed under the Statistics Ordinance with the mergence of the Statistics Sections in both Central Trade Registry and Department of Agriculture . The bureau was headed by the Registrar of Statistics with the functions to collect, process and publish data on foreign trade and production of crops in estates. This marked the beginning of a centralised data collection system in the country. The Statistics Ordinance 1949 was repealed when the Statistics Act 1965 was passed by the Parliament and the Bureau of Statistics was to be known as the Department of Statistics headed by a Chief Statistician. The new act was more comprehensive and under this act, the authority of the Department to collect statistics was further strengthen. Year 2009 – the review of the Statistics Act 1965 (Revised – 1989) Revision of the act Empower DOSM to have access of administrative data from OGAs To strengthen the statistical system by giving task to DOSM to coordinate statistical activities among OGAs in term of definition, methodology and analysis At present, DOSM is the main Government agency entrusted to generate official data for the country for the purpose of policy making and national planning.
DOSM currently has approximately 50 different applications, the NEWSS Project will focus on revamping the 6 major applications which are :- Labour Force Survey Monthly Manufacturing Survey 30 Economy Census/Surveys (SSE) Enterprise Frame Household Frame Codes and clasification With NEWSS, the features, functions and process flows of these disparate applications would be streamline into a single enterprise system which would support the entire survey process inline with the ISSF :-
ICT Strategic Planning (ISP) - A process for developing a strategy and plans for aligning enterprise level information systems with the business strategies of DOSM. It also inline with Our National’s ISP. NEWSS is a by-product from the ICT (Information and communication technologies) projects identified in DOSM’s ISP. In August 2008, NEWSS project commenced. The Integrated Statistical Systems Framework (ISSF) was formulated to provide a structured and standardized framework that would support the business functionalities for DOSM pre-collection, collection, processing, analysis and dissemination at both the HQ and State offices NEWSS would be DOSM’s first implementation in compliance to the ISSF and would lay the foundation of the enterprise architecture for all future DOSM systems . The Architecture includes a centralized depository of statistical information including standard codes and classifications that would be used by DOSM for all its processing needs. NEWSS would have alerts, reminders, escalation and other mechanism for DOSM to efficiently monitor, manage and control the surveys and census carried out by the department. Validation rules would also be built into the system to improve the accuracy of the stats information. With the monitoring and validation functionalities provided by NEWSS, DOSM would be able to improve the timeliness and reliability of the stats information. NEWSS would also be the conduit to increase the reach of dissemination of stats information to DOSM stakeholders
The framework – all the identified process involve in the survey or census build under same platform. So, others system will be ‘plug in’ to the frame work easily Centralised databases for metadata, frame and other codes are as a central look up reference for all DOSM survey Ensure the development of application produces quality statistic using standards mechanism, that shall increase efficiency and shorten the process of developing the census/surveys on the same platform. Provide a centralized repository to host the code and classification. Versioning of code and classification available for code that evolve overtime Pre-collection – aktiviti-aktiviti persediaan untuk pengumpulan data seperti menyediakan borang soal selidik, menyediakan rangka dan memilih set sampel dan maklumat untuk berhubung (alamat, nombor telefon dan lain-lain); b. Collection – merangkumi aktiviti-aktiviti pengumpulan data, validasi dan verifikasi data, tangkapan data, imputasi data dan pembetulan secara online dan menangani masalah non-respondent ; c. Processing – merangkumi aktiviti-aktiviti untuk menyediakan data bersih di peringkat nasional, seperti statistical editing and imputation, calendarization dan weighting and estimation ; d. Analysis – merangkumi aktiviti-aktiviti penjadualan dan semakan serta jaminan kualiti; dan e. Dissemination – aktiviti-aktiviti penyebaran maklumat melalui pelbagai saluran seperti penerbitan, laman web Jabatan, data dalam bentuk elektronik dan lain-lain serta menyimpan semua maklumat ini secara berpusat. Setiap proses mempunyai kaedah-kaedah yang boleh diguna oleh lebih daripada satu penyiasatan/ banci (atau perisian yang boleh dikongsi). Selain itu, pangkalan data digunakan untuk menghantar data dari satu proses ke proses seterusnya. Berikut adalah pangkalan-pangkalan data yang terdapat di sini: a. Survey Respondent Database – mengandungi maklumat berkenaan set sampel (responden) penyiasatan; b. Response Database – mengandungi maklum balas penyiasatan dan data sekunder; c. Clean Microdata Database – mengandungi data bersih (nasional) dari maklum balas penyiasatan dan data sekunder; d. Macrodata (Output) Database – mengandungi data yang telah diagregat; dan e. JP Data Warehouse – diperlukan untuk membantu analisis crosssectional dan time series . Information Support Infrastructure yang memastikan penggunaan metadata dan data rujukan yang sama dalam penghasilan statistik. Ia mempunyai komponen-komponen berikut: a. Metadata Databases yang mempunyai: i. Business metadata – mengandungi penerangan dan lain-lain maklumat mengenai sesuatu variable / label yang digunakan dalam penghasilan statistik dan yang disimpan di dalam pangkalan data. Ia juga boleh memastikan bahawa sesuatu konsep itu mempunyai maksud yang sama dalam semua penyiasatan dan banci yang dijalankan; ii. Technical metadata – digunakan semasa fasa pembangunan dan penyenggaraan projek; b. Reference registries - mengandungi rangka-rangka yang digunakan dalam penyiasatan di mana pada masa ini terdapat dua frame iaitu SIDAP dan SPDRB; dan c. Other survey-related databases - yang mengandungi kod-kod industri, pekerjaan, pendidikan dan lain-lain yang digunakan dalam penghasilan statistik serta pangkalan data mengandungi maklumat status sesuatu sistem penyiasatan atau banci yang sedang dijalankan. Untuk menyokong operasi penghasilan statistik ini terdapat juga Integrated Statistical Workflow database yang merakam prosedur operasi ( automated and manual ) sesuatu sistem. Aplikasi-aplikasi sedia ada yang dicadangkan untuk dinaik taraf dengan menggunakan Integrated Statistical Systems Framework adalah seperti berikut: 1. Sistem Akaun Kerajaan (GA); 2. Sistem Bersepadu Untuk Perangkaan Perkapalan dan Perdagangan (IPSTS); 3. Sistem Bersepadu Untuk Perangkaan Perkapalan dan Perdagangan (IPSTS) bagi Sabah dan Sarawak; 4. Sistem Penyiasatan Tenaga Buruh / Migrasi (SPTB/M); 5. Sistem Banci; 6. Sistem Banci Pedagangan Edaran 2002; 7. Sistem Tangkapan Data Penyiasatan Binaan Suku Tahunan (BST); 8. Sistem Indeks Import dan Eksport (SIIE)/ Indeks Perdagangan Luar Negeri (IPLN); 9. Sistem Penyiasatan Bulanan Perdagangan Edaran (MDT); 10. Sistem Penyiasatan Pembuatan Bulanan (SPPB); 11. Sistem Soalselidik Ekonomi; 12. Sistem Penyiasatan Pelaburan dan Perkhidmatan Antarabangsa (SPPPA); 13. Sisterm Penyusunan Akaun (SPAK); 14. Sistem Integrasi Perangkaan Penting; 15. Sistem Banci Getah Bulanan; 16. Sistem Pangkalan Data Alam Sekitar (PDAS); 17. Sistem Indeks Harga Pengguna; 18. Sistem Penyiasatan Perbelanjaan Isi Rumah (HES); 19. Sistem Banci Estet-estet Getah Tahunan (CEQ); 20. Sistem Akaun Negara Muktamad (FNA); 21. Sistem Jadual Perdagangan Akaun Negara (NATT); 22. Sistem Indeks Harga Pengeluar (IHPR) ( Base 89 ); 23. Sistem Penyiasatan Pendapatan Isi Rumah (HIS); 24. Sistem Suku Tahunan Akaun Negara (QNA); dan 25. Sistem Indeks Harga Pengeluar (IHPR) ( Base 2000 ).
Generation of the pre-survey data ( Pre-collection ) Online and offline collection of survey data ( Collection ) Processing of data , ( Processing ) Analysis of data ( Analysis ) Dissemination of the results ( Dissemination ) Maintenance of all the required services, code references and classification in the system would fall under the Centralization Maintenance Module in ISSF. It replaces the System Computer Assisted Coding (SCAC) and seeks to expand its functionalities and ease of use to maintain all reference codes and parameters used in the system. ISS – Information Support System The Supporting Systems within NEWS are :- The Enterprise Portal which would provide a single point to access the contents and records of the various NEWSS Systems. The Geographical Information system and Business Intelligence Solutions to support the analysis process. The Performance Management solution would allow DOSM to use a balanced scorecard KPI to assess the performance of all survey activities DOSM has requested that NEWSS be able to promote collaboration and sharing of information within the organization . The Knowledge Management solution seeks to enable this by providing a platform for NEWSS Users to collaborate and to access the NEWSS contents which can be in the format of : - Structured Data Unstructured Data Digital Maps
for pre collection - At the upper –- methodology division identify sampling, update survey project and also filter updated and selected data frame supplied by state office and other agency. All the information kept in the frame database Collection – doing the field work NEWSS is developed around an Integrated Statistical Systems Framework (ISSF) that focuses on reusable processes across different type of census or surveys, it provides the main business functionalities of DOSM on pre-collection, collection, processing, analysis and dissemination within a single platform that reduce tremendously on the time required in producing the quality statistics. Frame Management To produce quality statistics, right sample shall be produced that well represent the total population (be it establishment, enumeration block or living quarter). Frame management focuses on merging and consolidating information from others agencies (e.g. Registration of Company, Registration of Business, Employees Provident Fund etc) and collection of the information by the team within DOSM with the main vision of providing latest information to serve as the base in sample generation. Pre-Collection When a survey/census started, the project details and activities (with target completion date) are captured into the pre-collection module to facilitate the management and monitoring purposes. Comprehensive sampling techniques are built within the solution to allow the Methodology Division that utilizes the information in frame to produce the good sample in short timeframe. E.g. of in probability sampling technique - Dalenius Hodges method for obtaining the Optimum Points of Stratification, Neyman, Equal, Proportional, Optimum in sample calculation, random or systematic sample selection etc. The sample produced is directly accessed by the state officers the move forward with the collection activity. Collection Collection module focuses on managing the field work in collecting the survey data, be it from the end respondent or other agencies that provide could provide the data accordingly. It tracks the officers that handle the individual sample case and provide status update of the sample case that has been handled. From management perspective, the SMD able to visualize quickly whether the quality statistic can be produced via the response rate obtained from this cycle and take corrective/follow action quickly to rectify the situation. e.g 1. if the collection of the specify state officer doesn’t reach expected number of response cases, more details follow up can be done with the relevant state office. e.g.2. if the response rate not able to meet expected number due to many establishments no responding (e.g. not able to trace or not co-operating), extra sample could be added to ensure the response number meet the target (this apply to only sample produced via non-probability sampling – i.e. cut off technique) eSurvey also available in this cycle to allow the respondent to provide the survey data via online system, directly reduce the cost in collection activity. Processing Processing process focus on entering the answer in questionnaires to the system. The speed data entry component (offline Data entry component) is available to facilitate entering of data and avoid dependency of the network availability while able to integrate the data into the central repository. The data will go through a comprehensive validation process to ensure the cleanliness of data before being use to produce index or statistic. On survey that produce sample produced via probability sampling, the overall process of weighted data processing also carried here in producing the statistical figure that would represent the whole population. Analysis Analysis focus on analyzing the clean data with the support of Relative Standard Error (RSE) report. It produces the tabulation required publication and customized format required stakeholder. Dissemination Dissemination module focuses on managing and serving the publication and contents to the public and stakeholder. Provide auto notification of the publication produced to the interest parties. Provide the online purchase with online payment on statistical information that allow the outreach of the information to wider group of user and serving the information required in shorter timeframe. Provide subscription model of the publication, allowing the interest parties to download themselves when being notified on the release of the publication. Improve the time required in producing the requested data via online workflow processes that coordinate the parties who participate in the process of preparing the data. Provide a complete suite of managing and tracking the interest party on type of publication/data, subscribers, purchasers, stakeholders, transactions and delivery status. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Complementing the ISSF on assisting the user in performing the effective and efficient process is the ISS (information support systems) . ISS enrichment the capability of governance, digital mapping analysis, advance analysis and create the knowledge sharing environment within the department. Performance Management In the area of governance, it allow MBO (management by objectives). It measure qualitative (i.e. objective) achievement via quantitative (i.e. KPI) measurement. Individual objective will has a set of Key Performance Indicators (KPIs) defined that carry specific performance target. Periodically the automation engine will feed the required data to report the performance on KPIs and accordingly the achievement of the objectives could be obtained. It provide management group in visualizing at one glance on the total achievement of DOSM objectives and ensure that all level of officers are moving toward the vision and objectives of the department. Geographic Information Complementing numerical analysis that serve the analysis via the digital map. Multiple layer of map e.g. Based Map with State, Enumeration Block etc. could be used to analysis the exact boundary and location on individual analysis subject and assist the proper frame management. In fact it help in performing the proper assignment of individual field enumerator to relevant sample case. E.g. Field enumerator assign to multiple EB that shouldn’t be geographic disperse on the map to reduce the cost of travelling. Business Intelligence Provide the dynamic analysis and produce of analysis result without needing the help from IT division. This allow the generation of information or data from central repository in very short timeframe, and response quickly to the requester (stakeholder or public) when data is requested, directly improve the image of DOSM. Knowledge Management The total solution is not only focuses on operational and management processes but also create an collaboration environment that allow officers to work in more effective and efficient manner, it provide a platform for DOSM’s officers to exchange information and idea with capabilities and functionalities to encourage knowledge sharing. Enterprise Portal With lot of process, functionalities and information hosted within NEWSS, effective access to these resources are critical to ensure the full utilization of the system, Enterprise Portal provide a single access door to reach these resources with single login. With proper segregation of the internal user (DOSM’s user) and external user (public, stakeholder), it ease the user in performing operational and management activity on the system and facility the retrieval of relevant information that authorized to be accessed by every individual.
BI is often an area of friction between IT (who prepare the information) and the business users (who need it to do their jobs).
BI is often an area of friction between IT (who prepare the information) and the business users (who need it to do their jobs).
Information from Census / Survey updated to the Establishment / Enterprise Frame Sample selection will be carried out accurately
National Enterprise-Wide Statistical Systems (NEWSS)
National Enterprise-Wide Statistical Systems (NEWSS) NURUL EFFA AHMAD DEPARTMENT OF STATISTICS MALAYSIA 26 April 2010 Meeting on the Management of Statistical Information Systems (MSIS 2010)
Department of Statistics Malaysia (DOSM) At a Glance
DOSM Current Systems SIDAP SPDRB SPPB SSE PTB SCAC
Objective of NEWSS Efficiently monitor and manage Surveys and Census Improve the timeliness and reliability of the Statistical Information Integrated Statistical System Framework Extend the dissemination of Statistical Information
Integrated Statistical Systems Framework Source : DOSM NEWSS Vision
National Enterprise-Wide Statistical System ISS ENTERPRISE PORTAL Pre-Collection Collection Processing Analysis Dissemination ISSF BUSINESS INTELLIGENCE GEOGRAPHICAL INFORMATION KNOWLEDGE MANAGEMENT PERFORMANCE MANAGEMENT
ENTERPRISE PORTAL Pre-Collection Collection Processing Analysis Dissemination ISSF Enter Survey Project State Office (Operation Section) Extract Sample Enter Response/ Non-response Case Perform Analysis Publish Publication DOSM BKKP Set Performance Measure DOSM Officer Contribute Content PAYMENT GATEWAY Request Payment Return Payment Status View Content View Content DOSM Officer View Content PUBLIC/ STAKEHOLDER DOSM Officer Dynamic Analysis Digital Mapping Analysis Frame Management Perform Sampling Design Enter Survey Data State Office (Processing Section) DOSM BKKP/ SMD Monitor Performance State Office (Frame Section) Enter Data DOSM BKD/ SMD Perform Analysis ISS Request Data Manage Request Perform Analysis Other Agencies Provide Data Other Agencies Provide Data Perform Filtering Perform Data Processing Overview of System Flow Business Intelligence Knowledge Management DOSM BM DOSM SMD DOSM SMD DOSM BM DOSM BKD/ SMD Geographic Information Performance Management DOSM BKKP
Overview of Data Flow CENTRAL REPOSITORY Pre-Collection State/HQ SAMPLE Online Collection OPT DATA State/HQ/ Respondent Online Processing SURVEY DATA Offline State/HQ Offline Edit Spec Checking Data Entry State/HQ/OA Online WEIGHTED DATA WEIGHTED DATA CONTENT STORAGE Dissemination HQ Online DISSEMINTION DATA Analysis State/HQ Online Publication/ Tabulation Batch Analysis DATA WAREHOUSE Online Frame State/HQ Online Batch Other Agencies (OA) FRAME Frame Batch File DIGITAL MAPPING
Overview of Data Flow CENTRAL REPOSITORY Applications Platform FRAME DIGITAL MAPPING Frame State/HQ Online Batch Other Agencies (OA) FRAME Frame Batch File DIGITAL MAPPING Pre-Collection State/HQ SAMPLE Online Collection OPT DATA State/HQ/ Respondent Online Processing SURVEY DATA Offline State/HQ Offline Edit Spec Checking Data Entry State/HQ/OA Online WEIGHTED DATA WEIGHTED DATA CONTENT STORAGE Dissemination HQ Online DISSEMINTION DATA Analysis State/HQ Online Publication/ Tabulation Batch Analysis DATA WAREHOUSE Online Workflow Process Central Repository Operational Process & CCC Data Management Information Data Directory & Metadata Softcopy Contents Intranet Portal Internal Front End <ul><li>ISSF </li></ul><ul><li>KMS </li></ul><ul><li>PMS </li></ul><ul><li>BI </li></ul><ul><li>GIS </li></ul>External Portal External Front End <ul><li>Public </li></ul><ul><li>Content </li></ul><ul><li>Request </li></ul><ul><li>Feedback </li></ul><ul><li>Services </li></ul>
Centralized Integrated Statistical System <ul><li>Eliminate effort to transfer data between disparate Systems </li></ul><ul><li>Data Consistent and Up To Date </li></ul><ul><ul><li>Information captured at State Offices correctly reflected at the DOSM Divisions and vice versa </li></ul></ul><ul><ul><li>Information from the census / surveys updated to the Establishment / Enterprise Frame </li></ul></ul><ul><ul><li>Consistent set of codes and classifications used across all the Census / Survey </li></ul></ul><ul><li>Data at “finger tips” </li></ul><ul><ul><li>Current data, historical data, time series data </li></ul></ul><ul><ul><li>Mechanism for DOSM to perform Ad Hoc Analysis for customized data request </li></ul></ul>
Data Visualization <ul><li>Business Intelligence </li></ul>
Data Visualization <ul><li>Analyzing information using Maps </li></ul>
Data Visualization <ul><li>Performance Management System </li></ul>
More Effective Monitoring of the Census / Survey <ul><li>Complete information on the Census / Survey </li></ul><ul><ul><li>Activities </li></ul></ul><ul><ul><li>Officer in Charge </li></ul></ul><ul><ul><li>State and End Date (Planned and Actual) </li></ul></ul><ul><ul><li>Progress of the Activities </li></ul></ul><ul><li>Pro-active Monitoring mechanism </li></ul><ul><ul><li>Reminders on the activity start and completion date </li></ul></ul><ul><ul><li>Automatic update of the progress of the Census / Survey Collection and Processing activity </li></ul></ul><ul><li>Number of Sample Case assigned to each officer </li></ul>
Efficiency of Census / Survey Process <ul><li>Reduce the number of processes </li></ul><ul><ul><li>Merging of Survey Data received from all the State Offices </li></ul></ul><ul><li>Edit Spec Validation </li></ul><ul><ul><li>Validation carried out on NEWSS Server not on Officer’s local PC. Allows officer to continue on with other activities while system is validating the Census / Survey Data </li></ul></ul><ul><li>Census / Survey Data available for SMD to review after each officer submits the batch for validation. </li></ul><ul><li>Report Preparation </li></ul><ul><ul><li>Approx 1,200 tables, charts, publications, listings </li></ul></ul>
Efficiency of Census / Survey Process <ul><li>Automate Manual Processes </li></ul><ul><ul><li>Establishment / Enterprise Frame </li></ul></ul><ul><ul><ul><li>Monitoring and tracking of the Screening sent out to the Establishment / Enterprise </li></ul></ul></ul><ul><ul><ul><li>Reconciliation of the Establishment / Enterprise Information received from other agencies with the information in the Establishment / Enterprise Frame </li></ul></ul></ul><ul><ul><ul><li>Reconciliation of the information from the Household CQ Listing with the information in the Establishment / Enterprise Frame </li></ul></ul></ul><ul><ul><li>Economic Census / Survey (SSE) </li></ul></ul><ul><ul><ul><li>Update of the information from the Census / Survey into the Establishment / Enterprise Frame </li></ul></ul></ul><ul><ul><ul><li>Generation of Big Business Unit (BBU) and Small Business Unit (SBU) list </li></ul></ul></ul><ul><ul><ul><li>Reduce the number of steps for calculation of Adjusted Weight and Weighted Data </li></ul></ul></ul><ul><ul><li>Monthly Manufacturing Survey (MM) </li></ul></ul><ul><ul><ul><li>Update of the information from the Census / Survey into the Establishment / Enterprise Frame </li></ul></ul></ul><ul><ul><ul><li>Generation of Big Business Unit (BBU) and Small Business Unit (SBU) list </li></ul></ul></ul><ul><ul><ul><li>Conversion of the unit of measure from unit provided by the Respondent to standard required by SMD </li></ul></ul></ul>
Extend Dissemination of Statistical Information <ul><li>Mechanism for SMD to publish Census / Survey Reports on the DOSM website </li></ul><ul><ul><li>Increase number of publications available for download / purchase on the DOSM website </li></ul></ul><ul><ul><li>DOSM Stakeholders and General Public able to view and download the Census / Survey Reports from the DOSM website </li></ul></ul><ul><ul><li>Order the Census / Survey Reports Online </li></ul></ul><ul><li>Dissemination of Data Request </li></ul><ul><ul><li>Stakeholder and General Public to order online </li></ul></ul><ul><ul><li>DOSM to process and disseminate the Data Request </li></ul></ul><ul><li>Dissemination of Customized Census / Survey Reports </li></ul><ul><li>Feedback from the “Customer” </li></ul><ul><li>Customer Service </li></ul><ul><ul><li>Notifications of new publications </li></ul></ul><ul><ul><li>Manage subscription </li></ul></ul><ul><ul><li>Order tracking </li></ul></ul>
Knowledge Sharing <ul><li>Turning personal knowledge into corporate knowledge that can be shared throughout the organization </li></ul><ul><li>Platform for DOSM to collaborate, share and disseminate knowledge throughout the organization </li></ul><ul><ul><li>Community of Practice (forum) to promote a culture of free discussions and exchange of ideas </li></ul></ul><ul><ul><li>Question and Answer </li></ul></ul><ul><li>Knowledge Repository </li></ul><ul><ul><li>Centralized Document Repository where information can be stored and shared across the organization </li></ul></ul><ul><ul><li>Discussion thread in Community of Practice </li></ul></ul>Knowledge is Pooled, Retained, Shared and Developed