ONS business data and statistics user event

1. ONS business data and statistics user event 19 April 2023 slido #28361

2. Welcome Abbe Williams Assistant Deputy Director Surveys and Economic Indicators Office for National Statistics slido #28361

3. Agenda 10:00am to 10:05am Welcome – Abbe Williams, Assistant Deputy Director, Surveys and Economic Indicators, Office for National Statistics 10:05am to 10:15am Opening address – Mike Keoghan, deputy National Statistician, Office for National Statistics 10:15am to 10:45am Business survey data, their statistics and uses – Jon Gough, Office for National Statistics 10:45am to 11:00am Questions and Answers 11:00am to 11:10am Break 11:10am to 11:40am Transformation of business statistics – Craig Taylor, Deputy Director for Business Statistics Transformation, Office for National Statistics and Beck Keane, Office for National Statistics 11:40am to 11:55am Questions and Answers 11:55am to 12:05pm Break slido #28361

4. Agenda 12:05pm to 12:35pm ONS future plans – Andrew Allen, Office for National Statistics and Tony Chapple, Office for National Statistics 12:35pm to 12:50pm Questions and Answers 12:50pm to 1:20pm Economic microdata transformation – Rabiya Nasir, Russell Black and Yohannes Lemma, Office for National Statistics 1:20pm to 1:35pm Questions and Answers 1:35pm to 1:40pm Closing remarks – Darren Morgan, Director, Economic Statistics Production & Analysis Office for National Statistics slido #28361

5. Questions can be submitted via slido.com using code #28361. You can also access slido via the link in the chat box. slido #28361

6. Opening address Mike Keoghan Deputy National Statistician and Director General, Economic, Social and Environmental Statistics Group Office for National Statistics slido #28361

7. Business Survey data, their Statistics and Uses Jon Gough Office for National Statistics slido #28361

8. Business Surveys • Numerous surveys which help to measure the economy, labour market and socially • Business surveys cover all sized businesses covering a range of topics and frequencies • Financial Variables • Structural and short-term surveys • Feed into and meet user requirements

9. Short term surveys

10. Retail Sales Index • Retail sales data are collected from a sample of approx. 5,000 retailers across GB • The primary purpose of the Retail Sales Index (RSI) is to produce a short-term measure of the changes in the volume and value of sales of goods by retail businesses • The RSI feeds into estimates of Gross Domestic Product (GDP) • The value and volume measures of retail sales estimates are used by the private and public sector particularly by the Bank of England (BoE) and HM Treasury to assist in informed decision-making and policymaking

11. Monthly Business Survey • Monthly Business Survey (MBS) is a monthly sample of around 6,000 businesses • Feeds both the Index of Production (IoP) and Index of Services (IoS) which in turn is used to create monthly and quarterly GDP • Two of the primary data sources used in the compilation of monthly and quarterly GDP estimates; IoP and IoS account for approx. 93% of the economy • Used extensively by Government stakeholders to inform economic policy and future forecasts • Used by BoE at Monetary Policy Committee meeting to help inform interest rate decisions.

12. Construction • Surveys approx. 8k businesses per month • Publish by type of work – New and Repair & Maintenance • Nine sectors incl. housing, infrastructure and non-housing • Sub-sector and sub national estimates • Component of MGDP and QGDP • Used by other government departments such as BoE, HMT, BEIS and DLUHC

13. Business Insights and Conditions Survey (BICS)

14. What is BICS • The ONS created a new innovative fortnightly business survey, initially to monitor the impact of COVID-19, now geared towards assessing business sentiment • The BICS voluntary fortnightly business survey has provided us with a rapid and flexible survey to respond quickly to changing policy needs • Detailed results are published each fortnight with headline figures published each week in a Faster Indicators (now called RTI) bulletin Financial performance Workforce size Furlough Grants and Schemes EU Exit International trade & supply chains Net zero Shortage of workers Global supply chains Increase in prices (energy)

15. Business Insights and Conditions Survey • Fortnightly survey • Sample of around 39,000 businesses • Response rate 23-25% • ONS first ‘pulse’ business survey • Collects qualitative information • Initially collected the extent of trade during the pandemic and normal expectations for that time of year but expanded to cover range of topics: • Impact of EU Exit on trade • Environment and supply chains • Labour shortages and recruitment challenges • Government schemes to assist businesses during lockdowns • Price increases, including the energy prices

16. Capturing data from businesses to meet users diverse needs • Flexibility of the online survey and BICS processing systems allow us to add new questions quickly • The question bank of all the BICS questions for each wave is publicly available • We work closely with stakeholders on their priorities, rotations for the questions and this allows us to keep up to date with the latest emerging issues and gaps in data • With only a two-week window each wave we: • Agree content and design questions with the stakeholder • Work closely with ONS methodology teams to further refine questions • Test the survey before it goes live Agree Content Design Questionnaire Test

17. No longer a “COVID” survey, from April 2022, BICS moved towards business tendency survey with predictive power Regular question and comms reviews Introduction of core questions Adapting to free text responses provided businesses • Taken steps to transform BICS into a business tendency survey • Leading indicator on the real economy with potential for monitoring and forecasting business cycles • Collect consistent responses monthly over time (instead of 'normal times') • Indicators that have predictive power (output, prices employment) • Valuable for policy makers and forecasters • Focus on 'core' and 'non-core' themes fortnightly

18. Taken steps to transform BICS into a business tendency survey Core – leading indicators with predictive power Non-Core (Adhoc) – broad themes to help answer policy questions Exporting and Importing •Compared with a year ago •Increase in challenges/ disruption •Support received Business Concerns •Falling demand •Energy prices •Inflation Global Supply Chain •Disruption Workforce •Location changes •Worker shortage •Wages / redundancies Environment and climate •Concerns •Action taken Turnover •Turnover compared with previous month •Challenges impacting turnover •Turnover expectations Prices •bought/ sold compared with previous month •Price pass through to customers •Energy Stock and Capital expenditure •Stock levels •Expectations for capital expenditure Workforce •Number of employees compared with previous month •Expectations for staffing costs Business Confidence •Overall performance •Potential of insolvency •Cash flow

19. Use of BICS: Monitoring Cost of Living

20. BICS key findings

21. Use of BICS: BICS vs monthly GDP* Source: Office for National Statistics – Business Impact of Coronavirus (COVID-19) Survey Trends from businesses in BICS are also seen to be broadly reflect the published UK monthly GDP estimates, where BICS is published much earlier than the official monthly GDP estimate. Trend follows more closely since the change to BICS in April 2022

22. Use of BICS Recent trends in UK food and drink producer and consumer prices: January 2023 Nature and biodiversity risks monitored by 3% of UK businesses ahead of COP15 Rising business insolvencies and high energy prices Food and drink service firms are most likely to cut trading to tackle energy costs

23. Future developments • Reviewing content – questions and responses • Identifying where BICS can support with data gaps • Improving response rates where possible • Subnational estimates weighting methods • Exploring further predictive power of BICS to support Retail Sales Index, Construction

24. Structural Business Surveys 19 April 2023

25. Annual Business Survey and Annual Purchases Survey • Annual Business Survey (ABS) and Annual Purchases Survey (APS) form essential building blocks in the supply-use framework • Estimates from ABS make up three quarters of output by industry, and provide the majority of data on intermediate consumption • Provides estimates of Turnover, Purchases and aGVA at detailed industry level • In 2020, approximate gross value added at basic prices (aGVA) of the UK non-financial business economy was estimated to be £1,194.9 billion • APS gives the detail of which products are being consumed • They directly impact other areas of National Accounts including: • Gross operating surplus • Gross fixed capital formation • Taxes and subsidies • ABS & APS help inform policy decisions • Supplying data to other government departments for specific analysis e.g., identifying energy intensive industries

26. UK Manufacturers sales by product - PRODCOM • Products of the European Community (PRODCOM) is a long standing, large ONS business survey and is the primary source for measuring the UK manufacturing industry broken down by the products the industry produces • The total value of UK manufacturers’ product sales was £400.8 billion in 2021 • Purpose: Main uses of PRODCOM outputs are to inform the UK supply and use tables, it is also used to develop a sampling frame for the Producer Price Index (PPI) which is used to monitor inflation in sales at the “factory gate” • Key users: ONS (National accounts and Prices) Devolved Administrations, DIT, BEIS, MOD • Policy Impact: PRODCOM tracks manufacturing in the UK at product level and is a key source used when considering policy setting for the production industry • Supply chain issues being experienced in the UK, PRODCOM data has been used to help identify vulnerable products and when merged with other data, vulnerable countries

27. Annual Survey of Goods and Services (ASGS) • Relatively new ONS survey, considered as an experimental output, covers the UK services industry collecting annual turnover values broken down by the goods/services provided • Purpose: Main uses for ASGS outputs are to inform the UK supply and use tables, it is also used to develop a sampling frame for the Services Producer Price Index (PPI) • Key Users: ONS (National accounts and Prices), Devolved Administrations, DIT, BEIS • Policy Impact: As an experimental output, ASGS is not directly used to inform policy, more indirectly in that ASGS is used to inform wider ONS outputs that are used to inform and base policy making decisions on for example GDP and SPP outputs.

28. Q&A Questions can be submitted via slido.com using code #28361. You can also access slido via the link in the chat box. slido #28361

29. Break slido #28361

30. Transformation of business statistics Craig Taylor Deputy Director for Business Statistics Transformation Office for National Statistics Beck Keane Office for National Statistics slido #28361

31. What we will cover… • Update on transformation of Research and Development statistics progress • Update on plans for some of development of some of our larger structural statistics • High level summary of different phases of business statistics transformation

32. Transformation of Research and Development statistics Update on progress Craig Taylor Deputy director, Business Statistics Transformation slido #28361

33. Summary of the work Mission Vision Drivers Objectives Review & Development of R&D Statistics Completed by 2024 By 2024, UK R&D statistics are internationally recognised and trusted as high quality and responsive to user needs Differences between ONS/HMRC estimates New data sources Levelling up Need to conduct a review of R&D statistics Understand and explain differences in different data sources Use of available administrative data sources Provide more granular level statistics Be flexible and adaptive to user needs Expected outcomes / benefits Better informed decisions and improved policy evaluation related to R&D expenditure in the UK Increased confidence in R&D data Increased responsiveness and flexibility

34. R&D transformation timeline End2022 May2023 July2023 April2023 March 2024 Spring2024 Improved Research & Development statistics published (R&D) – phase 1 Transformed Business Enterprise Research & Development (BERD) survey despatched on EQ GovERD survey despatched with additional question/s and improved guidance Experimental Regional Government Expenditure on R&D Estimates published Transformed Research & Development statistics (phase 2) published (note aim for BERD to be published) Complete Further developed BERD survey (possibly including new questions) despatched

35. Reminder of R&D phase 1 Business Enterprise R&D • Identified BERD methodological improvement and incorporated into BERD (2014-2021) and GERD releases (2018-2020) • Reduced BERD release due to breakdown uncertainty (UK figure National Statistic; breakdowns NS status paused) and limited microdata available • Article explaining uplift approach and validation Higher education R&D • Secured use of TRAC data as alternative source  Improved HE R&D statistics including HE own funded and performed R&D (£4.9bn in 2020) and wider indirect costs of R&D and incorporated into GERD statistics in November 2022 • Changes in total HE R&D recorded as part of GERD statistics and taken back to 2018 TRAC data • Identified best practice on R&D collection and produced recommendations for depts • Developed short-term baseline for mission based around 3 large R&D funding departments  Baseline estimates published on 17 April 2023. Government R&D

36. Regional (ITL1) Public Funded R&D Expenditure Baseline: Background The Levelling Up White Paper R&D mission states: “By 2030, domestic public investment in R&D outside the Greater South East will increase by at least 40%, and over the Spending Review period by at least one third. This additional government funding will seek to leverage at least twice as much private sector investment over the long term to stimulate innovation and productivity growth.” In terms of Government expenditure on R&D, this implies the following data requirements: Short-term development of baseline data for Government spending on R&D at regional level. Improvement to the GOVerd survey to collect more accurate regional data on Government R&D expenditure (2024).

37. Sample design Survey design Take-on and validation Results processing Dissemination BERD: Sample design which addresses under coverage - Increase of sample size to 40k (from 5k) - Samples directly from business register - Improved reference list using HMRC tax data to target the largest business with R&D expenditure BERD: Sample design which optimises at sub-national level for as much as sample size allows - Increased use of long form to capture more detailed information BERD: Improved guidance for existing questions on survey form/s BERD: Move surveys to EQ BERD: Improved coverage of sub- national R&D information BERD: Improved coverage of R&D product information BERD: New take-on and validation system which has enabled streamlining of validation rules and move to strategic platform BERD: New methods for processing results e.g., estimation, imputation BERD: New results system which enables incorporation of new methods, delivers improvements to quality assurance approach BERD: Improvement to sub- national data through data collection and learning from baselining methods All: R&D outputs provide inputs into the National Accounts – part of core delivery BERD: Back series (required to support R&D publication and delivery into National Accounts) GovERD: New questions in response to stakeholder demand GovERD: changes to absorb question changes within data collection GovERD: changes to absorb question changes within results processing High level scope of R&D phase 2

38. Ongoing research areas While focus remains on delivering our challenging work for phase 2 work (predominantly transitioning our end to end system design onto new platforms to support the new sample design), there are some specific work that continues alongside, to help shape and inform future workstreams: • R&D international comparisons- carrying out some research into international approaches to measuring R&D, to understand what future improvements can be made • R&D back series development • R&D and the national accounts- understanding the current inclusion of R&D activity in the national accounts, how this is captured and how the new R&D work can be implemented • R&D additional analytical work- with the increase in sample size due to the new design, we are committed to increasing our analytical resource to make the best use of the existing sample size. It’s likely this work will be used for internal purposes, including refining future surveys and methods, as well as helping make sure the quality of our statistics can be maintained • GOVERD- learning from the one-off baseline data collection exercise to make improvements to our regular GOVERD publication • GOVERD- In conjunction with GO-Science, developing questions to be added to GOVERD to collect more granular information on purchased R&D by region, expenditure on Official Development Assistance (ODA) and technology areas i.e Artificial Intelligence • GOVERD- taking part in a pilot scheme in collaboration with GO-Science to help better prepare departments to complete GOVERD returns

39. User engagement Engagement so far Next Steps Huge amount of interest and user engagement, which has included:  User consultation  Presentations and engagement at vast array of events, conferences e.g. Better Statistics Conference  ONS Seminar on 6th December on measurement of R&D (BEIS; Go Science; OECD; British Academy presented) Engagement with stakeholders has been wide ranging and has included:  Stakeholder events such as Business Statistics User event  Continue to engage with stakeholders through established forums highlighted  Article outlining development progress (summer 2023)  Additional article including initial indications of impact to GDP etc (Autumn 2023)  ONS seminar focused on R&D (end 2023) Devolved Administrations British Academy, Universities UK HMRC, BEIS, HMT, DLUHC R&D Strategy Board Go Science Council Science and Technology (CST); Chief Scientific Advisors OSR Various academics inc LSE, ICL External R&D expert group inc OECD and Nesta BofE

40. Annual purchases survey (APS) Annual Business Survey (ABS) Update on plans Beck Keane Assistant deputy director, Business Statistics Transformation slido #28361

41. Aim Drivers Opportunities To review and develop the Annual Business Survey (ABS) and Annual Purchases Survey (APS) for an efficient, flexible and future proof solution Improve responder experience and streamline data collection Respond to emerging needs (including improved granularity) Produce national statistics as endorsed by OSR Strategic systems and efficient processes Improved methods and improved quality assurance New data sources Benefits Increased responsiveness and flexibility, reduce risk of errors Better informed decisions Improved policy evaluation Annual Business Survey (ABS) Mission Statement

42. APS and ABS progress to date • Understanding areas for redevelopment across the design • Using new technologies to move data collection online (2024 collection) • Reviewing end to end methods • Gathering and evaluating our user requirements following our consultation

43. APS where next… • Improving flexibility by rationalising questionnaires • Improving respondent experience through questionnaire design and automated validation • Configuration changes to processing systems to improve business processes for APS production

44. ABS where next… • Exploring strategic system to better support flexible response to user needs • Developing methods to support user requirements • Improving respondent experience through questionnaire design and automated validation

45. User engagement Engagement so far Next Steps Engagement activities have included: • OSR published ABS review • ONS consultation on ABS (June 2022) • Government user groups (next meeting 25th April 2023) Engagement has focussed on known users of ABS statistics including • ONS response to ABS consultation (May 2023) • ONS response to OSR report on ABS review (Summer 2023) • Sharing plans for next steps (summer 2023) • Identify additional users of ABS and APS Are you a user of our ABS and APS data? We would love to hear from you to make sure we communicate plans and understand requirements of our data abaps@ons.gov.uk Devolved Administrations Department for Trade BEIS, HMRC, HMT OSR DCMS Government User Group (GuG)

46. Business statistics transformation phases of delivery Craig Taylor Deputy director, Business Statistics Transformation slido #28361

47. R&D Phase 1 • Address under-coverage in BERD sample design by implementing an uplift to address known bias outside of core processes • Sub-national baseline work in response to Levelling Up White Paper 1 R&D Phase 2 • New sample design to address under- coverage & optimisation for agreed breakdowns • Move to electronic data collection for all R&D surveys, inc. improved guidance • Improved coverage of collecting information on sub-national & type of R&D conducted 2 Structural Surveys • Move of APS to electronic data collection • Revised ABS questionnaire as part of move to full electronic data collection • Full end to end transformation of ABS data take-on & results processes systems 3 Further phases of transformation 4 BST PROJECT – PHASED DELIVERY • New estimation & imputation methods • New results systems enabling incorporation of new methods & improved QA processes • Outputs able to be incorporated into wider National Accounts

49. Break slido #28361

50. ONS future plans Andrew Allen Office for National Statistics Tony Chapple Office for National Statistics slido #28361

51. Statistical Business Register (SBR) Andrew Allen Office for National Statistics slido #28361

52. 52 Background • Inter-Departmental Business Register (IDBR) was introduced in 1994. Improved constantly since, but technology limits further development. • IDBR provides sample frame for all business surveys and is used for analysis of UK business population. • Will be replaced by : • Statistical Business Register launch 2024

53. Statistical Business Register (SBR) Transformed register of UK businesses, covering the functionality of the IDBR Core functions: • Underpins Economic statistics outputs • Create statistical units • Create and manage frames • Create and manage samples • User interface and analytical extracts Benefits: • Modern technology. • New data sources – full companies house population, corporation tax and self assessment – will lead to full business population (additional +2m businesses). • More variables, improved sampling & analytics • Improved quality and granularity. • Greater use across Government.

54. Phase 0 (Discovery) Phase 1 (Research) Phase 2 (Alpha) Phase 3 (Beta 1) Phase 3 cont. (Beta 2 and 3) Phase 4 Migration SBR Discovery: An internal ONS Discovery Phase has taken place. Research Phase: Carry out a research phase to understand the needs of the users and system requirements and formulate solution options. Alpha Development: Create and demonstrate a prototype to verify that the solution is achievable and meets user needs. Prioritising Beta requirements and engaging with stakeholders to validate. Beta Development: Incrementally develop and regularly demonstrate a production-ready solution. Mostly technical environment set -up Beta Development & transition to DST: Incremental development and roll- out of new features. CGI will provide technical and project support, particularly at key points when rolling out the system. This will include during operational tests, when the system is launched to staff and managers and when the system is first used by users. Migration of Surveys and change management: 1. Enhanced support 2. Support business surveys to transition to the use of SBR by creating similar population as IDBR. 3. Expand the sample population for business surveys SBR Phases COMPLETE COMPLETE COMPLETE 30 Mar 23 B2: 16 Jan 24 B3: 29 Mar 24 30 Jun 24

55. 55 IDBR data sources • We aim to capture economic activity and not just business registration • IDBR is based on VAT and/or PAYE returns: company registration data used to assist matching. • Small companies without a VAT/PAYE are not held. • Hence IDBR has 2.0m Companies. Companies House has 4.6m effective registrations.

56. 56 SBR new data sources • HMRC Corporation tax dataset • will provide better financial variables • will facilitate the use of the full companies house population • HMRC Self Assessment • will allow better coverage of very small businesses. Giving us the full business population for the first time.

57. 57 Phased data source improvements • The SBR technical build will be complete in 2024 - which will allow us to flexibly add data sources. • New data sources will be added as they become available. • For example, ONS is still negotiating the Self-Assessment supply. Rules and processes will be needed to then adopt on the SBR.

58. 58 SBR benefits will feed into downstream products • SBR will improve Business Structures Database – extra business population and better time-stamping of variables etc. • Register based publications will transition to full business population e.g. UK Business and Business Demography • Timetable and details will be agreed later in the project.

59. 59 Business survey outputs • We will need to assess and work out how and when to use new register coverage. • The aim is to be able to replicate the IDBR population to enable continuity, while the impact of extra businesses is considered.

60. In partnership with Secure Research Service (SRS)Transition Tony Chapple – Head of SRS Transition tony.chapple@ons.gov.uk

61. Potted history VML <> SRS <> IDS VML Standalone / VML Tactical VML Private Cloud-based (PSN) SRS Private Cloud-based (Internet) SRS Technology Refresh 15+ years providing secure Research Access to deidentified unpublished data using 5-Safes Framework

62. Potted history VML <> SRS <> IDS VML Standalone / VML Tactical VML Private Cloud-based (PSN) SRS Private Cloud-based (Internet) SRS Technology Refresh 15+ years providing secure Research Access to deidentified unpublished data using 5-Safes Framework Advances in technology have opened the door for developments in the way users access data. IDS will build on the foundations provided by the SRS, and will, in time, replace the current service.

63. Potted history VML <> SRS <> IDS VML Standalone / VML Tactical VML Private Cloud-based (PSN) SRS Private Cloud-based (Internet) SRS Technology Refresh SRS Transition to IDS 15+ years providing secure Research Access to deidentified unpublished data using 5-Safes Framework IDS will represent a change from the way researchers access data in the SRS. Supporting users in making the change, a dedicated project team is in place to plan the transition and ensure that users move from the SRS to IDS as smoothly as possible.

64. Capabilities Capability Roadmap (Process & Technology) Clear outputs Supplementary Ingest Capabilities (Process & Tech) Tooling Roadmap Tools Data Roadmap Data Tools STATA SPSS R,Python Data Census LFS ASHE Key: SRS transition needs Wider IDS needs Context setting – Strategic approach to transition Integrated Data Service (IDS) Secure Research Service (SRS) Analysis IDS Strategic Roadmaps User Experience

65. Data Tools Capabilities (Process & Tech) SRS Transition Funnel Capabilities (Process & Tech) Tools Data Analysis Availability Tranche planning Context setting – Strategic approach to transition

66. SRS-IDS Transition: High Level Approach Initial Discovery Mobilisation Initial Proof of Concept Analysis & Planning Tranches Final Decommissioning & Handover Plan Migrate Accept Test OUTPUTS: • “As Is” • “To Be” Gap Analysis • High Level Requirements • Transition Strategy OUTPUTS: • Transition Kick Off • Identify “Air Traffic Controllers” • Detailed Planning & Requirements • Data Owner Engagement • Comms Kick Off • Plan E2E Pilot OUTPUTS: • Transitioned Data • Transition Project • Transition Users • Lesson Learnt OUTPUTS: •SRS Staff Transitioned •SRS Decommissioned •IDS Service Level Acceptance L0 SRS Decommission OUTPUTS: • Detailed Transition Plan • Lessons Learnt IDS Tooling Roadmap IDS Data Roadmap SRS-IDS Staff Transition & Service Development Demand Framework DISCOVERY / DESIGN & DEVELOPMENT TRANSITION TO SERVICE SRS CLOSURE IDS Capability (Process & Technology) Roadmap

67. SRS-IDS Transition: High Level Approach Initial Discovery Mobilisation Initial Proof of Concept Analysis & Planning Tranches Final Decommissioning & Handover Plan Migrate Accept Test OUTPUTS: • “As Is” • “To Be” Gap Analysis • High Level Requirements • Transition Strategy OUTPUTS: • Transition Kick Off • Identify “Air Traffic Controllers” • Detailed Planning & Requirements • Data Owner Engagement • Comms Kick Off • Plan E2E Pilot OUTPUTS: • Transitioned Data • Transition Project • Transition Users • Lesson Learnt OUTPUTS: •SRS Staff Transitioned •SRS Decommissioned •IDS Service Level Acceptance L0 SRS Decommission OUTPUTS: • Detailed Transition Plan • Lessons Learnt IDS Tooling Roadmap IDS Data Roadmap SRS-IDS Staff Transition & Service Development Demand Framework DISCOVERY / DESIGN & DEVELOPMENT TRANSITION TO SERVICE SRS CLOSURE IDS Capability (Process & Technology) Roadmap

68. Transition Complexities…… Staff Projects & Users Data Org Connectivity (Cloudflare) Tooling Projects & Users Data Owner Agreements DEA Data

69. High Level Project Transition 1. Engagement Willingness to transition ? Finish in SRS 2. IDS Onboarding Project / Account Setup Original Data made available Training & Guidance 3. Dual Running SRS IDS 3A. Code extraction Output from SRS Ingest into IDS 4. Accept into IDS Shutdown Project in SRS Continue in IDS 69

70. Some of the Differences … Connectivity & Access Cloudflare User Experience Cloud native <> Virtual Desktop Infrastructure Dataviews ‘Big Query/ SQL’ <> ‘flat files etc. Research Tooling Code & Package use

71. Summary This is the start of the Transition Journey • We will be providing regular communication updates to all users A commitment to minimise disruption as much as possible • Drive forward PoC (both data and project transition) • Front load the transition of Data, transition Projects when appropriate • Transition staff to IDS, according to ramp down / ramp up of SRS / IDS • Decommission SRS Next Steps

72. In partnership with Thank you ! Tony Chapple - tony.chapple@ons.gov.uk

74. Economic Microdata Transformation Rabiya Nasir, Russell Black and Yohannes Lemma Office for National Statistics slido #28361

75. Content • Role of the Economic Microdata Transformation team • Longitudinal Business Database • Annual Respondents Database X and Capital Stocks • HMRC Trade in Goods data in SRS • Microdata plans for 2023/24

76. • We are experts on transforming and analysing firm-level data. • Our research links and develops microdata sources to improve our understanding of the economy and how to measure it. • We make firm-level data more accessible to support external research in government and academia. • Work closely with Economic Statistics Centre of Excellence (ESCoE) Who are Economic Microdata Transformation

77. Motivating topics • Productivity: distribution and drivers at firm level • Business dynamism, competition and markups • Management and organisational structure • Role of businesses in Net Zero and ‘levelling up’ • Better and faster GDP estimates

78. Longitudinal Business Database (LBD) Economic Microdata Transformation LBD Team

79. The Longitudinal Business Database (LBD) • New set of firm-level spines of business references • It differs from existing Inter-Departmental Business Register (IDBR) linked dataset:  Quarterly cycle frequency  Uses information from two adjacent snapshots to establish longitudinality  Pertinent rules to capture and define economic criteria • First prototype to be made available on the Secure Research Service (SRS)

80. Benefits of the LBD project • Quarterly frequency  Provides researchers with tools to address wide ranging research questions • Enable multi-level linking of survey, administrative and novel data  A single entry of a firm in a given quarter displays the identity-links of all of its business units registered on the IDBR • Flexibility to accommodate to any research purpose • Give end users the capability to produce innovative and insightful research

81. Variables of interest • Scope for multi-level analysis  firm-group, firm, establishment-level unique identifiers • Activity status of the unit  unit births, deaths, incumbents per quarter • Turnover/Employment of the unit  productivity, job creation, destruction, job reallocation etc • Industrial Classification code of the unit  sectoral/industry analysis • Region code of the unit  regional analysis

82. Outputs from the LBD 0.0 0.5 1.0 1.5 2.0 2.5 3.0 1999Q1 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 Enterprises (millions) continuing_active death birth reactivation 0 0.5 1 1.5 2 2.5 3 1999Q1 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 Percentage Employment % of employment from births % of employment from deaths

83. • Linked dataset - LBD and PAYE to measure employment dynamics Main Findings • Business Dynamism has declined in the UK • This decline has taken place in all sectors of the economy. • Older micro firms are exiting the market less than before. Research using the LBD

84. Next steps • Data to be made available on the SRS • Kick off downstream analyses using the dataset • Develop and make available the other spines

85. Datasets in SRS: ARDX

86. Annual Respondents Database X (ARDx) • “Structural Business Survey” has changed several times. Annual Survey of Manufacturers, Annual Business Inquiry, Annual Business Survey… • ARDx: began in the 2000s. Effort to unite this data into one dataset for researchers, so there is an easy point-of-entry

87. ARDx – Annual Respondents Database X • “Structural Business Survey” has changed several times. Annual Survey of Manufacturers, Annual Business Inquiry, Annual Business Survey… • ARDx: began in the 2000s. Effort to unite this data into one dataset for researchers, so there is an easy point-of-entry • In practice: • The content/structure/coding of the surveys is very common, and the changes in processing are very minor

88. The original intention was for ABI1 and ABI2 to cover the same companies, to maximise availability of data when linked. This link was broken in 2006. While ONS will almost always attempt to be comprehensive for large companies, there are smaller companies with ABI2 but not ABI1 Pre 1998 1998-2008 2008/2009 onwards Employment data: Annual Employment Survey Annual Business Inquiry Part 1 Business Register Employment Survey Main economic performance data: [several previous surveys] Annual Business Inquiry Part 2 Annual Business Survey

89. ARDx 4.0 (April 2023) • Code completely revised • Bugs fixed around VAT in the calculation of GVA for retail, food and accommodation ARDx 4.1 (Later 2023) • Fixes for imputations

90. For researchers • Management of the SRS still by discrete datasets • Researchers should request Annual Business Survey and ARDx datasets for their project

91. Capital stocks • Official statistics surveys do not cover company-level capital stocks (ONS would rather take industry-level investment and use its own deprecation assumptions to calculate national balance sheet) • A key motivation for business survey panels going back to the past is to estimate company-level capital stocks. Company-level capital stocks are necessary for production function estimation and total factor productivity statistics, which are important in the economics literature.

92. Capital stock estimates in ARDx dataset • Will be updated to 2020 • Now at 3 asset level • Buildings, land and structures • Software and databases • Other • Small updates to imputation

93. Capital stocks estimates limitations • Business surveys are not the national accounts. From new years of data it will be more apparent of where they differ. • Small and medium companies: • Surveyed infrequently or only surveyed once. The information content is employment and industry history from the Inter-Departmental Business Register, and not capital-specific company data • Large companies: • In theory 2/3 of investment is by large companies, covered comprehensively by ABI2/ABS • In practice M&A, propcos, this is only the beginning of the problem of using this data in analysis • In any case, gap for company-level stocks for the intangible capitals

94. Datasets in SRS: Linked HMRC Trade-in- Goods/IDBR dataset

95. Inter-Departmental Business Register (IDBR) ONS-maintained database of companies, from Companies House, tax data and ONS surveys and reporting

96. Linked HMRC Trade-in-Goods/IDBR dataset • HRMC Trade-in-Goods dataset: collected by HMRC for VAT units at the 9 digit VAT reference, and stored in the HMRC data lab 1. Dataset linked to the IDBR 2. VAT units do not match cleanly to IDBR enterprises. Trade values have been apportioned across IDBR enterprises and reporting units 3. Made available over SRS

97. Limitations • Access cannot be given by Assured Organisational Connectivity. Whole SRS project will be limited to safe rooms. • Currently, commodity detail is 4-digit Harmonised System

98. Timing • June 2020: data for 2005-2016 • August 2022: data for 2008-2020, and small change in method to use more precise weights in apportionment • Timing for 2021 and 2022 data to be determined

99. Future plans • Deliver the next wave of the Management and Expectations Survey. • Contributions to international microdata projects: CompNet and OECD. • Using tax administrative data for firm-level statistics. • Analysis of business’ energy spending, using the Annual Business Survey and Annual Purchases Survey. • Supporting the Integrated Data Service team with migrating our datasets from SRS to IDS. • Collaborating with other business data and transformation teams across ONS to support their work with microdata insights

101. Closing remarks Darren Morgan Director Economic Statistics Production & Analysis Office for National Statistics slido #28361

102. Thank you for attending our user event You can keep up to date on all upcoming events via ons.gov.uk/economicevents If you would like to ask a question or provide any feedback, please do so via economic.engagement@ons.gov.uk slido #28361

ONS business data and statistics user event

ONS business data and statistics user event

Recommended

More Related Content

What's hot

What's hot(20)

Similar to ONS business data and statistics user event

Similar to ONS business data and statistics user event(20)

More from Office for National Statistics

More from Office for National Statistics(17)

Recently uploaded

Recently uploaded(20)

ONS business data and statistics user event