This document outlines examples of big time series data and techniques for visualizing and forecasting them. It discusses four examples of hierarchical and grouped time series data: Australian tourism demand, Australian labor market participation, PBS pharmaceutical sales, and spectacle sales. It then covers various visualization methods like kite diagrams, STL decomposition, seasonal stacked bar charts, and correlation graphs to explore patterns in large time series datasets.
Exploring the feature space of large collections of time seriesRob Hyndman
It is becoming increasingly common for organizations to collect very large amounts of data over time. Data visualization is essential for exploring and understanding structures and patterns, and to identify unusual observations. However, the sheer quantity of data available challenges current time series visualisation methods.
For example, Yahoo has banks of mail servers that are monitored over time. Many measurements on server performance are collected every hour for each of thousands of servers. We wish to identify servers that are behaving unusually.
Alternatively, we may have thousands of time series we wish to forecast, and we want to be able to identify the types of time series that are easy to forecast and those that are inherently challenging.
I will demonstrate a functional data approach to this problem using a vector of features on each time series, measuring characteristics of the series. For example, the features may include lag correlation, strength of seasonality, spectral entropy, etc. Then we use a principal component decomposition on the features, and plot the first few principal components. This enables us to explore a lower dimensional space and discover interesting structure and unusual observations.
MEFM: An R package for long-term probabilistic forecasting of electricity demandRob Hyndman
I will describe and demonstrate a new open-source R package that implements the Monash Electricity Forecasting Model, a semi-parametric probabilistic approach to forecasting long-term electricity demand. The underlying model proposed in Hyndman and Fan (2010) is now widely used in practice, particularly in Australia. The model has undergone many improvements and developments since it was first proposed, and these have been incorporated in this R implementation.
The package allows for ensemble forecasting of demand based on simulations of future sample paths of temperatures and other predictor variables. It requires the following data as inputs: half-hourly/hourly electricity demands; half-hourly/hourly temperatures at one or two locations; seasonal (e.g., quarterly) demographic and economic data; and public holiday data.
Peak electricity demand forecasting is important in medium and long-term planning of electricity supply. Extreme demand often leads to supply failure with consequential business and social disruption. Forecasting extreme demand events is therefore an important problem in energy management, and this package provides a useful tool for energy companies and regulators in future planning.
Automatic algorithms for time series forecastingRob Hyndman
Many applications require a large number of time series to be forecast completely automatically. For example, manufacturing companies often require weekly forecasts of demand for thousands of products at dozens of locations in order to plan distribution and maintain suitable inventory stocks. In these circumstances, it is not feasible for time series models to be developed for each series by an experienced analyst. Instead, an automatic forecasting algorithm is required.
In addition to providing automatic forecasts when required, these algorithms also provide high quality benchmarks that can be used when developing more specific and specialized forecasting models.
I will describe some algorithms for automatically forecasting univariate time series that have been developed over the last 20 years. The role of forecasting competitions in comparing the forecast accuracy of these algorithms will also be discussed.
Exploring the boundaries of predictabilityRob Hyndman
Why is it that we can accurately forecast a solar eclipse in 1000 years time, but we have no idea whether Yahoo's stock price will rise or fall tomorrow? Or why can we forecast electricity consumption next week with remarkable precision, but we cannot forecast exchange rate fluctuations in the next hour?
In this talk, I will discuss the conditions we need for predictability, how to measure the uncertainty of predictions, and the consequences of thinking we can predict something more accurately than we can.
I will draw on my experiences in forecasting Australia's health budget for the next few years, in developing forecasting models for peak electricity demand in 20 years time, and in identifying unpredictable activity on Yahoo's mail servers.
Exploring the feature space of large collections of time seriesRob Hyndman
It is becoming increasingly common for organizations to collect very large amounts of data over time. Data visualization is essential for exploring and understanding structures and patterns, and to identify unusual observations. However, the sheer quantity of data available challenges current time series visualisation methods.
For example, Yahoo has banks of mail servers that are monitored over time. Many measurements on server performance are collected every hour for each of thousands of servers. We wish to identify servers that are behaving unusually.
Alternatively, we may have thousands of time series we wish to forecast, and we want to be able to identify the types of time series that are easy to forecast and those that are inherently challenging.
I will demonstrate a functional data approach to this problem using a vector of features on each time series, measuring characteristics of the series. For example, the features may include lag correlation, strength of seasonality, spectral entropy, etc. Then we use a principal component decomposition on the features, and plot the first few principal components. This enables us to explore a lower dimensional space and discover interesting structure and unusual observations.
MEFM: An R package for long-term probabilistic forecasting of electricity demandRob Hyndman
I will describe and demonstrate a new open-source R package that implements the Monash Electricity Forecasting Model, a semi-parametric probabilistic approach to forecasting long-term electricity demand. The underlying model proposed in Hyndman and Fan (2010) is now widely used in practice, particularly in Australia. The model has undergone many improvements and developments since it was first proposed, and these have been incorporated in this R implementation.
The package allows for ensemble forecasting of demand based on simulations of future sample paths of temperatures and other predictor variables. It requires the following data as inputs: half-hourly/hourly electricity demands; half-hourly/hourly temperatures at one or two locations; seasonal (e.g., quarterly) demographic and economic data; and public holiday data.
Peak electricity demand forecasting is important in medium and long-term planning of electricity supply. Extreme demand often leads to supply failure with consequential business and social disruption. Forecasting extreme demand events is therefore an important problem in energy management, and this package provides a useful tool for energy companies and regulators in future planning.
Automatic algorithms for time series forecastingRob Hyndman
Many applications require a large number of time series to be forecast completely automatically. For example, manufacturing companies often require weekly forecasts of demand for thousands of products at dozens of locations in order to plan distribution and maintain suitable inventory stocks. In these circumstances, it is not feasible for time series models to be developed for each series by an experienced analyst. Instead, an automatic forecasting algorithm is required.
In addition to providing automatic forecasts when required, these algorithms also provide high quality benchmarks that can be used when developing more specific and specialized forecasting models.
I will describe some algorithms for automatically forecasting univariate time series that have been developed over the last 20 years. The role of forecasting competitions in comparing the forecast accuracy of these algorithms will also be discussed.
Exploring the boundaries of predictabilityRob Hyndman
Why is it that we can accurately forecast a solar eclipse in 1000 years time, but we have no idea whether Yahoo's stock price will rise or fall tomorrow? Or why can we forecast electricity consumption next week with remarkable precision, but we cannot forecast exchange rate fluctuations in the next hour?
In this talk, I will discuss the conditions we need for predictability, how to measure the uncertainty of predictions, and the consequences of thinking we can predict something more accurately than we can.
I will draw on my experiences in forecasting Australia's health budget for the next few years, in developing forecasting models for peak electricity demand in 20 years time, and in identifying unpredictable activity on Yahoo's mail servers.
Monal Daxini - Beaming Flink to the Cloud @ NetflixFlink Forward
http://flink-forward.org/kb_sessions/beaming-flink-to-the-cloud-netflix/
Netflix is a data driven company and we process over 700 billion streaming events per day with at-least once processing semantics in the cloud. To enable extracting intelligence from this unbounded stream easily we are building Stream Processing as a Service (SPaaS) infrastructure so that the user can focus on extracting value and not have to worry about boilerplate infrastructure and scale. We will share our experience in building a scalable SPaaS using Flink, Apache Beam and Kafka as the foundation layer to process over 1.3 PB of event data without service disruption.
Rapid growth in the elderly population and the rising incidence of chronic diseases, growing need for cost-effective healthcare delivery due to the increasing healthcare costs, and technological advancements of home care devices. In 2017, there were 3.8 million Australians aged 65 and over (comprising 15% of the total population). The number and proportion of older Australians are expected to continue to grow. The aging population demands more patient-centric healthcare services, which in return increases the demand for healthcare workers and agencies and is anticipated to drive market growth.
The growth of the aging population segment will boost the demand for healthcare and significantly increase the burden on governments and health systems as the aging population is more prone to chronic diseases. This will prove favorable to the market for home healthcare, which can enhance the overall reach of and access to healthcare while reducing unnecessary visits, hospital admissions & readmissions, and the time and costs involved in traveling to meet healthcare professionals.
Visit https://insights10.com/ for more healthcare industry insights.
Connect with us at info@insights10.com
UK Coronary Stents Market Analysis Sample ReportInsights10
The UK Coronary Stents Market is expected to witness growth from $294 Mn in 2022 to $365 Mn in 2030 with a CAGR of 2.75% for the forecasted year 2022-2030. In the UK, coronary artery disease affects a significant number of people, creating a serious health issue. The demand for coronary stents is anticipated to rise as the population ages and risk factors like obesity and diabetes become more common. To get a detailed report, contact us at - info@insights10.com
Overall, the report represents the Global Flexible endoscope market trends along with a market forecast that will help industry consultants, technology providers, existing players searching for expansion opportunities, new players searching possibilities and other stakeholders to align their market-centric strategies according to the ongoing and expected trends in the future.
Norway Medical and Diagnostic Laboratory Service Market Analysis Sample ReportInsights10
This report presents a strategic analysis of the Norway Medical and Diagnostic Laboratory Service Market and a forecast for its development in the medium and long term. It provides a broad overview of the market dynamics, trends and insights, growth drivers and restraints, segmentation, competitive landscape, healthcare policies and regulatory framework, reimbursement scenario, challenges and future outlook. To get a detailed report, contact us at - info@insights10.com
Coronavirus Diagnostics Market - Global Industry Size, Share, Trend, Opportun...TechSci Research
According to #TechSci Research report, Global Coronavirus Diagnostics Market was valued at USD73538.02 million in 2020 and is expected to reach USD15579.56 million by 2026.
Gain More Insight: https://bit.ly/3eh734y
Get Sample Report: https://bit.ly/3njrkuw
Press Release: https://bit.ly/33usYCr
Website: https://www.techsciresearch.com/
Market Research News: https://techsciblog.com/
Global Clinical Trial Patient Recruitment Services Market Analysis Sample ReportInsights10
The Global clinical trial patient recruitment services market is projected to grow from $ 0.84 Mn in 2022 to $1.56 Bn by 2030, registering a CAGR of 8% during the forecast period of 2022-30. The Global clinical trial patient recruitment services market is a rapidly growing market that provides services to support the recruitment of patients for clinical trials. To get a detailed report, contact us at - info@insights10.com
Global Veterinary Reference Laboratory (VRL) Market Research, 2018 to 2025Signitech
The global veterinary reference laboratory (VRL) market size was valued at USD 2.350 billion in 2016 and is expected to grow with CAGR of 9.1% during the forecast period.
Monal Daxini - Beaming Flink to the Cloud @ NetflixFlink Forward
http://flink-forward.org/kb_sessions/beaming-flink-to-the-cloud-netflix/
Netflix is a data driven company and we process over 700 billion streaming events per day with at-least once processing semantics in the cloud. To enable extracting intelligence from this unbounded stream easily we are building Stream Processing as a Service (SPaaS) infrastructure so that the user can focus on extracting value and not have to worry about boilerplate infrastructure and scale. We will share our experience in building a scalable SPaaS using Flink, Apache Beam and Kafka as the foundation layer to process over 1.3 PB of event data without service disruption.
Rapid growth in the elderly population and the rising incidence of chronic diseases, growing need for cost-effective healthcare delivery due to the increasing healthcare costs, and technological advancements of home care devices. In 2017, there were 3.8 million Australians aged 65 and over (comprising 15% of the total population). The number and proportion of older Australians are expected to continue to grow. The aging population demands more patient-centric healthcare services, which in return increases the demand for healthcare workers and agencies and is anticipated to drive market growth.
The growth of the aging population segment will boost the demand for healthcare and significantly increase the burden on governments and health systems as the aging population is more prone to chronic diseases. This will prove favorable to the market for home healthcare, which can enhance the overall reach of and access to healthcare while reducing unnecessary visits, hospital admissions & readmissions, and the time and costs involved in traveling to meet healthcare professionals.
Visit https://insights10.com/ for more healthcare industry insights.
Connect with us at info@insights10.com
UK Coronary Stents Market Analysis Sample ReportInsights10
The UK Coronary Stents Market is expected to witness growth from $294 Mn in 2022 to $365 Mn in 2030 with a CAGR of 2.75% for the forecasted year 2022-2030. In the UK, coronary artery disease affects a significant number of people, creating a serious health issue. The demand for coronary stents is anticipated to rise as the population ages and risk factors like obesity and diabetes become more common. To get a detailed report, contact us at - info@insights10.com
Overall, the report represents the Global Flexible endoscope market trends along with a market forecast that will help industry consultants, technology providers, existing players searching for expansion opportunities, new players searching possibilities and other stakeholders to align their market-centric strategies according to the ongoing and expected trends in the future.
Norway Medical and Diagnostic Laboratory Service Market Analysis Sample ReportInsights10
This report presents a strategic analysis of the Norway Medical and Diagnostic Laboratory Service Market and a forecast for its development in the medium and long term. It provides a broad overview of the market dynamics, trends and insights, growth drivers and restraints, segmentation, competitive landscape, healthcare policies and regulatory framework, reimbursement scenario, challenges and future outlook. To get a detailed report, contact us at - info@insights10.com
Coronavirus Diagnostics Market - Global Industry Size, Share, Trend, Opportun...TechSci Research
According to #TechSci Research report, Global Coronavirus Diagnostics Market was valued at USD73538.02 million in 2020 and is expected to reach USD15579.56 million by 2026.
Gain More Insight: https://bit.ly/3eh734y
Get Sample Report: https://bit.ly/3njrkuw
Press Release: https://bit.ly/33usYCr
Website: https://www.techsciresearch.com/
Market Research News: https://techsciblog.com/
Global Clinical Trial Patient Recruitment Services Market Analysis Sample ReportInsights10
The Global clinical trial patient recruitment services market is projected to grow from $ 0.84 Mn in 2022 to $1.56 Bn by 2030, registering a CAGR of 8% during the forecast period of 2022-30. The Global clinical trial patient recruitment services market is a rapidly growing market that provides services to support the recruitment of patients for clinical trials. To get a detailed report, contact us at - info@insights10.com
Global Veterinary Reference Laboratory (VRL) Market Research, 2018 to 2025Signitech
The global veterinary reference laboratory (VRL) market size was valued at USD 2.350 billion in 2016 and is expected to grow with CAGR of 9.1% during the forecast period.
Pulse Oximeters Market Size, Share & Trends Analysis Report By Type (Fingertip, Handheld), By End Use (Hospitals & Healthcare Facilities, Homecare), By Region, And Segment Forecasts, 2019 - 2025
Essay on Television for School Students & Children | Long, Short Essays. Essay about television - Selfguidedlife. Importance of television essay. 009 Essay Example Television Essays Thumb Regulation Free On Shows In .... Essay On Television | Television Essay for Students and Children in .... Essay on Television advantages and disadvantages in English. Positive And Negative Effects Of Television Essay Example for Free .... How To Write A Tv Show In An Essay - How to Write a Television Show .... The disadvantage about television Free Essay Example. Essay on Impact of Television Programmes on Children - YouTube.
Global immunity boosting products market is expected to witness growth over the next five years and surpass $ 17 billion by 2025, on account of growing urbanization, rising concerns for maintaining good health and lifestyle coupled with increasing personal disposable income. Click :https://bit.ly/2Fm2ess
Capsule Endoscopy Market Is Anticipated To Touch 886.3 Million In 2022: Grand...Amit M
The Global Capsule Endoscopy Market is expected to reach USD 886.3 million by 2022; according to a new study by Grand View Research, Inc. Increasing prevalence of colorectal cancer, GI diseases and growing demand for precise and accurate diagnostic tools are expected to drive the capsule endoscopy market.
Canada's clinical diagnostics is expected to witness significant growth from $3.9 Bn in 2022 to $4.9 Bn in 2030 with a CAGR of 3.1% for the year 2022-30. The main growth drivers of the Canadian clinical diagnostics market are growing awareness about early disease detection and the benefits of early treatment. Further, the increasing investments by the government in diagnostic services are responsible for the market growth.To get a detailed report, contact us at - info@insights10.com
A workshop hosted by the South African Journal of Science aimed at postgraduate students and early career researchers with little or no experience in writing and publishing journal articles.
A review of the growth of the Israel Genealogy Research Association Database Collection for the last 12 months. Our collection is now passed the 3 million mark and still growing. See which archives have contributed the most. See the different types of records we have, and which years have had records added. You can also see what we have for the future.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
Normal Labour/ Stages of Labour/ Mechanism of LabourWasim Ak
Normal labor is also termed spontaneous labor, defined as the natural physiological process through which the fetus, placenta, and membranes are expelled from the uterus through the birth canal at term (37 to 42 weeks
Safalta Digital marketing institute in Noida, provide complete applications that encompass a huge range of virtual advertising and marketing additives, which includes search engine optimization, virtual communication advertising, pay-per-click on marketing, content material advertising, internet analytics, and greater. These university courses are designed for students who possess a comprehensive understanding of virtual marketing strategies and attributes.Safalta Digital Marketing Institute in Noida is a first choice for young individuals or students who are looking to start their careers in the field of digital advertising. The institute gives specialized courses designed and certification.
for beginners, providing thorough training in areas such as SEO, digital communication marketing, and PPC training in Noida. After finishing the program, students receive the certifications recognised by top different universitie, setting a strong foundation for a successful career in digital marketing.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
2. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data Examples of big time series 2
3. 1. Australian tourism demand
Visualising and forecasting big time series data Examples of big time series 3
4. 1. Australian tourism demand
Visualising and forecasting big time series data Examples of big time series 3
Quarterly data on visitor night from
1998:Q1 – 2013:Q4
From: National Visitor Survey, based on
annual interviews of 120,000 Australians
aged 15+, collected by Tourism Research
Australia.
Split by 7 states, 27 zones and 76 regions
(a geographical hierarchy)
Also split by purpose of travel
Holiday
Visiting friends and relatives (VFR)
Business
Other
304 bottom-level series
5. 2. Labour market participation
Australia and New Zealand Standard
Classification of Occupations
8 major groups
43 sub-major groups
97 minor groups
– 359 unit groups
* 1023 occupations
Example: statistician
2 Professionals
22 Business, Human Resource and Marketing
Professionals
224 Information and Organisation Professionals
2241 Actuaries, Mathematicians and Statisticians
224113 Statistician
Visualising and forecasting big time series data Examples of big time series 4
6. 2. Labour market participation
Australia and New Zealand Standard
Classification of Occupations
8 major groups
43 sub-major groups
97 minor groups
– 359 unit groups
* 1023 occupations
Example: statistician
2 Professionals
22 Business, Human Resource and Marketing
Professionals
224 Information and Organisation Professionals
2241 Actuaries, Mathematicians and Statisticians
224113 Statistician
Visualising and forecasting big time series data Examples of big time series 4
8. 3. PBS sales
ATC drug classification
A Alimentary tract and metabolism
B Blood and blood forming organs
C Cardiovascular system
D Dermatologicals
G Genito-urinary system and sex hormones
H Systemic hormonal preparations, excluding sex hormones
and insulins
J Anti-infectives for systemic use
L Antineoplastic and immunomodulating agents
M Musculo-skeletal system
N Nervous system
P Antiparasitic products, insecticides and repellents
R Respiratory system
S Sensory organs
V Various
Visualising and forecasting big time series data Examples of big time series 6
9. 3. PBS sales
ATC drug classification
A Alimentary tract and metabolism14 classes
A10 Drugs used in diabetes84 classes
A10B Blood glucose lowering drugs
A10BA Biguanides
A10BA02 Metformin
Visualising and forecasting big time series data Examples of big time series 7
10. 4. Spectacle sales
Visualising and forecasting big time series data Examples of big time series 8
Monthly sales data from 2000 – 2014
Provided by a large spectacle manufacturer
Split by brand (26), gender (3), price range
(6), materials (4), and stores (600)
About a million bottom-level series
11. 4. Spectacle sales
Visualising and forecasting big time series data Examples of big time series 8
Monthly sales data from 2000 – 2014
Provided by a large spectacle manufacturer
Split by brand (26), gender (3), price range
(6), materials (4), and stores (600)
About a million bottom-level series
12. 4. Spectacle sales
Visualising and forecasting big time series data Examples of big time series 8
Monthly sales data from 2000 – 2014
Provided by a large spectacle manufacturer
Split by brand (26), gender (3), price range
(6), materials (4), and stores (600)
About a million bottom-level series
13. 4. Spectacle sales
Visualising and forecasting big time series data Examples of big time series 8
Monthly sales data from 2000 – 2014
Provided by a large spectacle manufacturer
Split by brand (26), gender (3), price range
(6), materials (4), and stores (600)
About a million bottom-level series
14. Hierarchical time series
A hierarchical time series is a collection of
several time series that are linked together in
a hierarchical structure.
Total
A
AA AB AC
B
BA BB BC
C
CA CB CC
Examples
Net labour turnover
Pharmaceutical sales
Tourism by state and region
Visualising and forecasting big time series data Examples of big time series 9
15. Hierarchical time series
A hierarchical time series is a collection of
several time series that are linked together in
a hierarchical structure.
Total
A
AA AB AC
B
BA BB BC
C
CA CB CC
Examples
Net labour turnover
Pharmaceutical sales
Tourism by state and region
Visualising and forecasting big time series data Examples of big time series 9
16. Hierarchical time series
A hierarchical time series is a collection of
several time series that are linked together in
a hierarchical structure.
Total
A
AA AB AC
B
BA BB BC
C
CA CB CC
Examples
Net labour turnover
Pharmaceutical sales
Tourism by state and region
Visualising and forecasting big time series data Examples of big time series 9
17. Hierarchical time series
A hierarchical time series is a collection of
several time series that are linked together in
a hierarchical structure.
Total
A
AA AB AC
B
BA BB BC
C
CA CB CC
Examples
Net labour turnover
Pharmaceutical sales
Tourism by state and region
Visualising and forecasting big time series data Examples of big time series 9
18. Grouped time series
A grouped time series is a collection of time
series that can be grouped together in a
number of non-hierarchical ways.
Total
A
AX AY
B
BX BY
Total
X
AX BX
Y
AY BY
Examples
Tourism by state and purpose of travel
Glasses by brand and store
Visualising and forecasting big time series data Examples of big time series 10
19. Grouped time series
A grouped time series is a collection of time
series that can be grouped together in a
number of non-hierarchical ways.
Total
A
AX AY
B
BX BY
Total
X
AX BX
Y
AY BY
Examples
Tourism by state and purpose of travel
Glasses by brand and store
Visualising and forecasting big time series data Examples of big time series 10
20. Grouped time series
A grouped time series is a collection of time
series that can be grouped together in a
number of non-hierarchical ways.
Total
A
AX AY
B
BX BY
Total
X
AX BX
Y
AY BY
Examples
Tourism by state and purpose of travel
Glasses by brand and store
Visualising and forecasting big time series data Examples of big time series 10
21. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data Time series visualisation 11
23. Kite diagrams
000
Line graph profile
Duplicate & flip
around the hori-
zontal axis
Fill the colour
Visualising and forecasting big time series data Time series visualisation 13
24. Kite diagrams: Victorian tourism
20002010
Holiday
20002010
VFR
20002010
Business
20002010
BAA
BAB
BAC
BBA
BCA
BCB
BCC
BDA
BDB
BDC
BDD
BDE
BDF
BEA
BEB
BEC
BED
BEE
BEF
Other
BEG
Victoria
Visualising and forecasting big time series data Time series visualisation 14
25. Kite diagrams: Victorian tourism
Visualising and forecasting big time series data Time series visualisation 14
26. Kite diagrams: Victorian tourism
20002010
Holiday
20002010
VFR
20002010
Business
20002010
BAA
BAB
BAC
BBA
BCA
BCB
BCC
BDA
BDB
BDC
BDD
BDE
BDF
BEA
BEB
BEC
BED
BEE
BEF
Other
BEG
Victoria: scaled
Visualising and forecasting big time series data Time series visualisation 14
27. An STL decomposition
STL decomposition of tourism demand
for holidays in Peninsula
5.06.07.0
data
−0.50.5
seasonal
5.86.16.4
trend
−0.40.0
2000 2005 2010
remainder
Visualising and forecasting big time series data Time series visualisation 15
28. Seasonal stacked bar chart
Place positive values above the origin
while negative values below the origin
Map the bar length to the magnitude
Encode quarters by colours
Visualising and forecasting big time series data Time series visualisation 16
29. Seasonal stacked bar chart
Place positive values above the origin
while negative values below the origin
Map the bar length to the magnitude
Encode quarters by colours
−1.0
−0.5
0.0
0.5
1.0
Holiday
BAA BABBACBBABCABCBBCCBDABDBBDCBDDBDEBDF BEA BEBBECBEDBEE BEFBEG
Regions
SeasonalComponent
Qtr
Q1
Q2
Q3
Q4
Visualising and forecasting big time series data Time series visualisation 16
30. Seasonal stacked bar chart: VIC
Visualising and forecasting big time series data Time series visualisation 17
31. Seasonal stacked bar chart: VIC
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
−1.0
−0.5
0.0
0.5
1.0
HolidayVFRBusinessOther
BAABABBACBBABCABCBBCCBDABDBBDCBDDBDEBDFBEABEBBECBEDBEEBEFBEG
Regions
SeasonalComponent
Qtr
Q1
Q2
Q3
Q4
Visualising and forecasting big time series data Time series visualisation 17
32. Corrgram of remainder
Visualising and forecasting big time series data Time series visualisation 18
Compute the correlations
among the remainder
components
Render both the sign and
magnitude using a colour
mapping of two hues
Order variables according to
the first principal component of
the correlations.
33. Corrgram of remainder: VIC
Visualising and forecasting big time series data Time series visualisation 19
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
BEEHolBEFOthBEEOthBDEOthBEBOthBEABusBEFBusBDCOthBACHolBEBBusBEAVisBBAHolBDEHolBABOthBAAVisBAAHolBDCHolBBABusBCBHolBEGBusBDDVisBABVisBDAVisBEAOthBDFHolBEEBusBAAOthBACOthBDAOthBDEBusBCBOthBACBusBEBVisBACVisBCAOthBEFVisBCBVisBEDHolBEGOthBDBHolBABBusBEBHolBDFBusBECHolBCAHolBDBOthBEAHolBDCBusBECVisBDBVisBCCHolBBAVisBABHolBBAOthBCCOthBCBBusBCCVisBEGVisBDDHolBECOthBDCVisBAABusBCCBusBECBusBCAVisBDFVisBEGHolBDDOthBEDOthBEDVisBDDBusBDEVisBEFHolBEEVisBDBBusBDABusBDAHolBCABusBDFOthBEDBus
BEEHolBEFOthBEEOthBDEOthBEBOthBEABusBEFBusBDCOthBACHolBEBBusBEAVisBBAHolBDEHolBABOthBAAVisBAAHolBDCHolBBABusBCBHolBEGBusBDDVisBABVisBDAVisBEAOthBDFHolBEEBusBAAOthBACOthBDAOthBDEBusBCBOthBACBusBEBVisBACVisBCAOthBEFVisBCBVisBEDHolBEGOthBDBHolBABBusBEBHolBDFBusBECHolBCAHolBDBOthBEAHolBDCBusBECVisBDBVisBCCHolBBAVisBABHolBBAOthBCCOthBCBBusBCCVisBEGVisBDDHolBECOthBDCVisBAABusBCCBusBECBusBCAVisBDFVisBEGHolBDDOthBEDOthBEDVisBDDBusBDEVisBEFHolBEEVisBDBBusBDABusBDAHolBCABusBDFOthBEDBus
34. Corrgram of remainder: VIC
Visualising and forecasting big time series data Time series visualisation 19
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
BDAHol
BDDHol
BEBHol
BEFHol
BECHol
BEDHol
BDFHol
BCCHol
BDCHol
BCAHol
BEAHol
BEGHol
BBAHol
BAAHol
BABHol
BDBHol
BDEHol
BACHol
BCBHol
BEEHol
BDAHol
BDDHol
BEBHol
BEFHol
BECHol
BEDHol
BDFHol
BCCHol
BDCHol
BCAHol
BEAHol
BEGHol
BBAHol
BAAHol
BABHol
BDBHol
BDEHol
BACHol
BCBHol
BEEHol
35. Corrgram of remainder: TAS
Visualising and forecasting big time series data Time series visualisation 20
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
FCAHol
FBBHol
FBAHol
FAAHol
FCBHol
FCAVis
FBBVis
FAAVis
FCBBus
FAAOth
FCAOth
FBBOth
FBABus
FBAOth
FCBVis
FCABus
FBAVis
FCBOth
FBBBus
FAABus
FCAHol
FBBHol
FBAHol
FAAHol
FCBHol
FCAVis
FBBVis
FAAVis
FCBBus
FAAOth
FCAOth
FBBOth
FBABus
FBAOth
FCBVis
FCABus
FBAVis
FCBOth
FBBBus
FAABus
36. Principal components decomposition
Visualising and forecasting big time series data Time series visualisation 21
−25−15−55
PC1
−50510
PC2
−50510
2000 2005 2010
PC3
Time
First three PCs
42. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
43. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
44. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
45. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
46. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
47. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
48. Feature analysis
Summarize each time series with a feature
vector:
strength of trend
summer seasonality
winter seasonality
Box-Pierce statistic on remainder of STL
Lumpiness (variance of annual variances of
remainder)
Do PCA on feature matrix
Visualising and forecasting big time series data Time series visualisation 23
51. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25
52. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
53. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
54. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
55. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
56. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
57. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
58. Hierarchical/grouped time series
Forecasts should be “aggregate
consistent”, unbiased, minimum variance.
Existing methods:
¢ Bottom-up
¢ Top-down
¢ Middle-out
How to compute forecast intervals?
Most research is concerned about relative
performance of existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26
59. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
60. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
61. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
62. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
63. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
64. Top-down method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27
Advantages
Works well in
presence of low
counts.
Single forecasting
model easy to
build
Provides reliable
forecasts for
aggregate levels.
Disadvantages
Loss of information,
especially
individual series
dynamics.
Distribution of
forecasts to lower
levels can be
difficult
No prediction
intervals
65. Bottom-up method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 28
Advantages
No loss of
information.
Better captures
dynamics of
individual series.
Disadvantages
Large number of
series to be
forecast.
Constructing
forecasting models
is harder because
of noisy data at
bottom level.
No prediction
intervals
66. Bottom-up method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 28
Advantages
No loss of
information.
Better captures
dynamics of
individual series.
Disadvantages
Large number of
series to be
forecast.
Constructing
forecasting models
is harder because
of noisy data at
bottom level.
No prediction
intervals
67. Bottom-up method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 28
Advantages
No loss of
information.
Better captures
dynamics of
individual series.
Disadvantages
Large number of
series to be
forecast.
Constructing
forecasting models
is harder because
of noisy data at
bottom level.
No prediction
intervals
68. Bottom-up method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 28
Advantages
No loss of
information.
Better captures
dynamics of
individual series.
Disadvantages
Large number of
series to be
forecast.
Constructing
forecasting models
is harder because
of noisy data at
bottom level.
No prediction
intervals
69. Bottom-up method
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 28
Advantages
No loss of
information.
Better captures
dynamics of
individual series.
Disadvantages
Large number of
series to be
forecast.
Constructing
forecasting models
is harder because
of noisy data at
bottom level.
No prediction
intervals
70. The BLUF approach
Hyndman et al (CSDA 2011) proposed a new
statistical framework for forecasting
hierarchical time series which:
1 provides point forecasts that are
consistent across the hierarchy;
2 allows for correlations and interaction
between series at each level;
3 provides estimates of forecast uncertainty
which are consistent across the hierarchy;
4 allows for ad hoc adjustments and
inclusion of covariates at any level.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 29
71. The BLUF approach
Hyndman et al (CSDA 2011) proposed a new
statistical framework for forecasting
hierarchical time series which:
1 provides point forecasts that are
consistent across the hierarchy;
2 allows for correlations and interaction
between series at each level;
3 provides estimates of forecast uncertainty
which are consistent across the hierarchy;
4 allows for ad hoc adjustments and
inclusion of covariates at any level.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 29
72. The BLUF approach
Hyndman et al (CSDA 2011) proposed a new
statistical framework for forecasting
hierarchical time series which:
1 provides point forecasts that are
consistent across the hierarchy;
2 allows for correlations and interaction
between series at each level;
3 provides estimates of forecast uncertainty
which are consistent across the hierarchy;
4 allows for ad hoc adjustments and
inclusion of covariates at any level.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 29
73. The BLUF approach
Hyndman et al (CSDA 2011) proposed a new
statistical framework for forecasting
hierarchical time series which:
1 provides point forecasts that are
consistent across the hierarchy;
2 allows for correlations and interaction
between series at each level;
3 provides estimates of forecast uncertainty
which are consistent across the hierarchy;
4 allows for ad hoc adjustments and
inclusion of covariates at any level.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 29
74. Hierarchical data
Total
A B C
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
75. Hierarchical data
Total
A B C
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
76. Hierarchical data
Total
A B C
yt = [Yt, YA,t, YB,t, YC,t] =
1 1 1
1 0 0
0 1 0
0 0 1
YA,t
YB,t
YC,t
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
77. Hierarchical data
Total
A B C
yt = [Yt, YA,t, YB,t, YC,t] =
1 1 1
1 0 0
0 1 0
0 0 1
S
YA,t
YB,t
YC,t
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
78. Hierarchical data
Total
A B C
yt = [Yt, YA,t, YB,t, YC,t] =
1 1 1
1 0 0
0 1 0
0 0 1
S
YA,t
YB,t
YC,t
Bt
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
79. Hierarchical data
Total
A B C
yt = [Yt, YA,t, YB,t, YC,t] =
1 1 1
1 0 0
0 1 0
0 0 1
S
YA,t
YB,t
YC,t
Bt
yt = SBt
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 30
Yt : observed aggregate of all
series at time t.
YX,t : observation on series X at
time t.
Bt : vector of all series at
bottom level in time t.
83. Grouped data
AX AY A
BX BY B
X Y Total
yt =
Yt
YA,t
YB,t
YX,t
YY,t
YAX,t
YAY,t
YBX,t
YBY,t
=
1 1 1 1
1 1 0 0
0 0 1 1
1 0 1 0
0 1 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
S
YAX,t
YAY,t
YBX,t
YBY,t
Bt
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 32
84. Grouped data
AX AY A
BX BY B
X Y Total
yt =
Yt
YA,t
YB,t
YX,t
YY,t
YAX,t
YAY,t
YBX,t
YBY,t
=
1 1 1 1
1 1 0 0
0 0 1 1
1 0 1 0
0 1 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
S
YAX,t
YAY,t
YBX,t
YBY,t
Bt
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 32
85. Grouped data
AX AY A
BX BY B
X Y Total
yt =
Yt
YA,t
YB,t
YX,t
YY,t
YAX,t
YAY,t
YBX,t
YBY,t
=
1 1 1 1
1 1 0 0
0 0 1 1
1 0 1 0
0 1 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
S
YAX,t
YAY,t
YBX,t
YBY,t
Bt
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 32
yt = SBt
86. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
87. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
88. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
89. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
90. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
91. Forecasting notation
Let ˆyn(h) be vector of initial h-step forecasts,
made at time n, stacked in same order as yt.
(They may not add up.)
Hierarchical forecasting methods of the form:
˜yn(h) = SPˆyn(h)
for some matrix P.
P extracts and combines base forecasts
ˆyn(h) to get bottom-level forecasts.
S adds them up
Revised reconciled forecasts: ˜yn(h).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 33
92. Bottom-up forecasts
˜yn(h) = SPˆyn(h)
Bottom-up forecasts are obtained using
P = [0 | I] ,
where 0 is null matrix and I is identity matrix.
P matrix extracts only bottom-level
forecasts from ˆyn(h)
S adds them up to give the bottom-up
forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 34
93. Bottom-up forecasts
˜yn(h) = SPˆyn(h)
Bottom-up forecasts are obtained using
P = [0 | I] ,
where 0 is null matrix and I is identity matrix.
P matrix extracts only bottom-level
forecasts from ˆyn(h)
S adds them up to give the bottom-up
forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 34
94. Bottom-up forecasts
˜yn(h) = SPˆyn(h)
Bottom-up forecasts are obtained using
P = [0 | I] ,
where 0 is null matrix and I is identity matrix.
P matrix extracts only bottom-level
forecasts from ˆyn(h)
S adds them up to give the bottom-up
forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 34
95. Top-down forecasts
˜yn(h) = SPˆyn(h)
Top-down forecasts are obtained using
P = [p | 0]
where p = [p1, p2, . . . , pmK
] is a vector of
proportions that sum to one.
P distributes forecasts of the aggregate to
the lowest level series.
Different methods of top-down forecasting
lead to different proportionality vectors p.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 35
96. Top-down forecasts
˜yn(h) = SPˆyn(h)
Top-down forecasts are obtained using
P = [p | 0]
where p = [p1, p2, . . . , pmK
] is a vector of
proportions that sum to one.
P distributes forecasts of the aggregate to
the lowest level series.
Different methods of top-down forecasting
lead to different proportionality vectors p.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 35
97. Top-down forecasts
˜yn(h) = SPˆyn(h)
Top-down forecasts are obtained using
P = [p | 0]
where p = [p1, p2, . . . , pmK
] is a vector of
proportions that sum to one.
P distributes forecasts of the aggregate to
the lowest level series.
Different methods of top-down forecasting
lead to different proportionality vectors p.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 35
98. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
99. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
100. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
101. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
102. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
103. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
104. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
105. General properties: bias
˜yn(h) = SPˆyn(h)
Assume: base forecasts ˆyn(h) are unbiased:
E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn]
Let ˆBn(h) be bottom level base forecasts
with βn(h) = E[ˆBn(h)|y1, . . . , yn].
Then E[ˆyn(h)] = Sβn(h).
We want the revised forecasts to be unbiased:
E[˜yn(h)] = SPSβn(h) = Sβn(h).
Result will hold provided SPS = S.
True for bottom-up, but not for any top-down
method or middle-out method.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 36
106. General properties: variance
˜yn(h) = SPˆyn(h)
Let variance of base forecasts ˆyn(h) be given
by
Σh = Var[ˆyn(h)|y1, . . . , yn]
Then the variance of the revised forecasts is
given by
Var[˜yn(h)|y1, . . . , yn] = SPΣhP S .
This is a general result for all existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 37
107. General properties: variance
˜yn(h) = SPˆyn(h)
Let variance of base forecasts ˆyn(h) be given
by
Σh = Var[ˆyn(h)|y1, . . . , yn]
Then the variance of the revised forecasts is
given by
Var[˜yn(h)|y1, . . . , yn] = SPΣhP S .
This is a general result for all existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 37
108. General properties: variance
˜yn(h) = SPˆyn(h)
Let variance of base forecasts ˆyn(h) be given
by
Σh = Var[ˆyn(h)|y1, . . . , yn]
Then the variance of the revised forecasts is
given by
Var[˜yn(h)|y1, . . . , yn] = SPΣhP S .
This is a general result for all existing methods.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 37
109. BLUF via trace minimization
Theorem
For any P satisfying SPS = S, then
min
P
= trace[SPΣhP S ]
has solution
P = (S Σ†
hS)−1
S Σ†
h.
Σ†
h is generalized inverse of Σh.
Equivalent to GLS estimate of regression
ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 38
110. BLUF via trace minimization
Theorem
For any P satisfying SPS = S, then
min
P
= trace[SPΣhP S ]
has solution
P = (S Σ†
hS)−1
S Σ†
h.
Σ†
h is generalized inverse of Σh.
Equivalent to GLS estimate of regression
ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 38
111. BLUF via trace minimization
Theorem
For any P satisfying SPS = S, then
min
P
= trace[SPΣhP S ]
has solution
P = (S Σ†
hS)−1
S Σ†
h.
Σ†
h is generalized inverse of Σh.
Equivalent to GLS estimate of regression
ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh).
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 38
112. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
113. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
114. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
115. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
116. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
117. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
118. Optimal combination forecasts
˜yn(h) = SPˆyn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Initial forecasts
Σ†
h is generalized inverse of Σh.
Var[˜yn(h)|y1, . . . , yn] = S(S Σ†
hS)−1
S
Problem: Σh hard to estimate.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 39
119. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
120. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
121. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
122. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
123. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
124. Optimal combination forecasts
˜yn(h) = S(S Σ†
hS)−1
S Σ†
hˆyn(h)
Revised forecasts Base forecasts
Solution 1: OLS
Assume εh ≈ SεB,h where εB,h is the
forecast error at bottom level.
Then Σh ≈ SΩhS where Ωh = Var(εB,h).
If Moore-Penrose generalized inverse used,
then (S Σ†
hS)−1
S Σ†
h = (S S)−1
S .
˜yn(h) = S(S S)−1
S ˆyn(h)
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 40
126. Optimal combination forecasts
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 41
˜yn(h) = S(S S)−1
S ˆyn(h)Total
A B C
Weights:
S(S S)−1
S =
0.75 0.25 0.25 0.25
0.25 0.75 −0.25 −0.25
0.25 −0.25 0.75 −0.25
0.25 −0.25 −0.25 0.75
129. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
130. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
131. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
132. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
133. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
134. Features
Covariates can be included in initial forecasts.
Adjustments can be made to initial forecasts
at any level.
Very simple and flexible method. Can work
with any hierarchical or grouped time series.
SPS = S so reconciled forcasts are unbiased.
Conceptually easy to implement: OLS on
base forecasts.
Weights are independent of the data and of
the covariance structure of the hierarchy.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 43
135. Challenges
Computational difficulties in big
hierarchies due to size of the S matrix and
singular behavior of (S S).
Need to estimate covariance matrix to
produce prediction intervals.
Ignores covariance matrix in computing
point forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 44
˜yn(h) = S(S S)−1
S ˆyn(h)
136. Challenges
Computational difficulties in big
hierarchies due to size of the S matrix and
singular behavior of (S S).
Need to estimate covariance matrix to
produce prediction intervals.
Ignores covariance matrix in computing
point forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 44
˜yn(h) = S(S S)−1
S ˆyn(h)
137. Challenges
Computational difficulties in big
hierarchies due to size of the S matrix and
singular behavior of (S S).
Need to estimate covariance matrix to
produce prediction intervals.
Ignores covariance matrix in computing
point forecasts.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 44
˜yn(h) = S(S S)−1
S ˆyn(h)
138. Optimal combination forecasts
Solution 1: OLS
Approximate Σ†
1 by cI.
Solution 2: Rescaling
Suppose we approximate Σ1 by its
diagonal.
Let Λ = diagonal Σ1
−1
contain inverse
one-step forecast variances.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 45
˜yn(h) = S(S Σ†
1S)−1
S Σ†
1ˆyn(h)
˜yn(h) = S(SΛS)−1
SΛˆyn(h)
139. Optimal combination forecasts
Solution 1: OLS
Approximate Σ†
1 by cI.
Solution 2: Rescaling
Suppose we approximate Σ1 by its
diagonal.
Let Λ = diagonal Σ1
−1
contain inverse
one-step forecast variances.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 45
˜yn(h) = S(S Σ†
1S)−1
S Σ†
1ˆyn(h)
˜yn(h) = S(SΛS)−1
SΛˆyn(h)
140. Optimal combination forecasts
Solution 1: OLS
Approximate Σ†
1 by cI.
Solution 2: Rescaling
Suppose we approximate Σ1 by its
diagonal.
Let Λ = diagonal Σ1
−1
contain inverse
one-step forecast variances.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 45
˜yn(h) = S(S Σ†
1S)−1
S Σ†
1ˆyn(h)
˜yn(h) = S(SΛS)−1
SΛˆyn(h)
141. Optimal combination forecasts
Solution 1: OLS
Approximate Σ†
1 by cI.
Solution 2: Rescaling
Suppose we approximate Σ1 by its
diagonal.
Let Λ = diagonal Σ1
−1
contain inverse
one-step forecast variances.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 45
˜yn(h) = S(S Σ†
1S)−1
S Σ†
1ˆyn(h)
˜yn(h) = S(SΛS)−1
SΛˆyn(h)
142. Optimal combination forecasts
Solution 1: OLS
Approximate Σ†
1 by cI.
Solution 2: Rescaling
Suppose we approximate Σ1 by its
diagonal.
Let Λ = diagonal Σ1
−1
contain inverse
one-step forecast variances.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 45
˜yn(h) = S(S Σ†
1S)−1
S Σ†
1ˆyn(h)
˜yn(h) = S(SΛS)−1
SΛˆyn(h)
143. Optimal reconciled forecasts
˜yn(h) = S ˆβn(h) = S(S ΛS)−1
S Λˆyn(h)
Easy to estimate, and places weight where
we have best forecasts.
Ignores covariances.
For large numbers of time series, we need
to do calculation without explicitly forming
S or (SΛS)−1
or SΛ.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 46
144. Optimal reconciled forecasts
˜yn(h) = S ˆβn(h) = S(S ΛS)−1
S Λˆyn(h)
Initial forecasts
Easy to estimate, and places weight where
we have best forecasts.
Ignores covariances.
For large numbers of time series, we need
to do calculation without explicitly forming
S or (SΛS)−1
or SΛ.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 46
145. Optimal reconciled forecasts
˜yn(h) = S ˆβn(h) = S(S ΛS)−1
S Λˆyn(h)
Revised forecasts Initial forecasts
Easy to estimate, and places weight where
we have best forecasts.
Ignores covariances.
For large numbers of time series, we need
to do calculation without explicitly forming
S or (SΛS)−1
or SΛ.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 46
146. Optimal reconciled forecasts
˜yn(h) = S ˆβn(h) = S(S ΛS)−1
S Λˆyn(h)
Revised forecasts Initial forecasts
Easy to estimate, and places weight where
we have best forecasts.
Ignores covariances.
For large numbers of time series, we need
to do calculation without explicitly forming
S or (SΛS)−1
or SΛ.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 46
147. Optimal reconciled forecasts
˜yn(h) = S ˆβn(h) = S(S ΛS)−1
S Λˆyn(h)
Revised forecasts Initial forecasts
Easy to estimate, and places weight where
we have best forecasts.
Ignores covariances.
For large numbers of time series, we need
to do calculation without explicitly forming
S or (SΛS)−1
or SΛ.
Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 46
148. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data Application: Australian tourism 47
150. Australian tourism
Visualising and forecasting big time series data Application: Australian tourism 48
Hierarchy:
States (7)
Zones (27)
Regions (82)
151. Australian tourism
Visualising and forecasting big time series data Application: Australian tourism 48
Hierarchy:
States (7)
Zones (27)
Regions (82)
Base forecasts
ETS (exponential
smoothing) models
152. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: Total
Year
Visitornights
1998 2000 2002 2004 2006 2008
600006500070000750008000085000
153. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: NSW
Year
Visitornights
1998 2000 2002 2004 2006 2008
18000220002600030000
154. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: VIC
Year
Visitornights
1998 2000 2002 2004 2006 2008
1000012000140001600018000
155. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: Nth.Coast.NSW
Year
Visitornights
1998 2000 2002 2004 2006 2008
50006000700080009000
156. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: Metro.QLD
Year
Visitornights
1998 2000 2002 2004 2006 2008
800090001100013000
157. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: Sth.WA
Year
Visitornights
1998 2000 2002 2004 2006 2008
400600800100012001400
158. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: X201.Melbourne
Year
Visitornights
1998 2000 2002 2004 2006 2008
40004500500055006000
159. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: X402.Murraylands
Year
Visitornights
1998 2000 2002 2004 2006 2008
0100200300
160. Base forecasts
Visualising and forecasting big time series data Application: Australian tourism 49
Domestic tourism forecasts: X809.Daly
Year
Visitornights
1998 2000 2002 2004 2006 2008
020406080100
162. Reconciled forecasts
Visualising and forecasting big time series data Application: Australian tourism 50
NSW
2000 2005 2010
180002400030000
VIC
2000 2005 2010
100001400018000
QLD
2000 2005 2010
1400020000
Other 2000 2005 2010
1800024000
163. Reconciled forecasts
Visualising and forecasting big time series data Application: Australian tourism 50
Sydney
2000 2005 2010
40007000
OtherNSW
2000 2005 2010
1400022000
Melbourne
2000 2005 2010
40005000
OtherVIC
2000 2005 2010
600012000
GCandBrisbane
2000 2005 2010
60009000
OtherQLD
2000 2005 2010
600012000
Capitalcities
2000 2005 2010
1400020000
Other
2000 2005 2010
55007500
164. Forecast evaluation
Select models using all observations;
Re-estimate models using first 12
observations and generate 1- to
8-step-ahead forecasts;
Increase sample size one observation at a
time, re-estimate models, generate
forecasts until the end of the sample;
In total 24 1-step-ahead, 23
2-steps-ahead, up to 17 8-steps-ahead for
forecast evaluation.
Visualising and forecasting big time series data Application: Australian tourism 51
165. Forecast evaluation
Select models using all observations;
Re-estimate models using first 12
observations and generate 1- to
8-step-ahead forecasts;
Increase sample size one observation at a
time, re-estimate models, generate
forecasts until the end of the sample;
In total 24 1-step-ahead, 23
2-steps-ahead, up to 17 8-steps-ahead for
forecast evaluation.
Visualising and forecasting big time series data Application: Australian tourism 51
166. Forecast evaluation
Select models using all observations;
Re-estimate models using first 12
observations and generate 1- to
8-step-ahead forecasts;
Increase sample size one observation at a
time, re-estimate models, generate
forecasts until the end of the sample;
In total 24 1-step-ahead, 23
2-steps-ahead, up to 17 8-steps-ahead for
forecast evaluation.
Visualising and forecasting big time series data Application: Australian tourism 51
167. Forecast evaluation
Select models using all observations;
Re-estimate models using first 12
observations and generate 1- to
8-step-ahead forecasts;
Increase sample size one observation at a
time, re-estimate models, generate
forecasts until the end of the sample;
In total 24 1-step-ahead, 23
2-steps-ahead, up to 17 8-steps-ahead for
forecast evaluation.
Visualising and forecasting big time series data Application: Australian tourism 51
168. Hierarchy: states, zones, regions
MAPE h = 1 h = 2 h = 4 h = 6 h = 8 Average
Top Level: Australia
Bottom-up 3.79 3.58 4.01 4.55 4.24 4.06
OLS 3.83 3.66 3.88 4.19 4.25 3.94
Scaling (st. dev.) 3.68 3.56 3.97 4.57 4.25 4.04
Level: States
Bottom-up 10.70 10.52 10.85 11.46 11.27 11.03
OLS 11.07 10.58 11.13 11.62 12.21 11.35
Scaling (st. dev.) 10.44 10.17 10.47 10.97 10.98 10.67
Level: Zones
Bottom-up 14.99 14.97 14.98 15.69 15.65 15.32
OLS 15.16 15.06 15.27 15.74 16.15 15.48
Scaling (st. dev.) 14.63 14.62 14.68 15.17 15.25 14.94
Bottom Level: Regions
Bottom-up 33.12 32.54 32.26 33.74 33.96 33.18
OLS 35.89 33.86 34.26 36.06 37.49 35.43
Scaling (st. dev.) 31.68 31.22 31.08 32.41 32.77 31.89
Visualising and forecasting big time series data Application: Australian tourism 52
169. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data Application: Australian labour market 53
170. ANZSCO
Australia and New Zealand Standard
Classification of Occupations
8 major groups
43 sub-major groups
97 minor groups
– 359 unit groups
* 1023 occupations
Example: statistician
2 Professionals
22 Business, Human Resource and Marketing
Professionals
224 Information and Organisation Professionals
2241 Actuaries, Mathematicians and Statisticians
224113 Statistician
Visualising and forecasting big time series data Application: Australian labour market 54
171. ANZSCO
Australia and New Zealand Standard
Classification of Occupations
8 major groups
43 sub-major groups
97 minor groups
– 359 unit groups
* 1023 occupations
Example: statistician
2 Professionals
22 Business, Human Resource and Marketing
Professionals
224 Information and Organisation Professionals
2241 Actuaries, Mathematicians and Statisticians
224113 Statistician
Visualising and forecasting big time series data Application: Australian labour market 54
172. Australian Labour Market data
Visualising and forecasting big time series data Application: Australian labour market 55
Time
Level0
7000900011000
Time
Level1
5001000150020002500
1. Managers
2. Professionals
3. Technicians and trade workers
4. Community and personal services workers
5. Clerical and administrative workers
6. Sales workers
7. Machinery operators and drivers
8. Labourers
Time
Level2
100200300400500600700
Time
Level3
100200300400500600700
Time
Level4
1990 1995 2000 2005 2010
100200300400500
173. Australian Labour Market data
Visualising and forecasting big time series data Application: Australian labour market 55
Time
Level0
7000900011000
Time
Level1
5001000150020002500
1. Managers
2. Professionals
3. Technicians and trade workers
4. Community and personal services workers
5. Clerical and administrative workers
6. Sales workers
7. Machinery operators and drivers
8. Labourers
Time
Level2
100200300400500600700
Time
Level3
100200300400500600700
Time
Level4
1990 1995 2000 2005 2010
100200300400500
Lower three panels
show largest
sub-groups at each
level.
174. Australian Labour Market data
Visualising and forecasting big time series data Application: Australian labour market 55
Time
Level0
7000900011000
Time
Level1
5001000150020002500
1. Managers
2. Professionals
3. Technicians and trade workers
4. Community and personal services workers
5. Clerical and administrative workers
6. Sales workers
7. Machinery operators and drivers
8. Labourers
Time
Level2
100200300400500600700
Time
Level3
100200300400500600700
Time
Level4
1990 1995 2000 2005 2010
100200300400500
Time
Level0
10800112001160012000
Base forecasts
Reconciled forecasts
Time
Level1
680700720740760780800
Time
Level2
140150160170180190200
Time
Level3
140150160170180
Year
Level4
2010 2011 2012 2013 2014 2015
120130140150160
175. Australian Labour Market data
Visualising and forecasting big time series data Application: Australian labour market 55
Time
Level0
7000900011000
Time
Level1
5001000150020002500
1. Managers
2. Professionals
3. Technicians and trade workers
4. Community and personal services workers
5. Clerical and administrative workers
6. Sales workers
7. Machinery operators and drivers
8. Labourers
Time
Level2
100200300400500600700
Time
Level3
100200300400500600700
Time
Level4
1990 1995 2000 2005 2010
100200300400500
Time
Level0
10800112001160012000
Base forecasts
Reconciled forecasts
Time
Level1
680700720740760780800
Time
Level2
140150160170180190200
Time
Level3
140150160170180
Year
Level4
2010 2011 2012 2013 2014 2015
120130140150160
Base forecasts
from auto.arima()
Largest changes
shown for each
level
177. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data Fast computation tricks 57
180. Fast computation: hierarchies
Think of the hierarchy as a tree of trees:
Total
T1 T2
. . . TK
Then the summing matrix contains k smaller summing
matrices:
S =
1n1
1n2
· · · 1nK
S1 0 · · · 0
0 S2 · · · 0
...
...
...
...
0 0 · · · SK
where 1n is an n-vector of ones and tree Ti has ni
terminal nodes.
Visualising and forecasting big time series data Fast computation tricks 60
181. Fast computation: hierarchies
Think of the hierarchy as a tree of trees:
Total
T1 T2
. . . TK
Then the summing matrix contains k smaller summing
matrices:
S =
1n1
1n2
· · · 1nK
S1 0 · · · 0
0 S2 · · · 0
...
...
...
...
0 0 · · · SK
where 1n is an n-vector of ones and tree Ti has ni
terminal nodes.
Visualising and forecasting big time series data Fast computation tricks 60
182. Fast computation: hierarchies
SΛS =
S1Λ1S1 0 · · · 0
0 S2Λ2S2 · · · 0
...
... ... ...
0 0 · · · SKΛKSK
+λ0 Jn
λ0 is the top left element of Λ;
Λk is a block of Λ, corresponding to tree Tk;
Jn is a matrix of ones;
n = k nk.
Now apply the Sherman-Morrison formula . . .
Visualising and forecasting big time series data Fast computation tricks 61
183. Fast computation: hierarchies
SΛS =
S1Λ1S1 0 · · · 0
0 S2Λ2S2 · · · 0
...
... ... ...
0 0 · · · SKΛKSK
+λ0 Jn
λ0 is the top left element of Λ;
Λk is a block of Λ, corresponding to tree Tk;
Jn is a matrix of ones;
n = k nk.
Now apply the Sherman-Morrison formula . . .
Visualising and forecasting big time series data Fast computation tricks 61
184. Fast computation: hierarchies
(SΛS)−1
=
(S1Λ1S1)−1
0 · · · 0
0 (S2Λ2S2)−1
· · · 0
...
...
...
...
0 0 · · · (SKΛKSK)−1
−cS0
S0 can be partitioned into K2
blocks, with the (k, )
block (of dimension nk × n ) being
(SkΛkSk)−1
Jnk,n (S Λ S )−1
Jnk,n is a nk × n matrix of ones.
c−1
= λ−1
0 +
k
1nk
(SkΛkSk)−1
1nk
.
Each SkΛkSk can be inverted similarly.
SΛy can also be computed recursively.
Visualising and forecasting big time series data Fast computation tricks 62
185. Fast computation: hierarchies
(SΛS)−1
=
(S1Λ1S1)−1
0 · · · 0
0 (S2Λ2S2)−1
· · · 0
...
...
...
...
0 0 · · · (SKΛKSK)−1
−cS0
S0 can be partitioned into K2
blocks, with the (k, )
block (of dimension nk × n ) being
(SkΛkSk)−1
Jnk,n (S Λ S )−1
Jnk,n is a nk × n matrix of ones.
c−1
= λ−1
0 +
k
1nk
(SkΛkSk)−1
1nk
.
Each SkΛkSk can be inverted similarly.
SΛy can also be computed recursively.
Visualising and forecasting big time series data Fast computation tricks 62
The recursive calculations can be
done in such a way that we never
store any of the large matrices
involved.
186. Fast computation
When the time series are not strictly
hierarchical and have more than two grouping
variables:
Use sparse matrix storage and arithmetic.
Use iterative approximation for inverting
large sparse matrices.
Paige & Saunders (1982)
ACM Trans. Math. Software
Visualising and forecasting big time series data Fast computation tricks 63
187. Fast computation
When the time series are not strictly
hierarchical and have more than two grouping
variables:
Use sparse matrix storage and arithmetic.
Use iterative approximation for inverting
large sparse matrices.
Paige & Saunders (1982)
ACM Trans. Math. Software
Visualising and forecasting big time series data Fast computation tricks 63
188. Fast computation
When the time series are not strictly
hierarchical and have more than two grouping
variables:
Use sparse matrix storage and arithmetic.
Use iterative approximation for inverting
large sparse matrices.
Paige & Saunders (1982)
ACM Trans. Math. Software
Visualising and forecasting big time series data Fast computation tricks 63
189. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data hts package for R 64
190. hts package for R
Visualising and forecasting big time series data hts package for R 65
hts: Hierarchical and grouped time series
Methods for analysing and forecasting hierarchical and grouped
time series
Version: 4.3
Depends: forecast (≥ 5.0)
Imports: SparseM, parallel, utils
Published: 2014-06-10
Author: Rob J Hyndman, Earo Wang and Alan Lee
Maintainer: Rob J Hyndman <Rob.Hyndman at monash.edu>
BugReports: https://github.com/robjhyndman/hts/issues
License: GPL (≥ 2)
191. Example using R
library(hts)
# bts is a matrix containing the bottom level time series
# nodes describes the hierarchical structure
y <- hts(bts, nodes=list(2, c(3,2)))
Visualising and forecasting big time series data hts package for R 66
192. Example using R
library(hts)
# bts is a matrix containing the bottom level time series
# nodes describes the hierarchical structure
y <- hts(bts, nodes=list(2, c(3,2)))
Visualising and forecasting big time series data hts package for R 66
Total
A
AX AY AZ
B
BX BY
193. Example using R
library(hts)
# bts is a matrix containing the bottom level time series
# nodes describes the hierarchical structure
y <- hts(bts, nodes=list(2, c(3,2)))
# Forecast 10-step-ahead using WLS combination method
# ETS used for each series by default
fc <- forecast(y, h=10)
Visualising and forecasting big time series data hts package for R 67
194. forecast.gts function
Usage
forecast(object, h,
method = c("comb", "bu", "mo", "tdgsf", "tdgsa", "tdfp"),
fmethod = c("ets", "rw", "arima"),
weights = c("sd", "none", "nseries"),
positive = FALSE,
parallel = FALSE, num.cores = 2, ...)
Arguments
object Hierarchical time series object of class gts.
h Forecast horizon
method Method for distributing forecasts within the hierarchy.
fmethod Forecasting method to use
positive If TRUE, forecasts are forced to be strictly positive
weights Weights used for "optimal combination" method. When
weights = "sd", it takes account of the standard deviation of
forecasts.
parallel If TRUE, allow parallel processing
num.cores If parallel = TRUE, specify how many cores are going to be
used
Visualising and forecasting big time series data hts package for R 68
195. Outline
1 Examples of big time series
2 Time series visualisation
3 BLUF: Best Linear Unbiased Forecasts
4 Application: Australian tourism
5 Application: Australian labour market
6 Fast computation tricks
7 hts package for R
8 References
Visualising and forecasting big time series data References 69
196. References
RJ Hyndman, RA Ahmed, G Athanasopoulos, and
HL Shang (2011). “Optimal combination forecasts for
hierarchical time series”. Computational statistics &
data analysis 55(9), 2579–2589.
RJ Hyndman, AJ Lee, and E Wang (2014). Fast
computation of reconciled forecasts for hierarchical
and grouped time series. Working paper 17/14.
Department of Econometrics & Business Statistics,
Monash University
RJ Hyndman, AJ Lee, and E Wang (2014). hts:
Hierarchical and grouped time series.
cran.r-project.org/package=hts.
RJ Hyndman and G Athanasopoulos (2014).
Forecasting: principles and practice. OTexts.
OTexts.org/fpp/.
Visualising and forecasting big time series data References 70
197. References
RJ Hyndman, RA Ahmed, G Athanasopoulos, and
HL Shang (2011). “Optimal combination forecasts for
hierarchical time series”. Computational statistics &
data analysis 55(9), 2579–2589.
RJ Hyndman, AJ Lee, and E Wang (2014). Fast
computation of reconciled forecasts for hierarchical
and grouped time series. Working paper 17/14.
Department of Econometrics & Business Statistics,
Monash University
RJ Hyndman, AJ Lee, and E Wang (2014). hts:
Hierarchical and grouped time series.
cran.r-project.org/package=hts.
RJ Hyndman and G Athanasopoulos (2014).
Forecasting: principles and practice. OTexts.
OTexts.org/fpp/.
Visualising and forecasting big time series data References 70
¯ Papers and R code:
robjhyndman.com
¯ Email: Rob.Hyndman@monash.edu