Event description: Why Tangent Works chooses Julia: The Two Language Problem TIM: Automatic Model Building for Energy Industry Julia and its major differences to other technical computing languages (R, Matlab, ...) - Why is vectorized code fast? - Why is it not as fast as it could be? Speaker: Ján Dolinský, Tangent Works (www.tangent.works) Language of the event: Julia, Slovak & English ------------------------------------ PyData Bratislava [Python Data Enthusiasts and Users, Data Scientists & Statisticians of all levels from Slovakia] ------------------------------------ -- This meetup group is for Data Scientists, Statisticians, Economists and Data Enthusiasts using Python for data analysis and data visualization. The goals are to provide Python enthusiasts a place to share ideas and learn from each other about how best to apply the language and tools to ever-evolving challenges in the vast realm of data management, processing, analytics, and visualization. -- PyData is a group for users and developers of data analysis tools to share ideas and learn from each other. We gather to discuss how best to apply Python tools, as well as those using R and Julia, to meet the evolving challenges in data management, processing, analytics, and visualization. PyData groups, events, and conferences aim to provide a venue for users acrossall the various domains of data analysis to share their experiences and their techniques. PyData is organized by NumFOCUS.org, a 501(c)3 non-profit in the United States. ------------------------------------ Our Facebook group here: https://www.facebook.com/groups/1813599648877946/ Our Twitter account here: https://twitter.com/PyDataBA Our LinkedIn group here: https://www.linkedin.com/groups/13506080 All materials from previous meetups on GitHub here: https://github.com/GapData/PyDataBratislava Recordings of previous meetups on our YouTube here: https://www.youtube.com/watch?v=XYpKpmapqjI&list=PLISV6olKXnd9pE-KPtPgwwLe6qPXvb9K7 ------------------------------------ Organizers: GapData Institute (https://www.gapdata.org/) (GDI) is a nonprofit nonpartisan research institution harnessing power of data & wisdom of economics for public good. || Data. Think. Change. || -- NumFOCUS (http://www.numfocus.org/) is a 501(c)(3) nonprofit that supports and promotes world-class, innovative, open source scientific computing. The mission of NumFOCUS is to promote sustainable high-level programming languages, open code development, and reproducible scientific research. We accomplish this mission through our educational programs and events as well as through fiscal sponsorship of open source data science projects. We aim to increase collaboration and communication within the scientific computing community.

1. TIM: Large-scale Forecasting
for Energy in Julia
Ján Dolinský, Tangent Works #PyDataBA19. 2. 2018
(PyData Bratislava Meetup #6, Nervosa)

3. The Two Language Problem
The Philosofical Part
Ján Dolinský
jan.dolinsky@tangent.works

4. Math & Stats
Information Geometry
Computer
Science
Domain
Expertise
Machine
Learning
Traditional
Software
Traditional
Research
VALUE

5. Math & Stats
Information Geometry
Computer
Science
Domain
Expertise
Machine
Learning
Traditional
Software
Traditional
Research
VALUE
Data Science Stack:
R, Matlab, Python, Ruby, ...
Production Stack:
C++, Java, Fortran, ...

6. The Two Language Problem
Prototyping Production
Mathematicians & Data Scientists
Getting the applied math right.
Performance engineers & Architects.

7. The Two Language Problem: Tangent Works in 2014
Prototyping Production
The Data Science Stack:
R, Matlab, Octave, Python, Ruby, ….
Production stack:
C++, Fortran, Java, ...
A lot of effort ...

8. The Two Language Problem: Tangent Works in 2018
Prototyping Production
Julia
Julia
Mathematicians
Getting the applied math right.
Performance engineers & Architects.

9. TIM: Automatic Model Building for
Time-series in Energy
The Product Part
Ján Dolinský
jan.dolinsky@tangent.works

10. TIM
Tangent Information Modeller
Ján Dolinský
jan.dolinsky@tangent.works

11. Content
• Time-series Problems in Energy Industry
• Data Science Process and Automatic Model Building
• Live Modeling Demonstration
• Large-scale Forecasting Systems & Why Automation Matters
• Julia Language
• GefCom 2014, 2017 results & Summary

12. Time-series Problems in Energy Industry
• Electricity Load (aggregated and individual)
• Technical Losses
• System Imbalance
• Gas Consumption (District Heating)
• District Cooling
• Wind Production
• Solar Production
• Electricity Price
Consumption side Production side

13. 1. Collect 2. Store 3. Transform 4. Model 5. Predict 6. Present
Scada Systems, Databases, Time Series Management
Oracle, MySQL, SAP HANA, Hadoop, ...
SAS, SPSS, R, ML libraries, Matlab Power BI, SAP,
QlickView etc ..
Automation Using TIM
Data Science Process

14. 4. Model 5. Predict
SAS, SPSS, R, ML libraries, Matlab
Automation Using TIM
Data Science Process
●
feature engineering
●
model selection
●
tedious
●
costs money

15. Data Science Process
y(k)=f (Temp(k−3),Temp(k−22)∗Wind(k−1), y(k−24))
NN, SVM, MARS, LASSO, RF, … ?
Which features and how many ?
// optional

16. Data Science Process
y(k)=f (Temp(k−3),Temp(k−22)∗Wind(k−1), y(k−24))
Temp(k-1), Temp(k-2), …, Temp(k-24)
Wind(k-1), Wind(k-2), …, Wind(k-24)
y(k-1), y(k-2), …, y(k-24)
Temp(k-1), Temp(k-2), …, Temp(k-96)
Wind(k-1), Wind(k-2), …, Wind(k-96)
y(k-1), y(k-2), …, y(k-96)
24 + 24 + 24 + 24*24 = 24*27 = 648 96 + 96 + 96 + 96*96 = 96*99 = 9504
// optional

17. TIM: As a Large-Scale Model Building Engine
TIM: As a Data Science Tool
Business User Mode produces high-quality models.
Advanced User Mode allows to explore new scenarios quickly.
Fine-tuning possible for critical assets.

18. Live Demonstration of TIM

19. GefCom 2017 ex-ante results
• Qualifying Match: 1st
place out of 177 teams
• Final Match: 2nd
place out of 12 pre-selected teams
Andritz Hackathon 2017
1st
place out of 7 ML companies

20. Large-scale Forecasting Systems
Platforms where thousands of different consumption and production models are produced.
This is not possible without full automation.

21. TIM Worker 1
CAPI
JAPI
TIM
WS
REST/SOAP
API
TIM ConnectorsData Source
fs2TIM_WS
fs2TIM_CAPI*
fs2TIM_JAPI*
JuliaDB2TIM_WS
...
File System
HAKOM TSM
JuliaDB
InfluxDB
MySQL
PosgreSQL
Oracle
TIM Worker N
CAPI
JAPI
TIM
....
Asset data &
model management
Weather data
Visualization & UI
Grafana
Genie
...

22. Julia Language
Walks like Python runs like C
●
A single platform for prototyping and production → significant gains in efficiency
of product development
●
Vectorized code runs equally fast as de-vectorized
●
Vectorization on a single thread level
●
SIMD out of the box
●
Direct calls to BLAS
●
Distributed parallel computation (multiple threads)

23. Summary
• Fully automated model building
• Accuracy often outperforms manually build models
• Quick insights into a time-series of interest
• Tedious model building is an option not necessity

24. Thank you.
jan.dolinsky@tangent.works
Tangent Works
Na Slavíne 1
81106 Bratislava
www.tangent.works

26. The Key Concepts in Julia
Julia Lectures for Tangent Works Employees
Ján Dolinský
jan.dolinsky@tangent.works

27. Lectures Block 1: Vectorization vs. De-vectorization
• Iterators: The Storage Concept
• On Demand Iteration: Generators
• MapReduce principle
• Broadcasting
• Syntactic Loop Fusion
 Why is vectorized code fast ?
 Why it is not as fast as it could be ?

28. Lectures Block 2: Linear Algebra
• Matrices storage and its consequences
• Column-major & SIMD
• BLAS calls

29. Lectures Block 3: Language Design Elements
• Multiple-dispatch
• Struct
• Mutable & Immutable concept
• Modules & Packages generation
• JIT and AOT compilation

30. Julia Language
Walks like Python runs like C
●
A single platform for prototyping and production → significant gains in efficiency
of product development
●
Vectorized code runs equally fast as de-vectorized
●
Vectorization on a single thread level
●
SIMD out of the box
●
Direct calls to BLAS
●
Distributed parallel computation (multiple threads)

31. 2 Years of Using Julia
• It is a real programming language
• In v0.2 to v0.4 still in heavy development
• v0.5 and v0.6 are very useful; static compilation possible
• Prototyping and production development is done using the same platform
• IDE not fully matured but already quite useful
• Debugger