SlideShare a Scribd company logo
A NEW PLATFORM FOR A NEW ERA
Driving the Future of Smart Cities

How to Beat the Traffic

Strata Santa Clara – February 13, 2013
Alexander Kagoshima, Data Scientist, @akagoshima
Noelle Sio, Senior Data Scientist, @noellesio
Ian Huston, Data Scientist, @ianhuston
@gopivotal
© Copyright 2014 Pivotal. All rights reserved.
2013

2
What Matters: Apps. Data. Analytics.
Apps power businesses, and
those apps generate data
Analytic insights from that data
drive new app functionality,
which in-turn drives new data
The faster you can move
around that cycle, the faster
you learn, innovate & pull
away from the competition
@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

3
Pivotal’s Opportunity
Uniquely positioned to help
enterprises modernize each
facet of this cycle today
Comprehensive portfolio of
products spanning Apps, Data
& Analytics
Converging these technologies
into a coherent, next-gen
Enterprise PaaS platform
@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

4
The Connected Car Drives Innovation
Telematics

Stolen vehicle
Remote
recovery
Behaviour
diagnosis
monitoring
Remote
Car2X
Driver
activation
assistance solutions
Floating
Real-time
eMobility
car data
parking info
solutions
Share my
trip

Traffic
updates
Car
search

Social Media

Responsive
Navigation
PoIs
Next gen
Navigation
Hybrid
Predictive
Navigation
traffic info Map/PoI
updates
Vehicle
Concierge
Fleet
Geo
Tracking
services
Management
fencing

Handsfree
telephony
Music
WiFi
streaming
hotspot
Pay as
Online
you drive Payment Web
games
solutions radio
Road
Environmental
Parking
tolls
browsing
VoD
space reservation
Car
sharing

eCommerce

City
toll

Rich media
comms
Car2X
comms

Communication

Entertainment

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

5
Possible Data Science Use-Cases
!  Predictive Car Maintenance

–  More accurately predict part failure
–  Optimize part repair and replacement schedule

!  Leveraging Driving Behavior

–  Useful to differentiate insurance pricing based on driving style
–  Optimize car design

!  Improving GPS Systems
–  Establish baseline for traffic congestion
–  Gain a detailed view on traffic
–  Create more meaningful metrics for routing

!  Predictive Power for Assistance Systems

–  Optimize fuel efficiency
–  Predict the future state of a car in the next 2 minutes
(starts, stops, emergency braking)

!  Traffic Light Assistance

–  Signal timing of traffic lights
–  Crowd sourcing of traffic signals

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

6
What does
traffic data
look like?

© Copyright 2014 Pivotal. All rights reserved.
2013

7
…like this?

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

8
How fast are vehicles moving?

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

9
How fast are vehicles moving?

0.015
0.000

0.005

0.010

density

0.020

0.025

0.030

Link 1000064869

0

50

100

km/h

150

200

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

10
When do disruptions happen?

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

11
[seconds]

When will the light change?

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

12
Taking Lessons From Other Disciplines
Change-Point Detection can
be used to uncover regimes in
wind-turbine data.
It can also be applied to uncover
regimes in traffic light
switching patterns.

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

13
Taking Lessons From Other Disciplines
Link 1000064869

Different Cell
Populations

0.015
0.010

density

0.020

0.025

0.030

Combined
Component 1
Component 2
Component 3
Component 4

0.000

0.005

Different Driving
Conditions

0

50

100

150

200

@noellesio, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

14
Understanding Traffic Flow
A dynamic, more detailed
understanding of traffic is now possible.
Can we answer both ‘What velocity?’
and ‘Why?’
Context
!  Current GPS systems are based on
average velocity over street segments
!  Real-time traffic information (e.g. Waze)
does not deliver detailed view nor
prediction
@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

15
Our Approach – Multi-step algorithm
From our experience, real-world data often requires multi-step procedures
Step 1: Answer ‘What velocity?’
First find distinct velocity
groups
Link 1000064869

Find influencing effects

?

?

?

?

?

?

0.000

0.005

0.010

density

0.015

0.020

0.025

0.030

Combined
Component 1
Component 2
Component 3
Component 4

Step 2: Answer ‘Why?’

0

50

100

150

200

km/h

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

16
Find Velocity Groups
!  Velocity distributions can be fit well
with Gaussians
!  An ‘overlay’ of multiple Gaussians is
called Gaussian Mixture Model

0.030
0.025
0.020
density

0.015
0.010
0.005

!  Shapes and positions of Gaussians
determine velocity groups

Combined
Component 1
Component 2
Component 3
Component 4

0.000

!  GMM fitting of the velocity
distribution is done by ExpectationMaximization algorithm

Link 1000064869

0

50

100

150

200

km/h

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

17
Gaussian Mixture Model
Link 1000064869

Link 1000064869
0.030

Combined
Component
Component
Component
Component

1
2
3
4

0.020
density

0.010
0.005
0.000
0

50

100

150

200

0

50

100

150

200

km/h

km/h

Link 1000064869

0.025

1
2
3
4

density

0.000

0.000

0.005

0.005

0.010

0.010

0.015

0.015

0.020

0.020

0.025

0.030

Combined
Component
Component
Component
Component

0.030

Link 1000064869

density

1
2
3
4

0.015

0.020
0.015
0.000

0.005

0.010

density

Combined
Component
Component
Component
Component

0.025

1
2
3
4

0.025

0.030

Combined
Component
Component
Component
Component

0

0

50

100

km/h

150

200

50

100

150

200

km/h

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

18
0.04

Predict Gaussians

0.00

0.01

"  Classification task!

0.02

density

0.03

!  The second step seeks to explain/predict
which Gaussian a data point belongs to

Combined
Component 1
Component 2
Component 3

0

50

100

150

200

250

!  Features for classification:
– 
– 
– 
– 
– 

Time of day, day of week
Weather
Direction
Special Events
…

?

?

?

?…

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

19
White and Black Boxes
!  Analyze correlations between features of a data point and its
assignment to a Gaussian
!  From a Machine Learning point of view, this is classification

!  Generate an interpretable model
description

!  Can capture more complex
correlations

" Explanation of behavior

" Prediction of Gaussian assignment

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

20
0.04

Putting it all together…
Combined
Component 1
Component 2
Component 3

0.02

Average velocity = 85 km/h
• 
Weekdays after 8pm
• 
Weekdays before 2pm, exiting
Average velocity = 120 km/h
• 
Weekends
• 
Weekdays before 2pm, not exiting

Two-Step algorithm:
GMM + Classification
•  Identified multiple velocity
profiles for every road segment
•  Intuitive and easily
interpretable results
•  Highly scalable for more
features and data

0.00

0.01

density

0.03

Average velocity = 45 km/h
• 
Weekdays between 2 – 8pm

0

50

100

km/h

150

200

250

@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

21
Better Travel Time Prediction
!  Traffic profiles emerged from data
–  Without using metadata, we uncovered road
segment traffic patterns

!  Identified Bias Effects
–  Inferring the impact of turns and day of week on velocity
–  Able to predict rush hour by day and time by road segment

!  Traffic Light Patterns
–  Infer public transportation effects on traffic
–  Automatically determine different switching patterns
@akagoshima, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

22
London Road Traffic
Disruptions
Can we predict when
unexpected incidents will end?
Publicly available data:
!  Transport for London traffic feed
(refreshed every 5 minutes)

!  Weather Underground reports

Photo by James Blunt Photography on
Flickr (CC BY-ND 2.0)

@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

23
@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

24
Major storm hits UK (Photo: BBC)

@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

25
@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

26
@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

27
Durations are very
different for
different types of
incident.
Mean duration for
Surface Damage
incidents is 107
hours!

@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

28
Rain affects
duration in a
surprising way.
Incidents which start
when it is raining
finish faster than
others.

@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

29
Models
Linear Regression
•  Disruption reports &
weather features
Random Forests
•  Rounded categorical
•  Regression
Category MAP
•  Only use category of
incident
•  Maximum Likelihood
estimate
@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

30
Live Predictions
http://ds-demo-transport.cfapps.io
Using:

@ianhuston, @gopivotal
© Copyright 2014 Pivotal. All rights reserved.

31
Summary
! Making use of a vibrant ecosystem of traffic data
! Innovative approaches needed to generate value
from abundant and complex sources
! Connecting predictive models to traffic in the
physical world is the future of smart cities

© Copyright 2014 Pivotal. All rights reserved.

32
Thank You!
Check out more of our Data Science use-cases at
www.goPivotal.com

© Copyright 2014 Pivotal. All rights reserved.
2013

33
A NEW PLATFORM FOR A NEW ERA

More Related Content

Similar to Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at Strata 2014)

Smart City: Intelligent Traffic Management System
Smart City: Intelligent Traffic Management SystemSmart City: Intelligent Traffic Management System
Smart City: Intelligent Traffic Management System
Suhas Motwani
 
Bruce Thompson on digital disruption and the environment
Bruce Thompson on digital disruption and the environment Bruce Thompson on digital disruption and the environment
Bruce Thompson on digital disruption and the environment
OCESAdmin
 
WIPAC Monthly - March 2023.pdf
WIPAC Monthly - March 2023.pdfWIPAC Monthly - March 2023.pdf
WIPAC Monthly - March 2023.pdf
Water Industry Process Automation & Control
 
Using Swarm Intelligence to Prepare for the Next Carmageddon
Using Swarm Intelligence to Prepare for the Next CarmageddonUsing Swarm Intelligence to Prepare for the Next Carmageddon
Using Swarm Intelligence to Prepare for the Next Carmageddon
Ness Digital Engineering
 
Internet of Things - Building a Smarter World
Internet of Things - Building a Smarter WorldInternet of Things - Building a Smarter World
Internet of Things - Building a Smarter World
Dr. Mazlan Abbas
 
6 use cases
6  use cases6  use cases
6 use cases
Derick Jose
 
6 iot cases
6 iot cases6 iot cases
6 iot cases
Flutura DS
 
Traffic Light Controller System using Optical Flow Estimation
Traffic Light Controller System using Optical Flow EstimationTraffic Light Controller System using Optical Flow Estimation
Traffic Light Controller System using Optical Flow Estimation
Editor IJCATR
 
IOT based fuel monitoring for future vehicles.
IOT based fuel monitoring for  future vehicles.IOT based fuel monitoring for  future vehicles.
IOT based fuel monitoring for future vehicles.
IRJET Journal
 
Combining Mobile Air Quality Sensors With Census Data
Combining Mobile Air Quality Sensors With Census DataCombining Mobile Air Quality Sensors With Census Data
Combining Mobile Air Quality Sensors With Census Data
Liz Derr
 
Smart Road System to Ensure Road Accidents & Traffic Flow: an Overview
Smart Road System to Ensure Road Accidents & Traffic Flow: an OverviewSmart Road System to Ensure Road Accidents & Traffic Flow: an Overview
Smart Road System to Ensure Road Accidents & Traffic Flow: an Overview
IRJET Journal
 
WIPAC Monthly - June 2023.pdf
WIPAC Monthly - June 2023.pdfWIPAC Monthly - June 2023.pdf
WIPAC Monthly - June 2023.pdf
Water Industry Process Automation & Control
 
Wipac monthly 49th edition october 2015
Wipac monthly 49th edition  october 2015Wipac monthly 49th edition  october 2015
Wipac monthly 49th edition october 2015
Water Industry Process Automation & Control
 
Design of intelligent traffic light controller using gsm & embedded system
Design of intelligent traffic light controller using gsm & embedded systemDesign of intelligent traffic light controller using gsm & embedded system
Design of intelligent traffic light controller using gsm & embedded system
Yakkali Kiran
 
The Dawn of Industry 4.0
The Dawn of Industry 4.0The Dawn of Industry 4.0
The Dawn of Industry 4.0
CPqD
 
Spectrum auction failure in India, October 2016
Spectrum auction failure in India, October 2016Spectrum auction failure in India, October 2016
Spectrum auction failure in India, October 2016
Coleago Consulting
 
IRJET- Smart Water Management
IRJET-  	  Smart Water ManagementIRJET-  	  Smart Water Management
IRJET- Smart Water Management
IRJET Journal
 
Mini-grids for Energy Access in Sub-Saharan Africa
Mini-grids for Energy Access in Sub-Saharan Africa Mini-grids for Energy Access in Sub-Saharan Africa
Mini-grids for Energy Access in Sub-Saharan Africa
World Resources Institute (WRI)
 
Wireless Network Optimization (2010)
Wireless Network Optimization (2010)Wireless Network Optimization (2010)
Wireless Network Optimization (2010)
Marc Jadoul
 
The Internet of Things Journey
The Internet of Things JourneyThe Internet of Things Journey
The Internet of Things Journey
Dr. Mazlan Abbas
 

Similar to Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at Strata 2014) (20)

Smart City: Intelligent Traffic Management System
Smart City: Intelligent Traffic Management SystemSmart City: Intelligent Traffic Management System
Smart City: Intelligent Traffic Management System
 
Bruce Thompson on digital disruption and the environment
Bruce Thompson on digital disruption and the environment Bruce Thompson on digital disruption and the environment
Bruce Thompson on digital disruption and the environment
 
WIPAC Monthly - March 2023.pdf
WIPAC Monthly - March 2023.pdfWIPAC Monthly - March 2023.pdf
WIPAC Monthly - March 2023.pdf
 
Using Swarm Intelligence to Prepare for the Next Carmageddon
Using Swarm Intelligence to Prepare for the Next CarmageddonUsing Swarm Intelligence to Prepare for the Next Carmageddon
Using Swarm Intelligence to Prepare for the Next Carmageddon
 
Internet of Things - Building a Smarter World
Internet of Things - Building a Smarter WorldInternet of Things - Building a Smarter World
Internet of Things - Building a Smarter World
 
6 use cases
6  use cases6  use cases
6 use cases
 
6 iot cases
6 iot cases6 iot cases
6 iot cases
 
Traffic Light Controller System using Optical Flow Estimation
Traffic Light Controller System using Optical Flow EstimationTraffic Light Controller System using Optical Flow Estimation
Traffic Light Controller System using Optical Flow Estimation
 
IOT based fuel monitoring for future vehicles.
IOT based fuel monitoring for  future vehicles.IOT based fuel monitoring for  future vehicles.
IOT based fuel monitoring for future vehicles.
 
Combining Mobile Air Quality Sensors With Census Data
Combining Mobile Air Quality Sensors With Census DataCombining Mobile Air Quality Sensors With Census Data
Combining Mobile Air Quality Sensors With Census Data
 
Smart Road System to Ensure Road Accidents & Traffic Flow: an Overview
Smart Road System to Ensure Road Accidents & Traffic Flow: an OverviewSmart Road System to Ensure Road Accidents & Traffic Flow: an Overview
Smart Road System to Ensure Road Accidents & Traffic Flow: an Overview
 
WIPAC Monthly - June 2023.pdf
WIPAC Monthly - June 2023.pdfWIPAC Monthly - June 2023.pdf
WIPAC Monthly - June 2023.pdf
 
Wipac monthly 49th edition october 2015
Wipac monthly 49th edition  october 2015Wipac monthly 49th edition  october 2015
Wipac monthly 49th edition october 2015
 
Design of intelligent traffic light controller using gsm & embedded system
Design of intelligent traffic light controller using gsm & embedded systemDesign of intelligent traffic light controller using gsm & embedded system
Design of intelligent traffic light controller using gsm & embedded system
 
The Dawn of Industry 4.0
The Dawn of Industry 4.0The Dawn of Industry 4.0
The Dawn of Industry 4.0
 
Spectrum auction failure in India, October 2016
Spectrum auction failure in India, October 2016Spectrum auction failure in India, October 2016
Spectrum auction failure in India, October 2016
 
IRJET- Smart Water Management
IRJET-  	  Smart Water ManagementIRJET-  	  Smart Water Management
IRJET- Smart Water Management
 
Mini-grids for Energy Access in Sub-Saharan Africa
Mini-grids for Energy Access in Sub-Saharan Africa Mini-grids for Energy Access in Sub-Saharan Africa
Mini-grids for Energy Access in Sub-Saharan Africa
 
Wireless Network Optimization (2010)
Wireless Network Optimization (2010)Wireless Network Optimization (2010)
Wireless Network Optimization (2010)
 
The Internet of Things Journey
The Internet of Things JourneyThe Internet of Things Journey
The Internet of Things Journey
 

More from Ian Huston

Cloud Foundry for Data Science
Cloud Foundry for Data ScienceCloud Foundry for Data Science
Cloud Foundry for Data Science
Ian Huston
 
Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud Foundry
Ian Huston
 
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural LanguagesData Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Ian Huston
 
Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)
Ian Huston
 
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field InflationCalculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Ian Huston
 
Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011
Ian Huston
 
Second Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-rollSecond Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-roll
Ian Huston
 
Inflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big BangInflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big Bang
Ian Huston
 
Cosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical SimulationsCosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical Simulations
Ian Huston
 
Cosmo09 presentation
Cosmo09 presentationCosmo09 presentation
Cosmo09 presentation
Ian Huston
 

More from Ian Huston (10)

Cloud Foundry for Data Science
Cloud Foundry for Data ScienceCloud Foundry for Data Science
Cloud Foundry for Data Science
 
Python on Cloud Foundry
Python on Cloud FoundryPython on Cloud Foundry
Python on Cloud Foundry
 
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural LanguagesData Science Amsterdam - Massively Parallel Processing with Procedural Languages
Data Science Amsterdam - Massively Parallel Processing with Procedural Languages
 
Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)Massively Parallel Processing with Procedural Python (PyData London 2014)
Massively Parallel Processing with Procedural Python (PyData London 2014)
 
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field InflationCalculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
Calculating Non-adiabatic Pressure Perturbations during Multi-field Inflation
 
Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011Second Order Perturbations - National Astronomy Meeting 2011
Second Order Perturbations - National Astronomy Meeting 2011
 
Second Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-rollSecond Order Perturbations During Inflation Beyond Slow-roll
Second Order Perturbations During Inflation Beyond Slow-roll
 
Inflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big BangInflation as a solution to the problems of the Big Bang
Inflation as a solution to the problems of the Big Bang
 
Cosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical SimulationsCosmological Perturbations and Numerical Simulations
Cosmological Perturbations and Numerical Simulations
 
Cosmo09 presentation
Cosmo09 presentationCosmo09 presentation
Cosmo09 presentation
 

Recently uploaded

Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Zilliz
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 

Recently uploaded (20)

Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 

Driving the Future of Smart Cities - How to Beat the Traffic (Pivotal talk at Strata 2014)

  • 1. A NEW PLATFORM FOR A NEW ERA
  • 2. Driving the Future of Smart Cities How to Beat the Traffic Strata Santa Clara – February 13, 2013 Alexander Kagoshima, Data Scientist, @akagoshima Noelle Sio, Senior Data Scientist, @noellesio Ian Huston, Data Scientist, @ianhuston @gopivotal © Copyright 2014 Pivotal. All rights reserved. 2013 2
  • 3. What Matters: Apps. Data. Analytics. Apps power businesses, and those apps generate data Analytic insights from that data drive new app functionality, which in-turn drives new data The faster you can move around that cycle, the faster you learn, innovate & pull away from the competition @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 3
  • 4. Pivotal’s Opportunity Uniquely positioned to help enterprises modernize each facet of this cycle today Comprehensive portfolio of products spanning Apps, Data & Analytics Converging these technologies into a coherent, next-gen Enterprise PaaS platform @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 4
  • 5. The Connected Car Drives Innovation Telematics Stolen vehicle Remote recovery Behaviour diagnosis monitoring Remote Car2X Driver activation assistance solutions Floating Real-time eMobility car data parking info solutions Share my trip Traffic updates Car search Social Media Responsive Navigation PoIs Next gen Navigation Hybrid Predictive Navigation traffic info Map/PoI updates Vehicle Concierge Fleet Geo Tracking services Management fencing Handsfree telephony Music WiFi streaming hotspot Pay as Online you drive Payment Web games solutions radio Road Environmental Parking tolls browsing VoD space reservation Car sharing eCommerce City toll Rich media comms Car2X comms Communication Entertainment @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 5
  • 6. Possible Data Science Use-Cases !  Predictive Car Maintenance –  More accurately predict part failure –  Optimize part repair and replacement schedule !  Leveraging Driving Behavior –  Useful to differentiate insurance pricing based on driving style –  Optimize car design !  Improving GPS Systems –  Establish baseline for traffic congestion –  Gain a detailed view on traffic –  Create more meaningful metrics for routing !  Predictive Power for Assistance Systems –  Optimize fuel efficiency –  Predict the future state of a car in the next 2 minutes (starts, stops, emergency braking) !  Traffic Light Assistance –  Signal timing of traffic lights –  Crowd sourcing of traffic signals @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 6
  • 7. What does traffic data look like? © Copyright 2014 Pivotal. All rights reserved. 2013 7
  • 8. …like this? @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 8
  • 9. How fast are vehicles moving? @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 9
  • 10. How fast are vehicles moving? 0.015 0.000 0.005 0.010 density 0.020 0.025 0.030 Link 1000064869 0 50 100 km/h 150 200 @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 10
  • 11. When do disruptions happen? @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 11
  • 12. [seconds] When will the light change? @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 12
  • 13. Taking Lessons From Other Disciplines Change-Point Detection can be used to uncover regimes in wind-turbine data. It can also be applied to uncover regimes in traffic light switching patterns. @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 13
  • 14. Taking Lessons From Other Disciplines Link 1000064869 Different Cell Populations 0.015 0.010 density 0.020 0.025 0.030 Combined Component 1 Component 2 Component 3 Component 4 0.000 0.005 Different Driving Conditions 0 50 100 150 200 @noellesio, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 14
  • 15. Understanding Traffic Flow A dynamic, more detailed understanding of traffic is now possible. Can we answer both ‘What velocity?’ and ‘Why?’ Context !  Current GPS systems are based on average velocity over street segments !  Real-time traffic information (e.g. Waze) does not deliver detailed view nor prediction @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 15
  • 16. Our Approach – Multi-step algorithm From our experience, real-world data often requires multi-step procedures Step 1: Answer ‘What velocity?’ First find distinct velocity groups Link 1000064869 Find influencing effects ? ? ? ? ? ? 0.000 0.005 0.010 density 0.015 0.020 0.025 0.030 Combined Component 1 Component 2 Component 3 Component 4 Step 2: Answer ‘Why?’ 0 50 100 150 200 km/h @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 16
  • 17. Find Velocity Groups !  Velocity distributions can be fit well with Gaussians !  An ‘overlay’ of multiple Gaussians is called Gaussian Mixture Model 0.030 0.025 0.020 density 0.015 0.010 0.005 !  Shapes and positions of Gaussians determine velocity groups Combined Component 1 Component 2 Component 3 Component 4 0.000 !  GMM fitting of the velocity distribution is done by ExpectationMaximization algorithm Link 1000064869 0 50 100 150 200 km/h @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 17
  • 18. Gaussian Mixture Model Link 1000064869 Link 1000064869 0.030 Combined Component Component Component Component 1 2 3 4 0.020 density 0.010 0.005 0.000 0 50 100 150 200 0 50 100 150 200 km/h km/h Link 1000064869 0.025 1 2 3 4 density 0.000 0.000 0.005 0.005 0.010 0.010 0.015 0.015 0.020 0.020 0.025 0.030 Combined Component Component Component Component 0.030 Link 1000064869 density 1 2 3 4 0.015 0.020 0.015 0.000 0.005 0.010 density Combined Component Component Component Component 0.025 1 2 3 4 0.025 0.030 Combined Component Component Component Component 0 0 50 100 km/h 150 200 50 100 150 200 km/h @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 18
  • 19. 0.04 Predict Gaussians 0.00 0.01 "  Classification task! 0.02 density 0.03 !  The second step seeks to explain/predict which Gaussian a data point belongs to Combined Component 1 Component 2 Component 3 0 50 100 150 200 250 !  Features for classification: –  –  –  –  –  Time of day, day of week Weather Direction Special Events … ? ? ? ?… @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 19
  • 20. White and Black Boxes !  Analyze correlations between features of a data point and its assignment to a Gaussian !  From a Machine Learning point of view, this is classification !  Generate an interpretable model description !  Can capture more complex correlations " Explanation of behavior " Prediction of Gaussian assignment @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 20
  • 21. 0.04 Putting it all together… Combined Component 1 Component 2 Component 3 0.02 Average velocity = 85 km/h •  Weekdays after 8pm •  Weekdays before 2pm, exiting Average velocity = 120 km/h •  Weekends •  Weekdays before 2pm, not exiting Two-Step algorithm: GMM + Classification •  Identified multiple velocity profiles for every road segment •  Intuitive and easily interpretable results •  Highly scalable for more features and data 0.00 0.01 density 0.03 Average velocity = 45 km/h •  Weekdays between 2 – 8pm 0 50 100 km/h 150 200 250 @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 21
  • 22. Better Travel Time Prediction !  Traffic profiles emerged from data –  Without using metadata, we uncovered road segment traffic patterns !  Identified Bias Effects –  Inferring the impact of turns and day of week on velocity –  Able to predict rush hour by day and time by road segment !  Traffic Light Patterns –  Infer public transportation effects on traffic –  Automatically determine different switching patterns @akagoshima, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 22
  • 23. London Road Traffic Disruptions Can we predict when unexpected incidents will end? Publicly available data: !  Transport for London traffic feed (refreshed every 5 minutes) !  Weather Underground reports Photo by James Blunt Photography on Flickr (CC BY-ND 2.0) @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 23
  • 24. @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 24
  • 25. Major storm hits UK (Photo: BBC) @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 25
  • 26. @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 26
  • 27. @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 27
  • 28. Durations are very different for different types of incident. Mean duration for Surface Damage incidents is 107 hours! @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 28
  • 29. Rain affects duration in a surprising way. Incidents which start when it is raining finish faster than others. @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 29
  • 30. Models Linear Regression •  Disruption reports & weather features Random Forests •  Rounded categorical •  Regression Category MAP •  Only use category of incident •  Maximum Likelihood estimate @ianhuston, @gopivotal © Copyright 2014 Pivotal. All rights reserved. 30
  • 32. Summary ! Making use of a vibrant ecosystem of traffic data ! Innovative approaches needed to generate value from abundant and complex sources ! Connecting predictive models to traffic in the physical world is the future of smart cities © Copyright 2014 Pivotal. All rights reserved. 32
  • 33. Thank You! Check out more of our Data Science use-cases at www.goPivotal.com © Copyright 2014 Pivotal. All rights reserved. 2013 33
  • 34. A NEW PLATFORM FOR A NEW ERA