An overview of current work in application of AI in intelligent infrastructure systems. Presented at an International Symposium Organized by the Institute of Scientific Research, Department of Transportation of China.

1. Mohammad Noori
International Institute for Urban Systems Engineering
Department of Highway Transportation Scientific Research Institute Forum on

2. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

3. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

4.  “…in situ, non-destructive
sensing and analysis of
system characteristics,
including structural
response, for the purpose
of detecting changes,
which may indicate
damage or degradation.”
(Housner, et-al 1997)
What is Structural Health Monitoring?

5.  A Paradigm Shift?
1) Unprecedented pace of urbanization- As of
2007, 50% of the world population live in cities.
Balance is tipped towards urbanization and
“Mega Cities.” Increased demand on
infrastructure and urban safety.
Why Structural Health Monitoring?- Context
o Development of global economy and
transportation market leads to the traffic growth.
o The European commission predicts a sustainable
annual growth ratio of truck traffic volume
between 1.5% and 2%.
o Bridge collapse due to accidental traffic
overloading happens occasionally.
o Long-span bridges suffer from heavier traffic loads
and simultaneous presence of multiple trucks1
1Lu, Naiwei, Beer, Michael, Noori, Mohammad, Liu, Yang. Lifetime deflection of long-span bridges under dynamic and
growing traffic loads. ASCE’s Journal of Bridge Engineering.

6. Why Structural Health Monitoring?- Context
 A Paradigm Shift?
2) Many large civil structures have been built or are being built throughout
the world.

7. Why Structural Health Monitoring?- Context
 A Paradigm Shift?
3) Many long bridges have been built or are being built throughout the world.

8. Civil Structures, on
the other hand, have
become increasingly
vulnerable to natural
& man-made
hazards.
Why Structural Health Monitoring?- Context
 A Paradigm Shift?
4. Protecting Lifeline Systems Against Natural Hazards-Increased random
occurrence of mega natural disasters. Renders more intelligent response to
catastrophes.
Natural Event Direct Loss Deaths
2005 Hurricane Katrina (Gulf Coast) $81 billion 1,836
2008 Wenchuan Earthquake (China) $100 billion 7,000
2011 Tohoku Earthquake (Japan) $700 billion 25,000
2011 Hurricane Irene (East Coast) $7 billion 35
2012 Hurricane Sandy (East Coast) $51B 110

9. More than
600,000
bridges in
U.S.
Pipelines – 2
million miles
of natural gas
lines in U.S.
9000
Commercial
Aircrafts in
use in U.S.
Wind Turbines
– 21,000 MW
capacity in the
U.S.
 A Paradigm Shift?
o Deteriorating Infrastructure-
 Next 15 years, 50% of the US’ bridges (~300,000) exceed their intended
50 year lives; requiring more vigilant inspection.
 Over 2000’s of commercial and Air Force aircrafts are aging
Why Structural Health Monitoring?- Context

10.  Dependence of Chinese and US economy on Infrastructure
● For example, highway system & Mississippi flood control system
in US and transportation systems in China
 Today, infrastructure systems are rapidly aging:
● ASCE estimates US requires $2.2 trillion to repair (16% of 2009 GDP)
● Economic resource for maintenance and repair is shrinking
Why Structural Health Monitoring?- Context
2009 ASCE
Report Card Grades

11.  Implications:
These societal/global challenges, in light of new technological
breakthroughs
Have brought about a new paradigm shift necessitating
transformative changes
Demand resilient infrastructure & more sustainable practices
Serviceability, safety and sustainability of large civil structures are
major concerns of global society
At stake is the economic prosperity of nations.
Recently developed health monitoring technology is an effective means
for monitoring serviceability, safety and sustainability.
Why Structural Health Monitoring?- Context

12. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

13.  Integrated Modules/Components of a long-term SHM System
Data
fusion/integration
& Processing
Sensing
System
Data
Acquisition/
Transmission
Data
Processing &
Analysis
System
Data Management
System (Information
Processing to
Define Current
State)
Decision
Making and
Management
Capability Analysis,
Prognosis, Maintenance
and SHM
Current State of SHM ……….
 Objectives:
 Monitor the system performance
(dynamic response)
 Detect damage
 Assess/diagnose the structural
health condition
 Make maintenance decision
Improvement of
Structural Performance
and Safety

14. An Efficient, Reliable, Integrated, Real-Time System for Predictive Maintenance & Control
Current State of SHM ……….

15. Current State of SHM ……….

16. What is Structural Health Monitoring?

17.  Two Major Components:
1. Data Acquisition- Sensing, Signal Generation/Storage and
Transmittal
2. Data Analysis/Diagnosis -
1. Data Acquisition/Sensing- Progress and Current State
Significant progress in sensing in more than 15 years:
o PZT Sensors (smart materials)
o Smart Sensors (Data Processing built-in)
o FBG- Point and Dist’d sensing (Simultaneous strain & temp)
o Fiber Optic Distributed Strain Sensor (Higher Spatial Resolution)
o Other Major Advances in Distributed Sensing Techniques-
 WSSN’s- Wireless Smart Sensor NetworksSingle connection, multiple sensing, with strain
and temp resolution as fine as 1 με and 0.1 oC.
Sensing length up to 70m providing 200
sensing pts/meter via single connection.
scatter recorded in segments of fiber
Current State of SHM ……….

18. 1. Data Acquisition/Sensing-
● Many of the above sensors adopted on major SHM Projects
Jindo Bridge
instrumented with dense
wireless sensor network
for SHM. The world’s
largest WSSN for civil
infrastructure to date.
includes solar energy
harvesting system.
Current State of SHM ……….

19. 1. Data Acquisition/Sensing-
● More recent or “futuristic” sensing technologies
MEMs accelerometers, low
cost devices for harsh field
environments
Bio-inspired/nano-structured
sensing skins designed to coat
structures. Will provide a
means of damage identification
in a true distributed manner.
Current State of SHM ……….

20. ▶ development of intelligent sensor technology for more than
15 years- Payoff: “smarter” bridges that self-identify health issues.
 I-35 Bridge , designed by FIGG Eng. Group. Includes an SHM system that
allows comprehensive monitoring throughout the bridge’s lifetime.
(Minnesota DOT & Univ. of Minnesota “Long-term I35 SHM Monitoring
Project”.)
Total of 500 permanent sensors
installed, designed for 100 year
service life (2007)
● strain gages
● thermistors
● fiber optic sensors
● linear potentiometers
● accelerometers
● corrosion sensors
Current State of SHM ….. Smart Bridges

21. Current State of SHM ….. Smart Bridges

22. Current State of SHM ….. Smart Bridges

23. Current State of SHM ….. Smart Bridges
Tsing Ma Bridge, HK Various Sensor Layouts

24. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

25. Emerging/Challenging Areas- What AI Can Do?
 Two Major Components:
1. Data Acquisition- Sensing, Signal Generation/Storage/Transmittal
2. Data Analysis/Diagnosis- Emerging/Challenging Area
 The complexity/size of the structure, number and types of the
sensing systems is rapidly increasing
 Massive data is generated, and/or, needed from the large
network of sensors
 Data transmission, fusion/integration and analysis are critical
areas
2 Challenging areas in
order to realize the
Intelligent
Infrastructure Systems:
 Big Data
 Intelligent Analysis - AI

26. Emerging/Challenging Areas- What AI Can Do?
 BIG DATA & Related Challenges-
o Virtually continuous stream (large volume) of measurements from
numerous heterogeneous sensors to enable tracking of damage evolution
o Data may also have variety (accel, strain, temp., etc), different complexity,
(e.g. complex degradation -Hysteresis), sensors not optimally placed, etc.
o Uncertainties: Sensor Measurements, Transmitted Data, Noise, Data Loss
o Data Fusion - to integrate data from a multitude of sensors to make more
confident damage detection decision
o Synchronization of sensing (e.g. SHM & Vibration Control), and
simultaneous processing
Specifically, when multiple sensors collect data, can we design
communication & routing schemes that perform “in-network”
aggregation to reduce the amount of data transported, yet, preserve
the information needed for certain tasks, e.g., detecting a damage?

27. SHM for I-35 Bridge (500 sensors)
Data acquired from fiber optic sensors over a period of 7 days
Emerging/Challenging Areas- What AI Can Do?

28. Current State of SHM ….. Smart Bridges

29. Current State of SHM ….. Smart Bridges
Total No of Sensors 1505

30. Emerging/Challenging Areas- What Can AI Do?
 What can AI do? - Current approaches for complex SHM systems (Big
DATA) are limited. New paradigms are direly needed.
Data driven SHM methods (such as Wavelet, Hilbert, or a large number of other tools)
o Take raw signals obtained from sensor networks,
o Process them to obtain features representative of the condition of the
structure.
o New measurements are then compared with baselines to detect damage.
Problems:
o Damage-sensitive features (e.g. Natl freq’s) exhibit variation due to
environmental and operational changes, etc
o These comparisons are not always straightforward
An automated/intelligent approach is necessary, particularly for large
scale sensor networks and taking into account the uncertainties.

31. Emerging/Challenging Areas- What Can AI Do?
 What can AI do? -
Artificial Intelligent & Cognitive Architecture Systems: A set of mathematical
and computational schemes that mimic the function of human brain and the
nervous system, i.e., sensing, learning, reasoning and decision making.
o Adapt to the environment, while working with insufficient knowledge and resources.
They rely on finite processing capacity, work in real time, handle unexpected tasks,
and learn from experience.
o Offer tools with Cognitive and Deep Learning ability. Can develop decision making
and consider data from multiple sources
o Can take into account Uncertainties and limited path and develop an intelligent, self
learning scheme
o Can best extract desired features in massive data
o AI tools are the building blocks for Cognitive Sensors
.

32.  What can AI do? -
Emerging/Challenging Areas- What Can AI Do?
SVM, Bayesian
NN,..
Big Data
AI
LS-SVM,
Bayesian, NN….

33. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

34. A Few Case Studies- CASE 1
 A Study of a Complex Damage Mechanism. Hysteretically Yielding
Structures (Ying Zhao and M. Noori, 2017)
Yielding Shear Panel Device
A magneto-rheological damper with a force-
lag phenomenon using BWBN model

35. Objectives- Utilized AI for:
1- System Identification of complex hysteretic system
2- Detecting hysteretic damage (gradual)- (Extension of S. Saadat PhD, 2003)
Developed an RBF based ANN called, IPV method
A Few Case Studies- CASE 1
2
ˆR2
u
g
x
3
ˆRx1
x2
x3
1st
Floor
2nd
Floor
3rd
Floor
Ground xg

36. A Few Case Studies- CASE 1
1- System ID: ANN-IPV and TCMC Based Bayesian Updating

37. Utilized a Machine
Learning (AI) approach,
instead of Finite
Element, to carry out
the model updating for a
prototype steel box
girder to assess fatigue
reliability of welded
joints and estimate the
service life.
A Few Case Studies- CASE 2

38. Utilized an SVM (AI) approach to estimate the structural limit state functions for a Truss
Bridge & a Suspension Bridge. Obtained “Failure” sequence for these structures (e.g.
strength failure of mid-span cables followed by bending failure of girders.)
A Few Case Studies- CASE 3

39.  10 Different AI based algorithms used to evaluate the superiority of AI
tools when dealing with enormous computation/data processing.
 Also two AI tools developed and used to reduce the size of data, through
meta modeling, yet maintaining the accuracy, and closely detect the
location and severity of damage in a large Truss structure and a
20mX20m double grid steel deck.
A Few Case Studies- CASE 4

40.  Use a huge data base of pavement images
 Identify a single image where the size of crack exceeds a threshold
 Build a pattern recognition network to label the data base, as crack or no crack
A Few Case Studies- CASE 5
A Quantitative Analysis Approach for Detecting and Measuring
Concrete Pavement Fracture Based on Deep Learning and Computer
Vision – Currently Conducted at IIUSE/SEU (Futao Ni, Jian Zhang)
Build a classifier to detect concrete crack from an image
Calculate the width of the crack near the
marked area, with the new method crack
is roughly 0.236mm

41. • CASE 6- Big-Data Enabled Multiscale Serviceability Analysis for
Aging Bridges (2016 Study) (A research Yu Liang, Dalei Wu, Guirong Liu , et al
University of Tennesee, 2016)
 A multi scale SHM based on Hadoop Ecosystem (MS-SHM).
 Apache Hadoop software library allows for the distributed processing of large
data sets across clusters of thousands of machines, each offering local
computation and storage.
A Few Case Studies- CASE 6

42.  Objectives:
 Real-time processing and integration of structure-related sensory data from
heterogeneous sensors
 Highly efficient storage and retrieval of SHM-related heterogeneous data(i.e., with
differences in format, durability, function, etc.) over a big-data platform
 Prompt while accurate evaluation about the safety of civil structures according to
historical and real-time sensory data
 Tasks:
1. Survey the nation-wide bridge information platform
2. Determine multiple performance indicators (PIs) to predict bridge performance
3. Data fetching and processing from Hadoop using PIs
4. Multi-scale structural dynamic modeling and simulation based on historical data of
sample bridges
5. Evaluate impact of innovative bridge construction methods on bridge performance by
instrumenting two new bridges
A Few Case Studies- CASE 6

43.  Infrastructure and Flow-chart of MS-Hadoop System:
A Few Case Studies- CASE 6

44.  Flow-chart of MS-Hadoop SHM Evaluation System:
A Few Case Studies- CASE 6

45.  Acquisition of sensory data and integrating structure-related data:
A Few Case Studies- CASE 6

46.  Flow-chart of Deep Learning (AI) analysis :
A Few Case Studies- CASE 6

47.  Global and Component Level Reliability Analysis:
A Few Case Studies- CASE 6

48.  Major contribution:
 As one of its major technical contributions, this system employs a
Bayesian network to formulate the integral serviceability of a
bridge according to component serviceability and inter-
component correlations.
 Enabled by deep learning and Hadoop techniques, a full-
spectrum, sustainable, and effective evaluation can be made to
cover the 600,000 nationwide bridges.
 Best example of the potential capabilities of AI for SHM in a very
large scale.
A Few Case Studies- CASE 6

49. Outline
1) Context and Overview of SHM
2) Current State of SHM and Reliability of
Major Infrastructure
3) Emerging/Challenging Areas & What AI
Can Address?
4) A few Case Studies
5) Prospects of AI Applications

50. Inventurist Separates Reality from
Hype in AI
50
Inventurist offers a structured approach for turning
AI into Business Value
• See article by Cirrus Shakeri published in 2016 on LinkedIn

51. Intelligent Infrastructure
Systems
Cirrus Shakeri, Ph.D.
Co-Founder & CEO
Discover the Next Breakthrough Solutions in:
Gil Heydari
Co-Founder & COO
July 2017
Professor Mohammad Noori
Head of Division,
Intelligent Infrastructures

52. Data Assets Data Ingestion AI Models Machine Reasoning Machine Intelligence
Web Content
(web sites, blogs, …)
Predict
(demand, inventory, …)
Learning from Usage Patterns
Semantic Inferencing
Social Networks
(twitter, Facebook, …)
Enterprise Apps
(ERP, CRM, …)
Internet of Things
(sensor data, device data, …)
Textual Content
(documents, reports, …)
Online Activities
(search, shopping, …)
Knowledge-bases
(taxonomies, ontologies, …)
Data Preparation
• Data integration
• Data enrichment
• Data imputation
• Data versioning
• Data provenance
• …
Natural Language
Processing
• Entity extraction
• Entity resolution
• Relationship extraction
• Taxonomy generation
• Knowledge based
population (slot filling)
• …
Context Engine
Sensemaking Engine
Semantic Search
Machine Learning
(classification,
clustering, anomaly
detection, …)
Design
(product, process, …)
Analyze
(performance, problem, …)
Detect
(incident, anomaly,
opportunity, …)
Find
(people, content, …)
Discover
(insight, pattern, …)
Compare
(products, companies,, …)
Processes
(process logs, server logs, …)
Automated Update Cycle
Rule Engine
Process Automation Engine
Semantic Query Engine
Inference Engine
Network of:
people, places,
organizations, processes,
rules, policies, events,
documents, devices, …
Recommendation Engine
……
…
Inventurist Uncovers Opportunities for New AI
Solutions
AI Solutions
(New Value Propositions)
52

53.  By 2010 number of cities with more than 5M population grew to more than
59, a 50% increase since 2000.
 By 2030 more than 60% of the world population will live in cities, including 29
more mega cities (with >10M population).
 While our immediate infrastructure needs are critical, it is shortsighted to
simply replicate more of what we have today. By doing so, we miss the
opportunity to create Intelligent Infrastructure that will be the foundation for:
 Increased safety, resilience, improved efficiencies and civic
services, broader economic opportunities, better services for the
citizens, and brings prosperity for high-tech global economy.
 Intelligent infrastructure is the deep embedding of sensing, computing, and
communications capabilities into traditional physical infrastructure, e.g.,
roads, buildings, and bridges.
Prospects of AI Applications

54.  AI is the cornerstone of Intelligent Infrastructure. Future of Intelligent
Infrastructure means V2I, I2V, I2I communication
Prospects of AI Applications
Long span bridge under railway,
cars, wind load

55. Paradigm Shifts that changed the world in immeasurable ways:
 Interstate Highway System - After World War II the US’ infrastructure
exploded to accommodate returning veterans and their families, and respond
to a huge economic growth. It forever changed how we transport our goods,
services and it allowed and contributed to the continued growth and expansion
of our cities and towns.
 Internet - The partnership of Government and the Private Sector played an
instrumental role shaping the scope and scale of Internet
 AI- The impact of AI is much more far reaching in reshaping the future of our
cities, the global urban development, and creating a true intelligent
infrastructure system going forward.
 The Government will decide the platform on which the national economy is
built going forward and AI can provide the foundation for that path.
Prospects of AI Applications

56. “AI can augment all modes of transportation to materially
impact safety for all types of travel. It can be used in structural
health monitoring and infrastructure asset management,
providing increased trust from the public and reducing costs of
repairs and reconstruction……”

57. THANK YOU & QUESTIONS?