MongoDB World 2019: Turkeys vs. Swans: Building Antifragile IT Systems for Disaster Response

Turkeys vs. Swans:
Building Antifragile IT Systems
for Disaster Response
MongoDB World 2019
Imanuel Portalatín
iportalatin@mitre.org
Approved for Public Release; Distribution Unlimited. Public Release Case Number 19-1264
©2019 The MITRE Corporation. ALL RIGHTS RESERVED.

About MITRE
• We work across the whole of government to tackle difficult problems
that challenge the safety, stability and well-being of our nation
through:
• Operation of Federally Funded R&D Centers (FFRDCs)
• Public-private partnerships
• We have a unique vantage point working across federal, state and
local governments, as well as industry and academia
• We work in the public interest to:
• Discover new possibilities and create unexpected opportunities
• Lead by pioneering together for public good to bring innovative ideas into
existence in national security and the public sector
Our mission-driven team is dedicated to solving problems for a safer world.

Agenda
• Black Swans, Turkeys and Antifragility
• OLAF Prototype Overview
• Lessons Learned
• Ansible MongoDB Deployment
• Tracking, Updating and Sharing Geo-Tagged Data
• OLAF Live Demo
• What’s Next?

Black Swans, Turkeys and Antifragility
An Introduction

Retrieved From: https://www.flickr.com/photos/noaasatellites/37224066361

Retrieved From: http://www.coastal.jp/tsunami2011/

Photo By: Michael C. Barton, Mass Communication Specialist 2nd Class, U.S. Navy, Released (100119-N-8878B-322)
Retrieved From: http://www.navy.mil/view_image.asp?id=80340

Retrieved From: https://www.maxpixel.net/Bad-Weather-Storm-Highway-Blizzard-Snow-Driving-2053388

Retrieved From: https://pixabay.com/photos/black-swan-bird-nature-water-1034891/

Retrieved From: https://www.pexels.com/photo/boy-crystal-ball-magic-magician-369384/

Turkey Problem:
“A turkey is fed by a butcher all its
life so every day that passes confirms
to the turkey and its team of analysts
that butchers love turkeys with
increased statistical confidence.
That is until a few days before
Thanksgiving when there is a big
surprise that the turkey never saw
coming.”
Retrieved From: https://www.maxpixel.net/Free-Range-Turkeys-Plumage-Birds-Poultry-2799462

Retrieved From: https://pxhere.com/en/photo/1334293

Retrieved From: https://www.dreamstime.com/stock-photo-exponential-growth-image18365370#res22058481

Promoting Antifragility
• Follow simple heuristics (rules of thumb)
• Take small risks with high payoff potential
• Avoid risks that could lead to catastrophic outcomes
• Build in redundancy and multiple layers
• Keep your options open (optionality)
• Use small incremental steps

MongoDB and Antifragility
• Document Database
• Dynamic schema supports fluent polymorphism
• Documents (i.e. objects) correspond to native data types in many
programming languages
• High Availability (Replica Sets)
• Automatic failover
• Data redundancy
• Horizontal Scalability
• Sharding distributes data across a cluster of machines
• Compatible with commodity servers and virtualized elastic environments

Offline Application Facilitation (OLAF)
Prototype Overview

Problem
• Communication networks are
devastated by regional disasters
• Effective response requires close
coordination between diverse relief
agencies
• Backup communications cannot
handle the volume of data generated
during these disasters
Experts consider communications a basic necessity, like food and water!
Photo By: Michael C. Barton, Mass Communication Specialist 2nd Class, U.S. Navy, Released (100119-N-8878B-322)
Retrieved From: http://www.navy.mil/view_image.asp?id=80340

1. Special software is automatically
installed on inexpensive low-power
computers, called edge gateways
2. Edge gateways automatically relay
data through alternate channels
when main networks are down
3. First responders and the public
connect to edge gateways via local
Wi-Fi networks on their own
personal devices
4. Critical information is uploaded via
gateways with active connection to
the internet, called master nodes
5. Up-to-date information is made
available via online services,
improving command & control
capabilities and public awareness
Idea
Retrieved From: https://www.maxpixel.net/Computer-Pi-Technology-Raspberry-
Pi-Electronics-1719219
Photo By: © Arne9001 | Dreamstime.com

OLAF Architecture
Affected
Population
First Responders
Command &
Control
Relief
Organizations
General Public
Local Wi-Fi
Edge Gateway
Containerized
app
Local data
Mesh Protocol
Update Agent
Master Node
Automated
Deployment
Shared data
Mesh Protocol
Update Server
DRMS/GeoQ
Other Services

Lessons Learned
Ansible MongoDB Deployment

Benefits of Ansible
• Lightweight agent-less IT automation tool
• Designed for simplicity and ease of use
• Minimum moving parts
• Uses SSH for secure transport
• Only depends on Python and OpenSSH
• Easy to understand template language (YAML)

Ansible Inventory
all:
vars:
ansible_user: olaf
ansible_ssh_private_key_file: ~/.ssh/mesh_node
ansible_become: yes
ansible_become_pass: olaf
children:
master_nodes:
hosts:
master:
ansible_host: 192.168.0.101
mesh_nodes:
hosts:
mesh_node1:
ansible_host: 192.168.0.1
Variables for All Hosts
Hosts and Groups

Dockerized MongoDB Deployment
---
- name: Deploy MonogDB to all hosts
hosts: all
tasks:
- name: Start Mongo DB container
docker_container:
name: database
image: mongo
state: started
recreate: yes
published_ports:
- 27017:27017
register: docker_container
$ ansible_playbook mongo_deploy.yml –i hosts
mongo_deploy.yml:
Command Line:
Target Hosts or Groups
Tasks to be run on Host

Node.js App Configuration
- name: Configure Node.js app
hosts: mesh_node
vars:
- mongoConfig:
url: "mongodb://localhost:27017/"
tasks:
- name: Get MongoDB ip from docker_start output
set_fact:
mongodb_ip: "{{ docker_container.ansible_facts.docker_container.NetworkSettings.IPAddress }}”
when: docker_container is defined
- name: Override default MongoDB config
set_fact:
mongoConfig: "{{ mongoConfig | combine({ 'url' : 'mongodb://' + mongodb_ip + '/' }) }}”
when: mongodb_ip is defined
- name: Write Javascript configuration file
copy:
content: "module.exports = {{ appMongoConfig | to_nice_json }}"
dest: /opt/olaf/webapp/config/mongoConfig.js
mongo_deploy.yml (continued):

Node.js App Configuration (Continued)
module.exports = {
url: "mongodb://172.17.0.2:27017/"
}
./config/mongoConfig.js (auto-generted):
'use strict’;
const mongoConfig = require('./config/mongoConfig’);
MongoClient.connect(mongoConfig.url, function(err, client) {
assert.equal(null, err);
…
});
app.js:

Lessons Learned
Tracking, Updating and Sharing Geo-Tagged Data

Geo-Marker Document
{
"_id":"5b0dcb214ff9980016c119f6",
"type":"Feature",
"geometry": {
"type":"Point",
"coordinates": [
-66.03899002075195,
18.398795619534134
]
},
"properties": {
"name":"fire”
}
}
MongoDB ObjectId
GeoJSON Data Fields

Creating 2dsphere Index in Node.js
…
const db = client.db(mongoConfig.dbName);
const collection = db.collection(mongoConfig.makerCollection);
collection.createIndex( { “geometry.coordinates” : “2dsphere” }, function(err, result) {
if (err) {
console.log('Error while creating indexes: '+err);
} else {
console.log('All 2dsphere indexes were created successfully!');
client.close();
}});
});
server.js:

Tracking Updates with MongoDB ObjectID
…
const ObjectID = require('mongodb').ObjectID;
var lastObjectId = ObjectID.createFromTime(0);
setInterval(function() {
var objectId = new ObjectID();
let query = { '$and' : [ { '_id' : { '$gte' : lastObjectId } }, {'_id' : { '$lt' : objectId } } ] };
db.collection(col).find(query).toArray(function(err,docs) {
// Process query results …
lastObjectId = objectId;
});
});
},delay);
update.js:

Graceful Recovery after Disconnect
…
setInterval(function() {
var objectId = new ObjectID();
let query = { '$and' : [ { '_id' : { '$gte' : lastObjectId } }, {'_id' : { '$lt' : objectId } }, { update : { '$exists' : false } } ] };
db.collection(col).find(query).toArray(function(err,docs) {
// Process query results …
db.collection(col).update(query, { '$set’ : { 'update’ : 1 } }, { 'multi' : true }, function(err,updated) {
lastObjectId = objectId;
});
});
});
},delay);
update.js:

Sharing Geo-Tagged Data
DRMS/GeoQ
OLAF Master Node
Automated
Deployment
Shared data
Mesh Protocol
Update Server
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [0, 0]
},
"properties": {
"popupContent": "flood"
}
}
]
}

Sharing FeatureCollection over HTTP
…
router.route('/getMarkers').get(function(req,res) {
var query = getQuery(req);
var filter = { '_id' : 0 };
…
collection.find(query,filter).toArray(function(err, docs) {
res.setHeader('content-type',"application/geo+json");
// Build the GeoJSON FeatureCollection object
var featureCollection = {
'type' : 'FeatureCollection',
'features’ : docs
}
res.end(JSON.stringify(featureCollection));
});
});
server.js:
Process URL Query String
Query Markers Collection
Set HTTP Response MIME Type
Package Feature Array in
FeatureCollection Object
Remove ObjectID from Results
Send HTTP Response

Live Demo
OLAF Prototype

What’s Next?
OLAF Prototype

1. OLAF Prototype and AWS Greengrass used to deploy
trained ML models and mesh network software to
edge gateways hosted on Raspberry Pi’s.
2. UAVs with on-board OLAF gateways
can transmit overhead imagery and
relay data from other sensors.
4. Local inference on human pose estimation
models and custom analytics at edge
gateways classify images of victims that
are trapped or need of immediate help.
3. Edge gateways are
deployed to autonomous
platforms able to operate
in hard to access and
hazardous areas.
5. Geotagged images of
victims trapped or in
need of assistance are
sent to OLAF master
nodes at forward
command center and
displayed at DMRS
console.
DRMS Console
6. Analysts use DRMS to
validate automatic
classification and
identify cells where
resources need to be
deployed immediately.

MongoDB World 2019: Turkeys vs. Swans: Building Antifragile IT Systems for Disaster Response

Recommended

Recommended

More Related Content

Similar to MongoDB World 2019: Turkeys vs. Swans: Building Antifragile IT Systems for Disaster Response

Similar to MongoDB World 2019: Turkeys vs. Swans: Building Antifragile IT Systems for Disaster Response (20)

More from MongoDB

More from MongoDB (20)

Recently uploaded

Recently uploaded (20)

MongoDB World 2019: Turkeys vs. Swans: Building Antifragile IT Systems for Disaster Response