An innovative approach for utilizing Azure cloud service in nationwide Census surveys

1. CENSUS AS A SERVICE
A cloud based approach for Nationwide census projects

2. The
“As A Service”
Era
What if we can bring the
magic of the cloud to the
world of surveying systems?

3. • Runs every 10 Years
• Runs for few months only
• Massive Data ingestion
• Heavy Analytics(Real-time and Batch
based)
• Nationwide project
The Case
• Near real-time results
• Real time progress tracking
• Dissemination results just after 1
month
• Fully digitized
• Location Aware
202x Modern Requirements
THE NATIONWIDE CENSUS

4. • Huge IT CAPEX for program to run every 10 Years.
• Centralized compacted execution time(few months) which needs high-end
scaled environment.
• Real-time heavy data ingestion (Tens thousands per second) which needs
high concurrency requirements within only 8 working hours per day.
• Real-time workflow and monitoring(up to 100k field-worker).
• Near real-time analytics.
• Speedy Results dissemination process which means a lot of analytics and
insights to be available for the community just after the census.
Challenges
THE ON-PREMISES TRADITION, IS IT
EASY?

5. • with pay-as-you-go model instead of investing in massive IT
Services To run for a few months
Challenge: Massive IT service for a short
time
Answer : COPEX model
• Elastic in-memory managed Data services(SQLNoSQL) for fast ingestion
and near real-time progress analysis
• Big data Batch processing tools for daily progress analysis
• Elastic Data Warehouse store for final result dissemination and ad-hoc
analytics
Challenge: Massive real-time and batch
analytics
Answer : Enterprise Data Hub
• with well controlled throughputegress rate to avoid back-end AppsAPIs
and Resources from being flooded with field messageseventsnotifications.
• can digest thousands of ingress real-time eventsmessages per second
with scale updown rules based on the current in-field load
Challenge: Massive data ingestion
Answer : Enterprise Data pipeline
• Serverless Computing model with pay per consumption model
• Estimate and Pay for the current real-time load instead of estimating for the
maximum expected load
• Elasticity on all level(Apps, Services, Data, Infrastructure)
Challenge: uncontrolled workload up to 100k users
Answer : Elasticity with pay per
useconsumption
• Scalable Network Bandwidth, ingress and egress rates.
• Scalable storage IOPS, Throughput(MBS) with minimal latency.
• Scalable VMs using load balanced Scale Sets
Challenge: uncontrolled workload up to
100k users
Answer : Scalable Infrastructure
Optimized Approach? Cloud is the answer…

6. THE WORKFLOW
Post-
Census
Census
Pre-
Census
Data
Preparation
Data
Collection
& Real-time
Analytics
Data Insights
Dashboards

7. HOLISTIC MULTI-TIER ARCHITECTURE
Enterprise Data hub
Real-time Analytics | Batch analytics
Services Tier
Web APIs| Mobile APIs
Apps Tier
Web Apps| Mobile Apps |Reporting & analytics
Data Pipeline
Stream Processing| Services Bus |Notification hub
Administration&Security

8. SOLUTION BUILDING BLOCKS
Pre-Census Phase
Data Preparation batch Jobs
• Enumeration areascontent
• Tasks Assigning
Enterprise Mobility
management
• MDMMAM Services
• Content Provisioning
Services
• Mobile device Security
services
Enterprise
Service Bus
Notification hub
Stream Processing
In-Memory OLTP (SQLNo-SQL)
Data Pipeline
Data Hub
Services
Tier
Batch Analytics
Operations
Room Apps
Web APIs
Monitoring Executive dashboardsAdministration
Inspectors Supervisors
VPN
Customer
Network
ExecutivesOperators
Site-to-Site
During Census Phase Post-Census Phase
Public
Dissemination portal
Web APIs
Data WareHouse
Mobile APIs

9. CLOUD BASED ARCHITECTURE
Pre-Census Phase
Data Preparation batch Jobs
• Enumeration areas
• Tasks Assigning
Enterprise Mobility
management
• MDMMAM Services
• Content Provisioning
Services
• Mobile device Security
services
Cosmos DB
Data
Pipeline
Data
Hub
Services
Tier
Azure HDinisghts
Operations
Room Apps
Real-time Analytics
APIs
Monitoring
Web Apps
PowerBI
Dashboards
Administration
Web Apps
Inspectors Supervisors
VPN
Customer
Network
ExecutivesOperators
Site-to-Site
During Census Phase Post-Census Phase
Public
Dissemination portal
Web APIs
SQL DatabWareHouse
Azure
Functions
Event
Hub
Notification
Hub
Stream Analytics
Polybase
Periodic Progress
Insights
Charting
Apps
Dashboards
Location Aware
Mobile
APIs
APN

10. DATA COLLECTION FLOW
Data
Ingestion
Event Hub
Notification
Hub
Mobile APIs
Security
& Content
Stream
Processing
Data Hub Presentation
Submit
& Data
Fast
Data
Slow
Data
Long-Term
Storage
Real-time
Apps
Analytics
Cosmos DB
Databricks
Storage
Mobile APIs
Alternatives:
HDINISGHT KAFKA
Alternatives:
HDINISGHT
Storm
Azure functions
Alternatives:
MemSQL for RT
HDINISGHT
Alternatives:
Tableau
Qlik

11. SOLUTION BEST PRACTICES
• PaaS over IaaS, whenever possible PaaS is considered the first option and
this allows to:
• Optimized cost as pay per usageconsumption models.
• Rely on Azure managed services hence better focus on business.
• More satisfactory SLAs for critical operations
• Multi-tenancy, for such nationwide projects with centralized business model
the use of multi-tenancy is very effective from performance point of view
where:
• Each regiongovernorate can be treated as a tenant(separate service plans, separate
DB instance, separate storage accounts)
• This avoid issues like throttling due to IO overload.
• Avoid performance impact of heavy load tenants on less load tenants.
• Security and regulation
• Follow Azure security center analysis and recommendations.
• Apply Data masking encryption to avoid any individuals data (GDPR compliance)

12. FOCUS ON THE VALUE,
• Value oriented ,and let the cloud keep the lights on for you
• Optimized Cost, Estimate and pay for the exact usage and load rather than the
maximum expected
• No high CAPEX.
• Pay for the exact consumption and time you use the service
• Holistic approach, One ecosystem rather than multiple service providers in
addition to holistic System integrator
• Business Agility, respond to changing business demands in timely manner using:
• the “As-a-service” model
• Robust DevOPS for continuous delivery with confidence.
• Business continuity,
• Achieve your business continuity requirements with instant site recovery services
• Detailed Diagnostics and Log analytics for protective actions
rather than the IT services.
provide on-demand insights about the country population characteristics for nation planning and development activities
The Value

13. COST OPTIMIZATION,
Pay for the exact not the maximum.
• IT dept. is no more a cost center, using the cloud built-in reporting and charge back
capabilities the IT dept. can collect budget from business owners instead of being a cost
center.
• Instant Recovery services, Better Business reputation
• Robust DevOps, better time to market
• Continuous Integration
• Continuous Delivery
• Better ROI,
• Noticed reduction in time to market
• Faster market penetration
• Optimized Cost (pay for the exact usage rather than the maximum estimated)
• As per the “Total economic
Impact of Azure PaaS”
conducted by Forrester:

14. • Thank you for attending
• Enjoy & Have Fun with MS Azure
• Open for Questions
By:
Shady Kamel
Solutions Architect