European
Civil Protection
Data Processing
Gisli Olafsson – Consultant
Sources:
UNDAC Methodology
OCHA IM Strategy
Outline
• Principles of data processing
• Plan of Action (PoA) and IM
• Internal information flow within
team/OSOCC – prac...
The IM chain
Input Throughput Output
Collect Analyse DisseminateProcess
Why we need information
noise becomes data when it has a cognitive pattern
data becomes information when its assembled int...
Well organized information
Incoming
data
Incoming
phone
calls
Incoming
emails
Data Processing
What are some issues you can
foresee?
• Information coming from multiple sources
• Information coming through multiple
pat...
Sources of Information
New Sources?
Format
• Enter data
• Reformat
• Clean
Collate
• Compile
• Collate
Verify
quality
• Cross-
reference
• Request
information...
Format
• Structured vs. Unstructured
• Electronic vs. Paper vs. Word of Mouth
• Spelling
• Number format
• Date format
• C...
Group Exercise
• You have been assigned to the PAHO
Emergency Operation Center
• Your task is to go through reports coming...
Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENT
http://twitter.com/vidipblip de #cholera en puerto principe
...
Cholera Outbreak (cont‘d)
Dr Marie-Carmel Charles First human death of Anthrax ("Charbon" in french") is confirmed in Leog...
Formatted data
Collation
• Identify different sources of data to compile from
• Agree methodology with ALL team members
• Identify Priori...
Consumer
social
networks
E-mail Fax
Face-to-face
File shares Paper-based
Messaging
Phone
Different sources of data
OPERATIONAL
IMPORTANCE
TIME / EFFORT INVOLVED TO PROCESS DATA
High
Low
Easy Medium Hard
PRIORITY #1
PRIORITY #2
PRIORITY #...
Group Exercise
• Identify a process to collate and compile
information that comes from
– Meetings
– Phone calls
– Reports ...
PoE
electronic
Hard
Copy
Phone Person
Data Capture
on Template
Physical
Inbox
-Email
-Websites
-file Transfers
(eg.USB FTP...
Quality Assurance
• Verification
• Cross-reference
• Request information
• Triangulate
Verification
• Make sure to record source & other
metadata
• Add a ranking to assess credibility (1-
High; 5 low)
• Differ...
Cross-reference
• Requires baseline data
• Are we all using the same common
baseline data?
• Are we seing similar data fro...
Feedback loop
• If something doesn‘t seem right – can you
reach back to the source?
• If you publish the data to those who...
Triangulate
• Are there more people affected than live
in that area?
• Do the numbers add up?
Storage
• Where does the data reside?
• Do we share data with others?
• Do we track what data we are keeping?
• Do we trac...
Storage/Tracking
• A good file structure
– If using Groove, there is a standard storage format suggested within UNDAC
– If...
Archiving
• Do we keep backups of original data and
processed data?
• How can we get back to the original data?
• Do we tr...
Archiving
• Follow folder system set up by storage team.
• Compile comprehensive directory structure / catalogue / map of
...
Lessons learned
When
Processing/Collating…
• Focus on the value added
• Be aware of data volume
• Consider remote processing
• Create (or ...
Convince partner to share
Add value to
information
Disseminate
quickly
Partner sees net
benefit
IM as a Value Added Service
Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENT
http://twitter.com/vidipblip de #cholera en puerto principe
...
Visualized Data
Consider the volume of data
DATA VOLUME =
# of Forms Collected x Number/Type of Questions x Periodicity
(Frequency of data...
Remote Processing
Simple things…
 Mission Log
 Key events flipchart
 Inbox/Outbox
 Assigned ownership of certain tasks
 Deliverables ov...
Experience from missions
• Information can take hours to get from one
end of an OSOCC tent to the other
• Crucial informat...
Some key things to remember
• Stress the importance of good information
management practices to everyone
– Meeting minutes...
Source: xkcd,
http://xkcd.com/386/
And finally...
© European Commission
Thank you
gisli.olafsson@nethope.org
Gisli Olafsson
Skype: disasterexpert
Upcoming SlideShare
Loading in …5
×

Information Management - Data Processing

1,621
-1

Published on

Training on data processing given as part of the Information Management Course of the European Union.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,621
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • This slides should build, so that the issues we have listed does not show in the beginning. Use the buzz-group approach for 5 minutes (3 per group) to get input on what issues arise. Then collect them onto a flip-chartAt the end, let the rest of this slide build and compare to the ones we have on the flip-chart
  • We ask one of the presenters to take down notes and then ask the audience to provide us with some examples of things that worked well in missions/exercises they have done.Once we have the list from the audience we go through the list on the slide (it should build with only the heading before the q&a).Mission Log:Tracking of all things that happen into a mission log that is “timestamped”. The log should be accessible by everyoneKey events flipchart:A flipchart at a prominent place where key events from the mission log are registered and it is placed where everyone Etc.
  • Information Management - Data Processing

    1. 1. European Civil Protection Data Processing Gisli Olafsson – Consultant Sources: UNDAC Methodology OCHA IM Strategy
    2. 2. Outline • Principles of data processing • Plan of Action (PoA) and IM • Internal information flow within team/OSOCC – practical examples • IM strategies • Group exercise
    3. 3. The IM chain Input Throughput Output Collect Analyse DisseminateProcess
    4. 4. Why we need information noise becomes data when it has a cognitive pattern data becomes information when its assembled into a coherent whole which can be related to other information information becomes knowledge when its integrated with other information in a form that is useful for making decisions and determining actions, knowledge becomes understanding when related to other knowledge in a manner useful in anticipating, judging and acting, understanding becomes wisdom when its informed by purpose, ethics, principals, memory and projection Dee Hock, 1996
    5. 5. Well organized information Incoming data Incoming phone calls Incoming emails Data Processing
    6. 6. What are some issues you can foresee? • Information coming from multiple sources • Information coming through multiple paths • Information coming in multiple formats • Some information digital, other analog • Reliability of information • Conflicting information • Information Gaps • Getting information to everyone is hard
    7. 7. Sources of Information
    8. 8. New Sources?
    9. 9. Format • Enter data • Reformat • Clean Collate • Compile • Collate Verify quality • Cross- reference • Request information • Triangulate Storage • Track • Store • Assign metadata Archiving • Back-up • Retrieval Data Processing Framework
    10. 10. Format • Structured vs. Unstructured • Electronic vs. Paper vs. Word of Mouth • Spelling • Number format • Date format • Currency format • Database vs. Spreadsheet • Geo-tagging
    11. 11. Group Exercise • You have been assigned to the PAHO Emergency Operation Center • Your task is to go through reports coming in of a Cholera outbreak in Haiti • Your task is to clean up the data • Think about what data you would like to capture about each report
    12. 12. Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENT http://twitter.com/vidipblip de #cholera en puerto principe http://twitter.com/firesideint Death confirmed near Cite Soleil alertnet.org 1500 cases in Haiti with http://alertnet.org/thenews/newsdesk/N2216021 5.htm Haiti's government and its aid partners fought on Friday to contain a cholera epidemic that has killed at least 140 people in the nation's worst medical emergency since the Jan. 12 earthquake. http://twitter.com/mediahacker 6 suspected cases http://twitter.com/mediahacker Report of child dying http://twitter.com/gaetantguevara Sant hospitalier lafiteau,gen posiblite pou gen yon moun ki gen #cholera ki ap pran swen.(Center Hospital lafiteauare likely to have a person with cholera # will take care. http://twitter.com/mediahacker Just been told of another death in Lafito, not far from the huge Canaraan camp - man who came from St. Marc. #haiti http://twitter.com/melindayiti Just witnessed a death from #cholera in Lafito/Lafiteau #Haiti only a few minutes from camps Kanaran and Corail, not far from Cite Soleil http://biosurveillance.typepad.com/haiti_operatio nal_biosurv/2010/10/confirmed-cholera-in- arcahaie-ouest-haiti.html Confirmed Cholera in Arcahaie, Ouest http://twitter.com/MissionMANNA cholera reported in Archai and Lagonave http://twitter.com/dodof2 40 people infected in Akaye http://twitter.com/renewal4haiti In other news... 6 cows reported dead of Anthrax in #Leogane. Confirmed by several doctors @iceaxe5 @biosurveillance #whatsgoingon http://twitter.com/realhope4haiti 6 cases reported--First heard it on radio and then a family member from that area came to our village http://twitter.com/biosurveillance Discovery of suspect cholera in Lafito/Lafiteau places the lead edge of the epidemic in direct proximity to Cite Soliel. cholera reported in Archai and Lagonave http://twitter.com/Just_my2cents 15 in Lagonave are at the clinic, very sick, all say they ate fish yesterday! #Cholera International Medical Corps California-based aid group, International Medical Corps, said they had confirmed cases in Croix-des-Bouquet. http://twitter.com/Shayners78 42 now dead in Mirebalais! 28 more than yesterday. I heard from someone who works in our hospital here just up the street from our compound! PAHO http://twitter.com/tonycece 17 dead in remote village of Dauphin. Tried to go further to Grande-Saline but couldn't get there http://twitter.com/newhopehaiti The cholera outbreak has claimed the lives of two of Jean-Claubert's cousins.Artibonite (Gonaives) http://community.ob.org/site/PageServer water purification system installed in Babou Laporte near St Marc on Oct 23 http://twitter.com/CharlesHBaker Petite Riviere is hard Hit. Just left St Michel,it has 2 communal sections w problems; Plantana & Another w 15 dead 26 sick(6 babies) HealthMap 2 cases of cholera in Bel-Air http://twitter.com/pih 10 cases reported in Gonaives http://twitter.com/biosurveillance Two Suspect Cases of #Cholera Seen at HUEH / PAP General Hospital From Artibonite Valley http://twitter.com/biosurveillance Confirmation of Cholera at Multiple Sites in Port au Prince, We now have confirmation of multiple cholera cases inside Port-au-Prince in the Petionville area
    13. 13. Cholera Outbreak (cont‘d) Dr Marie-Carmel Charles First human death of Anthrax ("Charbon" in french") is confirmed in Leogane http://twitter.com/renewal4haiti Hep A, ten cases this week, 1 already treated, 9 at CAMEJO as of Oct 23 http://twitter.com/CharlesHBaker We are Hopital de Secours des Gonaives, One Dead from #cholera #Haiti 70 infected http://twitpic.com/308as4 Samaritans Purse First water filtration system is installed. Word is out&many people have come w/buckets to fill up http://twitter.com/lionelstpierre Report NON CONFIRMED from source of mine!! Kolerat has been detected in "les nippes" South of Haiti ​From a journalist working for ABC http://twitter.com/KarlJeanJeune @KarlJeanJeune Just witnessed the 10th death today at Petite Rivière http://twitter.com/sovelaviorg we have had no deaths. many on IV's. (Sove Lavi Clinic) lcyarrington@email.wm.edu Justinian Hospital lcyarrington@email.wm.edu Sacre Coeur Hospital lcyarrington@email.wm.edu Planned Health Facility http://twitter.com/stevehelling Four cases of cholera detected overnight in a tent city near Delmas 33 http://twitter.com/louis_press Oxfam will provide clean h2o to 25k today http://twitter.com/Renewal4haiti 2 suspect cases at #MSF in #Leogane. Docs working on confirmation now. http://twitter.com/operationbless We've distributed LifeSaver Jerrycans in Dauphin and Luben, as well as water purification unit in St. Marc. Hope this helps! http://twitter.com/operationbless Jerrycans in Luben http://twitter.com/operationbless Purification unit
    14. 14. Formatted data
    15. 15. Collation • Identify different sources of data to compile from • Agree methodology with ALL team members • Identify Priorities EARLY • Capture information about people and organizations into 3W • Collect information into charts, matrixes on walls etc • Ensure enough human resources to process • Establish a reporting systems / forms • Enter ONLINE if possible • Document what you are collating • Try to compile information into one repository
    16. 16. Consumer social networks E-mail Fax Face-to-face File shares Paper-based Messaging Phone Different sources of data
    17. 17. OPERATIONAL IMPORTANCE TIME / EFFORT INVOLVED TO PROCESS DATA High Low Easy Medium Hard PRIORITY #1 PRIORITY #2 PRIORITY #3 Prioritization Graph
    18. 18. Group Exercise • Identify a process to collate and compile information that comes from – Meetings – Phone calls – Reports via radio – Email • How do you deal with important information
    19. 19. PoE electronic Hard Copy Phone Person Data Capture on Template Physical Inbox -Email -Websites -file Transfers (eg.USB FTP) TO DO Data Entry Defined Categories Assessment Data Contacts 3W SITREPS Resources
    20. 20. Quality Assurance • Verification • Cross-reference • Request information • Triangulate
    21. 21. Verification • Make sure to record source & other metadata • Add a ranking to assess credibility (1- High; 5 low) • Different quality control needed on different types of data
    22. 22. Cross-reference • Requires baseline data • Are we all using the same common baseline data? • Are we seing similar data from other sources?
    23. 23. Feedback loop • If something doesn‘t seem right – can you reach back to the source? • If you publish the data to those who report it they will usually provide you feedback • Social media has been shown to be very self-correcting due to its openess with the data
    24. 24. Triangulate • Are there more people affected than live in that area? • Do the numbers add up?
    25. 25. Storage • Where does the data reside? • Do we share data with others? • Do we track what data we are keeping? • Do we track information about the data? • Technology options
    26. 26. Storage/Tracking • A good file structure – If using Groove, there is a standard storage format suggested within UNDAC – If not using Groove, need to agree upon a structure to use • Track and share information – Step 1: How many excel sheets, word docs, pdfs, etc – Step 2: Excel sheets can be pasted on google docs – Step 3: Word docs can be pasted on google docs = Work on own computer as well as internet • Need to back up original data as well as collated data base • • How do we track the data and file/label data so that the right people get it? – Protocol format for files received, priority, flagging for clusters/units – Creation of virtual mailboxes for each cluster/sector, multiple copies of each document to tag it for the correct audience – Mark each item with: time of receipt, source, who is it for – Who stores the data – the recipient or the sender?
    27. 27. Archiving • Do we keep backups of original data and processed data? • How can we get back to the original data? • Do we trace decisions back to information back to reports?
    28. 28. Archiving • Follow folder system set up by storage team. • Compile comprehensive directory structure / catalogue / map of archive • Per day • Per cluster • Back-up of archive – e.g. secure remote server. • Changes to archive backed-up automatically to remote server. • During versus post-mission. • During mission back-up every document • After mission – archive original sources and final products. • During a two week mission – storing, tracking and archiving/retrieving are the sane thing for a short-duration UNDAC mission. • Simple logic important to facilitate handover (e.g. UNDAC mission software).
    29. 29. Lessons learned
    30. 30. When Processing/Collating… • Focus on the value added • Be aware of data volume • Consider remote processing • Create (or make use of) standard products and avoid ad hoc ones • Consider speed over perfection • Balance resources and time management
    31. 31. Convince partner to share Add value to information Disseminate quickly Partner sees net benefit IM as a Value Added Service
    32. 32. Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENT http://twitter.com/vidipblip de #cholera en puerto principe http://twitter.com/firesideint Death confirmed near Cite Soleil alertnet.org 1500 cases in Haiti with http://alertnet.org/thenews/newsdesk/N2216021 5.htm Haiti's government and its aid partners fought on Friday to contain a cholera epidemic that has killed at least 140 people in the nation's worst medical emergency since the Jan. 12 earthquake. http://twitter.com/mediahacker 6 suspected cases http://twitter.com/mediahacker Report of child dying http://twitter.com/gaetantguevara Sant hospitalier lafiteau,gen posiblite pou gen yon moun ki gen #cholera ki ap pran swen.(Center Hospital lafiteauare likely to have a person with cholera # will take care. http://twitter.com/mediahacker Just been told of another death in Lafito, not far from the huge Canaraan camp - man who came from St. Marc. #haiti http://twitter.com/melindayiti Just witnessed a death from #cholera in Lafito/Lafiteau #Haiti only a few minutes from camps Kanaran and Corail, not far from Cite Soleil http://biosurveillance.typepad.com/haiti_operatio nal_biosurv/2010/10/confirmed-cholera-in- arcahaie-ouest-haiti.html Confirmed Cholera in Arcahaie, Ouest http://twitter.com/MissionMANNA cholera reported in Archai and Lagonave http://twitter.com/dodof2 40 people infected in Akaye http://twitter.com/renewal4haiti In other news... 6 cows reported dead of Anthrax in #Leogane. Confirmed by several doctors @iceaxe5 @biosurveillance #whatsgoingon http://twitter.com/realhope4haiti 6 cases reported--First heard it on radio and then a family member from that area came to our village http://twitter.com/biosurveillance Discovery of suspect cholera in Lafito/Lafiteau places the lead edge of the epidemic in direct proximity to Cite Soliel. cholera reported in Archai and Lagonave http://twitter.com/Just_my2cents 15 in Lagonave are at the clinic, very sick, all say they ate fish yesterday! #Cholera International Medical Corps California-based aid group, International Medical Corps, said they had confirmed cases in Croix-des-Bouquet. http://twitter.com/Shayners78 42 now dead in Mirebalais! 28 more than yesterday. I heard from someone who works in our hospital here just up the street from our compound! PAHO http://twitter.com/tonycece 17 dead in remote village of Dauphin. Tried to go further to Grande-Saline but couldn't get there http://twitter.com/newhopehaiti The cholera outbreak has claimed the lives of two of Jean-Claubert's cousins.Artibonite (Gonaives) http://community.ob.org/site/PageServer water purification system installed in Babou Laporte near St Marc on Oct 23 http://twitter.com/CharlesHBaker Petite Riviere is hard Hit. Just left St Michel,it has 2 communal sections w problems; Plantana & Another w 15 dead 26 sick(6 babies) HealthMap 2 cases of cholera in Bel-Air http://twitter.com/pih 10 cases reported in Gonaives http://twitter.com/biosurveillance Two Suspect Cases of #Cholera Seen at HUEH / PAP General Hospital From Artibonite Valley http://twitter.com/biosurveillance Confirmation of Cholera at Multiple Sites in Port au Prince, We now have confirmation of multiple cholera cases inside Port-au-Prince in the Petionville area
    33. 33. Visualized Data
    34. 34. Consider the volume of data DATA VOLUME = # of Forms Collected x Number/Type of Questions x Periodicity (Frequency of data collection)
    35. 35. Remote Processing
    36. 36. Simple things…  Mission Log  Key events flipchart  Inbox/Outbox  Assigned ownership of certain tasks  Deliverables overview (when, how, to whom)  Meeting schedule overview (where, when, who)  Scrum (interrupt everyone for 5 minutes every few hours hours to run very quickly through newest status)  Situational overview map
    37. 37. Experience from missions • Information can take hours to get from one end of an OSOCC tent to the other • Crucial information often does not get processed • Information Management people often get very overloaded • Technology often becomes a hurdle – especially when there is no process in place • …
    38. 38. Some key things to remember • Stress the importance of good information management practices to everyone – Meeting minutes for all minutes – Geospatial coding of data collected whenever possible • Assign ownership – divide and conquer • Use the tools available in an efficient manner • But above all Put in place an Information Management Process!!! Remember to keep it simple, stupid!!
    39. 39. Source: xkcd, http://xkcd.com/386/ And finally...
    40. 40. © European Commission Thank you gisli.olafsson@nethope.org Gisli Olafsson Skype: disasterexpert

    ×