Neo4j Partner Tag Berlin - Investigating the Panama Papers connections with neo4j
•
0 likes•248 views
The document discusses using Neo4j and graph databases to analyze documents from the Panama Papers investigation. It describes the steps taken by journalists to acquire, classify, and analyze over 3 million documents. Key aspects included developing a data model to represent entities and relationships, extracting metadata and parsing documents to populate the graph. Neo4j was highlighted as a tool to efficiently store and query the connected relational data in the documents. An example graph showing relationships in documents related to Azerbaijan's president was also presented.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Neo4j Partner Tag Berlin - Investigating the Panama Papers connections with neo4j
Similar to Neo4j Partner Tag Berlin - Investigating the Panama Papers connections with neo4j (20)
More from Neo4j
More from Neo4j (20)
Recently uploaded
Recently uploaded (20)
Neo4j Partner Tag Berlin - Investigating the Panama Papers connections with neo4j
- 3. Source Material taken from • the ICIJ presenta1on • the Reddit AMA • online publica1ons (SZ, Guardian, TNW et.al.) • the ICIJ website • hFps://panamapapers.icij.org/ • The Power Players • Key Numbers & Figures
- 6. +190 journalists in more than 65 countries 12 staff members (USA, Costa Rica, Venezuela, Germany, France, Spain) 50% of the team = Data & Research Unit
- 7. raw files metadata author; sender... database search and discovery raw text
- 8. 3 million files x 10 seconds per file = 347 Days Inves1gators used Nuix’s op1cal character recogni1on to make millions of scanned documents text-searchable. They used Nuix’s named en1ty extrac1on and other analy1cal tools to iden1fy and cross- reference the names of Mossack Fonseca clients through millions of documents.
- 9. Lucene syntax queries with proximity matching! 400 users
- 10. Unstructured data extrac1on ● Nuix professional OCR service ● ICIJ Extract (open source, Java: hFps://github.com/ICIJ/extract), leverages Apache Tika, Tesseract OCR and JBIG2-ImageIO. Structured data extrac1on ● A bunch of Python Database ● Apache Solr (open source, Java) ● Redis (open source, C) ● Neo4j (open source, Java) App ● Blacklight (open source, Rails) ● Linkurious (closed source, JS) Stack
- 15. The world is a graph – everything is connected • people, places, events • companies, markets • countries, history, poli1cs • sciences, art, teaching • technology, networks, machines, applica1ons, users • sodware, code, dependencies, architecture, deployments • criminals, fraudsters and their behavior
- 18. NODE key: “value” proper1es Property Graph Model Nodes • The en11es in the graph • Can have name-value proper%es • Can be labeled Rela&onships • Relate nodes by type and direc1on • Can have name-value proper%es RELATIONSHIP NODE NODE key: “value” proper1es key: “value” proper1es key: “value” proper1es
- 19. Your friend Neo4j An open-source graph database • Manage and store your connected data as a graph • Query rela&onships easily and quickly • Evolve model and applica&ons to support new requirements and insights • Built to solve rela&onal pains
- 22. Neo4j: All about PaTerns (:Person { name:"Dan"} ) -[:KNOWS]-> (:Person {name:"Ann"}) KNOWS Dan Ann NODE NODE LABEL PROPERTY hTp://neo4j.com/developer/cypher LABEL PROPERTY
- 23. Cypher: Find PaTerns MATCH (:Person { name:"Dan"} ) -[:KNOWS]-> (who:Person) RETURN who KNOWS Dan ??? LABEL NODE NODE LABEL PROPERTY ALIAS ALIAS hTp://neo4j.com/developer/cypher
- 25. Ge]ng Data into Neo4j Load JSON with Cypher • Load JSON via procedure • Deconstruct the document • Into a non-duplicated graph model {} {} {} CALL apoc.load.json("url") yield value as doc UNWIND doc.items as item MERGE (:Contract {title:item.title, amount:toFloat(item.amount)});
- 26. Ge]ng Data into Neo4j CSV Bulk Loader neo4j-import • For ini1al database popula1on • For loads with 10B+ records • Up to 1M records per second ,,, ,,, ,,, bin/neo4j-import –-into people.db --nodes:Person people.csv --nodes:Company companies.csv --relationship:STAKEHOLDER stakeholders.csv
- 28. The Steps Involved in the Document Analysis 1. Acquire documents 2. Classify documents • Scan / OCR • Extract document metadata 3. Whiteboard domain and ques&ons, determine • en&&es and their rela&onships • poten1al en1ty and rela1onship proper&es • sources for those en11es and their proper1es
- 29. The Steps Involved in the Document Analysis 4. Develop analyzers, rules, parsers and named en1ty recogni1on 5. Parse and store metadata, document and en1ty rela1onships • Parse by author, named en11es, dates, sources and classifica1ons 6. Infer en1ty rela1onships 7. Compute similari1es, transi1ve cover and triangles 8. Analyze data using graph queries and visualiza1ons
- 30. We need a Data Model Meta Data En&&es • Document, Email, Contract, DB- Record • Meta: Author, Date, Source, Keywords • Conversa1on: Sender, Receiver, Topic • Money Flows Actual En&&es • Person • Representa1ve (Officer) • Address • Client • Company • Account Either based on our use cases & ques1ons On the en11es present in our meta-data and data.
- 31. Data Model – Rela&onships Meta-Data • sent, received, cc‘ed • men1oned, topic-of • created, signed • aFached • roles • family rela1onships Ac&vi&es • open account • manage • has shares • registered address • money flow
- 33. The ICIJ Data Model • Simplis1c Datamodel with 4 En11es and 5 Rela1onships • We only know the published model • Missing • Documents, Metadata • Family Rela1onships • Connec1ons to Public Record Databases • Contains Duplicates • Rela1onship informa1on stored on en11es • Could use richer labeling
- 34. Example Dataset - Azerbaijan’s President Ilham Aliyev • was already previously inves1gated • whole family involved • different shell companies & involvements hFp://neo4j.com/graphgist/ec65c2fa-9d83-4894-bc1e-98c475c7b57a