Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Guerrilla Analytics
Introduction and Case Study
Enda Ridge, PhD
Copyright Enda Ridge 2015 0#GuerrillaAnalytics http://guer...
What we are told about Data Science
1#GuerrillaAnalytics http://guerrilla-analytics.net
“the sexy job in the next 10 years...
Hi, we need an update on the
insurance policy classification
work.
It’s going to the Head of
Underwriting this afternoon.
...
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 3
Those were the droids
I was looking for ...
My Journey to Guerrilla Analytics
Mechanical
Engineer
PhD
Computer
Science
Boutique
Consultancy
Forensic
Data
Analytics
Se...
Common format
#GuerrillaAnalytics http://guerrilla-analytics.net 5
Data Analytics Insight
Copyright Enda Ridge 2015
Misconception
#GuerrillaAnalytics http://guerrilla-analytics.net 6
Shearer C., The CRISP-
DM model: the new
blueprint for ...
Reality is Guerrilla Analytics
#GuerrillaAnalytics http://guerrilla-analytics.net 7
Data
• Extraction
• Receipt
• Loading
...
Maintain Data Provenance
#GuerrillaAnalytics http://guerrilla-analytics.net 8Copyright Enda Ridge 2015
Maintaining Data Pr...
7 Principles of Guerrilla Analytics
• Space is cheap, confusion is expensive1
• Prefer simple, visual project structures2
...
Guerrilla Analytics
#GuerrillaAnalytics http://guerrilla-analytics.net 10
Data
• Extraction
• Receipt
• Loading
Analytics
...
Guerrilla Analytics Case Study
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 11
Client Reta...
System Landscape
Customer Contact
Card System 1 Card System 2
Collections
Manual
Intervention
Copyright Enda Ridge 2015 #G...
Data Receipt
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 13
Guerrilla Analytics Environme...
Data Receipt
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 14
Guerrilla Analytics Approach
...
Data Load
File System
Crazy-name spreadsheet 1
Crazy-name spreadsheet 2
Crazy-name spreadsheet 3
FNU810A
long_named_file_v...
Data Load
Data
Crazy-name spreadsheet
1
Crazy-name spreadsheet
2
FNU810A
long_named_file_v0.2.1.
pdf
Analytics Environment...
Guerrilla Analytics
#GuerrillaAnalytics http://guerrilla-analytics.net 17
Data
• Extraction
• Receipt
• Loading
Analytics
...
Analytics: Guerrilla Analytics Environment
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 18...
Analytics: Guerrilla Analytics Approach
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 19
• ...
Analytics: Guerrilla Analytics Approach
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 20
• ...
Analytics: Guerrilla Analytics Approach
File System
WP_024
010_Accounts_Cleaned.SQL
030_Transaction_Profiles.R
050_Sample_...
Data Manipulation: Guerrilla Environment
Account_ID Statement_ID Min_Payment Transaction_ID Amount Type
A 15 30.00 1 50.00...
Data Manipulation: Guerrilla Analytics Approach
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.n...
Guerrilla Analytics
#GuerrillaAnalytics http://guerrilla-analytics.net 24
Data
• Extraction
• Receipt
• Loading
Analytics
...
Reporting – what is a report?
#GuerrillaAnalytics http://guerrilla-analytics.net 25Copyright Enda Ridge 2015
Reporting – Guerrilla Analytics Environment
#GuerrillaAnalytics http://guerrilla-analytics.net 26Copyright Enda Ridge 2015
Reporting – Guerrilla Analytics Environment
#GuerrillaAnalytics http://guerrilla-analytics.net 27Copyright Enda Ridge 2015...
Guerrilla Analytics
#GuerrillaAnalytics http://guerrilla-analytics.net 28
Data
• Extraction
• Receipt
• Loading
Analytics
...
Why consolidate?
#GuerrillaAnalytics http://guerrilla-analytics.net 29
Raw
Duplicates
Customers Clean_Cust
Deduped New_dup...
Why consolidate?
#GuerrillaAnalytics http://guerrilla-analytics.net 30
Raw
Duplicates
Customers Clean_Cust
Deduped New_dup...
Guerrilla Analytics Approach: Builds
#GuerrillaAnalytics http://guerrilla-analytics.net 31
Deduped
Clean_cust
New_dupes
Du...
Summary
Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 32
A Realistic Workflow
• Guerrilla A...
Keep in Touch!
#GuerrillaAnalytics http://guerrilla-analytics.net 33
@Enda_Ridge
http://guerrilla-analytics.net
Copyright ...
Upcoming SlideShare
Loading in …5
×

Guerrilla Analytics - Introduction and Case Study

351 views

Published on

What is Guerrilla Analytics and how does it help you do Data Science in real world project environments? Example case study highlighting the application of the 7 Guerrilla Analytics Principles.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Guerrilla Analytics - Introduction and Case Study

  1. 1. Guerrilla Analytics Introduction and Case Study Enda Ridge, PhD Copyright Enda Ridge 2015 0#GuerrillaAnalytics http://guerrilla-analytics.net
  2. 2. What we are told about Data Science 1#GuerrillaAnalytics http://guerrilla-analytics.net “the sexy job in the next 10 years will be statisticians” “Data Scientist: The Sexiest Job of the 21st Century” “Information is the oil of the 21st century, and analytics is the combustion engine.” http://www.gapminder.org/ http://www.statistics.com/data-science-quotes/ https://github.com/mbostock/d3/wiki/Gallery Copyright Enda Ridge 2015
  3. 3. Hi, we need an update on the insurance policy classification work. It’s going to the Head of Underwriting this afternoon. Um. Which work? I think Jo did that but Jo’s on holidays. I’ll check my mailbox and send you my spreadsheet from last week. Err.....the population changed with the extra system extract on Tuesday. And we added a bunch of business rules to accommodate that.... so we can’t go back to the earlier numbers. The Reality 2#GuerrillaAnalytics http://guerrilla-analytics.netCopyright Enda Ridge 2015
  4. 4. Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 3 Those were the droids I was looking for ...
  5. 5. My Journey to Guerrilla Analytics Mechanical Engineer PhD Computer Science Boutique Consultancy Forensic Data Analytics Senior Manager #GuerrillaAnalytics http://guerrilla-analytics.net 4Copyright Enda Ridge 2015 Constraints Constraints + Dynamic Reproducible Constraints + Dynamic Reproducible + Tested Constraints + Dynamic Reproducible + Tested + Audit
  6. 6. Common format #GuerrillaAnalytics http://guerrilla-analytics.net 5 Data Analytics Insight Copyright Enda Ridge 2015
  7. 7. Misconception #GuerrillaAnalytics http://guerrilla-analytics.net 6 Shearer C., The CRISP- DM model: the new blueprint for data mining, J Data Warehousing (2000); 5:13—22 Copyright Enda Ridge 2015
  8. 8. Reality is Guerrilla Analytics #GuerrillaAnalytics http://guerrilla-analytics.net 7 Data • Extraction • Receipt • Loading Analytics • Transform • Algorithms • Consolidate Insight • Reporting • Work Products Disruptions Copyright Enda Ridge 2015
  9. 9. Maintain Data Provenance #GuerrillaAnalytics http://guerrilla-analytics.net 8Copyright Enda Ridge 2015 Maintaining Data Provenance mitigates disruptions
  10. 10. 7 Principles of Guerrilla Analytics • Space is cheap, confusion is expensive1 • Prefer simple, visual project structures2 • Prefer automation with program code3 • Link data on the file system, analytics environment, and work products4 • Version control data and code5 • Consolidate team knowledge in builds6 • Prefer code that runs end to end7 Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 9 ~100 practice tips
  11. 11. Guerrilla Analytics #GuerrillaAnalytics http://guerrilla-analytics.net 10 Data • Extraction • Receipt • Loading Analytics • Transform • Algorithms • Consolidate Insight • Reporting • Work Products Disruptions Copyright Enda Ridge 2015
  12. 12. Guerrilla Analytics Case Study Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 11 Client Retail Bank Situation Error in credit card customer mailing process Failure to comply with regulations, potential fines Mission • Understand system landscape & get the right data • Rebuild full customer history • Identify system errors and start of non-compliance • Quantify effected customers and cost to bank Timeline 6-8 weeks
  13. 13. System Landscape Customer Contact Card System 1 Card System 2 Collections Manual Intervention Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 12
  14. 14. Data Receipt Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 13 Guerrilla Analytics Environment • Lost Data • Multiple Copies of data • Limited supporting information • Local copies of data • Renamed data • ä~ delimited data
  15. 15. Data Receipt Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 14 Guerrilla Analytics Approach • Have 1 Data location • Data Unique Identifiers • Data log • Supporting material near data • Never modify the data
  16. 16. Data Load File System Crazy-name spreadsheet 1 Crazy-name spreadsheet 2 Crazy-name spreadsheet 3 FNU810A long_named_file_v0.2.1.pdf Analytics Environment Credit_Card_Samples DBO.Accounts Customer_Letters #GuerrillaAnalytics http://guerrilla-analytics.net 15 Guerrilla Environment • Renamed files • Scattered inconsistent locations • Multiple versions of files • Replacements of files Copyright Enda Ridge 2015
  17. 17. Data Load Data Crazy-name spreadsheet 1 Crazy-name spreadsheet 2 FNU810A long_named_file_v0.2.1. pdf Analytics Environment D010.Crazy-name spreadsheet 1 D026.Crazy-name spreadsheet 2 D040.FNU810A D051.long_name d_file_v0.2.1.pdf #GuerrillaAnalytics http://guerrilla-analytics.net 16 Guerrilla Analytics Approach • One-to-one mapping from files to datasets – Keep crazy names • Minimize prep work • Put the Data Identifier in the path Copyright Enda Ridge 2015 D010 D026 D040 D051
  18. 18. Guerrilla Analytics #GuerrillaAnalytics http://guerrilla-analytics.net 17 Data • Extraction • Receipt • Loading Analytics • Transform • Algorithms • Consolidate Insight • Reporting • Work Products Disruptions Copyright Enda Ridge 2015
  19. 19. Analytics: Guerrilla Analytics Environment Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 18 My Documents/Transactions Accounts_Formatted.SQL TransProf_FINAL.R Trans_DO_NOT_USE.R TransProf_v2.R Sample_accounts.SQL • Many code files/languages • Variety of output types • Data manipulation – on file system – in analytics environment • Combinations of tools • Many users • Many iterations
  20. 20. Analytics: Guerrilla Analytics Approach Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 19 • One folder for all team work products • Give every work product an identifier • Keep a work product log • Clear running order of files • No dead/orphaned files Work_Products Work_products.xls WP_024 • 010_Accounts_Cleaned.SQL • 030_Transaction_Profiles.R • 050_Sample_accounts.SQL WP_96 WP_97
  21. 21. Analytics: Guerrilla Analytics Approach Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 20 • Keep older versions in subfolder • Keep related information in a subfolder WP_024 010_Accounts_Cleaned.SQL 030_Transaction_Profiles.R 050_Sample_accounts.SQL supporting archive
  22. 22. Analytics: Guerrilla Analytics Approach File System WP_024 010_Accounts_Cleaned.SQL 030_Transaction_Profiles.R 050_Sample_accounts.SQL Analytics Environment WP_024.ACCOUNTS_CLEANED WP_024.TRANSACTION_PROFILES WP_024.SAMPLE_ACCOUNTS Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 21
  23. 23. Data Manipulation: Guerrilla Environment Account_ID Statement_ID Min_Payment Transaction_ID Amount Type A 15 30.00 1 50.00 Expense A 15 30.00 2 25.00 Expense A 15 30.00 3 -75.00 Payment A 15 30.00 4 20.00 Expense Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 22 ID Stmnt_ID Min_Payment Balance Min_Paym_Made A 15 30.00 20.00 No ... ... ...
  24. 24. Data Manipulation: Guerrilla Analytics Approach Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 23 Account_ID Statement_I D Min_Paym ent Transaction _ID Amount Type Running Payments Min Paym Made A 15 30.00 1 50.00 Expense 0.00 No A 15 30.00 2 25.00 Expense 0.00 No A 15 30.00 3 -75.00 Payment 75.00 Yes A 15 30.00 4 20.00 Expense 75.00 Yes Account_ID Statement_ID Min_Payment Transaction_ID Amount Type A 15 30.00 1 50.00 Expense A 15 30.00 2 25.00 Expense A 15 30.00 3 -75.00 Payment A 15 30.00 4 20.00 Expense
  25. 25. Guerrilla Analytics #GuerrillaAnalytics http://guerrilla-analytics.net 24 Data • Extraction • Receipt • Loading Analytics • Transform • Algorithms • Consolidate Insight • Reporting • Work Products Disruptions Copyright Enda Ridge 2015
  26. 26. Reporting – what is a report? #GuerrillaAnalytics http://guerrilla-analytics.net 25Copyright Enda Ridge 2015
  27. 27. Reporting – Guerrilla Analytics Environment #GuerrillaAnalytics http://guerrilla-analytics.net 26Copyright Enda Ridge 2015
  28. 28. Reporting – Guerrilla Analytics Environment #GuerrillaAnalytics http://guerrilla-analytics.net 27Copyright Enda Ridge 2015 Select min/max of transaction_time WP_030 •010_Late payments.SQL •030_Late payments.py WP_042
  29. 29. Guerrilla Analytics #GuerrillaAnalytics http://guerrilla-analytics.net 28 Data • Extraction • Receipt • Loading Analytics • Transform • Algorithms • Consolidate Insight • Reporting • Work Products Disruptions Copyright Enda Ridge 2015
  30. 30. Why consolidate? #GuerrillaAnalytics http://guerrilla-analytics.net 29 Raw Duplicates Customers Clean_Cust Deduped New_dupes Work Product Copyright Enda Ridge 2015
  31. 31. Why consolidate? #GuerrillaAnalytics http://guerrilla-analytics.net 30 Raw Duplicates Customers Clean_Cust Deduped New_dupes Duplicates_02 Customers_02 Duplicates Deduped Clean_cust New_dupes Work Product Copyright Enda Ridge 2015
  32. 32. Guerrilla Analytics Approach: Builds #GuerrillaAnalytics http://guerrilla-analytics.net 31 Deduped Clean_cust New_dupes Duplicates_02 Duplicates Customers_02 Dupes_latest Cust_Latest Raw Latest Clean Rules Interface Version Controlled Code and Data Copyright Enda Ridge 2015 WP_030
  33. 33. Summary Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net 32 A Realistic Workflow • Guerrilla Analytics Principles • Guerrilla Analytics Practice Tips Case Study • Data receipt and load • Analytics • Reporting and work products • Consolidation with Builds Why Data Science is Difficult • Disruptions, Constraints • These break Data Provenance Those were the droids I was looking for ...
  34. 34. Keep in Touch! #GuerrillaAnalytics http://guerrilla-analytics.net 33 @Enda_Ridge http://guerrilla-analytics.net Copyright Enda Ridge 2015 Or contact me for 50% discount

×