Automated data analysis with Python
Upcoming SlideShare
Loading in...5
×
 

Automated data analysis with Python

on

  • 1,983 views

Talk at #pyconindia2012

Talk at #pyconindia2012

Statistics

Views

Total Views
1,983
Views on SlideShare
1,636
Embed Views
347

Actions

Likes
7
Downloads
40
Comments
1

2 Embeds 347

http://in.pycon.org 329
https://twitter.com 18

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Automated data analysis with Python Automated data analysis with Python Presentation Transcript

  • AUTOMATED DATA ANALYSIS WITH PYTHON (PART II) S.Anand@Gramener.com
  • DO WE FOLLOW PEP8?
  • author repo filename errno name countMichael0x2a axe interpreter-parsing_rules.py E231 missing whitespace after , 14177egirault googleplay api-googleplay_pb2.py E121 continuation line indentation is not a multiple of four 12953mdwrigh2 pyice parsetab.py E231 missing whitespace after , 5452steviesteveo projecteuler euler22.py E231 missing whitespace after , 5162xiongchiamiov Mirage mirage.py W191 indentation contains tabs 4593Ariel team Ariel-compiler-Sintactic.py E231 missing whitespace after , 4489albertz PyCParser cparser.py W191 indentation contains tabs 3041pombredanne PyCParser cparser.py W191 indentation contains tabs 3025cshen PyCParser cparser.py W191 indentation contains tabs 3025bohr PyCParser cparser.py W191 indentation contains tabs 2988fj PyCParser cparser.py W191 indentation contains tabs 1863steviesteveo projecteuler euler42.py E231 missing whitespace after , 1786steviesteveo projecteuler wordlist.py E231 missing whitespace after , 1785aixp pycoco Core.py E111 indentation is not a multiple of four 1760mdoege 3NewsFeed newsfeed.py W191 indentation contains tabs 1738mdoege 3NewsFeed newsfeed.py E101 indentation contains mixed spaces and tabs 1738ebegoli EMLPy w3c_ir_assertions.py W191 indentation contains tabs 1650AvsPmod AvsPmod AvsP.py E501 line too long (80 > 79 characters) 1598chikuzen AvsPmod AvsP.py E501 line too long (80 > 79 characters) 1598tweetr python twitter-twitterapi.py E101 indentation contains mixed spaces and tabs 1507duartebarbosa googletranslate Languages.py E101 indentation contains mixed spaces and tabs 1422duartebarbosa googletranslate Languages.py W191 indentation contains tabs 1422nrub python twitter-twitterapi.py E101 indentation contains mixed spaces and tabs 1297danudey python twitter-twitterapi.py E101 indentation contains mixed spaces and tabs 1278idan python twitter-twitterapi.py E101 indentation contains mixed spaces and tabs 1278
  • import sysimport pandas as pddata = pd.read_csv(sys.argv[1])
  • import sysimport pandas as pddata = pd.read_csv(sys.argv[1])print data.groupby(name).sum().sort(count)tab after keyword 1blank lines found after function decorator 2tab after operator 28tab before keyword 31unexpected indentation 41expected an indented block 78multiple spaces after keyword 120...blank line contains whitespace 40543no spaces around keyword / parameter equals 41858indentation is not a multiple of four 44109missing whitespace around operator 47286indentation contains mixed spaces and tabs 52633line too long (80 > 79 characters) 78201missing whitespace after , 91612indentation contains tabs 168842
  • LET’S TAKE MARKS
  • DIST_CODE DOB Day Caste B/G Med Cond Total SCHOOL_NAME Kannada English Hindi Maths Science SocialCHIKKABALLAPUR 13-Jul-95 Thu ST G K N 111 PRIYADHARSHINI HIGH SCHOOL 46 7 10 30 8 10GADAG 09-Feb-95 Thu OTHERS B E N 458 LOYALA HIGH SCHOOL GADAG 86 69 52 70 90 91MANGALORE 27-Oct-95 Fri OTHERS B K N 390 GOVT.HIGH SCHOOL KOKKADA 105 35 65 76 67 42BELGAUM 15-Jun-95 Thu ST B M N 151 MADYAMIKA VIDYALAYA BELAVATTI 14 23 25 26MADHUGIRI 11-Sep-95 Mon OTHERS B K N 240 SRI KALIDASA VIDYAVARDHAKA H.S. 57 35 35 48 30 35KOLAR 08-May-95 Mon OTHERS B E N 363 DR.AMBEDKAR HIGH SCHOOL 57 63 60 61 62 60BIJAPUR 24-May-95 Wed OTHERS B K N 451 LOYOLA HIGH SCHOOL STATION BACK 90 51 87 79 81 63UDUPI 05-Feb-96 Mon SC B K N 239 GOVT JUNIOR COLLEGE BAILOOR 54 30 65 30 30 30BANGALORE NORTH 20-Oct-95 Fri OTHERS G E N 530 ST MARYS HIGH SCHOOL NO 1 T 92 78 69 77GULBARGA 03-Jan-95 Tue OTHERS G K N 397 GOVERNMENT HIGH SCHOOL ANDOLA, 96 47 61 65 67 61BELGAUM 10-May-94 Tue CAT-1 B K N 111 GOVERNMENT HIGH SCHOOL SULEBHAVI 21 35 9 22 18 6BIJAPUR 10-Jul-95 Mon OTHERS B K N 380 H G P U COLLEGE SINDAGI BIJAPUR 87 43 69 65 60 56CHIKODI 25-Apr-95 Tue OTHERS B K N 408 GOVERNMENT HIGH SCHOOL 94 54 85 47 63 65SHIMOGA 18-Dec-95 Mon SC G K N 215 SAHYADRI HIGH SCHOOL SHIMOGA 44 35 40 31 30 35BIJAPUR 18-Nov-93 Thu SC B K N 157 TILAGUL HIGH SCHOOL TILAGUL 29 12 35 20 31 30KOLAR 26-Sep-93 Sun SC B K N 237 GOVERNMENT HIGH SCHOOL MEDIHAL 55 30 37 30 38 47KOPPAL 01-Jun-93 Tue OTHERS B K N 254 GOVERNMENT HIGH SCHOOL HIRE 38 42 37 53 49 35CHIKKABALLAPUR 21-Apr-96 Sun OTHERS B K N 251 GOVT. HIGH SCHOOL KADALAVENI 77 40 53 40 26 15CHIKODI 25-Nov-95 Sat OTHERS B M N 477 ARUN SHAMARAO PATIL HIGH SCHOOL 70 80 66 77BELGAUM 16-Feb-95 Thu OTHERS G U N 307 BEGUM LATIFA GIRLS HIGH SCHOOL 44 9 50 56
  • import sysimport pandas as pddata = pd.read_csv(sys.argv[1])print data.groupby(DIST_CODE).means().sort(TOTAL_MARKS) TOTAL_MARKS Kannada ... Social ScienceDIST_CODEBIDAR 245.018650 56.594794 ... 40.368867YADGIR 285.778553 63.193738 ... 48.891916MADHUGIRI 291.869219 73.725051 ... 43.854291......CHIKODI 354.548775 79.675186 ... 58.088485SIRSI 356.859926 82.086493 ... 56.168686UDUPI 358.532346 82.697818 ... 50.479084
  • KANNADA ENGLISH HINDIMATHS SCIENCE SOCIAL SCIENCE
  • HOW DO WE GENERALISE?
  • Groups Things you can group by(Dimensions) Place, Categories, Attributes string, datetime, intNumbers Things you can measure (Metrics) Sizes, Values, Growth, Frequencies float, int
  • category title kJ ratedairy Activia Pouring Natural Yogurt 1X950g 216 0.21dairy Activia Pouring Strawberry Yogurt 1X950g 250 0.21dairy Activia Pouring Vanilla Yogurt 1X950g 263 0.21icecream Almondy Daim 400G 1804 0.75icecream Almondy Toblerone 400G 1850 0.5cereals Alpen 10 Pack Lite Summer Fruits Cereal Bars 210G 1222 1.57cereals Alpen 10Pk Fruit Nut And Chocolate Cereal Bars 290G 1812 1.14cereals Alpen Coconut And Chocolate Cereal Bars 5Pk 145G 1863 1.24cereals Alpen Fruit And Nut With Chocolate Cereal Bar 5X29g 1812 1.24cereals Alpen High Fruit 650G 1439 0.4cereals Alpen Light Bars Chocolate And Orange 5X21g 1246 1.71cereals Alpen Light Chocolate And Fudge Bar 5X21g 1264 1.71cereals Alpen Light Sultana & Apple Bars 5Pk 105G 1197 1.71cereals Alpen Light Summer Fruits Bars 5Pk 105G 1222 1.71cereals Alpen No Added Sugar 1.3Kg 1488 0.31cereals Alpen No Added Sugar 560G 1488 0.46cereals Alpen Original 1.5Kg 1509 0.27cereals Alpen Original Muesli 750G 1509 0.35cereals Alpen Raspberry And Yoghurt Cereal Bars5x29g 1748 1.24cereals Alpen Strawberry With Yoghurt Cereal Bar 5X29g 1756 1.24dairy Alpro Natural Yofu 500G 0.28dairy Alpro Raspberry Vanilla Yofu 4X125g 0.35dairy Alpro Strawberry And Fof Soya Yofu 4X125g 0.35dairy Alpro Vanilla Yofu 500G 0.28 Which categories of food are light? Which are inexpensive?
  • import sysimport pandas as pddata = pd.read_csv(sys.argv[1])groups = data.dtypes[data.dtypes != float].indexnumbers = data.dtypes[data.dtypes == float].index>>> groupsIndex([category, title], dtype=object)>>> numbersIndex([kJ, rate], dtype=object)
  • import sysimport pandas as pddata = pd.read_csv(sys.argv[1])groups = data.dtypes[data.dtypes != float].indexnumbers = data.dtypes[data.dtypes == float].indexfor group in groups: ave = data.groupby(group).mean() for num in numbers: print ave.sort(num, ascending=False)
  • LET’S APPLY THIS MARKS TRAINS CRICKET
  • Afghanistan’s s/r Australia’s s/r Difference is large compared to the spread High probability that s/r is different55 60 65 70 75 Average probability that s/r is different55 60 65 70 75 Low probability that s/r are different55 60 65 70 75
  • WELCOME TO STATS 201 scipy.stats.mstats.ttest_ind scipy.stats.mstats.f_oneway
  • import sysimport pandas as pdfrom scipi.stats.mstats import f_onewaydata = pd.read_csv(sys.argv[1])groups = data.dtypes[data.dtypes != float].indexnumbers = data.dtypes[data.dtypes == float].indexfor group in groups: grouped = data.groupby(group) ave = grouped.mean() for num in numbers: F, prob = f_oneway(*grouped[number].values) print prob print ave.sort(num, ascending=False)
  • LET’S APPLY THIS GROCERIES CRICKET TRAINS
  • import sysimport pandas as pdfrom scipi.stats.mstats import f_onewaydata = pd.read_csv(sys.argv[1])groups = data.dtypes[data.dtypes != float].indexnumbers = data.dtypes[data.dtypes == float].indexfor group in groups: grouped = data.groupby(group) ave = grouped.mean() for num in numbers: F, prob = f_oneway(*grouped[number].values) improvement = (ave[number].max() / data[number].mean() – 1) print improvement, prob # print ave.sort(num, ascending=False)
  • LET’S APPLY THIS GROCERIES CRICKET MARKS TRAIN
  • Hypotheses Data Insight Data Autolysis TAKE ANY DATASET THROW IT AT A PROGRAM GET INSIGHTS
  • DIRECTIONSCROSS TABULATIONS CORRELATIONS OUTLIERS HULLS
  • A data analytics and visualisation companyWe handle terabyte-size data via non-traditional analytics and visualise it in real-time. We’re recruiting S.Anand@Gramener.com