Your SlideShare is downloading. ×
0
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
email transaction network analysis
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

email transaction network analysis

921

Published on

Thesis defense presentation - SDSU Computational Sciences, 2013. Use a number of network analysis tools, including community detection, pagerank, eigenvector centrality, etc., to determine key metrics …

Thesis defense presentation - SDSU Computational Sciences, 2013. Use a number of network analysis tools, including community detection, pagerank, eigenvector centrality, etc., to determine key metrics of graphs determined by key word searches. These custom graphs and the associated metrics are then presented in interactive graphics and tables.

Published in: Education, Business, Technology
2 Comments
0 Likes
Statistics
Notes
  • The animation of page 31 posted here: http://youtu.be/dGa-TXpoPz8
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Note that page 31 is a quicktime movie. it appears that slideshare only shows it as an image.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total Views
921
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
2
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ELECTRONIC MAIL TRANSACTION NETWORK ANALYSIS Keith P. Jolley
  • 2. INTRODUCTION • Deliverable of this project: – Provide a means to identify who to turn to for more information on a topic, or several topics – Provide better insight into project organization • Use Network Analysis tools – Augment, not replace, existing enterprise search tools – Social Network Analysis • Community detection algorithm • PageRank, …
  • 3. WHAT WAS DONE • Used several datasets as input: – Archived “public” Qualcomm mail-lists: • 136 mailboxes, 10.0 GB • 20393 vertices, 940403 edges – Enron email: • 158 mailboxes, 1.3 GB • 90026 vertices, 3715056 edges – Test datasets: • Karate Club: 34 vertices, 78 edges • Les Misérables: 77 vertices, 820 edges • Created an interactive web client: – User search term input – Interactive graphics and tables of metrics – Perl CGI, R, C++, Javascript, D3 (Data Driven Documents)
  • 4. MAILBOX INPUT Enron dataset: user based • Wide range of topics per inbox • All emails in each mailbox are all “from:”, “to:”, or “cc:” the same person • Examples: • Jeff Skilling • Kenneth Lay Qualcomm dataset: topic based • All emails have, for the most part, a common theme • Emails in each mailbox are from multiple senders • All emails include the mail-list as a recipient • Emails may include other recipients, including other mail-lists • Examples: • Photography • Android • Hiking
  • 5. User Interface - URL Enron: Test: Qualcomm:
  • 6. User Interface – mail-list Enron: Test: Qualcomm:
  • 7. User Interface – search Enron: Search term parsing: user input: ‘San Diego Power’ regex: ‘(san|diego|power)’ All Fields: searches all dataset columns Topic: search only the “subject” column People: only search “to:” and “from:” columns Maillist: matches only on the “mail-list” column*
  • 8. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Transform Email
  • 9. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Transform Email Into “net.lists”
  • 10. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic"
  • 11. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Lots of email:
  • 12. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Lots of email: 517424 Enron emails
  • 13. PREPROCESSING Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Message-ID: <25313859.1075851942346.JavaMail.evans@thyme> Date: Wed, 7 Mar 2001 01:56:00 -0800 (PST) From: jmckechnie@cabotlng.com To: dan.masters@enron.com Subject: Re: Feb 12 delivery of "Methane Arctic" Cc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: cskinner@cabotlng.com, jaime.sanabria@enron.com, barbo@enron.com, todd.peterson@enron.com, radamiak@cabotlng.com, jmichalek@cabotlng.com X-From: "Joseph McKechnie" <jmckechnie@cabotlng.com> X-To: Dan.Masters@enron.com X-cc: "Christopher Skinner" <cskinner@cabotlng.com>, Jaime.Sanabria@enron.com, Paul.Y'Barbo@enron.com, Todd.Peterson@enron.com, "Rudy Adamiak" <radamiak@cabotlng.com>, "Jane Michalek" <jmichalek@cabotlng.com> X-bcc: X-Folder: Paul_Ybarbo_Nov2001Notes FoldersCabot X-Origin: YBARBO-P X-FileName: pybarbo.nsf ... Source Target Mailbox Subject jmckechnie dan.masters ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie cskinner@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jaime.sanabria ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie barbo ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie todd.peterson ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie radamiak@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" jmckechnie jmichalek@cabotlng.com ybarbo-p re: feb 12 delivery of "methane arctic" Lots of email: 517424 Enron emails 3715056 edges
  • 14. SOFTWARE DESIGN & OPERATION Custom Directed Graph Creation Step 1 – parse dataset • Create edge list • Create adjacency matrix
  • 15. SOFTWARE DESIGN & OPERATION Custom Directed Graph Creation Step 2 - simplify • Remove self-loops • Remove vertices without edges
  • 16. PREPROCESSING Edge weight Two methods compared: “bytes sent” and “message count”. Bytes sent is the sum of the number of characters in each matching message, minus any included emails in each message. Message count is the total number of matching messages. Analysis and intuition says either method will provide similar results. Message count chosen because it is simpler and faster. Enron dataset 19k vertices, 3.5M vertices
  • 17. COMMUNITY DETECTION • Communities are clusters of vertices that have more interconnections within the cluster than outside of the cluster • Any non-trivial social network will have communities • The metric associated with communities is “modularity”, which ranges from -1 to 1, and is defined as:
  • 18. COMMUNITY DETECTION • Ai,j is the edge weight between vertices i and j • ki is the sum of the weights attached to vertex i • m is the ½ the sum of all the weights in the graph (for compatibility with earlier definitions of modularity) • δ(ci,cj) is equal to 1 if vertices i and j are in the same community, and 0 if not. • A completely random network has modularity ~ 0
  • 19. COMMUNITY DETECTION • Modularity detection algorithms seek to maximize Q • Exact solutions are computational expensive, particularly for large networks • Various network detection algorithms exist • For this project the “Louvain” method was used • Authors: • Vincent Blondel • Jean-Loup Guillaume • Renaud Lambiotte • Etienne Lefebvre
  • 20. COMMUNITY DETECTIONZachary − Karate Club Step 1 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Starting graph
  • 21. COMMUNITY DETECTIONZachary − Karate Club Step 2 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Modularity: -0.04980276Assign each vertex to a unique community
  • 22. COMMUNITY DETECTIONZachary − Karate Club Step 2 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Modularity: -0.04980276 Pick a vertex and place in the community of each of its neighbors. Measure change in modularity at each step. Place the vertex in the community with greatest positive change.
  • 23. COMMUNITY DETECTION Modularity: -0.04980276 Continue to pick vertices at random and swapping until a minimum increase is found after a complete cycle. Zachary − Karate Club Step 3 through N swap communities 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
  • 24. COMMUNITY DETECTION Modularity: 0.2483563 Continue to pick vertices at random and swapping until a minimum increase is found after a complete cycle. Zachary − Karate Club Step 4 through N 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
  • 25. COMMUNITY DETECTION Modularity: 0.2483563 Continue to pick vertices at random and swapping until a minimum increase is found after a complete cycle. Zachary − Karate Club Step 4 through N 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
  • 26. COMMUNITY DETECTION Modularity: 0.2483563 Combine all nodes and edges in each new community into “super- communities”. Repeat previous steps. Zachary − Karate Club Step 5 through N 1 2 3 4 5 6 7
  • 27. COMMUNITY DETECTION Collapse all nodes and edges in each new community into “super- communities”. Repeat previous steps. 1 2 3 4 5 6 7
  • 28. COMMUNITY DETECTION Repeat previous steps until no further increase in modularity can be gained, or an upper limit on iterations is reached 1 2 3 4
  • 29. COMMUNITY DETECTIONZachary − Karate Club Final 1 2 3 4 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Modularity: 0.3532216
  • 30. COMMUNITY DETECTION
  • 31. COMMUNITY DETECTION
  • 32. METRICS • Eigenvector Centrality – Favors nodes that are highly connected to other highly connected nodes • PageRank – Favors nodes that are connected from other highly connected nodes – Strongly biased towards mail-lists
  • 33. METRICS • Eigenvector Centrality – Favors nodes that are highly connected to other highly connected nodes
  • 34. METRICS • PageRank 0.0 0.1 0.2 0.3 A B C D E vertex value ranking pagerank eigenvector
  • 35. METRICS • Degree – the number of other vertices connected to a vertex • Strength (in, out, total) – sum of the weights of inbound, outbound, edges • Betweenness Centrality: – The ratio of the shortest paths traversing through a vertex divided by the total number of shortest paths in the network
  • 36. METRICS • Closeness Centrality – The sum of the inverse of the shortest paths that traverse a vertex
  • 37. SOFTWARE DESIGN • Testing done by using known datasets and comparing values to other published values • First step of the CGI is to run PageRank and keep only the top 750 nodes • Most searches likely only want the top few ranked vertices • Keep processing on local machine manageable • Prevents “hairballs”
  • 38. SOFTWARE DESIGN • Perl CGI running under either Apache or python CGIHTTPServer • R does all the heavy lifting for the analysis • Force-Directed Graph from D3, a javascript library, is used for interactive graphics • DataTables creates interactive html tables for sorting and filtering • The size of the vertices is an average of the PageRank and Eigenvector values • Color is assigned by community
  • 39. SOFTWARE USAGE
  • 40. SOFTWARE USAGE http://localhost/~kjolley/cgi-bin/tw-qcom.pl
  • 41. SOFTWARE USAGE http://localhost/~kjolley/cgi-bin/tw-qcom.pl
  • 42. SOFTWARE USAGE • Results from a chip design project – Dark blue: configuration management – Light blue: hardware design – Dark green: senior leadership – Light green: test and design for test – Salmon: not exactly sure • One of the senior leads recognized this as a good visualization of the organization of the team and said this would be of value to Qualcomm
  • 43. FUTURE STEPS • More datasources – ClearCase, email, communities, Perforce, HR databases • Better search • Deeper search • Make Gephi more scriptable • Commercial products

×