Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
MESSY WEB, MEET
CLEAN(ER) WEB:
AN EXPERIMENT
IN SOCIAL MEDIA ARCHIVING
Archive-It Annual Partners Meeting
8/2/2016
Atlanta...
THE WEB IS MESSY
TWITTER
FACEBOOK
WIKIPEDIA
INSTAGRAM
WHAT A PERSON THINKS A TWEET IS
WHAT A CRAWLER THINKS A TWEET IS
There is no such thing as a “tweet.”
All I see is HTML & hyperlinks.
WHAT AN APP THINKS A TWEET IS
TIMESTAMP
TWEET
TEXT
TWITTER HANDLE
(CREATOR)
TWITTER HANDLE
(MENTION)
URL, EXPANDED
SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
JSON
Render JSON
ELK (w/ Kibana)
DEMO
SOCIAL FEED MANAGER (SFM)
ELK
PYWB
SFM VIEWER
(screenshots if web fails us)
SFM
JSON
Process and
visualize data
API Heritrix
ELK (w/ Kibana)
(screenshots if web fails us)
ELK IS GREAT!
(FOR USERS WHO DO DATA ANALYSIS)
WHAT ABOUT OUR OTHER USERS?
- linked web content
- media (images, videos)
-...
SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
Render JSON
ELK (w/ Kibana)
DEMO
SOCIAL FEED MANAGER (SFM)
ELK
PYWB
SFM VIEWER
(screenshots if web fails us)
SFM
PyWB
SFM Viewer
WARC
JSON
Process and
visualize data
Render websites
JSON
API Heritrix
Render JSON
ELK (w/ Kibana)
WHAT’S THE TAKEAWAY?
It’s possible to collect messy-web & social-media-web
. . . without creating access silos
. . . using...
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving
Upcoming SlideShare
Loading in …5
×

Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving

128 views

Published on

Archive-It Annual Partners Meeting, 8/2/2016
Atlanta, GA
(Society of American Archivists)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Messy Web, Meet Clean(er) Web: An Experiment in Social Media Archiving

  1. 1. MESSY WEB, MEET CLEAN(ER) WEB: AN EXPERIMENT IN SOCIAL MEDIA ARCHIVING Archive-It Annual Partners Meeting 8/2/2016 Atlanta, GA Rachel Trent Digital Services Manager George Washington University Libraries, Special Collections
  2. 2. THE WEB IS MESSY
  3. 3. TWITTER FACEBOOK WIKIPEDIA INSTAGRAM
  4. 4. WHAT A PERSON THINKS A TWEET IS
  5. 5. WHAT A CRAWLER THINKS A TWEET IS There is no such thing as a “tweet.” All I see is HTML & hyperlinks.
  6. 6. WHAT AN APP THINKS A TWEET IS
  7. 7. TIMESTAMP TWEET TEXT TWITTER HANDLE (CREATOR) TWITTER HANDLE (MENTION) URL, EXPANDED
  8. 8. SFM PyWB SFM Viewer WARC JSON Process and visualize data Render websites JSON API Heritrix JSON Render JSON ELK (w/ Kibana)
  9. 9. DEMO SOCIAL FEED MANAGER (SFM) ELK PYWB SFM VIEWER (screenshots if web fails us)
  10. 10. SFM JSON Process and visualize data API Heritrix ELK (w/ Kibana) (screenshots if web fails us)
  11. 11. ELK IS GREAT! (FOR USERS WHO DO DATA ANALYSIS) WHAT ABOUT OUR OTHER USERS? - linked web content - media (images, videos) - browsing
  12. 12. SFM PyWB SFM Viewer WARC JSON Process and visualize data Render websites JSON API Heritrix Render JSON ELK (w/ Kibana)
  13. 13. DEMO SOCIAL FEED MANAGER (SFM) ELK PYWB SFM VIEWER (screenshots if web fails us)
  14. 14. SFM PyWB SFM Viewer WARC JSON Process and visualize data Render websites JSON API Heritrix Render JSON ELK (w/ Kibana)
  15. 15. WHAT’S THE TAKEAWAY? It’s possible to collect messy-web & social-media-web . . . without creating access silos . . . using existing technologies . . . to make more users happy

×