Technology of the New News Workflow


Published on

My presentation at WordCamp Miami - Feb 20, 2010.

Covering True/Slant's enhancements to WordPress MU - Editorial and cross-network / multi-site.

Get more details on this topic at

Published in: Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This site is built on WordPress MU, with about 300 contributors, each with their own blog (“site”). If you’re familiar with WordPress MU, you’ll understand that out of the box, no plugins, each blog/site stands apart. You can’t search across blogs, you can’t query posts across blogs. It’s great for each blog – nice fast pageloads. But to integrate the network installation into one presence, you need to build on this great WordPress platform. I’m going to walk you through some of the things True/Slant can do, including editorial workflows – not everything you see is automated. And then I’ll get into techniques and methods to combine multiple blogs into something bigger than the sum of its parts.
  • This is one blog. This particular one by Matt Taibbi who also writes for Rolling Stone. The functionality here is pretty simple, just a few sidebar pieces to show content from the rest of the site, and what we call an Activity Feed showing what the contributor has done in the past few weeks. We started this before BuddyPress was really available, and haven’t had a chance to check it out – in case you were wondering if there’s any BPress in True/Slant. Not yet…
  • Here’s another contributor. This is the main page – very similar, and a common theme.
  • And here’s another. They are all the same theme, just different content and writers. So why not stop here? Because it’s more than just one writer, just like any newspaper is more than just one column.
  • Here are some of the bits we put on the home page. With 300 contributors, there are a lot of viewpoints. We have basically one full-time editor picking content to feature. And to keep him productive, we’ve created a few tools. But some things are automated.
  • Many users arrive on our site at a specific post. Our chance to engage them starts on the single post page. These sidebar modules will attract users to other posts on the site.
  • Here’s our footer. The most popular posts, recent contributors, and the last few posts and comments wind up here. This is all automatic.
  • Rather than just posts from this contributor, we feature related posts from all blogs. We also have what we call topic streams. They aren’t exactly categories or tags; a post can be assigned to multiple topics; this exposes them to the site-wide topic streams.
  • We call these Live Streams, but they’re more like topics. Posts can be assigned to multiple topics, based on - which category they are in in their home blog, or - an editor’s manual selection per post (either forcing a post into a topic, or possibly excluding it) Inside the streams, we can have posts and various status updates – such as this bit that lists recently recommended posts in the stream. In the sidebar, we can feature a post. All the editor needs to do is hover over the story, and click a link.
  • This is how an editor manually overrides the automatic topic assignment. Here, the contributor’s Category placed this post into the World topic, but the editor manually added the post to the Haiti Earthquake topic as well.
  • A user can follow contributors and topics, once you follow them, you can receive email alerts, and this handy drop-down shows you the latest updates.
  • We modularized our code, so that we can hook in additional features later. When we added Facebook Share, and Email alerts, we did not have to modify the Follow code or change the Follow workflow for the user. If a user is not connected to Facebook, the Facebook part doesn’t appear.
  • Everything a user does on the site can be displayed to other users.
  • We implemented this with a special master table, listing all events. We indexed it and made some keys unique, so that there’s only one record, with the latest timestamp.
  • We created several editorial tools, which add to the automation. The idea is to get the best content exposed, We have data to help us – eg post views, trends, comments. But don’t make it hard for the editor, And certainly don’t make anyone jump through hoops We use a lot of AJAX, which is only exposed to privileged users
  • This is one of the first, and simplest customizations we did.
  • The only thing you need to do here is increment comment_karma in the comment table by 1, and use that to set a class on the comment. Also, alerts hooks into the comment being called out action, so we can send a note to the original commenter.
  • We also enabled an interface on the backend to curate comments from the comment list; green here indicates a called-out comment.
  • Our home page needed a bit fancier set of tools, to create and arrange posts in panels.
  • This network header editor allows editors to prioritize panels, hide them, change or re-crop the thumbnails and headlines, etc.
  • To make search results fast across all blogs, we Added a new column to our shadow posts and comments table with just the searchable text of the post Made a fulltext index of that Intercepted the standard search query and queried our shadow table instead We also derive topics and contributors to provide refinement tools and links in the sidebar
  • We also give contributors some tools – this is my dashboard Blog stats, top posts, referers all via internal stats (which we use in many other ways)
  • Needs work
  • One big issue we faced early was combining the multiple blogs into one. Does it make sense to use RSS feeds if the database is right there? What other information do we need? And – we’d still need to pull 300 feeds! Forget that!
  • This is a normal, single “site” or blog.
  • When you have two blog sites, you have to use a JOIN to query both tables at the same time (eg to find the latest post)
  • When you have 300 blogs/sites, a normalized database is hard to deal with for reads. It would kill performance.
  • Solution: de-normalize the data; we hooked into the post update and comment update actions and copied the data into this shadow table. Adding a blog_id column.
  • No point in using this for each blog; WordPress handles things fine.
  • Technology of the New News Workflow

    1. 1. Technology of the New News Workflow* Roger Theriault True/Slant *featuring WordPress MU
    2. 2. True/Slant
    3. 3. Contributors - Taibbi
    4. 4. Contributors – Colin Horgan
    5. 5. Contributors – Miles Obrien
    6. 6. Content surfaces on main site
    7. 7. Inbound Traffic <ul><li>>80% of inbound traffic is to individual posts </li></ul><ul><li>We also need to expose these users to True/Slant via the single page template </li></ul>
    8. 8. Widgets
    9. 9. Related Posts and Streams
    10. 10. Topic Streams How to combine? <ul><li>Every author has a different way to organize categories, and a different opinion of what belongs </li></ul><ul><li>We devised a ½ editorial, ½ automated Topic </li></ul><ul><li>Fed by designating categories from specific contributors </li></ul><ul><li>Fine tuned by manipulating on a post by post basis </li></ul>
    11. 11. Streams
    12. 12. Streams <ul><li>Editorial overrides </li></ul><ul><li>Default based on post categories </li></ul><ul><li>AJAX of course </li></ul>
    13. 13. Following
    14. 14. Following <ul><li>Simple list of contributor ids (and topics) stored for each user </li></ul><ul><li>AJAX controls </li></ul><ul><li>Other modules use the follow data (eg: email alerts, Facebook) </li></ul>
    15. 15. User Activity Feed <ul><li>Constant stream of user activity on the site </li></ul><ul><li>Posts </li></ul><ul><li>Comments by user </li></ul><ul><li>Comments on user’s post </li></ul><ul><li>Headline Grabs </li></ul><ul><li>New follows </li></ul><ul><li>Etc. </li></ul>
    16. 16. User Activity Feed <ul><li>Master activity table </li></ul><ul><li>Multiple unique indexes </li></ul><ul><ul><li>Event type </li></ul></ul><ul><ul><li>Acting user </li></ul></ul><ul><ul><li>Receiving user </li></ul></ul><ul><ul><li>Timestamp </li></ul></ul><ul><ul><li>Blog id </li></ul></ul><ul><ul><li>Post id or Link id </li></ul></ul><ul><ul><li>Comment id </li></ul></ul><ul><li>Reciprocal events for receiver </li></ul>
    17. 17. Editorial <ul><li>Surface compelling content </li></ul><ul><li>Leverage existing information </li></ul><ul><li>Simple interface </li></ul><ul><li>Don’t get in anyone’s way </li></ul>
    18. 18. Comment Curation <ul><li>Surface the best comments </li></ul><ul><li>But permit the entire thread to be seen </li></ul><ul><li>Show in streams and activity feeds </li></ul><ul><li>Keeps conversations civilized </li></ul>
    19. 19. Comment Curation <ul><li>Simple AJAX controls to “Call Out” a comment </li></ul><ul><li>Alerts hook into it to send user notifications </li></ul><ul><li>Rest is templates </li></ul>
    20. 20. Comment Curation
    21. 21. News Now
    22. 22. Network Header
    23. 23. Search
    24. 24. Contributor Tools
    25. 25. Technical Challenges <ul><li>How to integrate so many blogs? </li></ul><ul><li>Consistent experience </li></ul><ul><li>Fast performance </li></ul>
    26. 26. Caveat – what we are doing is OK <ul><li>No need to write everything as a separate plugin </li></ul><ul><li>Need flexibility </li></ul><ul><li>For easy scalability and maintenance, build hooks and filters in </li></ul><ul><li>Be ready to throw a bunch of code away </li></ul><ul><li>Don’t hack core! Copy and change functions if needed. </li></ul>
    27. 27. Editorial - AJAX <ul><li>Editors can’t always remember which tab in Admin to go to, so we put everything right in front of them </li></ul><ul><li>Lots of hover – click elements for editors </li></ul><ul><li>Some even for contributors </li></ul><ul><ul><li>Curate Comments </li></ul></ul><ul><li>On backend, simple option storage usually is sufficient for editorial data </li></ul><ul><li>Consistent UI elements eliminate complexity </li></ul><ul><li>Conditional loading of editor elements and scripts </li></ul>
    28. 28. RSS – no
    29. 29. WordPress tables Posts Table
    30. 30. WordPress MU tables Posts Table Posts Table
    31. 31. WordPress MU tables Posts Table Posts Table Posts Table Posts Table X 300 ?
    32. 32. Shadow tables Posts Table Posts Table Posts Table Posts Table X 300 Master Posts Table
    33. 33. Shadow Tables <ul><li>Topic Streams </li></ul><ul><li>Search </li></ul><ul><li>Related Posts </li></ul><ul><li>Activity Feed data </li></ul><ul><li>Stats data </li></ul><ul><li>not individual blog posts </li></ul>
    34. 34. Thank You <ul><li>Roger Theriault [email_address] </li></ul><ul><li> </li></ul><ul><li>True/Slant </li></ul><ul><li>“ News is more than what happens” </li></ul>