Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Become Efficient or Die: The Story of BackType


Published on

How BackType does a lot with a little.

Presented at POSSCON ’11.

Become Efficient or Die: The Story of BackType

  1. Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
  2. BackTypeBackType helps businesses understand social media and make use of it
  3. BackTypeData Services (APIs)Social Media Analytics Dashboard
  4. APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
  5. URL Profiles
  6. Site comparisons
  7. Influencer Profiles
  8. Twitter Account Analytics
  9. Topic Analysis
  10. Topic Analysis
  11. BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
  12. BackType stats• 3 full time employees• 2 interns• 1.4M in funding
  13. How? Avoid wasteInvest in efficiency
  14. Development philosophy
  15. Development philosophy• Waterfall
  16. Development philosophy• Waterfall• Agile
  17. Development philosophy• Waterfall• Agile• Scrum
  18. Development philosophy• Waterfall• Agile• Scrum• Kanban
  19. Development philosophySuffering-oriented programming
  20. Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
  21. Example• Growing from 2 people to 3 people
  22. Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
  23. Example• Growing from 3 people to 5 people
  24. Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
  25. Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
  26. Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
  27. Example• Batch processing
  28. Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
  29. Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
  30. Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
  31. OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
  32. Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
  33. Knowledge debt 20 15 10 5 0 Your productivity Your potential
  34. Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
  35. Knowledge debtUse small, independent projects toexperiment with new technology
  36. Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
  37. Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
  38. Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
  39. Knowledge debt (Crappy job ad)
  40. Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
  41. Technical debtTechnical debt builds up in a codebase
  42. Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
  43. Technical debtNever high enough priority to work on, but these issues built up and slow you down
  44. BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
  45. BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
  46. What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
  47. How do you decide what to work on?
  48. Don’t want to waste three monthsbuilding a feature no one cares about
  49. Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
  50. Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
  51. Example
  52. ExamplePro product didn’t actually exist yet
  53. Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
  54. Example
  55. Hypothesis #1Customers want analytics on topics being discussed on Twitter
  56. Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
  57. Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
  58. Hypothesis #2Customers want to know how often topics are mentioned over time
  59. Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
  60. Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
  61. Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
  62. Lean Startup
  63. Questions? Twitter: @nathanmarzEmail: Web: