Become Efficient or Die    The Story of BackType                      Nathan Marz                      @nathanmarz
BackTypeBackType helps businesses understand   social media and make use of it
BackTypeData Services (APIs)Social Media Analytics      Dashboard
APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
URL Profiles
Site comparisons
Influencer Profiles
Twitter Account   Analytics
Topic Analysis
Topic Analysis
BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
BackType stats• 3 full time employees• 2 interns• 1.4M in funding
How?   Avoid wasteInvest in efficiency
Development philosophy
Development         philosophy• Waterfall
Development         philosophy• Waterfall• Agile
Development         philosophy• Waterfall• Agile• Scrum
Development         philosophy• Waterfall• Agile• Scrum• Kanban
Development       philosophySuffering-oriented programming
Suffering-oriented  ProgrammingDon’t add process until you feel   the pain of not having it
Example• Growing from 2 people to 3 people
Example• Founders were essentially “one brain”• Cowboy coding led to communication  mishaps• Added biweekly meeting to syn...
Example• Growing from 3 people to 5 people
Example• Moving through tasks a lot faster with 5  people• Needed more frequent prioritization of  tasks• Changed biweekly...
Suffering-oriented  ProgrammingDon’t build new technology untilyou feel the pain of not having it
Suffering-oriented  Programming   First, make it possible.  Then, make it beautiful.      Then, make it fast.
Example• Batch processing
Make it possible• Hack things out using MapReduce/  Cascading• Learn the ins and outs of batch processing
Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
OverengineeringAttempting to create beautiful software without a   thorough understanding of problem domain
Premature optimizationOptimizing before creating “beautiful” design,     creating unnecessary complexity
Knowledge debt 20 15 10  5  0      Your productivity Your potential
Knowledge debt 20 15                                         Knowledge 10                                           debt  ...
Knowledge debtUse small, independent projects toexperiment with new technology
Example• Needed to write a small server to collect  records into a Distributed Filesystem• Wrote it using Clojure programm...
Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and  rewrote it late...
Example• Needed an automated deploy for a  distributed stream processing system• Wrote it using Pallet• Massive win: antic...
Knowledge debt   (Crappy job ad)
Knowledge debtInstead of hiring people who share your skill set, hire      people with completely different skill sets    ...
Technical debtTechnical debt builds up in a codebase
Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
Technical debtNever high enough priority to work on, but these       issues built up and slow you down
BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items  from that wiki page
BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt  issues when don’t have time to deal with  them...
What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extrem...
How do you decide what to work on?
Don’t want to waste three monthsbuilding a feature no one cares about
Don’t want to waste three monthsbuilding a feature no one cares about      This could be fatal!
Product development                                     Keep                           Valid?Form hypothesis   Test hypoth...
Example
ExamplePro product didn’t actually exist yet
Example• We tested different feature combinations  and measured click through rate• Clicking on “sign up” went to a survey...
Example
Hypothesis #1Customers want analytics on topics    being discussed on Twitter
Testing hypothesis #1• Fake feature -> clicking on topic goes to  survey page
Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
Hypothesis #2Customers want to know how often topics       are mentioned over time
Testing hypothesis #2• Build topic mentions over time graph for  “big topics” our private beta customers are  interested i...
Hypothesis #3• Customers want to see who’s talking about  a topic on a variety of dimensions: recency,  influence, num foll...
Testing hypothesis #3• Create search index on last 24 hours of  data that can sort on all dimensions
Lean Startup
Questions?   Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Upcoming SlideShare
Loading in...5
×

Become Efficient or Die: The Story of BackType

91,152

Published on

How BackType does a lot with a little.

Presented at POSSCON ’11.

7 Comments
94 Likes
Statistics
Notes
No Downloads
Views
Total Views
91,152
On Slideshare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
766
Comments
7
Likes
94
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Become Efficient or Die: The Story of BackType

    1. 1. Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
    2. 2. BackTypeBackType helps businesses understand social media and make use of it
    3. 3. BackTypeData Services (APIs)Social Media Analytics Dashboard
    4. 4. APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
    5. 5. URL Profiles
    6. 6. Site comparisons
    7. 7. Influencer Profiles
    8. 8. Twitter Account Analytics
    9. 9. Topic Analysis
    10. 10. Topic Analysis
    11. 11. BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
    12. 12. BackType stats• 3 full time employees• 2 interns• 1.4M in funding
    13. 13. How? Avoid wasteInvest in efficiency
    14. 14. Development philosophy
    15. 15. Development philosophy• Waterfall
    16. 16. Development philosophy• Waterfall• Agile
    17. 17. Development philosophy• Waterfall• Agile• Scrum
    18. 18. Development philosophy• Waterfall• Agile• Scrum• Kanban
    19. 19. Development philosophySuffering-oriented programming
    20. 20. Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
    21. 21. Example• Growing from 2 people to 3 people
    22. 22. Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
    23. 23. Example• Growing from 3 people to 5 people
    24. 24. Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
    25. 25. Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
    26. 26. Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
    27. 27. Example• Batch processing
    28. 28. Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
    29. 29. Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
    30. 30. Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
    31. 31. OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
    32. 32. Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
    33. 33. Knowledge debt 20 15 10 5 0 Your productivity Your potential
    34. 34. Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
    35. 35. Knowledge debtUse small, independent projects toexperiment with new technology
    36. 36. Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
    37. 37. Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
    38. 38. Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
    39. 39. Knowledge debt (Crappy job ad)
    40. 40. Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
    41. 41. Technical debtTechnical debt builds up in a codebase
    42. 42. Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
    43. 43. Technical debtNever high enough priority to work on, but these issues built up and slow you down
    44. 44. BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
    45. 45. BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
    46. 46. What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
    47. 47. How do you decide what to work on?
    48. 48. Don’t want to waste three monthsbuilding a feature no one cares about
    49. 49. Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
    50. 50. Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
    51. 51. Example
    52. 52. ExamplePro product didn’t actually exist yet
    53. 53. Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
    54. 54. Example
    55. 55. Hypothesis #1Customers want analytics on topics being discussed on Twitter
    56. 56. Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
    57. 57. Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
    58. 58. Hypothesis #2Customers want to know how often topics are mentioned over time
    59. 59. Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
    60. 60. Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
    61. 61. Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
    62. 62. Lean Startup
    63. 63. Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×