Your SlideShare is downloading. ×
0
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Become Efficient or Die: The Story of BackType
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Become Efficient or Die: The Story of BackType

91,053

Published on

How BackType does a lot with a little. …

How BackType does a lot with a little.

Presented at POSSCON ’11.

7 Comments
94 Likes
Statistics
Notes
No Downloads
Views
Total Views
91,053
On Slideshare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
765
Comments
7
Likes
94
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
    • 2. BackTypeBackType helps businesses understand social media and make use of it
    • 3. BackTypeData Services (APIs)Social Media Analytics Dashboard
    • 4. APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
    • 5. URL Profiles
    • 6. Site comparisons
    • 7. Influencer Profiles
    • 8. Twitter Account Analytics
    • 9. Topic Analysis
    • 10. Topic Analysis
    • 11. BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
    • 12. BackType stats• 3 full time employees• 2 interns• 1.4M in funding
    • 13. How? Avoid wasteInvest in efficiency
    • 14. Development philosophy
    • 15. Development philosophy• Waterfall
    • 16. Development philosophy• Waterfall• Agile
    • 17. Development philosophy• Waterfall• Agile• Scrum
    • 18. Development philosophy• Waterfall• Agile• Scrum• Kanban
    • 19. Development philosophySuffering-oriented programming
    • 20. Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
    • 21. Example• Growing from 2 people to 3 people
    • 22. Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
    • 23. Example• Growing from 3 people to 5 people
    • 24. Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
    • 25. Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
    • 26. Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
    • 27. Example• Batch processing
    • 28. Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
    • 29. Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
    • 30. Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
    • 31. OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
    • 32. Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
    • 33. Knowledge debt 20 15 10 5 0 Your productivity Your potential
    • 34. Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
    • 35. Knowledge debtUse small, independent projects toexperiment with new technology
    • 36. Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
    • 37. Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
    • 38. Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
    • 39. Knowledge debt (Crappy job ad)
    • 40. Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
    • 41. Technical debtTechnical debt builds up in a codebase
    • 42. Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
    • 43. Technical debtNever high enough priority to work on, but these issues built up and slow you down
    • 44. BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
    • 45. BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
    • 46. What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
    • 47. How do you decide what to work on?
    • 48. Don’t want to waste three monthsbuilding a feature no one cares about
    • 49. Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
    • 50. Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
    • 51. Example
    • 52. ExamplePro product didn’t actually exist yet
    • 53. Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
    • 54. Example
    • 55. Hypothesis #1Customers want analytics on topics being discussed on Twitter
    • 56. Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
    • 57. Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
    • 58. Hypothesis #2Customers want to know how often topics are mentioned over time
    • 59. Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
    • 60. Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
    • 61. Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
    • 62. Lean Startup
    • 63. Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com

    ×