Become Efficient or Die: The Story of BackType

  • 90,639 views
Uploaded on

How BackType does a lot with a little. …

How BackType does a lot with a little.

Presented at POSSCON ’11.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • after i download,how to open the file......it's xxx.key,not ppt
    Are you sure you want to
    Your message goes here
  • 创业的Rule:Idea - build - code - measure - data - learn -idea

    痛苦式开发,只有当问题真正出现时才去做,不要随意添加流程和一些不必要的事儿。

    先实现,后美化,再优化
    Are you sure you want to
    Your message goes here
  • good one...!
    Are you sure you want to
    Your message goes here
  • Awesome! Really Awesome.
    Are you sure you want to
    Your message goes here
  • I like how you describe knowledge debt and tackle technical debt
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
90,639
On Slideshare
0
From Embeds
0
Number of Embeds
22

Actions

Shares
Downloads
756
Comments
7
Likes
93

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
  • 2. BackTypeBackType helps businesses understand social media and make use of it
  • 3. BackTypeData Services (APIs)Social Media Analytics Dashboard
  • 4. APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
  • 5. URL Profiles
  • 6. Site comparisons
  • 7. Influencer Profiles
  • 8. Twitter Account Analytics
  • 9. Topic Analysis
  • 10. Topic Analysis
  • 11. BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
  • 12. BackType stats• 3 full time employees• 2 interns• 1.4M in funding
  • 13. How? Avoid wasteInvest in efficiency
  • 14. Development philosophy
  • 15. Development philosophy• Waterfall
  • 16. Development philosophy• Waterfall• Agile
  • 17. Development philosophy• Waterfall• Agile• Scrum
  • 18. Development philosophy• Waterfall• Agile• Scrum• Kanban
  • 19. Development philosophySuffering-oriented programming
  • 20. Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
  • 21. Example• Growing from 2 people to 3 people
  • 22. Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
  • 23. Example• Growing from 3 people to 5 people
  • 24. Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
  • 25. Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
  • 26. Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
  • 27. Example• Batch processing
  • 28. Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
  • 29. Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
  • 30. Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
  • 31. OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
  • 32. Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
  • 33. Knowledge debt 20 15 10 5 0 Your productivity Your potential
  • 34. Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
  • 35. Knowledge debtUse small, independent projects toexperiment with new technology
  • 36. Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
  • 37. Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
  • 38. Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
  • 39. Knowledge debt (Crappy job ad)
  • 40. Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
  • 41. Technical debtTechnical debt builds up in a codebase
  • 42. Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
  • 43. Technical debtNever high enough priority to work on, but these issues built up and slow you down
  • 44. BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
  • 45. BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
  • 46. What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
  • 47. How do you decide what to work on?
  • 48. Don’t want to waste three monthsbuilding a feature no one cares about
  • 49. Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
  • 50. Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
  • 51. Example
  • 52. ExamplePro product didn’t actually exist yet
  • 53. Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
  • 54. Example
  • 55. Hypothesis #1Customers want analytics on topics being discussed on Twitter
  • 56. Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
  • 57. Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
  • 58. Hypothesis #2Customers want to know how often topics are mentioned over time
  • 59. Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
  • 60. Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
  • 61. Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
  • 62. Lean Startup
  • 63. Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com