Become Efficient or Die: The Story of BackType
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Become Efficient or Die: The Story of BackType

on

  • 91,346 views

How BackType does a lot with a little.

How BackType does a lot with a little.

Presented at POSSCON ’11.

Statistics

Views

Total Views
91,346
Views on SlideShare
33,067
Embed Views
58,279

Actions

Likes
91
Downloads
754
Comments
7

61 Embeds 58,279

http://nathanmarz.com 49723
http://tech.backtype.com 5521
http://softwarestrategiesblog.com 795
http://feeds.feedburner.com 748
http://1.bigdata.sinaapp.com 501
http://blog.stevecoast.com 147
http://www.michevan.id.au 130
http://blog.derekperez.com 106
http://nathanmarz.com. 84
http://ryanhoover.me 84
http://www.scoop.it 75
http://theoldreader.com 36
https://twitter.com 36
http://nmarz.squarespace.com 31
http://nmarz.squarespace.com 31
http://paper.li 29
http://webcache.googleusercontent.com 28
http://bigdata.sinaapp.com 20
http://twitter.com 18
http://localhost 11
http://lanyrd.com 10
http://xianguo.com 9
http://translate.googleusercontent.com 8
http://www.newsblur.com 7
http://www.slideshare.net 6
http://www.onlydoo.com 6
http://www.techgig.com 6
http://omgatch.blogspot.com 5
https://softwarestrategiesblog.com 5
https://si0.twimg.com 5
http://static.slidesharecdn.com 5
http://dashboard.bloglines.com 4
http://localhost:1891 4
http://a0.twimg.com 4
http://trunk.ly 4
http://rrhoover.tumblr.com 3
http://www.mongodb.org 3
https://p.yammer.com 3
http://pinterest.com 2
http://boutofcontext.com 2
http://ranksit.com 2
http://www.surfanon.net 2
http://www.twylah.com 2
http://www.pinterest.com 1
http://web.archive.org 1
http://www.goread.io 1
http://reader.googleusercontent.com 1
url_unknown 1
http://www.ofertadeviaje.org 1
http://www.ofertadeviajes.eu 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 7 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • after i download,how to open the file......it's xxx.key,not ppt
    Are you sure you want to
    Your message goes here
    Processing…
  • 创业的Rule:Idea - build - code - measure - data - learn -idea

    痛苦式开发,只有当问题真正出现时才去做,不要随意添加流程和一些不必要的事儿。

    先实现,后美化,再优化
    Are you sure you want to
    Your message goes here
    Processing…
  • good one...!
    Are you sure you want to
    Your message goes here
    Processing…
  • Awesome! Really Awesome.
    Are you sure you want to
    Your message goes here
    Processing…
  • I like how you describe knowledge debt and tackle technical debt
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Become Efficient or Die: The Story of BackType Presentation Transcript

  • 1. Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
  • 2. BackTypeBackType helps businesses understand social media and make use of it
  • 3. BackTypeData Services (APIs)Social Media Analytics Dashboard
  • 4. APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
  • 5. URL Profiles
  • 6. Site comparisons
  • 7. Influencer Profiles
  • 8. Twitter Account Analytics
  • 9. Topic Analysis
  • 10. Topic Analysis
  • 11. BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
  • 12. BackType stats• 3 full time employees• 2 interns• 1.4M in funding
  • 13. How? Avoid wasteInvest in efficiency
  • 14. Development philosophy
  • 15. Development philosophy• Waterfall
  • 16. Development philosophy• Waterfall• Agile
  • 17. Development philosophy• Waterfall• Agile• Scrum
  • 18. Development philosophy• Waterfall• Agile• Scrum• Kanban
  • 19. Development philosophySuffering-oriented programming
  • 20. Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
  • 21. Example• Growing from 2 people to 3 people
  • 22. Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
  • 23. Example• Growing from 3 people to 5 people
  • 24. Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
  • 25. Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
  • 26. Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
  • 27. Example• Batch processing
  • 28. Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
  • 29. Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
  • 30. Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
  • 31. OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
  • 32. Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
  • 33. Knowledge debt 20 15 10 5 0 Your productivity Your potential
  • 34. Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
  • 35. Knowledge debtUse small, independent projects toexperiment with new technology
  • 36. Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
  • 37. Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
  • 38. Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
  • 39. Knowledge debt (Crappy job ad)
  • 40. Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
  • 41. Technical debtTechnical debt builds up in a codebase
  • 42. Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
  • 43. Technical debtNever high enough priority to work on, but these issues built up and slow you down
  • 44. BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
  • 45. BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
  • 46. What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
  • 47. How do you decide what to work on?
  • 48. Don’t want to waste three monthsbuilding a feature no one cares about
  • 49. Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
  • 50. Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
  • 51. Example
  • 52. ExamplePro product didn’t actually exist yet
  • 53. Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
  • 54. Example
  • 55. Hypothesis #1Customers want analytics on topics being discussed on Twitter
  • 56. Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
  • 57. Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
  • 58. Hypothesis #2Customers want to know how often topics are mentioned over time
  • 59. Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
  • 60. Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
  • 61. Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
  • 62. Lean Startup
  • 63. Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com