Become Efficient or Die: The Story of BackType
Upcoming SlideShare
Loading in...5
×
 

Become Efficient or Die: The Story of BackType

on

  • 91,098 views

How BackType does a lot with a little.

How BackType does a lot with a little.

Presented at POSSCON ’11.

Statistics

Views

Total Views
91,098
Views on SlideShare
32,876
Embed Views
58,222

Actions

Likes
90
Downloads
754
Comments
7

61 Embeds 58,222

http://nathanmarz.com 49707
http://tech.backtype.com 5521
http://softwarestrategiesblog.com 776
http://feeds.feedburner.com 748
http://1.bigdata.sinaapp.com 501
http://blog.stevecoast.com 147
http://www.michevan.id.au 130
http://blog.derekperez.com 106
http://nathanmarz.com. 84
http://www.scoop.it 75
http://ryanhoover.me 65
http://theoldreader.com 36
https://twitter.com 35
http://nmarz.squarespace.com 31
http://nmarz.squarespace.com 31
http://paper.li 29
http://webcache.googleusercontent.com 28
http://bigdata.sinaapp.com 20
http://twitter.com 18
http://localhost 11
http://lanyrd.com 10
http://xianguo.com 9
http://translate.googleusercontent.com 8
http://www.newsblur.com 7
http://www.slideshare.net 6
http://www.techgig.com 6
http://www.onlydoo.com 6
http://omgatch.blogspot.com 5
https://si0.twimg.com 5
http://static.slidesharecdn.com 5
http://a0.twimg.com 4
http://dashboard.bloglines.com 4
http://localhost:1891 4
http://trunk.ly 4
http://rrhoover.tumblr.com 3
https://softwarestrategiesblog.com 3
http://www.mongodb.org 3
https://p.yammer.com 3
http://pinterest.com 2
http://boutofcontext.com 2
http://ranksit.com 2
http://www.surfanon.net 2
http://www.twylah.com 2
http://www.pinterest.com 1
http://web.archive.org 1
http://www.goread.io 1
http://reader.googleusercontent.com 1
url_unknown 1
http://www.ofertadeviaje.org 1
http://www.ofertadeviajes.eu 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • after i download,how to open the file......it's xxx.key,not ppt
    Are you sure you want to
    Your message goes here
    Processing…
  • 创业的Rule:Idea - build - code - measure - data - learn -idea

    痛苦式开发,只有当问题真正出现时才去做,不要随意添加流程和一些不必要的事儿。

    先实现,后美化,再优化
    Are you sure you want to
    Your message goes here
    Processing…
  • good one...!
    Are you sure you want to
    Your message goes here
    Processing…
  • Awesome! Really Awesome.
    Are you sure you want to
    Your message goes here
    Processing…
  • I like how you describe knowledge debt and tackle technical debt
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Become Efficient or Die: The Story of BackType Become Efficient or Die: The Story of BackType Presentation Transcript

  • Become Efficient or Die The Story of BackType Nathan Marz @nathanmarz
  • BackTypeBackType helps businesses understand social media and make use of it
  • BackTypeData Services (APIs)Social Media Analytics Dashboard
  • APIs• Conversational graph for url• Comment search• #Tweets / URL• Influence scores• Top sites• Trending links stream• etc.
  • URL Profiles
  • Site comparisons
  • Influencer Profiles
  • Twitter Account Analytics
  • Topic Analysis
  • Topic Analysis
  • BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec
  • BackType stats• 3 full time employees• 2 interns• 1.4M in funding
  • How? Avoid wasteInvest in efficiency
  • Development philosophy
  • Development philosophy• Waterfall
  • Development philosophy• Waterfall• Agile
  • Development philosophy• Waterfall• Agile• Scrum
  • Development philosophy• Waterfall• Agile• Scrum• Kanban
  • Development philosophySuffering-oriented programming
  • Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it
  • Example• Growing from 2 people to 3 people
  • Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up
  • Example• Growing from 3 people to 5 people
  • Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments
  • Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it
  • Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.
  • Example• Batch processing
  • Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing
  • Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data
  • Make it fast• Use it in production• Profile and identify bottlenecks• Optimize
  • OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain
  • Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity
  • Knowledge debt 20 15 10 5 0 Your productivity Your potential
  • Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential
  • Knowledge debtUse small, independent projects toexperiment with new technology
  • Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems
  • Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx
  • Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure
  • Knowledge debt (Crappy job ad)
  • Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)
  • Technical debtTechnical debt builds up in a codebase
  • Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation
  • Technical debtNever high enough priority to work on, but these issues built up and slow you down
  • BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page
  • BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”
  • What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries
  • How do you decide what to work on?
  • Don’t want to waste three monthsbuilding a feature no one cares about
  • Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!
  • Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard
  • Example
  • ExamplePro product didn’t actually exist yet
  • Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page
  • Example
  • Hypothesis #1Customers want analytics on topics being discussed on Twitter
  • Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page
  • Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis
  • Hypothesis #2Customers want to know how often topics are mentioned over time
  • Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers
  • Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, influence, num followers, or num retweets
  • Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions
  • Lean Startup
  • Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com