Become Efficient or Die: The Story of BackType

Become Efﬁcient or Die The Story of BackType Nathan Marz @nathanmarz

BackTypeBackType helps businesses understand social media and make use of it

BackTypeData Services (APIs)Social Media Analytics Dashboard

APIs• Conversational graph for url• Comment search• #Tweets / URL• Inﬂuence scores• Top sites• Trending links stream• etc.

URL Proﬁles

Site comparisons

Inﬂuencer Proﬁles

Twitter Account Analytics

Topic Analysis

BackType stats• >30 TB of data• 100 to 200 machine cluster• Process 100M messages per day• Serve 300 req/sec

BackType stats• 3 full time employees• 2 interns• 1.4M in funding

How? Avoid wasteInvest in efﬁciency

Development philosophy

Development philosophy• Waterfall

Development philosophy• Waterfall• Agile

Development philosophy• Waterfall• Agile• Scrum

Development philosophy• Waterfall• Agile• Scrum• Kanban

Development philosophySuffering-oriented programming

Suffering-oriented ProgrammingDon’t add process until you feel the pain of not having it

Example• Growing from 2 people to 3 people

Example• Founders were essentially “one brain”• Cowboy coding led to communication mishaps• Added biweekly meeting to sync up

Example• Growing from 3 people to 5 people

Example• Moving through tasks a lot faster with 5 people• Needed more frequent prioritization of tasks• Changed biweekly meeting to weekly meeting• Added “chat room standups” to facilitate mid-week adjustments

Suffering-oriented ProgrammingDon’t build new technology untilyou feel the pain of not having it

Suffering-oriented Programming First, make it possible. Then, make it beautiful. Then, make it fast.

Example• Batch processing

Make it possible• Hack things out using MapReduce/ Cascading• Learn the ins and outs of batch processing

Make it beautiful• Wrote (and open-sourced) Cascalog• The “perfect interface” to our data

Make it fast• Use it in production• Proﬁle and identify bottlenecks• Optimize

OverengineeringAttempting to create beautiful software without a thorough understanding of problem domain

Premature optimizationOptimizing before creating “beautiful” design, creating unnecessary complexity

Knowledge debt 20 15 10 5 0 Your productivity Your potential

Knowledge debt 20 15 Knowledge 10 debt 5 0 Your productivity Your potential

Knowledge debtUse small, independent projects toexperiment with new technology

Example• Needed to write a small server to collect records into a Distributed Filesystem• Wrote it using Clojure programming language• Huge win: now we use Clojure for most of our systems

Example• Needed to implement social search• Wrote it using Neo4j• Ran into lot of problems with Neo4j and rewrote it later using Sphinx

Example• Needed an automated deploy for a distributed stream processing system• Wrote it using Pallet• Massive win: anticipate dramatic reduction in complexity in administering infrastructure

Knowledge debt (Crappy job ad)

Knowledge debtInstead of hiring people who share your skill set, hire people with completely different skill sets (food for thought)

Technical debtTechnical debt builds up in a codebase

Technical debt• W needs to be refactored• X deploy should be faster• Y needs more unit tests• Z needs more documentation

Technical debtNever high enough priority to work on, but these issues built up and slow you down

BackSweep• Issues are recorded on a wiki page• We spend one day a month removing items from that wiki page

BackSweep• Keeps our codebase lean• Gives us a way to defer technical debt issues when don’t have time to deal with them• “Garbage collection for the codebase”

What is a startup? A startup is a human institution designed to deliver a newproduct or service under conditions of extreme uncertainty. - Eric Ries

How do you decide what to work on?

Don’t want to waste three monthsbuilding a feature no one cares about

Don’t want to waste three monthsbuilding a feature no one cares about This could be fatal!

Product development Keep Valid?Form hypothesis Test hypothesis Learn Invalid? Discard

Example

ExamplePro product didn’t actually exist yet

Example• We tested different feature combinations and measured click through rate• Clicking on “sign up” went to a survey page

Example

Hypothesis #1Customers want analytics on topics being discussed on Twitter

Testing hypothesis #1• Fake feature -> clicking on topic goes to survey page

Testing hypothesis #1• Do people click on those links?• If not, need to reconsider hypothesis

Hypothesis #2Customers want to know how often topics are mentioned over time

Testing hypothesis #2• Build topic mentions over time graph for “big topics” our private beta customers are interested in (e.g. “nike”, “microsoft”, “apple”, “kodak”)• Talk to customers

Hypothesis #3• Customers want to see who’s talking about a topic on a variety of dimensions: recency, inﬂuence, num followers, or num retweets

Testing hypothesis #3• Create search index on last 24 hours of data that can sort on all dimensions

Lean Startup

Questions? Twitter: @nathanmarzEmail: nathan.marz@gmail.com Web: http://nathanmarz.com

http://nathanmarz.com	16394
http://tech.backtype.com	5521
http://1.bigdata.sinaapp.com	501
http://feeds.feedburner.com	281
http://softwarestrategiesblog.com	250
http://blog.stevecoast.com	147
http://www.michevan.id.au	93
http://blog.derekperez.com	87
http://www.scoop.it	36
http://paper.li	29
http://nmarz.squarespace.com	28
http://nmarz.squarespace.com	28
http://webcache.googleusercontent.com	27
http://bigdata.sinaapp.com	20
http://twitter.com	18
http://ryanhoover.me	13
http://lanyrd.com	7
http://www.slideshare.net	6
http://localhost	6
http://www.onlydoo.com	6
https://twitter.com	5
http://omgatch.blogspot.com	5
http://static.slidesharecdn.com	5
http://trunk.ly	4
http://a0.twimg.com	4
http://localhost:1891	4
http://www.mongodb.org	3
http://dashboard.bloglines.com	3
http://translate.googleusercontent.com	3
http://www.techgig.com	3
http://rrhoover.tumblr.com	3
https://p.yammer.com	3
http://www.newsblur.com	2
http://www.twylah.com	2
http://tweetedtimes.com	1
http://www.ofertadeviaje.org	1
http://www.appspot.co.za	1
http://www.ofertadeviajes.eu	1
http://cache.baidu.com	1
url_unknown	1
http://us-w1.rockmelt.com	1
http://nuevospowerpoints.blogspot.com	1
http://wix.lamalamp.com	1
http://wix.local.host:3000	1
http://bummerware.tumblr.com	1
https://portal.oversee.net	1

Become Efficient or Die: The Story of BackType

by nathanmarz on Mar 28, 2011

Accessibility

Tags

Upload Details

Usage Rights

46 Embeds 23,559

Statistics

Become Efficient or Die: The Story of BackType — Presentation Transcript