Director of Engineering Derek Rose's presentation on IDX Broker Research and Development.
Learn more about the IDX Broker Developer Partner program: http://bit.ly/DeveloperProgram
2. Started December 2005.
First paid employee who was not also a founder.
Certified Amazon Web Services Developer.
My mom says I’m cool.
Who Am I?
3. We handle the MLS the client handles the rest*.
The Basic Ideas
Be as flexible as possible
Leads are our currency
2015 V2 New Lead Stats
Nearly 350,000 Network-wide
Over 200,000 with Dev-Partner
4. We build and launched V1 in 5 months with 3 coders
Secret to our success: not knowing any better
Beta was rough but overall: huge success
R&D Phase 1: Head down get it done
5. Reactionary development.
Idea to implementation often in less than one week
Many of great ideas – like our Partner program
Long term code-base viability more of an afterthought
Phase 2: What should we do next
6. Refactoring and code creep was getting out of hand
We had to get a lot more deliberate with our code planning
Started living 6 months ahead
How do we stay ahead of demand? How do today’s decisions
impact us 2 quarters from now? How do we milk what we
have for everything it can give us?
Phase 3: Planning ahead
7. First used about 6 months after it launched for images
Previously used a combination of 1&1 and Dreamhost
Before there were SDKs
Quickly grew to photo storage to over 1 TB
Enter AWS
8. • 24 Terabytes of S3/Glacier storage
• > 40 EC2 servers
• 50 EBS Volumes
• 40 RDS servers
• 20 Hosted Zones
• 16 CDN Endpoints
• …
• Major R&D challenge
• Just keeping up on the updates requires diligence
IDX and AWS in 2015
9. AWS, dedicated colo, and rebuild efforts.
Keeping up with our code issues and our popularity was
proving to be too much
Servers were constantly swamped
This is when I became a DevOp… before it was cool
Phase 4: The dark days
10. This lead to a period of me saying no… a lot
Every new idea went from something fun and interesting
to just another load on our resources
Research was all about keeping us afloat and me from
going crazy
Phase 5: No
11. In a lot of was this is the stage we’re still in.
Constant chicken / egg decisions.
How do we offer the latest and greatest and not
damage features people already count on?
Phase 6: Later
12. Well what if we didn’t have the legacy problem?
Lets build in our biggest wish list items from the start
Lets build a platform many products can be built on
Lets build a platform for our dev partners
V2
13. Started about three years ago with a realization.
Still having efficiency problems but not with our code
Horizontal scaling would carry us a long time but…
New problems = Same problems
$ $
$ $
$ $ $ $
$ $ $ $
$ $ $ $
$ $ $
14. Technology currently in our dev pipeline didn’t exist
three years ago…
Demand can grow quickly to 500+ reqs/sec
Have to start measuring infrastructure responses in
nanoseconds instead of milliseconds.
“Webscale” tech is fast evolving
Base Usage (V2)
110 hits/second
200 million hits/month
275 million w/ CDN
15. • NoSQL
• Containerization, micro-services, and batch workloads
• Columnar databases
• Package based code-bases
• Hybrid networks
• Real time Analytics
• More buzzwords!
In our sights
17. Free Day
A day for exploring and learning
Three basic rules:
1. Free day isn’t a day off
2. Your research should be for the betterment of IDX
3. Critical issues come first
Editor's Notes
When we were first planning V1 there was, of course, lots of talk of specific must haves and killer features but from my prospective it boiled down to these three ideas and the sea of technical challenges they embody.
It was an epic task to launch V1 in only 5 months. I attribute our success only to knowing that we shouldn't have tried such a thing.
For the first year or so we basically planned and implemented new features as they were suggested. We paid attention to ActiveRain, blogs, emails, and other sources to see what it sort of things were missing not just from our product but from the IDX industry as a whole. Jeff and Chad would suggest something, we’d argue about how to implement it, then we’d just do it. There was very little attention paid to what this was going to do to running cost or system stability.
Our CEO, Chad, has said that one of his saddest days was the day I came to his office and told him we had to better at planning our features and dealing with available resources.
Hacking together a photo system from affordable services taught us a lot about adaptive uses of DNS and multihost services.
Just some of our stats
Over capacity and what to do. Toyed with hiring an operations specialist, even had some consultants come in.
Still or maybe “back” in”
If we were only dealing with growth scale would be linear and doable but the latest must have features make the resource demand curve much steeper. SQL is great for structured search – which is technically speaking our primary feature – but the degree of freedom we offer has made it a performance sink.