Open Source at


Published on

The Salesforce core stack embraces Open Source Software. Join one of our leading engineers and learn about how we tackle enterprise-grade challenges for our customers using Hadoop, HBase, Jetty, Solr, and Apache QPID. We'll also discuss the process of opening internal libraries like Aura and Phoenix, as well as Salesforce's place in the larger Open Source community.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Open Source at

  1. 1. How Open Source Software Embiggens What We Use, And What We Contribute To Ian Varley,, Principal Member of Technical Staff @thefutureian
  2. 2. Safe Harbor Safe harbor statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available., inc. assumes no obligation and does not intend to update these forward-looking statements.
  3. 3. Who am I? Ian Varley 3.5 years at Core Database Team, Big Data Team @thefutureian
  4. 4. Show of hands ... ● Use OSS? ● Contribute to OSS? ● Write their own OSS projects?
  5. 5. Developers at spend all day in open source software.
  6. 6. engineers work on an OSS stack ... ▪ Linux (Ubuntu, RHL) ▪ Java ▪ Eclipse (+ IntelliJ, vim, emacs) ▪ Guava, Apache Commons, more ▪ JUnit, Mockito, Selenium ▪ Git (+ p4) ▪ Memcached
  7. 7. What’s so great about open source software?
  8. 8. A rising tide ... … lifts all boats.
  9. 9. It’s a win-win situation. ▪Everyone gets more out than they put in ▪You have control over your own destiny ▪You can attract the industry’s best minds ● The smartest devs seem to gravitate towards open source ● So if you raised your hand before, give yourself a pat on the back.
  10. 10. So, what do we use?
  11. 11. Servlet Container Servlet containers handle routing HTTP requests to code. ▪Started w/ commercial product ▪Feature: “steal” work from overloaded servers ▪(Code name: Hamburglar) But! Show stopper bug, and no way to fix it ...
  12. 12. Solution: Jetty ▪ ▪Year-long migration process ▪Tricky with 10+ years of legacy code! ▪Now running Jetty (almost) everywhere.
  13. 13. Search Indexing Indexer takes text (e.g. chatter posts, etc.), makes it searchable. ▪Original implementation: Lucene (forked) ▪But, scale keeps increasing! ▪Bottleneck: single-writer QFS on a SAN ▪Needed solution to scale horizontally
  14. 14. Solution: Solr ▪Horizontally scalable, REST interface ▪Query / index on same host, no more SAN ▪New features, core library is (latest) Lucene We’ve also contributed some small fixes, and contracted a big fix to allow handling indexers with many cores (10K+!).
  15. 15. Contributing is a win / win.
  16. 16. Message Queue Decouple calling code from its execution. ▪Originally: 10-15 devs had rolled their own ▪Centralized on a transactional queue (Vijay) ▪Commercial product, deeply coupled to DB ▪OK until: the “600” error. 3 years of back and forth. ▪Eventually rewrote our layer to work around it! ▪Scale problems: 50 -> 500 queues ▪CPU contention at head of queue
  17. 17. Solution: QPID ▪Apache project, good reputation ▪Separate tier from the DB ▪Ran into bugs … ▪… and fixed them. ▪40% memory savings on client (QPID-4873; thanks Helen Kwong & Brian Toal)
  18. 18. Open source lets us bring our experts to help everybody.
  19. 19. Build: Ant Build tools help get you from “code written” to “code running”. ▪Used Apache Ant for years ▪But, as the # of devs has grown … ▪scale and maintainability problems.
  20. 20. Solution: Maven ▪Moving core build to Apache Maven ▪Goal is a more modular and decoupled build structure ▪Declarative dependencies FTW ▪OSGi: Apache Felix
  21. 21. Deployment: Home Grown Deployment tools let you get code out to servers. ▪ has always used home-grown tool, “ReleaseRunner” ▪Required for tight security model (no passwordless ssh, root) ▪But as we scale out, manual methods aren’t cutting it
  22. 22. Solution: Puppet, Salt, Razor, Rundeck Get code out to lots of servers with little manual involvement. ▪Razor: automated machine inventory ▪Puppet: deployment of bits and configuration ▪Salt, Rundeck: service orchestration for restarts All of this still very much WIP; nobody else does it w/ this level of security
  23. 23. Batch Processing ▪Salesforce == RDBMS ▪No great approach for batch processing ▪Especially on sets that don’t fit in memory ▪Working outside relational model very hard
  24. 24. Solution: Hadoop Map/Reduce: ship computations to your data instead vice versa ▪Walter Macklem (Platform CTO); Codename: Gridforce ▪+HDFS (distributed file storage) ▪+Pig (a higher level language) ▪Features: recommendations, search relevance, machine learning ▪Log export pilot ... (Ask your CSR/CSM/AM to get nominated for the pilot!)
  25. 25. Big Data Relational databases are powerful ... ▪But, started looking at cost model ▪Lower than average (multi-tenancy) ▪Model is so rich, prohibitive for really large data ▪RDBMS has strict scalability limits per object ▪Hard to scale out because, runs on big iron What if we could store vast numbers of records, but with fewer capabilities and assumptions? Scale horizontally, but with the same safety guarantees?
  26. 26. Big Data: HBase Horizontally scalable NoSQL database. ▪Fewer capabilities (no joins, transactions) ▪Scales by adding machines ▪Fault tolerant (on HDFS) ▪Features? Initially, audit & compliance, event tracking ▪Eventually, a lot more: really big objects ▪Got a lot of field history? Join the FHR retention pilot! (Talk to you CSM) This is my team, so I could talk for hours. But go see Lars Hofhansl’s talk!
  27. 27. OK, that’s cool. But, does Salesforce contribute new projects?
  28. 28. Historically: no, not many. But, this is changing.
  29. 29. Aura: UI Framework Basis for new generation of Salesforce UI ▪High performance client-server architecture ▪Event-driven, MVC architecture ▪
  30. 30. Phoenix: a SQL Skin for HBase “We put the SQL back in NoSQL” ▪A proper subset of SQL ▪Familiar interface, scalable storage ▪Unlike Hive, uses the HBase client API ▪Blazing fast; queries in milliseconds ▪Tons of contribution since we opened it ▪
  31. 31. Mobile SDK All SDK dev for Salesforce done in open source ▪ ▪ ▪Also: heavy use of Apache Cordova, to blend web & native components
  32. 32. Lots more! So far, we’ve only been talking about Salesforce core. ▪Many companies use tons of Open Source: •Heroku - •Radian6,, ExactTarget - you name it, we probably use it somewhere ▪And lots of open source stuff on the platform, too! • •You can search github for Apex & Salesforce
  33. 33. isn’t just an OSS user. We’re an OSS pusher.
  34. 34. Committers on dozens of big projects actively supports a lot of people who primarily contribute to open source projects (not just a side thing). ▪ Postgres: Tom Lane (Project Lead) ▪ Ruby: Matz (Project Lead) ▪ Maven: Jason Van Zyl (Project Lead) ▪ HBase: Lars Hofhansl (PMC, release manager); Jesse Yates ▪ Phoenix: James Taylor (Project Lead) ▪ Aura: Doug Chasman (Project Lead) ▪ Pig: Prashant Kommeredi
  35. 35. Is Open Source right for everything? No.
  36. 36. It’s great for ... ▪Core components ▪Databases ▪Common algorithms ▪Reusable UI libraries & abstractions And any case where “the source isn’t the secret sauce”.
  37. 37. It’s not great for ... ▪Code entangled with your business model ▪Code you didn’t write with a plan to open up ▪Software that’s “all things to all people” ▪Getting other people to do your work But, these are kind of anti-patterns anyway, right … ?
  38. 38. Most return on investment is from open sourcing “the interesting bits”, rather than the whole stack.
  39. 39. And embracing the Open Source approach, particularly in the last 3 years, has been a sea change.
  40. 40. In conclusion ...
  41. 41. In contributing, we all gain. Look for more OSS involvement from in the future!
  42. 42. Ian Varley @thefutureian
  43. 43. We want to hear from YOU! Please take a moment to complete our session survey Surveys can be found in the “My Agenda” portion of the Dreamforce app