Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

We open sourced our trade secrets, and so can you!


Published on

AppNeta TraceView is distributed tracing software as a service. Our Ruby instrumentation is a gem that hooks into popular frameworks, clients, and drivers to capture performance data from a running application – and if you think storing data in the cloud is a hard sell, just wait until you mention that you're modifying executing code to capture it! Bad instrumentation could have all kinds of adverse impacts on your application, from crashes to slowdowns to subtle corruptions, so our guiding principle is "first, do no harm". Plenty of customers have found our gem to be fast and stable under real-world conditions, but until recently, they had to take us at our word that we weren't shipping backdoors, ticking time bombs, or dark magic.

About a year ago, we decided that it was time to open source our gem. Starting off proprietary meant that we'd never planned for the world to read our code, so we didn't yet understand what risks we were exposing ourselves and our customers to. It was also internal-focused, so we weren't even sure whether anyone who wasn't an AppNeta engineer would get anything out of it. But we wanted to make it happen, and for it to happen to more code in the future, so we developed an internal process for open-sourcing code sustainably. Before launch day, we got buy-in from involved departments, vetted the license with our lawyers, figured out how our support team would support it, audited code and removed information that needed to stay private, wrote tests that anyone could run, turned internal documentation into "TFM", figured out our internal workflow, and developed a framework for community contributions. It took a few months to get everything in order, but the day finally came when we switched over our RubyGems build to the new repo and opened it to the world. (I wrote the announcement blog post – yay!)

Our gem has been on GitHub at appneta/oboe_ruby for over six months now, and we're very pleased with the results we've seen. We've had no problem continuing to sell our product, our Ruby customers have been much more engaged, and we've even been able to release new instrumentation based on their contributed code. We're glad we open sourced our trade secrets – what's stopping you?

Published in: Technology
  • Be the first to comment

We open sourced our trade secrets, and so can you!

  1. 1. We open sourced our trade secrets, and so can you! James Meickle Developer evangelist, AppNeta @jmeickle Boston Rails Meetup August 26, 2014
  2. 2. AppNeta • Application performance management (APM) – Client – Network – Server • Software as a service • Cloud hosted
  3. 3. A brief history of APM • Closed source monitoring for closed source applications • Oriented around mainframes and J2EE • Incredibly high cost, not production-ready • Today: rapid value, easy deployment, low cost
  4. 4. AppNeta TraceView • Distributed tracing (Zipkin, Dapper) • Bytecode instrumentation • Java, .NET, PHP, Ruby, Python, node.js • Definitely not Tracelytics!
  5. 5. THE GEM Implementing TraceView’s functionality in Ruby
  6. 6. Our maintainer • Peter Lombardo came on a few years ago • Exclusively focused on the oboe gem • Works remote from Italy, or else he’d be giving this talk!
  7. 7. oboe • Installed like a normal gem (with C extension) • No configuration required • Provides: – Application performance management – Software as a service – Distributed tracing – Bytecode instrumentation – Trace API
  8. 8. Application performance management • Collect performance data: – Database queries – Cache requests – Remote calls – Templating time • Without causing problems: – Identical behavior – Overhead below 1% – Stable
  9. 9. Software as a service • Don’t do any processing locally • Report data to the agent (UDP) • Report data to collector (SSL, on Heroku)
  10. 10. Distributed tracing • Follow calls from one application to another – Different servers – Multiple languages – Asynchronous • Common request/response protocols: – HTTP – Thrift – EJB (for JRuby)
  11. 11. Bytecode instrumentation • Support common components: – Frameworks – Libraries – Drivers/clients • Without code modification – Ruby, Python: monkey patching – Java: classloader – PHP: 
  12. 12. Trace API • Same API that we use when adding layers • Our backend is built around supporting it • Duck typing for events • Report anything!* *please don’t report credit card numbers, thanks
  13. 13. Example 1: memcache
  14. 14. Example 2: rack
  15. 15. Example 2: Net::HTTP
  16. 16. Example 4: resque
  17. 17. THE PROCESS Opening the oboe gem to the world
  18. 18. Why did we open source oboe? • We <3 open source • Larger customers want to audit our code • Increased community contributions • Better capture of reported issues • Good publicity 
  19. 19. Why couldn’t we “flip the switch”? • The code was not appropriately documented • The test suite needed work • There was no contribution guide • It wasn’t licensed • Not branded properly! • Issues contained customer-specific information
  20. 20. Documentation • Added
  21. 21. Tests • “Before opensourcing, we had to move away from manual testing via nosetests (30 something stack variations, manual boot and manually run nosetests, rinse and repeat for the next hour). We went with Minitest. Today with Minitest, we now test 7 Ruby versions: each with 202 tests, 1979 assertions - biggest win beyond coverage is that it’s automated with Travis for each git commit.” – Peter Lombardo
  22. 22. Contribution guide
  23. 23. Branding • We were already AppNeta TraceView • But we still were Tracelytics in our code • Not as simple as a s/foo/bar/; lots of unexpected places (support URLs!)
  24. 24. Licensing • Open source != free software • Competitive advantage • Still an unfathomable change from where we were 10 years ago 
  25. 25. Issues with issues • We didn’t want to lose historical issues • New issues would also need to reference specific customers and post about their data
  26. 26. The solution • “The key was to use two repos (1 private and 1 public) using two git remotes. Now our default is that all issues are filed in the public repository unless it contains customer specific data. If a public issue, turns out to need a pointer to some internal data or resource, we file a corresponding internal issue that points to the public issue. We now always prefer the public repo over the private and only use the private when absolutely necessary.” – Peter Lombardo
  27. 27. The final countdown
  28. 28. Marketing helped too! • Pre-launch – Implications of licensing – How to reward and publicize contributors – Info page for repo • Post-launch – Launch blog post – Social media – Existing customers!
  29. 29. Launch day • Set up the public repo as a private one • Switch our build process (rubygems) to use the right repo • Post the new build as a GitHub Release too • Publish the blog post!
  30. 30. THE RESULT What we’ve seen after open sourcing
  31. 31. appneta / oboe-ruby • Not many visitors  • But they were incredibly engaged: more than 25% of them cloned the repo. • And about a third of them contributed code!
  32. 32. More contributions • 80%: post an issue • 15%: post a fix/snippet • 5%: an entire feature! – Grape API: Todd Lunter at Swipely – EventMachine: Diogo Benica at Abril Midia
  33. 33. Happier maintainer • “Not sure if this part is going to make sense but working on an opensource project is like having a weight lifted off of your shoulders. When it’s private, proprietary code, the burden is completely on you to assure quality and to not screw up. When the code is opensource, there is a slightly calming feel because you know that your code is public and is available to be reviewed and improved upon by anyone. We love the contributions we’ve gotten so far and are always looking for more.” – Peter Lombardo
  34. 34. GitHub issues • Work being done lives in GitHub issues on the public repo • Anyone can post there!
  35. 35. Supporting contributors • Most people use the trace API incorrectly • We only notice when they file support tickets • Post custom code on GH, and we can help while you write it! 
  36. 36. Thank you! Come work for us! We’re a mostly Python shop  You can meet us at: • Sep 15-17: Velocity New York • Sep 18: WebPerfDays New York • Oct 21: TechBreakfast New York • October 24: Surge (DC) • Nov 11-14: AWS re:Invent (Vegas!) Or just try us out: