Overview of key open source projects at Pivotal.LHS is our apps stack, RHS is our data stack. [Redis & Rabbit really play in both]We’ve been involved with building successful open source projects and communities for many years.Programming languages such as Groovy and AspectJDevelopment frameworks such as Grails and Spring (defacto standard, ~3M devs etc.)Modern messaging & middleware: rabbitmq, redis, apache tomcat, apache http serverCloud Platforms: Cloud FoundryUnderpinning some of the biggest enterprises in the world, both in a traditional enterprise context, and in the new consumer-grade internet companies (Netflix, Twitter, Pinterest, ….) [See Pivotal P.O.V. for some great case studies]On the data side we’re fully committed to Hadoop and increasing our investment and contributions to the Hadoop projects [get the latest phrasing & stats here from Roman]We’re using open source to collaborate with leading academic researchers on MADlib, a big data machine learning library in SQL (Pivotal, Berkeley, Stanford, Florida)And we’re collaborating in open source to create collaborative environments for data explanation and analysis with Open ChorusSome of these projects are hosted at the Apache Software Foundation (tomcat, httpd, hadoop) . Even more of them use the Apache License [CLICK]And we’ve been doing it for a long time – over ten years, despite Pivotal as a company having just celebrated our first birthday.During that time we’ve learned a lot about the contributing factors to successful open source: code, community, collaboration, and commerce!
Many people starting out in open source think that it’s just about the code. That it is sufficient for example to simply license a body of code under the Apache License and collaborators will flock to help you, and a huge user community appear over the horizon.Some are motivated by freedoms – free as in speech, free in spirit.Some want to lower barriers to adoption – free as in beerOf course having an open source & appropriately licensed code base is fundamentally important and underpins everything else we do with open source. For many organisations involved in open source, the motivations go beyond just setting the code free. They may be hoping to drive adoption – to build a big user community. They may be hoping to share development effort and expertise across a number of collaborators. And increasingly all of this is driven by underlying commercial considerations.
Open sourcing your code base can certainly lower barriers to adoption, but building a strong and vibrant community around an open source project takes a lot more than just good code.In fact, some good advice is to think of building a user community a bit like building a customer base – except that you don’t get paid! Building a community around an open source project is an investment: you will need good documentation, tutorials, a website. You will need to interact on forums and in user groups, to give talks at conferences such as this one. And then you want to empower your community so that they can start to do some of these things for you too. The close interaction with a user community that well-managed open source projects foster also creates a wonderful feedback loop, helping to quickly create projects that have very strong community (market) fit. This rapid feedback loop can be very difficult for closed source alternatives to compete with. In fact, in many sectors of our industry, being an open source project is no longer a competitive advantage when it comes to community adoption – it’s become a cost of doing business. So we see competition between open source projects for community mindshare. And the benchmark can be very high – a typical VC backed open source project may be using A/B testing and a whole range of analytics to optimize their site and community experience.
Open source has become an essential foundation for collaboration on a shared code base. This is an area in which the Apache Software Foundation has excelled, providing a neutral place for individuals and organizations to work together towards shared goals. Collaboration can be motivated by a desire to pool expertise – with each party bringing something unique to the picture and creating a whole that no-one organisation or individual could do on their own. Collaboration can also be motivated by a desire to share costs – particularly when many stakeholders gain advantage from something existing, but aren’t necessarily looking for individual competitive advantage through unique implementations. The open source project can become a shared industry asset – developed in collaboration by the industry, for the benefit of multiple industry players. But what prevents this situation from degrading into a Tragedy of the Commons? Why not stand by and let everyone else invest in building out the open source component, and then just come along and use it after the fact? For the answer to this question, we need to look into our fourth C, commerce….
When I look at all of the open source projects that Pivotal is involved in, the truth is that the vast majority of development effort is undertaken by developers who are paid full-time to work on these projects. And this holds true for many successful open source projects today. Full-time developers are paid a salary, and someone has to pay that salary. So commercial considerations underpin open source. What commercial motivations might there be? We can divide them into three basic camps: Cost reduction: Collaborating in order to share development costs or gain access to expertise you do not have (and would otherwise have to pay for) Commercial enablement: building a business directly on top of, or incorporating, open source components. This is what prevents a Tragedy of the Commons meltdown in many situations – committers have become a valuable commercial commodity and the ability to support open source with deep expertise (“from the source”), and to be able to influence roadmap and features is commercially differentiating in a world where the software itself is available to anyone for free Indirect – for example, a company may get involved with open source to demonstrate its technical credentials and aid in recruitmentThe commercial underpinning of open source is vital to the world of open source as we know it today. This is also something for which we very much have the ASF to thank. The Apache License enables commercial freedom and innovation to flourish alongside open source. It has been instrumental in supporting business models that are necessary to keep the whole machine working smoothly. In fact, this mechanism is so powerful that it has fundamentally transformed the industry.
In the beginning people used to say that open source couldn’t innovate. That it was only good for commoditizing existing capabilities. That it was a race to the bottom which would destroy industry value. Bullshit.We never believed that. Still don’t. Open source is the best way to innovate because of the short feedback cycles it can create that we talked about previously. In the last few weeks, we’ve seen a subset of the vendors in the Hadoop space, which is itself just a part of the over 140 projects at the Apache Software Foundation, achieve a combined market valuation of $3Bn (Cloudera = $2Bn, HW = $1Bn). That’s a whole lot of industry value being created through open source. What happened?First open source became a means of overcoming proprietary lock-in,. Then it replaced standards at the leading edge of industry adoption, fueled by a rate of innovation that standards could never keep up with. Then it became a strategic asset and an integral part of corporate strategy.
The fear of vendor lock-in is buried deep in the enterprise psyche. And for good reason – lock-in can be very expensive! So if you’re a vendor, especially in the enterprise application development, integration, middleware and data spaces that Pivotal is involved in, it’s very hard to create commercial adoption and success around a proprietary interface. One of the ways that the industry solves this problem is through standards. For example, the SQL standard, or the Java Enterprise Edition platform specification. Multiple vendors can then compete on implementations of a standard, and perhaps differentiate through qualities of services, installation, configuration, management, and so forth. Innovation at the interface layer is discouraged under such a model. Open source has emerged as an industry accepted alternative to standards as a means of combating vendor lock-in. This allows open source projects – and the vendors behind them – to innovate freely, making deep improvements and new advances that are indeed – of necessity - reflected in the interfaces that an enterprise will ultimately bind to. Much of this transformation has been driven at and by the Apache Software Foundation.Open source has become the new Open standard.
The rate and pace of change in our industry only continues to rise. We’re undergoing a transition to a whole new set of platforms and a whole new set of concerns. Cloud platforms, big data, real-time analytics, mobile delivery, systems of engagement and more are all bringing in waves of change and innovation. The old model of committee-based collaboration to create a standard specification that can then be implemented by multiple vendors is just too slow for this world. Not only is it too slow, but it also does not produce solutions that are as good as those created by refining open source projects in the fire of real user feedback. So the industry has mostly replaced a standards-first approach with an open-source first approach. In many cases, with an open-source *only* approach, but there are also examples of open standards being created behind the bow-wave of open source, ratifying what have become defacto standards. [This recently happened with the Spring Batch project for example, where the lag between the first open source version being used in production, and the JEE standard being complete was about 5 years! That’s an eternity in today’s world of IT].
Open source is also at the heart of many corporate strategies. Today there are a lot of VC-backed open source companies that treat open source as an integral, central part of their strategy. This is pushing the notion of professional open source to new levels – not just full-time developers, but the full gamut of business development tools and strategies being deployed around the creation of successful open source projects in the pursuit of commercial success. In hotly contested areas, the table stakes in the open source game have become very high indeed. So if open source is the leading edge of innovation, and a core part of many strategies and business models, what are some of the choices facing us when it comes to competing and collaborating around open source?
In which we use the four C’s model to analyse some common options in the world of strategic OSS.[ALT – each of these could also be broken out into a picture slide following the format of the previous sections]I want to talk about three common choices for running an open source project. The most basic option these days is simply to host your code and project wiki / website at GitHub. This is can be a great way of getting your code out there, and of starting to build a user community. It is often chosen when the open source project is very largely developed by committers working for a single company. This option is less attractive if you want to enable development collaboration with other stakeholders – you have to manage all of the IP issues associated with that for yourself, and often need to provide a way to reassure other contributors that they can be equal partners. It does enable product-oriented business models (commerce), but makes it harder to support some other business models where the open source is intended to be the shared foundation for a platform and ecosystem.There are many advantages to hosting a project at a shared foundation – The Apache Software Foundation in particular – especially if fostering genuine collaboration is an important goal. Some projects, for example Twitter’s Storm, start off with the GitHub only model and then later on apply to move to a foundation. Storm is currently being incubated at Apache. Apache enables collaboration in two fundamental ways :- it can hold intellectual property in a neutral way such that the foundation and its users need not fear any lock-in, and it establishes a meritocracy giving everyone an equal opportunity to participate. The Apache Software Foundation checks all four of our boxes – it enables open distribution of code (with strong intellectual property management;, it has proven time and time again its ability to build communities; it provides a solid footing for collaboration; and it has licensing that facilities commercial backing and exploitation. With venture-backed companies looking to use open source for strategic advantage, the importance of a level playing field for collaboration has never been greater. So why did we not choose to go this route with Cloud Foundry? We elected for the third way. The creation of a dedicated foundation. This is an expensive and resource-intensive option and certainly not to be taken lightly! A dedicated foundation provides collaboration opportunities that go beyond just the code and related artifacts. It can facilitate collaboration around marketing and ecosystem development. It’s an option that only makes sense when you’re looking to establish broad ecosystem adoption around a collection of projects that collectively define a platform. Eclipse was an early example of this approach, and OpenStack a more recent one. We recently announced that we are collaborating with a number of partners to create a Cloud Foundry Foundation around the extensive set of projects that now make up Cloud Foundry. All under the Apache License version 2 of course.
Transcript of "Apachecon 2014 Keynote: The Apache Way in the Cloud with Cloud Foundry "