Keynote at Geode Summit 2016 by Dr. Justin Erenkrantz, Bloolmberg LP. Creating the Future of Big Data Through "The Apache Way" and why this matters to the community
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
#GeodeSummit Keynote: Creating the Future of Big Data Through 'The Apache Way"
1. >>>>>>>>>>>>>>>>>>>>>
CREATING THE FUTURE
OF BIG DATA THROUGH
"THE APACHE WAY”
WHY THIS MATTERS TO THE
COMMUNITY
Dr. Justin R. Erenkrantz, Bloomberg LP
justin@erenkrantz.com / @jerenkrantz
2. WHY SHOULD I PAY ATTENTION?
» Mentor to Apache Geode and HAWQ
» Commi5er to Apache HTTP Server, APR, Subversion, Serf
» Former President and Director of The Apache SoBware
FoundaDon
» Ph.D. from University of California, Irvine
» DissertaDon: "ComputaDonal REST: A New Model for
Decentralized, Internet-Scale ApplicaDons”
» Head of Compute Architecture at Bloomberg LP
» ~50 billion Dcks DAILY flow through our systems
2
3. TECH @ BLOOMBERG: OPEN SOURCE
3
» The core of our Bloomberg Professional plaorm has evolved away
from proprietary code
» FoundaDons of our next-generaDon infrastructure - OpenStack, Ceph,
Hadoop, Spark, Solr, Chromium, Chef - are all open-source
» No longer can vendors tell us that they won’t fix a criDcal bug
» Places a lot of pressure on our partners to collaborate openly
» Giving back to the community - h"ps://github.com/bloomberg/
» Allows us to innovate at the higher levels – helping our customers
make sense of the firehose of informaDon that is available to them
6. PHILOSOPHY OF THE APACHE SOFTWARE FOUNDATION
6
» Let the contributors do what they do best: contribute.
FoundaDon exists to do the rest.
» Does not pay for contribuDons
» Many are sponsored by a third-party
» Staff ASF has are focused on infrastructure/PR/etc
» Does not pick “winners” or “losers”
» “CompeDDon” between ASF projects perfectly
acceptable as long as there are healthy
communiDes…think Geode and Ignite (!)
8. ROLE OF APACHE INCUBATOR
8
» Each project (TLP) is run relaDvely autonomously
» Project karma does not automaDcally carry over
» If I can commit to Geode, it doesn’t mean I can
commit to Ignite! (But, I could likely earn it easily!)
» Incubator was formed in 2003 as we were struggling to
scale the foundaDon and repeat the model. It worked.
» If a podling does not have a healthy community, it’ll
never graduate. That’s OK. If the podling does
become a TLP, but later loses its community, it’ll end
up in the Arc. That’s OK, too.
9. TRANSPARENCY & MERITOCRACY
9
» Roy’s Mantra: "If it's not on the list, it didn't happen.”
» Apache in the age of GitHub, JIRA, ReviewBoard, etc.
» Is the mailing list doomed?
» Generation gap may mean email isn’t preferred
» Tools are always secondary to process
» Transparency is the aim: allows others to have a voice
» The tools and process are never about prohibiting face-to-
face contact - but, ensuring that there is equal access for
participation and permitting asynchronous decision making
» Making decisions in a synchronous echo chamber (Slack,
IRC, etc.) is not conducive to transparency
10. MAKING DECISIONS
10
» Voting is the way contributors are (and feel) empowered
» “Binding” votes from recognized contributors (PMC)
» Vote on code, ideas, and, most importantly, releases
» Minimum acceptable quorum: 3 voters
» Minimum acceptable time frame: 72 hours
» The power of the dreaded “-1” (veto)
» Code can be vetoed, but not releases
» Veto should be cast as a last resort; used to foster
discussion
14. NORMS OF THE COMMUNITY
14
» Over the years, most disputes I have seen come down
to norms that were not agreed upon or documented
» Forming an explicit consensus on release versioning
and compaDbility rules up-front is so incredibly helpful.
» Projects always have a tension between “new
features” and compaDbility. Decide where the
community wants to be early on.
» The Geode wiki secDon is great. Keep it up!