Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Contributing to Apache Spark 3


Published on

A quick tour of how to get started contributing to Apache Spark 3

Published in: Internet
  • Be the first to comment

Contributing to Apache Spark 3

  1. 1. @holdenkarau Effectively Contributing to Apache Spark Spark BCN 2019 I am on the PMC but this represents my own personal views
  2. 2. @holdenkarau Where you can find the slides for this talk
  3. 3. @holdenkarau Who am I? Holden ● Prefered pronouns: she/her ● Co-author of the Learning Spark & High Performance Spark books ● OSS Big Data Developer advocate @ Google ● Spark PMC & Committer ● Twitter @holdenkarau ● Live stream code & reviews: ●
  4. 4. @holdenkarau
  5. 5. @holdenkarau What we are going to explore together! Getting a change into Apache Spark & the components involved: ● The current state of the Apache Spark dev community ● Reason to contribute to Apache Spark ● Different ways to contribute ● Places to find things to contribute ● Tooling around code & doc contributions Torsten Reuschling
  6. 6. @holdenkarau Who I think you wonderful humans are? ● Nice* people ● Don’t mind pictures of cats ● May know some Apache Spark? ● Want to contribute to Apache Spark
  7. 7. @holdenkarau Why I’m assuming you might want to contribute: ● Fix your own bugs/problems with Apache Spark ● Learn more about distributed systems (for fun or profit) ● Improve your Scala/Python/R/Java experience ● You <3 functional programming and want to trick more people into using it ● “Credibility” of some vague type ● You just like hacking on random stuff and Spark seems shiny
  8. 8. @holdenkarau What’s the state of the Spark dev community? ● Really large number of contributors ● Active PMC & Committer’s somewhat concentrated ○ Better than we used to be ● Also a lot of SF Bay Area - but certainly not exclusively so gigijin
  9. 9. @holdenkarau How can we contribute to Spark? ● Direct code in the Apache Spark code base ● Code in packages built on top of Spark ● Code reviews ● Yak shaving (aka fixing things that Spark uses) ● Documentation improvements & examples ● Books, Talks, and Blogs ● Answering questions (mailing lists, stack overflow, etc.) ● Testing & Release Validation Andrey
  10. 10. @holdenkarau Which is right for you? ● Direct code in the Apache Spark code base ○ High visibility, some things can only really be done here ○ Can take a lot longer to get changes in ● Code in packages built on top of Spark ○ Really great for things like formats or standalone features ● Yak shaving (aka fixing things that Spark uses) ○ Super important to do sometimes - can take even longer to get in romana klee
  11. 11. @holdenkarau Which is right for you? (continued) ● Code reviews ○ High visibility to PMC, can be faster to get started, easier to time box ○ Less tracked in metrics ● Documentation improvements & examples ○ Lots of places to contribute - mixed visibility - large impact ● Advocacy: Books, Talks, and Blogs ○ Can be high visibility romana klee
  12. 12. @holdenkarau Contributing Code Directly to Spark ● Maybe we encountered a bug we want to fix ● Maybe we’ve got a feature we want to add ● Either way we should see if other people are doing it ● And if what we want to do is complex, it might be better to find something simple to start with ● It’s dangerous to go alone - take this uting+to+Spark Jon Nelson
  13. 13. @holdenkarau The different pieces of Spark Apache Spark “Core” SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Graph X MLLib Community Packages Spark on Yarn Spark on Mesos Standalone Spark
  14. 14. @holdenkarau The different pieces of Spark: 2.0+ Apache Spark “Core” SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Graph X MLLib Community Packages Structured Streaming
  15. 15. @holdenkarau The different pieces of Spark: 3+? Apache Spark “Core” SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Graph X MLLib Community Packages Structured Streaming Spark on Yarn Spark on Mesos Spark on Kubernetes Standalone Spark
  16. 16. @holdenkarau Choosing a component? ● Core ○ Conservative to external changes, but biggest impact ● ML / MLlib ○ ML is the home of the future - you can improve existing algorithms - new algorithms face uphill battle ● Structured Streaming ○ Current API is in a lot of flux so it is difficult for external participation ● SQL ○ Lots of fun stuff - very active - I have limited personal experience ● Python / R ○ Improve coverage of current APIs, structural change hard ● GraphX - Dead see GraphFrames instead Rikki's Refuge
  17. 17. @holdenkarau Choosing a component? (cont) ● Kubernetes ○ New, lots of active work and reviewers ● YARN ○ Old faithful, always needs a little work. Hadoop 3 support ● Mesos ○ Needs some love, probably easy-ish-path to committer (still hard) ● Standalone ○ Not a lot going on Rikki's Refuge
  18. 18. @holdenkarau Onto JIRA - Issue tracking funtimes ● It’s like bugzilla or fog bugz ● There is an Apache JIRA for many Apache projects ● You can (and should) sign up for an account ● All changes in Spark (now) require a JIRA ● ● Check it out at: ○
  19. 19. @holdenkarau What we can do with ASF JIRA? ● Search for issues (remember to filter to Spark project) ● Create new issues ○ search first to see if someone else has reported it ● Comment on issues to let people know we are working on it ● Ask people for clarification or help ○ e.g. “Reading this I think you want the null values to be replaced by a string when processing - is that correct?” ○ @mentions work here too
  20. 20. @holdenkarau What can’t we do with ASF JIRA? ● Assign issues (to ourselves or other people) ○ In lieu of assigning we can “watch” & comment ● Post long design documents (create a Google Doc & link to it from the JIRA) ● Tag issues ○ While we can add tags, they often get removed
  21. 21. @holdenkarau
  22. 22. @holdenkarau Finding a good “starter” issue: ● There are explicit starter tags in JIRA we can search for ● But often the starter tag isn’t applied ● Read through and look for simple issues ● Pick something in the same component you eventually want to work in ○ And or consider improving the non-Scala language API for the component(s) you want to work on. ● Look at the reporter and commenters - is there a committer or someone whose name you recognize? ● Leave a comment that says you are going to start working on this
  23. 23. @holdenkarau Find an issue you want to work on Also grep for TODO in components you are interested in (e.g. grep -r TODO ./python/pyspark or grep -R TODO ./core/src) Look between language APIs and see if anything is missing that you think is interesting - .apache.spark.package neko kabachi
  24. 24. @holdenkarau Explore things that make sense to revisit Consider looking for issues which we couldn't fix due to our compatibility requirements and should revisit for 3+ Maurizio Zanetti
  25. 25. @holdenkarau Finding SPIPs: t%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Pro gress%22%2C%20Reopened)%20AND%20text%20~%20%22SPIP%22 Large pieces of work Not the easiest to contribute to, but can see design Warrick Wynne
  26. 26. @holdenkarau
  27. 27. @holdenkarau But before we get too far: ● Spark wishes to maintain compatibility between releases ● We're working on 3 though so this is the time to break things Meagan Fisher
  28. 28. @holdenkarau Getting at the code: yay for GitHub :) ● ● Make a fork of it ● Clone it locally dougwoods
  29. 29. @holdenkarau
  30. 30. @holdenkarau Building Spark ./build/sbt or ./build/mvn Working in Python? Make sure to build the package target so your Python code will run :) You can quickly verify build with the Spark Shell :) Kara
  31. 31. @holdenkarau What about documentation changes? ● Still use JIRAs to track ● We can’t edit the wiki :( ● But a lot of documentations lives in docs/*.md Kreg Steppe
  32. 32. @holdenkarau Building Spark’s docs ./docs/ has a lot of info - but quickly: SKIP_API=1 jekyll build SKIP_API=1 jekyll serve --watch *Requires a recentish jekyll - install instructions assume ruby2.0 only, on debian based s/gem/gem2.0/
  33. 33. @holdenkarau Finding your way around the project ● Organized into sub-projects by directory ● IntelliJ is very popular with Spark developers ○ The free version is fine ● Some people like using emacs + ensime or magit too ● Language specific code is in each sub directory
  34. 34. @holdenkarau Testing the issue The spark-shell can often be a good way to verify the issue reported in the JIRA is still occurring and come up with a reasonable test. Once you’ve got a handle on the issue in the spark-shell (or if you decide to skip that step) check out ./[component]/src/test for Scala or doctests for Python
  35. 35. @holdenkarau While we get our code working: ● Remember to follow the style guides ○ ide ● Please always add tests ○ For development we can run scala test with “sbt [module]/testOnly” ○ In python we can specify module with ./python/run-tests ● ./dev/lint-scala & ./dev/lint-python check for some style ● Changing the API? Make sure we pass or you update MiMa! ○ Sometimes its OK to make breaking changes, and MiMa can be a bit overzealous so adding exceptions is common
  36. 36. @holdenkarau A bit more on MiMa ● Spark wishes to maintain binary compatibility ○ in non-experimental components ○ 3.0 can be different ● MiMa exclusions can be added if we verify (and document how we verified) the compatibility ● Often MiMa is a bit over sensitive so don’t feel stressed - feel free to ask for help if confused Julie Krawczyk
  37. 37. @holdenkarau Making the change: No arguing about which editor please - kthnx Making a doc change? Look inside docs/*.md Making a code change? grep or intellij or github inside project codesearch can all help you find what you're looking for.
  38. 38. @holdenkarau Python API change parity update?
  39. 39. @holdenkarau Yay! Let’s make a PR :) ● Push to your branch ● Visit github ● Create PR (put JIRA name in title as well as component) ○ Components control where our PR shows up in ● If you’ve been whitelisted tests will run ● Otherwise will wait for someone to verify ● Tag it “WIP” if its a work in progress (but maybe wait) [puamelia]
  40. 40. @holdenkarau Code review time ● Note: this is after the pull request creation ● I believe code reviews should be done in the open ○ With an exception of when we are deciding if we want to try and submit a change ○ Even then should have hopefully decided that back at the JIRA stage ● My personal beliefs & your org’s may not align ● If you have the time you can contribute by reviewing others code too (please!) Mitchell Joyce
  41. 41. @holdenkarau And now onto the actual code review... ● Most often committers will review your code (eventually) ● Other people can help too ● People can be very busy (check the release schedule) ● If you don’t get traction try pinging people ○ Me ( @holdenkarau - I'm not an expert everywhere but I can look) ○ The author of the JIRA (even if not a committer) ○ The shepherd of the JIRA (if applicable) ○ The person who wrote the code you are changing (git blame) ○ Active committers for the component Mitchell Joyce
  42. 42. @holdenkarau What does the review look like? ● LGTM - Looks good to me ○ Individual thinks the code looks good - ready to merge (sometimes LGTM pending tests or LGTM but check with @[name]). ● SGTM - Sounds good to me (normally in response to a suggestion) ● Sometimes get sent back to the drawing board ● Not all PRs get in - its ok! ○ Don’t feel bad & don’t get discouraged. ● Mixture of in-line comments & general comments ● You can see some videos of my live reviews at Phil Long
  43. 43. @holdenkarau
  44. 44. @holdenkarau
  45. 45. @holdenkarau
  46. 46. @holdenkarau
  47. 47. @holdenkarau
  48. 48. @holdenkarau
  49. 49. @holdenkarau
  50. 50. @holdenkarau
  51. 51. @holdenkarau
  52. 52. @holdenkarau That’s a pretty standard small PR ● It took some time to get merged in ● It was fairly simple ● Review cycles are long - so move on to other things ● Only two reviewers ● Apache Spark Jenkins comments on build status :) ○ “Jenkins retest this please” is great ● Big PRs - like making PySpark pip installable can have > 10 reviewers and take a long time ● Sometimes it can be hard to find reviewers - tag your PRs & ping people on github James Joel
  53. 53. @holdenkarau Don’t get discouraged David Martyn Hunt It is normal to not get every pull request accepted Sometimes other people will “scoop” you on your pull request Sometimes people will be super helpful with your pull request
  54. 54. @holdenkarau Don’t get discouraged David Martyn Hunt If you don’t hear anything there is a good chance it is a “soft no” - but you can ping me and I can try and help. The community has been trying to get better at explicit “Won’t Fix” or saying no on PRs
  55. 55. @holdenkarau So who was that “Spark QA”/SparkJenkins/etc.? ● Automated pull request builder ● Jenkins based ● Runs all of the tests & style checks ● Lives in Berkeley ● Test logs live on, artifacts not so much ●
  56. 56. @holdenkarau Some changes require even more testing ● spark-perf (common for ML changes) ● spark-sql-perf (common for SQL changes) ● spark-integration-tests (integration testing) Image of FLG by Eric Kilby
  57. 57. @holdenkarau While we are waiting: ● Keep merging in master when we get out of sync ● If we don’t jenkins can’t run :( ● We get out of sync surprisingly quickly! ● If our pull request gets older than 30 days it might get auto-closed ● If you don’t here anything try pinging the dev list to see if it's a “soft no” (and or ping me :)) Moyan Brenn
  58. 58. @holdenkarau In review: Where do we get started? ● Search for “starter” on JIRA ● Look on the mailing list for problems ● Stackoverflow - lots of questions some of which are bugs ● grep TODO broken FIXME ● Compare APIs between languages ● Customer/user reports? Serena
  59. 59. @holdenkarau What about doing reviews? ● You don't need to be an expert (just will be slower) ● It's OK to leave suggestions like "I think does X but it's a little confusing maybe add a comment" ● First pass reviews from others are super useful ● Helping people find the right reviewers is useful ● We have over 450 open pull request (> 150 "active") ● You can drill down by component in
  60. 60. @holdenkarau What about when we want to make big changes? ● Talk with the community ○ Developer mailing list ○ User mailing list ● Consider if it can be published as a spark-package ● Create a public design document (google doc normally) ● Be aware this will be somewhat of an uphill battle (I’m sorry) ● You can look at SPIPs (Spark's versions of PEPs)
  61. 61. @holdenkarau Other resources: ● “Contributing to Apache Spark” - uting+to+Spark ● Programming guide (along with JavaDoc, PyDoc, ScalaDoc, etc.) - ● Developer list - /
  62. 62. @holdenkarau What things can be good Spark packages? ● Input formats (especially Spark SQL, Streaming) ● Machine learning pipeline components & algorithms ● Testing support ● Monitoring data sinks ● Deployment tools frankieleon
  63. 63. @holdenkarau Making your a package ● Relatively simple - need to publish to maven central ● Listed on ● Cross building (Spark versions) not super easy ○ I use a perl script (don’t tell on me) ● If your building with sbt check out to make it easy to publish ● Used to do API compatibility checks ● Sometimes flakey - just republish if it doesn’t go through frankieleon
  64. 64. @holdenkarau How about writing a book? ● Can be lots of fun ● Can also take up 100% of your “free” time ● Can get you invited to more nerd parties ● Most of the publisher are looking to improve/broaden their Spark book line up ● Like an old book that hasn’t been updated? Talk to the publisher about updating it. Kreg Steppe
  65. 65. @holdenkarau How about yak shaving? ● Lots of areas need shaving ● JVM deps are easier to update, Python deps are not :( ● Things built on top are a great place to go yak shaving ○ Jupyter etc. Jason Crane
  66. 66. @holdenkarau Testing/Release Validation ● Join the dev@ list and look for [VOTE] threads ○ Check and see if Spark deploys on your environment ○ If your application still works, or if we need to fix something ○ Great way to keep your Spark application working with less work ● Adding more automated tests is good too ○ Especially integration tests
  67. 67. @holdenkarau Spark Videos ● Apache Spark Youtube Channel ● My Spark videos on YouTube - ○ ● Spark Summit 2014 training ● Paco’s Introduction to Apache Spark Paul Anderson
  68. 68. @holdenkarau Learning Spark Fast Data Processing with Spark (Out of Date) Fast Data Processing with Spark (2nd edition) Advanced Analytics with Spark Spark in Action High Performance SparkLearning PySpark
  69. 69. @holdenkarau High Performance Spark! You can buy it today! On the internet! Cats love it* *Or at least the box it comes in. If buying for a cat, get print rather than e-book.
  70. 70. @holdenkarau Sign up for the mailing list @
  71. 71. @holdenkarau And some upcoming talks: ● March ○ Dataworks Barcelona -- tomorrow ○ Strata San Francisco -- next week ● April ○ Spark Summit ● May ○ KiwiCoda Mania ● June ○ "Secret" (for another week or so) ● July ○ OSCON Portland ○ Skills Matter in London
  72. 72. @holdenkarau k thnx bye :) If you care about Spark testing and don’t hate surveys: . Will tweet results “eventually” @holdenkarau Do you want more realistic benchmarks? Share your UDFs! It’s performance review season, so help a friend out and fill out this survey with your talk feedback