This talk "Precise, cross-project code navigation at GitHub scale", was presented at FOSDEM 2020 by Douglas Creager, Manager of Semantic Code team at GitHub, in the Devroom Session "Dependency Management"

1. Precise, cross-project
code navigation
at GitHub scale
FOSDEM 2020
Douglas Creager — @dcreager February 1, 2020

2. Precise, cross-project
code navigation
at GitHub scale
FOSDEM 2020
Douglas Creager — @dcreager February 1, 2020

3. Back in November...Back in November...

4. github.com

5. github.com

6. github.com

7. github.com

8. LimitationsLimitations

9. Only three languages are GA
Go Python Ruby
(JavaScript, TypeScript, and PHP are in beta; more on the way!)

10. “Fuzzy” (or ctags-like) symbol matching

11. github.com

12. github.com

13. Can only follow links within a repository

14. LimitationsLimitations

15. Takeaways
Local development ≠
Hosted service
Incremental processing
is a must
Within-repo = Cross-repo

16. Local development Hosted service≠

18. Local development
User choice is paramount
Context is a single workspace
Interactive

19. Local development
User choice is paramount
Context is a single workspace
Interactive
LSP

20. Local development
User choice is paramount
Context is a single workspace
Interactive
LSP
M × N → M + N
Long-running sidecar process
Maintains long-lived in-memory state

21. Hosted service
Lots of simultaneous contexts
Looks more like a database
Not as interactive
...but latency still counts!

22. Hosted service
Lots of simultaneous contexts
Looks more like a database
Not as interactive
...but latency still counts!
Everything is hidden behind an API

23. Hosted service
Lots of simultaneous contexts
Looks more like a database
Not as interactive
...but latency still counts!
Everything is hidden behind an API
...and we carry a pager

24. Code generation

25. tree-sitter parser for the language

26. machine-readable description of the generated parser

27. automatically generate AST data types from that grammar description

28. including automatically generated API documentation!

29. pattern-matching rules to pull out the definitions and references

30. some boilerplate but not too bad

31. We can produce fuzzy symbol matches incrementally.

32. Incremental processing is a must

33. Can we do that for precise symbol matches too?

37. Within repository Across repositories=

38. Takeaways
Local development ≠
Hosted service
Incremental processing
is a must
Within-repo = Cross-repo

39. Thank you!
@dcreager
https://github.com/github/semantic/
https://github.com/tree-sitter/tree-sitter/
Image credits (CC-BY-2.0 unless otherwise noted)
Sundial, liz west. https://flic.kr/p/EWBd4
Rain chain, lo fidelion. https://flic.kr/p/F12EUN
go, Luis de Bethencourt. https://flic.kr/p/4c5RaR
Ball Python, Renee Grayson. https://flic.kr/p/KTMCBL
Halsschmuck mit rotem Edelstein auf dunklem Hintergrund, Ivan Radic. https://flic.kr/p/2fBnsE6
Out of Focus Beach Scene, jmclellon. https://flic.kr/p/28wcV4Z (Public domain)
Han Pottery Pigpen, Gary Todd. https://flic.kr/p/gGuDug (Public domain)
Mangshan National Park, cattan2011. https://flic.kr/p/NjJUYx
laptop-iphone-desk, Coffee Channel. https://flic.kr/p/23rikHK
Computer racks, Tristan Schmurr. https://flic.kr/p/22uf6Ax
Clones, Manuel. https://flic.kr/p/WYQJQL (Public domain)
Inchworm, Katja Schulz. https://flic.kr/p/PJMP4w
Red Arrows Spaghetti Burst, Clint Budd. https://flic.kr/p/WmLc1n