About a year ago data streams team at Wix has released to open-source its Kafka client SDK wrapper called Greyhound. Greyhound offers rich functionality like message processing parallelisation and batching, various fault tolerant retry policies and much more. This talk will show how the team designed Greyhound with a layered architecture to allow both public and private parts and also different levels of flexible configuration. How it automatically syncs only relevant code from private repo to public one and also how it securely accepts public PRs back to the private repo. Outline: * Quick intro on what Greyhound is and its history at Wix * Greyhound layered architecture design to allow both public and private parts and also different levels of flexible configuration. * How it automatically syncs only relevant code from private repo to public one using Copybara tool * how it securely accepts public PRs back to the private repo.

1. Open Sourcing a
Successful Internal Project
Natan Silnitsky
Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

2. To developers
of the World
Open
Source
a successful
project

3. Wix developers
To developers
of the World
Open
Source
a successful
project

4. 1st Way To
Open Source
Public
Private
move
Open Sourcing a Successful Internal Project

5. 2nd Way To
Open Source
Public
Private
sync
Open Sourcing a Successful Internal Project

6. Open Sourcing a Successful Internal Project
What is
Greyhound?
2000

7. Greyhound
Wraps Kafka
Open Sourcing a Successful Internal Project
Kafka
Consumer
Kafka
Producer
Kafka Broker
Wix Microservices

8. Abstract
so that it’s easy to change for
everyone
Simplify
APIs, with additional features
Kafka Broker
Open Sourcing a Successful Internal Project
Greyhound
Kafka
Consumer
Kafka
Producer
Wix Microservices

9. Greyhound
Kafka
Consumer
Kafka
Producer
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service ❤
Wix developers
Kafka Broker
Open Sourcing a Successful Internal Project

10. To developers
of the World
2015
Greyhound
2019
Open
Source
Greyhound
Self Service ❤
Rewrite!

11. Open Sourcing
Greyhound A Layered
Architecture
------ 2019
Open Sourcing a Successful Internal Project
Allow different
access to
different parts

12. Kafka
Producer
Kafka
Consumer
Greyhound
Kafka Broker
Multiple APIs
for Java, Scala and Wix devs
Scala Future ZIO Java
Zio Core
Open Sourcing a Successful Internal Project

13. Kafka
Producer
Kafka
Consumer
Greyhound
Kafka Broker
Scala Future ZIO Java
Zio Core
Wix Interop
Open Sourcing a Successful Internal Project
Public
Private

14. Open Sourcing
Greyhound A Layered
Architecture
------ 2019
Auto Sync
Private → Public
end of 2019
Open Sourcing a Successful Internal Project
I don’t want to
worry about tech
debt

15. Kafka
Producer
Kafka
Consumer
Greyhound
Kafka Broker
Scala Future ZIO Java
Zio Core
Wix Interop
Wix Public
Github Repo
Open Sourcing a Successful Internal Project
Wix Private
Github Repo
Sync!

16. Wix Public
Github Repo
Open Sourcing a Successful Internal Project
Wix Private
Github Repo
Sync!
Merge PR to master
How oSync Works
→ Transforms part of it with Copybara
→ Creates a (new) PR
→ Runs Tests > Merges

17. Open Sourcing
Greyhound A Layered
Architecture
------ 2019
Auto Sync
Private → Public
end of 2019
Accept Public PRs
------ 2020
Open Sourcing a Successful Internal Project
I want to be open
to contributions

18. Wix Public
Github Repo
Open Sourcing a Successful Internal Project
Wix Private
Github Repo
Sync!
→ New PR (close original PR)
→ Reviewer > Merge
PR opened in public
→ Slack alert
→ Reviewer > Slack approval

19. Can make
frequent Changes
Private
(auto) sync
Public
developers of the World
Wix developers
Open Sourcing a Successful Internal Project

20. Private
(auto) sync
Public
developers of the World
Wix developers
Open Sourcing a Successful Internal Project
Layers &
Public PR
checks
Contributions

21. Open Sourcing a Successful Internal Project

22. Thank
You!
👉 slideshare.net/NatanSilnitsky
Any questions?
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

23. Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?