Git with Swift Atlanta 2014

447 views

Published on

Published in: Internet, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
447
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Git with Swift Atlanta 2014

  1. 1. Chmouel Boudjnah! OpenStack Summit 2014 Atlanta OpenStack Swift as a Backend for Git Chmouel Boudjnah - @chmouel - eNovance
  2. 2. Chmouel Boudjnah - @chmouel - eNovance The mandatory about me
  3. 3. Chmouel Boudjnah - @chmouel - eNovance About me I am not the author, Fabien Boucher aka: @bzho Chmouel Boudjnah - @chmouel - eNovance
  4. 4. Chmouel Boudjnah - @chmouel - eNovance About me We both work for eNovance
  5. 5. Chmouel Boudjnah - @chmouel - eNovance About me OpenStacker for quite some time now
  6. 6. Chmouel Boudjnah - @chmouel - eNovance So what is Swift?
  7. 7. Chmouel Boudjnah - @chmouel - eNovance • Object Storage • Not a file system • Eventually Consistent • HTTP REST API • Very Scalable • Software defined Storage What is Swift
  8. 8. Chmouel Boudjnah - @chmouel - eNovance Why would you want to use GIT with Swift?
  9. 9. Chmouel Boudjnah - @chmouel - eNovance Why Swift with GIT Why not?
  10. 10. Chmouel Boudjnah - @chmouel - eNovance Why Swift with GIT When you use a file system you have to : Perform backups of your repositories regularly Perform complex and risky operations to extend the storage capabilities for you repositories. You have mostly have to manage it yourself It’s an another piece in your infrastructure You have to manage the downtime Not fun
  11. 11. Chmouel Boudjnah - @chmouel - eNovance So what are the benefits?
  12. 12. Chmouel Boudjnah - @chmouel - eNovance Why Swift with GIT The advantages of using Swift Safety: The inherent replication of a data store. Capability: easily extend storage capacity. High Availability: no single point of failure. Maintainability: allow operations teams to maintain the cluster without any downtime. In an OpenStack environment it’s usually already there.
  13. 13. Chmouel Boudjnah - @chmouel - eNovance Let’s see how git internal works
  14. 14. Chmouel Boudjnah - @chmouel - eNovance
  15. 15. Chmouel Boudjnah - @chmouel - eNovance
  16. 16. Chmouel Boudjnah - @chmouel - eNovance So how did we plug that to Swift?
  17. 17. Chmouel Boudjnah - @chmouel - eNovance We did not want to modify the GIT client binaries.
  18. 18. Chmouel Boudjnah - @chmouel - eNovance We used a pure python GIT library called Dulwich
  19. 19. Chmouel Boudjnah - @chmouel - eNovance • Create, read, manage loose objects (blob, tree, commit, tag). • Create, read, manage pack files, pack indexes. • Implement the Git smart protocol through git- upload-pack and git-receive-pack. • Implement the Git, HTTP, SSH listeners to start Dulwich as a Git server. • Provide a well designed interface to the storage backend. (Repo, MemoryRepo). Dulwich
  20. 20. Chmouel Boudjnah - @chmouel - eNovance How do we use Dulwhich We implemented a custom repo called SwiftRepo as a backend to the git listener of Dulwich which communicate to OpenStack Swift
  21. 21. Chmouel Boudjnah - @chmouel - eNovance So about that SwiftRepo backend how does it work?
  22. 22. Chmouel Boudjnah - @chmouel - eNovance SwiftRepos backend to Dulwich It doesn’t use the standard way to store Git references (one file by reference). It doesn’t store loose objects but instead uses the pack format. In addition to the .index that relate to each pack file we added .info object It makes the Request to Swift concurrently.
  23. 23. Chmouel Boudjnah - @chmouel - eNovance How’s the benchmark then?
  24. 24. Chmouel Boudjnah - @chmouel - eNovance Benchmark Setup Benchmark performed on two architectures : Github : ● Git client from eNovance network <-> GitHub ! Dulwich/Swift : ●· Git client from eNovance network <-> Dulwich (Rax compute) <-> Rax Cloudfiles ! Three different repos : ! ●· Swiftsync (873 objects with 293 commits) ●· Edeploy (2246 objects with 618 commits) ●· Swift (23480 objects with 3212 commit)
  25. 25. Chmouel Boudjnah - @chmouel - eNovance Full repository push
  26. 26. Chmouel Boudjnah - @chmouel - eNovance Clone
  27. 27. Chmouel Boudjnah - @chmouel - eNovance That’s it, hope you enjoyed.
  28. 28. Chmouel Boudjnah - @chmouel - eNovance Eventual Consistentie Why Eventual Consistentie window is not an issue

×