Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Rubygems - behind the gems.

on

  • 6,951 views

My talk about the current state of the rubygems infrastructure, problems, possible solutions. ...

My talk about the current state of the rubygems infrastructure, problems, possible solutions.

The intention behind this talk is to make people care about a problem and join forces to fix them. It's not about blaming anyone who spents her/his time for doing open source work!

Statistics

Views

Total Views
6,951
Views on SlideShare
6,524
Embed Views
427

Actions

Likes
13
Downloads
45
Comments
1

9 Embeds 427

http://railsmagazin.de 324
http://feeds2.feedburner.com 72
https://twimg0-a.akamaihd.net 11
http://eatmarkit.dev 7
https://si0.twimg.com 7
http://www.onlydoo.com 2
http://duckduckgo.com 2
http://webcache.googleusercontent.com 1
https://twitter.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Nice slide deck! We definitely need to figure this out. We've discussed using mirrorbrain among other things. Hop in #rubygems at some point so we can discuss this, or shoot me an email.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

Rubygems - behind the gems. Rubygems - behind the gems. Presentation Transcript

  • RUBYGEMS ...behind the gems Ruby User Group München 26.10.2010 Roland Moriz ~ Moriz GmbH http://moriz.de/
  • http:// Rubygems behind the gems. moriz.de/ Hello blaaa bla Moriz GmbH bla bla Software Development Services bla bla bla bla Consulting bla blaaaa bla blaaaaaa bla Infrastructure Services bla bla Roland bla bla bla bla professional software development since 1999 bla bla Amazon Marketplace Deutschland bla bla bla Tiscali Games bla bla FIFA WM 2006 bla Yahoo.de bla bla bla bla two billion pageviews bla bla blala Allianz24.de/Allsecur.de bla bla bla Ruby User Group München bla bla blabla http://moriz.de/ bla blaaaba http://rails.io bla http:// boot.io blablabla recently hetzner-api gem bla bla bla and the slides will be available @ http://moriz.de/ talks/rubygems. ;-)
  • http:// Rubygems behind the gems. moriz.de/ RUBYGEMS MOVING rubygems / cli gemcutter gem code / library, app, data, meta
  • http:// Rubygems behind the gems. moriz.de/ RUBYGEMS MOVING rubygems / cli gemcutter $ gem http:// rubygems.org/ require ”rubygems“ (and extensions to the rubygems client) creation, download, setup, distribution usage (index building, server)
  • http:// Rubygems behind the gems. moriz.de/ RUBYGEMS FACTS • used by nearly every ruby project • the core of the ruby ecosystem • standard lib (with MRI 1.9.x) • 17.000+ gem projects • 81.000+ gem files • 23 GB+
  • http:// Rubygems behind the gems. moriz.de/ RUBYGEMS FACTS started at RubyConf 2003 by: • Rich Kilmer • Chad Fowler • David Black • Paul Brannan • Jim Weirch > http://rubyforge.org/projects/rubygems/
  • http:// Rubygems behind the gems. moriz.de/ GEM FACTS • described by a .gemspec • gem build my.gemspec easier ways: • bundler, jewler, newgem(?), ...
  • http:// Rubygems behind the gems. moriz.de/ GEM FACTS contents: tar xvf rails-3.0.1.gem x data.tar.gz x metadata.gz
  • http:// Rubygems behind the gems. moriz.de/ GEM FACTS metadata.gz > gzipped YAML data.tar.gz > payload
  • http:// Rubygems behind the gems. moriz.de/ GEMCUTTER FACTS • started in April 2009 • is now rubygems.org (rubygems 1.3.6+) • replaced rubyforge • manages uploads & downloads • rails app using PostgreSQL + rack middleware with sinatra • by Nick Quaranto (@qrush) of Thoughtbot > http://github.com/rubygems/gemcutter
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: UPLOAD RELEASE gem release $ gem release hetzner-api.gemspec Successfully built RubyGem Name: hetzner-api Version: 1.0.0 File: hetzner-api-1.0.0.gem Pushing gem to RubyGems.org...
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: UPLOAD RELEASE cli rubygems.org (gemcutter)
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: UPLOAD RELEASE cli rubygems.org gem file AWS S3
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: UPLOAD RELEASE cli rubygems.org spec files AWS S3 update specs > database > spec files to s3
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: UPLOAD RELEASE cli rubygems.org AWS S3 update specs webhooks, http://rubygems.org/pages/api_docs
  • http:// Rubygems behind the gems. moriz.de/ BIG PICTURE: DOWNLOAD cli rubygems.org AWS S3
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ no specific version => latest $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ Gem.marshal_version => Marshal::MAJOR_VERSION Marshal::MINOR_VERSION $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ irb(main):001:0> x = {} => {} irb(main):002:0> x['farbe'] = 'ananasblau' => "ananasblau" irb(main):003:0> Marshal.dump x => "004b{006"nfarbe"017ananasblau" etc. > http://ruby-doc.org/core/classes/Marshal.html
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ latest_specs: lists the latest release number of all gems (~150 KB / 570 KB) latest_specs = Marshal.load open 'latest_specs.4.8' latest_specs.size => 17501 specs: list of all gem releases (380 KB / 2.2 MB) specs = Marshal.load open 'specs.4.8'; specs.size => 83490 (there‘s also a pre-release spec (remember „gem install rails --pre“) and others: see rubygems source lib/rubygems/commands/generate_index_command.rb)
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ $sudo gem install rails -V load + parse spec x dependencies GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ Marshal.load Gem.inflate File.read 'rails-3.0.1.gemspec.rz' Gem::Specification.new do |s|   s.authors = ["David Heinemeier Hansson"]   s.date = Time.utc(2010, 10, 14)   s.dependencies = [Gem::Dependency.new("activesupport",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("actionpack",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("activerecord",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("activeresource",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("actionmailer",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("railties",     Gem::Requirement.new(["= 3.0.1"]),     :runtime),    Gem::Dependency.new("bundler",     Gem::Requirement.new(["~> 1.0.0"]),     :runtime)]   s.description = "Ruby on Rails is a full-stack web framework optimized
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz deps with explicit 302 Found GET version requirement => http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK require full spec list GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ...
  • http:// Rubygems behind the gems. moriz.de/ SPECS AKA „THE INDEX“ $sudo gem install rails -V GET http://gems.rubyforge.org/latest_specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/latest_specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/rails-3.0.1.gemspec.rz 200 OK GET http://gems.rubyforge.org/specs.4.8.gz 302 Found GET http://production.s3.rubygems.org/specs.4.8.gz 200 OK GET http://gems.rubyforge.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz 302 Found GET http://production.s3.rubygems.org/quick/Marshal.4.8/activesupport-3.0.1.gemspec.rz ... for each dependency then download and install the .gem files
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: WHAT IF? cli rubygems.org AWS S3
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: WHAT IF? cli Temporary Outage: no new gem releases rubygems.org no gem downloads (index missing) new app deployments? ! new server deployments? AWS S3
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: WHAT IF? cli Fatal Outage, reasons: • Hardware • Software (attack, fs corruption) rubygems.org • Amazon • account „deactivation“ • account deletion • S3 data loss AWS S3 • S3 bucket account theft/crack • Sunny day kills all the clouds. • Jeff Bezos‘ new bicy^Segw^Rocket.
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: WHAT IF? cli Fatal Outage: ALL GEMS LOST rubygems.org AWS S3
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: WHAT IF? cli Fatal Outage: ALL GEMS LOST again. rubygems.org try ^_^ AWS S3
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: MIRRORING Fallback for rubygems.org desaster? > Use a public mirror > Start your own mirror Infrastructure independence to save your business from a rubygems desaster: > Start your own mirror
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: PUBLIC MIRRORS Comprehensive Perl Archive Network 2010-10-25 online since 1995-10-26 7770 MB 228 mirrors 8463 authors 18582 modules 228 independent public and free mirrors!
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: PUBLIC MIRRORS Debian Mirror Sites: 445 http://www.debian.org/mirror/list
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: PUBLIC MIRRORS „The Python Package Index is a repository of software for the Python programming language. There are currently 11801 packages here“
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: PUBLIC MIRRORS
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: PUBLIC MIRRORS 0 active, public, free mirrors. lost in migration (rubyforge > gemcutter)
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: MIRRORING Mirroring stuff in rubygems is currently broken: • „gem mirror“ misses some gems & slow downloads: one gem at a time. • index building is broken (see #362) • reliability (#362, too) http://help.rubygems.org/discussions/problems/362-cant-mirror-rubygems- repo-incorrect-header-check Gemcutter already lost gems: http://help.rubygems.org/discussions/problems/212-some-gems-and-specs-missing-that-are-in-the-index
  • http:// Rubygems behind the gems. moriz.de/ PROBLEMS: MIRRORING There is also no easy way to mirror a S3 bucket: • no ftp • no rsync • no file-list to use with e.g. wget = you cannot even run a reliable private mirror :-(
  • http:// Rubygems behind the gems. moriz.de/ SOLUTION Provide rsync on master for sync-ability. On EC2, Rackspace, does not matter if it‘s fast... > NO custom mirroring software! > most FOSS mirror sites use rsync > use rsync, ask mirrors, problem solved. > AWS cloudfront is NOT a solution > not mirrorable, same vendor SPOFs.
  • http:// Rubygems behind the gems. moriz.de/ SOLUTION Provide rsync on master for sync-ability. On EC2, Rackspace, does not matter if it‘s fast... > a realiable base for (private) mirroring > NO custom mirroring software needed! Provide a DNS based distribution (GeoDNS) > speed & latency improvements > saves money (AWS and Rackspace fees) > make use of the new mirrors!
  • http:// Rubygems behind the gems. moriz.de/ SOLUTION Why not? > no „instant deploy“ (real-time mirroring) > no download stats
  • http:// Rubygems behind the gems. moriz.de/ SOLUTION Why not? > no „instant deploy“ (real-time mirroring) > no download stats Rubygems CLI could fallback to the rubygems.org master if a gem version is not on the used mirror. It already does if you configure it. (current downside: d/l spec-lists from master everytime, looks fixable to me)
  • http:// Rubygems behind the gems. moriz.de/ THINGS WILL FAIL... just make sure you‘ve a working plan B AND: KISS & YAGNI. Keep it simple. less moving parts > less things that will break. Don‘t over-engineer.
  • http:// Rubygems behind the gems. moriz.de/ HELP OpenSource projects need your support. Gemcutter/Rubygems, too. Go contribute if you care about your ruby business. The Gemcutter source is really awesome, a good read for every developer.