Piotr Wasiak XI 2022
AWS Lambda
How to upload and load gems fast
Agenda
AWS loading code and gems
• Short intro to AWS Lambda
• How to load gems
• Problem with native extensions
• What's the load time?
• Deploying layers with no downtime?
About.me/Piotr.Wasiak
Ruby (and Rails) developer
• I work currently in SpaceOS.io prop-tech startup
• lambda is used in production to sync sequentially data with external systems
AWS Lambda
What is the fuss about
• Reduced complexity. There’s a
fi
xed operating system,
fi
xed software
language, and a
fi
xed version.
• No infrastructure to manage. AWS owns and manages the infrastructure.
You only pay for the time your code runs.
• Implicit scaling. AWS will handle scaling, no matter how much you use your
functions.
AWS Lambda
function sca
ff
old
• We can trigger lambda function by:
• Amazon Simple Queue Service
• HTTP Request
• Other Amazon events
What about loading libraries
3 ways
• pushing gem sources with code (naive way)
• pushing gems as a layer (pack the sources as zip)
• bundle install --deployment
• zip -FSr gem_layer.zip ./ruby/
• aws lambda publish-layer-version ....
• dockerize lambda image (but hmm it should be without additional setup)
Can we use bundler for loading gems?
bundler is by default shiped with ruby
• yes, we can use Gem
fi
le for loading
• on lambda function start:
require 'bundler'
Bundler.require(:default)
• bundler will raise on start if some gem version is missing or wrong
• but you can always require 'gem_name' wherever do you want it
First problem with layers
What if we require gems
with native extensions?
dependency from Savon
• inside Nokogiri there is also racc
• so there are external libs per
architecture/distributions
Nokogiri
Solution
• use aws-lambda docker to generate libs
• bundle con
fi
g set --local clean 'true'
• zip & push it as layer
Nokogiri depedency
Second problem with ruby
What is really the load time
of lambda function
Before adding native extentions
Loading time ~ 1050 ms
After adding native extentions
Loading time ~ 1550 ms
Our Gemfile right now
> bundle install --without test
(...)
Using activesupport 7.0.4
Using nokogiri 1.13.9 (arm64-darwin)
Using aws-sdk-core 3.166.0
Using faraday 2.6.0
Using graphql 2.0.15
Using savon 2.13.1
(...)
Using sentry-lambda 0.2.1
Using warning 1.3.0
Using zeitwerk 2.6.4
Bundle complete! 14 Gem
fi
le dependencies, 38 gems now installed.
Gems in the group 'test' were not installed.
Autoloader to load only required files
Zeitwerk Solution
• Convention to store required classes in proper folders
• load on usage and don't load twice (even if it has di
ff
erent require path)
After adding Zeitwerk (code autoloader)
Loading time ~ 1350 ms
Load gems only if needed
Solution
• skip requiring Nokogiri and GraphQL in Gem
fi
le (on boot)
• manual require in service
fi
les
With autoloader skipping Savon, GraphQL
Loading time ~ 820 ms Faster than 6 month ago :O
Third problem with layers
How to ensure no downtime
with releasing new libs and code
How to release new layer with code
Solutions
1. Combine together previous layer with new one and ship combined layer
fi
rst
• can be incompatible with API changes (test it
fi
rst)
• no downtime at all
• have to use bundler (require does not know which gem version should be loaded)
2. Push new layer
and switch it just before pushing the new function code
• there is no way to push the code and switch to new layer in one transaction
• very small downtime (already started functions will be
fi
ne during that change)
3. Use lambda function versions and just switch event trigger to new code version with layer
Take aways
• Lambda loading time
for ruby is more 0,5 sec
• Loading gems with layers
provides some challenges
• If you have complex app
then go with lambda docker
Sources
• https://stackify.com/aws-lambda-with-ruby-getting-started-guide
• https://docs.aws.amazon.com/lambda/latest/dg/lambda-ruby.html
• github.com aws-lambda developer-guide/sample-ruby
• https://github.com/fxn/zeitwerk#awards
• https://bundler.io
Thanks for listening

AWS Lambda How to upload and load gems fast

  • 1.
    Piotr Wasiak XI2022 AWS Lambda How to upload and load gems fast
  • 2.
    Agenda AWS loading codeand gems • Short intro to AWS Lambda • How to load gems • Problem with native extensions • What's the load time? • Deploying layers with no downtime?
  • 3.
    About.me/Piotr.Wasiak Ruby (and Rails)developer • I work currently in SpaceOS.io prop-tech startup • lambda is used in production to sync sequentially data with external systems
  • 4.
    AWS Lambda What isthe fuss about • Reduced complexity. There’s a fi xed operating system, fi xed software language, and a fi xed version. • No infrastructure to manage. AWS owns and manages the infrastructure. You only pay for the time your code runs. • Implicit scaling. AWS will handle scaling, no matter how much you use your functions.
  • 5.
    AWS Lambda function sca ff old •We can trigger lambda function by: • Amazon Simple Queue Service • HTTP Request • Other Amazon events
  • 6.
    What about loadinglibraries 3 ways • pushing gem sources with code (naive way) • pushing gems as a layer (pack the sources as zip) • bundle install --deployment • zip -FSr gem_layer.zip ./ruby/ • aws lambda publish-layer-version .... • dockerize lambda image (but hmm it should be without additional setup)
  • 7.
    Can we usebundler for loading gems? bundler is by default shiped with ruby • yes, we can use Gem fi le for loading • on lambda function start: require 'bundler' Bundler.require(:default) • bundler will raise on start if some gem version is missing or wrong • but you can always require 'gem_name' wherever do you want it
  • 9.
    First problem withlayers What if we require gems with native extensions?
  • 10.
    dependency from Savon •inside Nokogiri there is also racc • so there are external libs per architecture/distributions Nokogiri
  • 11.
    Solution • use aws-lambdadocker to generate libs • bundle con fi g set --local clean 'true' • zip & push it as layer Nokogiri depedency
  • 12.
    Second problem withruby What is really the load time of lambda function
  • 13.
    Before adding nativeextentions Loading time ~ 1050 ms
  • 14.
    After adding nativeextentions Loading time ~ 1550 ms
  • 15.
    Our Gemfile rightnow > bundle install --without test (...) Using activesupport 7.0.4 Using nokogiri 1.13.9 (arm64-darwin) Using aws-sdk-core 3.166.0 Using faraday 2.6.0 Using graphql 2.0.15 Using savon 2.13.1 (...) Using sentry-lambda 0.2.1 Using warning 1.3.0 Using zeitwerk 2.6.4 Bundle complete! 14 Gem fi le dependencies, 38 gems now installed. Gems in the group 'test' were not installed.
  • 16.
    Autoloader to loadonly required files Zeitwerk Solution • Convention to store required classes in proper folders • load on usage and don't load twice (even if it has di ff erent require path)
  • 17.
    After adding Zeitwerk(code autoloader) Loading time ~ 1350 ms
  • 18.
    Load gems onlyif needed Solution • skip requiring Nokogiri and GraphQL in Gem fi le (on boot) • manual require in service fi les
  • 19.
    With autoloader skippingSavon, GraphQL Loading time ~ 820 ms Faster than 6 month ago :O
  • 20.
    Third problem withlayers How to ensure no downtime with releasing new libs and code
  • 21.
    How to releasenew layer with code Solutions 1. Combine together previous layer with new one and ship combined layer fi rst • can be incompatible with API changes (test it fi rst) • no downtime at all • have to use bundler (require does not know which gem version should be loaded) 2. Push new layer and switch it just before pushing the new function code • there is no way to push the code and switch to new layer in one transaction • very small downtime (already started functions will be fi ne during that change) 3. Use lambda function versions and just switch event trigger to new code version with layer
  • 22.
    Take aways • Lambdaloading time for ruby is more 0,5 sec • Loading gems with layers provides some challenges • If you have complex app then go with lambda docker
  • 23.
    Sources • https://stackify.com/aws-lambda-with-ruby-getting-started-guide • https://docs.aws.amazon.com/lambda/latest/dg/lambda-ruby.html •github.com aws-lambda developer-guide/sample-ruby • https://github.com/fxn/zeitwerk#awards • https://bundler.io
  • 24.