Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximise the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!
5. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
3
6. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
Co-Author of Node.js Design Patterns 👉
3
7. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
Co-Author of Node.js Design Patterns 👉
Connect with me:
(blog)
(twitter)
(twitch)
(github)
loige.co
@loige
loige
lmammino 3
8. We are business focused technologists that
deliver.
| |
Accelerated Serverless AI as a Service Platform Modernisation
We are hiring: do you want to ?
work with us
loige 4
9. loige
In the previous episodes...
SLIC WATCH:
SLIDES:
fth.link/slic-watch
fth.link/o11y-no-pain
5
10. loige
⚠ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)
Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was
greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK ->
ALARM transition).
@wake_me_up_bot
6
15. for event in payload['logEvents']:
# ...
cw_client.put_metric_data(...)
loige 10
16. for event in payload['logEvents']:
# ...
cw_client.put_metric_data(...)
loige
Sending 1 metric per log line... 🙈
10
17. [
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]
loige 11
18. [
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]
loige
⚠Payload size
limit: 40 KB
11
53. loige
Other potential use cases
Log data before/after it is sent to AWS
Add additional headers to AWS requests (tracing)
Additional input validation (your custom rules)
Payload enrichment (e.g. make sure certain tags exist)
25
54. loige
What if you don't use Python?
Customizing the AWS SDK for Go V2 Client Requests
Introducing Middleware Stack in Modular AWS SDK for
JavaScript
Handlers and Middleware in the AWS SDK for PHP Version 3
26
55. loige
What did we learn?
Always have alarms for your lambdas!
(have you checked already? 😉)
You can use Gzip compression to send big payloads to AWS
Boto3 is extensible through its event system!
SLIC Watch
27
56. loige
Do you think boto3 should make it easier to
send gzipped payloads?
👍UPVOTE this issue: github.com/boto/botocore/issues/2425
28