Create a serverless architecture for data collection with Python and AWS

Create a serverless architecture for data collec1on
with Python and AWS
9 Apr 2017
David Santucci

david.santucci@cloudacademy.com
About me
David Santucci
Data scien;st @ CloudAcademy.com
@davidsantucci
linkedin.com/in/davidsantucci/

Agenda
• Introduc;on
• Architecture
• Amazon Kinesis Stream
• Amazon Lambda
• Dead LeGer Queue (DLQ)
• Conclusions
• Q&A

Introduc4on
Challenges:
• Collect events from diﬀerent sources
• Backend applica;ons
• Frontend applica;ons
• Mobile apps
• Store events to diﬀerent des4na4ons
• Data Warehouse
• Third-party services
• e.g., Hubspot, Mixpanel, GTM, …
• Avoid data loss

A serverless architecture
AWS services:
• Kinesis Stream
• Lambda Func;ons
• SQS
• S3
• Amazon API Gateway

Manage events from mul4ple sources

Amazon Kinesis Stream
What is Amazon Kinesis Stream?
• Collect and process large streams of data records in real ;me.
• Typical scenarios for using Streams:
• Manage mul;ple producers that push their data feed directly into a stream;
• Collect real-;me analy;cs and metrics;
• Process applica;on logs;
• Create pipeline with other AWS services (the consumers).

from time import gmtime, strftime 
import boto3 
 
client = boto3.client( 
service_name="kinesis", 
region_name="us-east-1", 
) 
 
for i in xrange(300): 
print "sending event {}".format(i+1) 
response = client.put_record( 
StreamName="data-collection-stream", 
Data='{"name":"event-%d","data":{"payload":%d}}' % (i, i), 
PartitionKey=strftime("PK-%Y%m%d-%H%M%S", gmtime()), 
) 
print "response for event {}: {}".format(i+1, response) 
Amazon Kinesis Stream

Amazon Kinesis Stream - Tips
• Use API Gateway as entry point for front-end and mobile.
• Start with a single shard and increase only when needed.
• Output events one by one to avoid data loss.
• Generate Par44onKey using uuid (e.g., for test purpose).

Amazon Lambda
What is AWS Lambda?
• It processes a single event at real-;me without managing servers.
• Highly scalable.
• Fallback strategy in case of errors.

Amazon Lambda - Events rou4ng
It works as router and it
is directly triggered by
Kinesis Streams.

[ 
{ 
"destination_name": "mixpanel", 
"destination_arn": "arn:aws:lambda:region:account-id:function:function-name:prod", 
"enabled_events": [ 
"page_view", 
"search", 
"button_click", 
"page_scroll", 
] 
}, 
{ 
"destination_name": "hubspotcrm", 
"login", 
"logout", 
"registration", 
"page_view", 
"search", 
"email_sent", 
"email_open", 
] 
}, 
{ 
"destination_name": "datawarehouse", 
"login", 
"logout", 
"registration", 
"page_view", 
"search", 
"button_click", 
"page_scroll", 
"email_sent", 
"email_open", 
] 
} 
]

…
{ 
"destination_name": "datawarehouse", 
"destination_arn": "arn:aws:lambda:region:id:function:name:prod", 
"login", 
"logout", 
"registration", 
"page_view", 
"search", 
"button_click", 
"page_scroll", 
"email_sent", 
"email_open", 
] 
}

Amazon Lambda - events rou4ng
It provides the logic to
connect to the des4na4on
services (e.g., HubSpot,
Mixpanel, etc … ).
Custom retry strategy (with
exponen;al delay).

Amazon Lambda - Retry strategy
def lambda_handler(event, context=None): 
try: 
hub_id = os.environ['HUBSPOT_HUB_ID'] 
except KeyError: 
raise DoNotRetryException('HUBSPOT_HUB_ID') 
event = format_event_data(event, hub_id) 
process_event(event['data']) 
return "ok" 
 
def format_event_data(event, hub_id): 
event_id = event["name"].split(".")[-1].replace("_", " ").title() 
event['data'].update({ 
'_a': hub_id, 
'_n': event_id, 
'email': event['data']['_email'], 
}) 
return event 
 
@retry 
def process_event(params): 
url = 'http://track.hubspot.com/v1/event?{}'.format(urllib.urlencode(params)) 
urllib2.urlopen(url)

Amazon Lambda - Retry strategy
def retry(func, max_retries=3, backoff_rate=2, scale_factor=.1): 
def func_wrapper(*args, **kwargs): 
attempts = 0 
while True: 
attempts += 1 
if attempts >= max_retries: 
raise 
try: 
return func(*args, **kwargs) 
except DoNotRetryException: 
raise 
except: 
time.sleep(backoff_rate ** attempts * scale_factor) 
return func_wrapper 
 
 
class DoNotRetryException(Exception): 
def __init__(self, *args, **kwargs): 
Exception.__init__(self, *args, **kwargs)

Amazon Lambda - Our 4ps
• Enable Kinesis Stream as a trigger for other AWS services.
• To preserve the priority Configure trigger with Batch size: 1 and Star;ng posi;on: Trim Horizon
• An S3 file can be used to define the rou;ng rules.
• Invoke Lambda Func;ons that work as connector asynchronously
• Always create aliases and versions for each Func;on.
• Use environment variables for configura;ons.
• Create a custom IAM role for each Func;on.
• Detect delays in stream processing monitoring IteratorAge metric 
in the Lambda console’s monitoring tab.

Dead LeIer Queues (DLQ) - Avoid event loss

DLQ - Simple Queue Service (SQS)
What is AWS SQS?
• Lambda automa4cally retries failed execu;ons for asynchronous invoca;ons.
• Conﬁgure Lambda (advanced secngs) to forward payloads that were not
processed to a dead-leIer queue (an SQS queue or an SNS topic).
• We used a SQS.

def get_events_from_sqs( 
sqs_queue_name, 
region_name='us-west-2', 
purge_messages=False, 
backup_filename='backup.jsonl', 
visibility_timeout=60): 
""" 
Create a json backup file of all events in the SQS queue with the
given 'sqs_queue_name'. 
:sqs_queue_name: the name of the AWS SQS queue to be read via boto3 
:region_name: the region name of the AWS SQS queue to be read via boto3 
:purge_messages: True if messages must be deleted after reading, False otherwise 
:backup_filename: the name of the file where to store all SQS messages 
:visibility_timeout: period of time in seconds (unique consumer window) 
:return: the number of processed batch of events 
""" 
forwarded = 0 
counter = 0 
sqs = boto3.resource('sqs', region_name=region_name) 
dlq = sqs.get_queue_by_name(QueueName=sqs_queue_name)
# continues to next slide ..

# continues from previous slide ..
with open(backup_filename, 'a') as filep: 
while True: 
batch_messages = dlq.receive_messages( 
MessageAttributeNames=['All'], 
MaxNumberOfMessages=10, 
WaitTimeSeconds=20, 
VisibilityTimeout=visibility_timeout, 
) 
for msg in batch_messages: 
try: 
line = "{}n".format(json.dumps({ 
'attributes': msg.message_attributes, 
'body': msg.body, 
})) 
print("Line: ", line) 
filep.write(line) 
if purge_messages: 
print('Deleting message from the queue.') 
msg.delete() 
forwarded += 1 
except Exception as ex: 
print("Error in processing message %s: %r", msg, ex) 
counter += 1 
print('Batch %d processed', counter)

DLQ - Our 4ps
• Set a DLQ on each Lambda Func;on that can fail.
• Re-process events sent to DLQ with a custom script.
• Tune DLQ conﬁg directly from Lambda Func;on panel.

Conclusions
Why a serverless architecture?
• scalability - prevent data loss - full control on each step - costs
Open points:
• Integrate a custom CloudWatch dashboard.
• Conﬁgure Firehose for a Backup.
• Write a script that manages events sent to DLQs.
• Create a listener for anomaly detec;on with Kinesis Analy;cs.
• Amazon StepFunc;ons.

WE’RE HIRING!
clad.co/fullstack-dev

Useful links
These slides:
Create a serverless architecture for data collec4on with Python and AWS
—> hGp://clda.co/pycon8-serverless-data-collec;on
Blog post with code snippets:
Building a serverless architecture for data collec4on with AWS Lambda
—> hGp://clda.co/pycon8-data-collec;on-blogpost
Serverless Learning Path:
GeJng Started with Serverless Compu4ng
—> hGp://clda.co/pycon8-serverless-LP

Create a serverless architecture for data collection with Python and AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Create a serverless architecture for data collection with Python and AWS

Similar to Create a serverless architecture for data collection with Python and AWS (20)

Recently uploaded

Recently uploaded (20)

Create a serverless architecture for data collection with Python and AWS