Step by step instructions on how to create an AWS Lambda python function to get files from SFTP and save it to AWS S3

1. AWS Lambda
Scheduled file transfer sftp to s3
python

2. Steps
1. Create a new Administrator user in the IAM
2. Create a Role and allow Lambda execution and permissions for S3
operations
3. Take note of the User ARN
4. Deploy 64-bit Amazon Linux EC2 instance
5. SSH in, make project directory/folder
6. Create requirements.txt file, list python packages (except boto3)
7. Write the Handler
8. Zip up code and Dependencies
9. Create Lambda Function
10. Manually invoke the Lambda Function
11. Create the schedule

3. Handlerimport paramiko
import boto3
import datetime
import os
LOCALFILE='/tmp/invbatch.txt'
s3 = boto3.client('s3')
def connect_to_sftp():
transport = paramiko.Transport((os.environ['HOST'],int(os.environ['PORT'])))
transport.connect(username=os.environ['USERNAME'],
password=os.environ['PASSWORD'])
sftp = paramiko.SFTPClient.from_transport(transport)
return sftp
def handler(event, context):
sftp = connect_to_sftp()
sftp.get(os.environ['REMOTEFILE'], LOCALFILE)
now = datetime.datetime.today()
S3FILE='test/' + now.strftime('%Y%m%d-%H%M%S') + '_invbatch.txt'
s3.upload_file(Filename=LOCALFILE, Bucket='nufin-test-bucket', Key=S3FILE)

4. Makefile
Zip up Code and Dependencies
packageDependencies:
zip -r9 $(FUNCTION).zip sftp_to_s3.py
mkdir -p site-packages
virtualenv $(FUNCTION)
. $(FUNCTION)/bin/activate; pip install -r requirements.txt;
cd site-packages; cp -r $$VIRTUAL_ENV/lib/python2.7/site-packages/ ./;
cp -r $$VIRTUAL_ENV/lib64/python2.7/site-packages/ ./;
cd site-packages; zip -r9 ../../$(FUNCTION).zip .
PROJECT = sftp2s3
FUNCTION = $(PROJECT)
REGION = ap-southeast-1
ROLE = arn:aws:iam::864962608331:role/service-role/myLambdaRole
Makefile Variables

5. Makefile
Create the Lambda Function
createFunction:
aws lambda create-function
--region $(REGION)
--function-name $(PROJECT)
--zip-file fileb://$(FUNCTION).zip
--role $(ROLE)
--environment Variables=“{HOST='12.123.123.12',PORT=22,USERNAME='lhftp',PASSWORD='123456789012',REMOTEFILE='lhftp_uploads/invb
--handler sftp_to_s3.handler
--runtime python2.7
--profile Administrator
--timeout 10
--memory-size 1024

6. Makefile
manualInvoke:
aws lambda invoke
--invocation-type RequestResponse
--function-name $(FUNCTION)
--region $(REGION)
--profile Administrator
outputfile.txt
Manually Invoke the Lambda function to test it

7. Makefile
# Schedule the Lambda Function
# Step 1:
createRule:
aws events put-rule
--region $(REGION)
--name $(FUNCTION)-rule
--schedule-expression 'cron(0/5 8-17 ? * MON-FRI *)'
--profile Administrator
Schedule the Lambda function : Create the rule

8. Makefile
# $(ARNRULE) is taken from the put-rule target output and parsed in
# ARNRULE=arn:aws:events:.....
# Step 2:
addPermission:
aws lambda add-permission
--region $(REGION)
--function-name $(FUNCTION)
--statement-id $(FUNCTION)-event
--action 'lambda:InvokeFunction'
--principal events.amazonaws.com
--source-arn $(ARNRULE)
--profile Administrator
Schedule the function : Give permission to the event

9. Makefile
Schedule the function : Associate the Lambda with the Event
putTargets:
aws events put-targets
--region $(REGION)
--rule $(FUNCTION)-rule
--targets file://targets.json
--profile Administrator
[
{
"Id": "1",
"Arn":"arn:aws:lambda:ap-southeast-1:864962608331:function:sftp2s3"
}
]
targets.json