PNNL uses AWS to enable collaborative research between researchers and engineers. This allows for faster delivery of research projects that meet researchers' needs. An example is described of building an image classification pipeline on AWS that retrieves images from S3, processes them through Lambda, and classifies them for researchers to access. The pipeline demonstrates how AWS services like S3, SNS, Lambda, and Nifi can work together to build flexible research tools while avoiding infrastructure management.

1. PNNL-SA-126544
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ralph Perko, Senior Software Engineer & Technical Team Lead for Computing and Analytics
Mike Giardinelli, Senior Software Engineer & Technical Team Lead for Computing and Analytics
June 11, 2017
How Pacific Northwest National Laboratory
Uses AWS to Enable Research

2. PNNL-SA-126544
“Invention requires two things: the ability to
try a lot of experiments, and not having to
live with the collateral damage of failed
experiments.”
— Andy Jassy, AWS CEO

3. PNNL-SA-126544
Who We Are
• Mike Giardinelli – Senior Software Engineer & Technical Team Lead for
Computing and Analytics
• Ralph Perko – Senior Software Engineer & Technical Team Lead for
Computing and Analytics

4. PNNL-SA-126544
Agenda Outline
• Introduction
• PNNL Overview
• Software Engineering at PNNL
• Data Flows
• Research example
• Wrap Up
• Questions?

5. PNNL-SA-126544
The National Laboratory System

6. PNNL-SA-126544
PNNL at a Glance

7. PNNL-SA-126544
PNNL Leverages its Science Base to Provide National
Leadership
UNCLASSIFI
ED

8. PNNL-SA-126544
Software Engineering at PNNL
• Staff focus is research and innovation, not operations
• Developers work with scientists to enable research
• Limited space and resources for hardware
• Big driver for moving to AWS!

9. PNNL-SA-126544
Problem: Isolated Research
• Researchers work independently
• Lack of collaboration with engineers
• Long delivery times
• Product usually wasn’t what customer had envisioned

10. PNNL-SA-126544
Solution: Collaborative Research

11. PNNL-SA-126544
Data Flow
Raw

12. PNNL-SA-126544
Data Flow “Toolbox”

13. PNNL-SA-126544
Use Case - Image Classification Pipeline
The goal is to provide:
• An image retrieval pipeline
• Image storage repository
• Mechanism for distributing images to classifier
• Storage for image classifications and image reference
• A means for other collaborators to access the system

14. PNNL-SA-126544
Image Retrieval and Classification

15. PNNL-SA-126544
Image Retrieval and Classification
Engineering Research
Collaboration

16. PNNL-SA-126544
Image Retrieval and Classification – Apache Nifi

17. PNNL-SA-126544
Nifi Overview
• Message/data routing is very flexible and robust
• ETL is painless
• Easy to install, scale, configure, and extend
• Visually see what is going on with your pipelines
(metrics/bottlenecks)
• Backpressure and queueing are baked into the flows –
excellent for systems that have brittle end points
• Low barrier to entry, broadens user audience

18. PNNL-SA-126544
Image Cache Data Flow

19. PNNL-SA-126544
Nifi Logic
{"image":"http://somewebsite.com/puppies.jpeg"},{"image":"http://
somewebsite.com/kittens.jpeg"},{"image":"http://somewebsite.com/k
oalas.jpeg"}
Read data from Kafka topic
Filter
“We only care about Koala bears!”
Create new message
and push to Amazon
SNS
{
"url":"http://somewebsite.com/koalas.jpeg",
"hash" : "092f6b17d186adb2e121afcdc7e5470b0c6f82a5",
"name" : "koalas.jpeg",
"type" : "jpeg",
"bucket” : “image-classifier-test”
}
{"image":"http://somewebsite.com/koalas.jpeg"}

20. PNNL-SA-126544
Image Retrieval and Classification – Amazon SNS

21. PNNL-SA-126544
Example Incoming Amazon SNS Message
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:EXAMPLE",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": null,
"Signature": "EXAMPLE",
"SigningCertUrl": "EXAMPLE",
"MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
"Message": "{"hash": "092f6b17d186adb2e121afcdc7e5470b0c6f82a5",
"url": "http://somewebsite.com/koalas.jpeg", "bucketname": "image-
classifier-test", "name" : "koalas.jpeg", "type" : "jpeg",}",
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
},

22. PNNL-SA-126544
Image Retrieval and Classification – AWS Lambda

23. PNNL-SA-126544
AWS Lambda

24. PNNL-SA-126544
Lambda Function
• Written in Java
• Use Standard Libraries
• Apache HTTP Components to download image
• ImageMagick to convert and strip image
public void handleRequest(SNSEvent snsEvent,
Context context) throws Exception {…}

25. PNNL-SA-126544
Image Retrieval and Classification – Collaboration
Engineering Research
Collaboration

26. PNNL-SA-126544
Collaboration

27. PNNL-SA-126544
Researchers and Engineers Are Happy

28. PNNL-SA-126544
Thanks!
Cindy Hewitt – AWS
Jed Sundwall – AWS
Justin Almquist – PNNL
Court Corley – PNNL
Michael Henry – PNNL
Donald Jorgensen - PNNL
Ben Killinger – PNNL
Shalonne Luke - PNNL
Steve Rysavy – PNNL

29. PNNL-SA-126544
Thank you!