This document discusses using serverless architecture for real-time analysis. It describes using API Gateway and Lambda functions to process streaming data from Kinesis and index it into Elasticsearch. The architecture provides real-time analysis at a low cost by automatically scaling Lambda as event volume increases and decreases. Key benefits of the serverless approach include flexible scaling, increased efficiency, simplified deployment and management, and focusing only on code.

1. Serverless
Real-Time Analysis
Minyoung Jeong
CTO, The Beatpacking Company

2. Minyoung Jeong
CTO, The Beatpacking
Company
Founder, AWSKRUG
AWS Community Hero
Love Python!

3. Agenda
• Real-Time Analysis ?
• Why Serverless?
• Architecture Overview
• AWS Products

4. Real-Time
Analysis

5. BEAT
Ad supported Streaming Radio

6. To sell ads,
need Real-time analysis

7. To detect abusing,
use Real-time analysis.

8. What's Real-time?

9. Differences between Unit

10. Day, Hour, Minute

11. Day, Hour, Minute

12. Second

13. 충분히 적은 오차를 감수
지연을 줄이는 분석

14. Why Serverless?

15. –William of Occam
“Pluralitas non est ponenda sine neccesitate”

16. Everything seems nails when you hold a hammer?

17. Flexible by demand
Increase efficiency

18. Deploy, Scaling, Capacity
Fault tolerance,, ….

19. Works great with
Microservice Architecture

20. Service that covers smaller area
Quick develop, higher reliability

21. Monitoring
Load dispersion and
management

22. Forget everything other
than codes.

23. Forget everything other
than codes ?

24. Lambda has inconvenient
development model
Apex, Serverless

25. Architecture
Overview

33. Peak: 1K RPS
Avg: 300 RPS

34. API Gateawy+Lambda+ES
<= $400/m
< m4.2xlarge 1ea/m

35. API Gateway

36. REST Endpoint
Auth & Cache &
Monitoring

37. HTTP Proxy
Lambda, AWS Service

41. By using Cloufront,
reduce latency and
prevent DDOS

42. Automatically expands
within designated Range,
then throttling works.

43. Utilizing Stage feature,
Test/Verification

45. Request/Response Template

46. Kinesis

47. Platform for streaming
data on AWS

48. Managed Kafka
Queue/ETL/Analysis

49. Scale In/Out by Shard
Explore forward, backward, and
certain point of stream
Configurable Retention Period

50. Stream
Shard
Shard
Shard
Shard

51. TRIM_HORIZON LATESTAT(AFTER)_SEQUENCE

52. Connect various consumers to one stream
Invocation is available within retaining
period
Idempotence

53. Lambda Integration(w/Event Source)
Provide KCL/KPL SDK
REST API

54. Lambda

55. Code in the Cloud

56. Execute by receiving event and
context
CRON/AWS Event/API Gateway
Cloudwatch Log & Monitoring

58. Set only memory and execution
timeout
Autoscale by throughput
Charge by 100ms

59. Throtting and latency spike exist
Execute invocation and assure idempotence

60. event

61. function endpointCollect(event, context) {
// api-gateway context
let apiContext = event.context;
event.body._context = {
es_name: apiContext['es-name'],
es_index: apiContext['es-index'],
es_doc_type: apiContext['es-doc-type'],
s3_bucket: apiContext['s3-bucket'],
s3_prefix: apiContext['s3-prefix']
}
putStream(apiContext['stream-name'],
event).then((resolve, reject) => {
context.succeed();
}).catch((err) => {
context.fail(err);
});
}

62. function putStream(streamName, event) {
// inbound from AWS API Gateway
let injected = injectContext(event);
let record = {
Data: JSON.stringify(injected),
PartitionKey: `${event.body.timestamp}`,
StreamName: streamName
};
return new Promise((resolve, reject) => {
kinesis.putRecord(record, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}

63. const ESEndpoint = new AWS.Endpoint(es_host);
const req = new AWS.HttpRequest(ESEndpoint);
req.method = 'POST';
req.body = body;
req.path = '/_bulk';
req.region = 'ap-northeast-1';
req.headers['presigned-expires'] = false;
req.headers['Host'] = ESEndpoint.host;
if (AWSCredential.accessKeyId) {
const signer = new AWS.Signers.V4(req, 'es');
signer.addAuthorization(AWSCredential, new Date());
}

64. Demo

65. Thanks!