오라클 클라우드에서는 카프카를 관리형 서비스로 제공합니다. 밋업 세션에서는 관리형 카프카 서비스의 편의성을 소개하고 카프카 서비스의 데모를 진행합니다. 또한 MSA, 빅데이터 및 Blockchain의 인프라로 카프카가 핵심 위치를 갖는 것 뿐만 아니라 오라클 클라우드의 통합 핵심 컴포넌트로 카프카는 중요한 의미를 갖습니다. 오라클 클라우드의 통합 컴포넌트로 카프카의 역할과 주요 서비스의 구성을 소개합니다. * 본 세션은 “입문자/초급자/중급자” 분들께 두루 적합한 세션입니다.

1. 1
Managed Kafka Service
Oracle Event Hub CS
6th Oracle Developer Meetup

2. 2
+ =
Managed Kafka Service
Oracle Event Hub CS

3. 3
About: Me
• 1999 ~ 2006 Java (SI, Developer)
• 2007 ~ 2013 Middleware (Oracle, SC)
– WAS, Memory Cache, CEP, DB Replication
• 2013 ~ 2014 NoSQL, CouchDB (SE)
• 2015 ~ Big Data, ML, Deep Learning(SC)
민수
수학
Analysis
컨텐츠

4. 4
Compute
Networking
Storage
Load Balancing
Edge Services
Container
Container Pipline
FastConnect
Ravello
Compute Classic
Networking Classic
Storage Classic
Infrastructure Service Platform Service
Database
Database Backup
Big Data
Big Data Cloud
Event Hub
MySQL
Autonomous NoSQL Database Cloud
Data Hub
Autonomous Data Warehouse Cloud
Data Management Application Development
Java
Application Container
Mobile and Chatbots
Autonomous Visual Builder Cloud
AI Platform
Blockchain
Developer
Messaging
API Catalog
Autonomous Integration Cloud
Oracle Integration Cloud
Data Integration
Internet of Things
API Management
Self-Service Integration
Self-Service Integration
SOA
Integration
Process Automation
Application Performance Monitoring
Infrastructure Monitoring
Log Analytics
Orchestration
IT Analytics
Management
Content and Experience
Content and Experience
WebCenter Portal Cloud
DIVA Cloud
Business Analytics
Autonomous Analytics Cloud
Analytics Cloud
Business Intelligence
Data Visualization
Essbase
Security
CASB
Identity
Configuration and Compliance
Security Monitoring and Analytics

5. 5
Compute
Networking
Storage
Load Balancing
Edge Services
Container
Container Pipline
FastConnect
Ravello
Compute Classic
Networking Classic
Storage Classic
Infrastructure Service Platform Service
Database
Database Backup
Big Data
Big Data Cloud
Event Hub
MySQL
Autonomous NoSQL Database Cloud
Data Hub
Autonomous Data Warehouse Cloud
Data Management Application Development
Java
Application Container
Mobile and Chatbots
Autonomous Visual Builder Cloud
AI Platform
Blockchain
Developer
Messaging
API Catalog
Autonomous Integration Cloud
Oracle Integration Cloud
Data Integration
Internet of Things
API Management
Self-Service Integration
Self-Service Integration
SOA
Integration
Process Automation
Application Performance Monitoring
Infrastructure Monitoring
Log Analytics
Orchestration
IT Analytics
Management
Content and Experience
Content and Experience
WebCenter Portal Cloud
DIVA Cloud
Business Analytics
Autonomous Analytics Cloud
Analytics Cloud
Business Intelligence
Data Visualization
Essbase
Security
CASB
Identity
Configuration and Compliance
Security Monitoring and Analytics

6. 6
Compute
Networking
Storage
Load Balancing
Edge Services
Container
Container Pipline
FastConnect
Ravello
Compute Classic
Networking Classic
Storage Classic
Infrastructure Service Platform Service
Database
Database Backup
Big Data
Big Data Cloud
Event Hub
MySQL
Autonomous NoSQL Database Cloud
Data Hub
Autonomous Data Warehouse Cloud
Data Management Application Development
Java
Application Container
Mobile and Chatbots
Autonomous Visual Builder Cloud
AI Platform
Blockchain
Developer
Messaging
API Catalog
Autonomous Integration Cloud
Oracle Integration Cloud
Data Integration
Internet of Things
API Management
Self-Service Integration
Self-Service Integration
SOA
Integration
Process Automation
Application Performance Monitoring
Infrastructure Monitoring
Log Analytics
Orchestration
IT Analytics
Management
Content and Experience
Content and Experience
WebCenter Portal Cloud
DIVA Cloud
Business Analytics
Autonomous Analytics Cloud
Analytics Cloud
Business Intelligence
Data Visualization
Essbase
Security
CASB
Identity
Configuration and Compliance
Security Monitoring and Analytics

7. 7
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL Is Dead, Long Live Streams:
real-time streams w/ Apache Kafka

8. 8
Apps Apps Apps
NoSQL
Search
RDBMS
Monitoring
RT
Analysis
DWH
Hadoop
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL Is Dead, Long Live Streams:
real-time streams w/ Apache Kafka

9. 9
Apps Apps Apps
NoSQL
Search
RDBMS
Monitoring
RT
Analysis
DWH
Hadoop
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL Is Dead, Long Live Streams:
real-time streams w/ Apache Kafka

10. 10
Producers
Consumers

11. 11
Producers
Consumers
Stream
APIs

12. 12
Producers
Stream
APIsConnectors
Consumers

13. 13
나의 아저씨

14. 14
나의 카프카!
1
https://youtu.be/I32hmY4diFY?t=500

15. 15
나의 카프카!
Databus (Lambda/Kappa)
Data Vault Model 2.0MSA Big Data

16. 16
In Oracle Cloud
Self Managed Platform Managed
Apache Kafka on IAAS Compute Event Hub Cloud Service (Dedicated) Event Hub Cloud Service (Topics)
$0.06 OCPU/hour
(69원)
$0.27 OCPU/hour
(290원)

17. 17
Step 01: 클러스터 생성

18. 18

19. 19

20. 20
2018. 5. 17 오전 11시 35분 33초 UTC Activity Submitted
2018. 5. 17 오전 11시 35분 33초 UTC Activity Started
2018. 5. 17 오전 11시 35분 33초 UTC Started operation to create service [demokafka] in identity domain [idcs-c69311285cdb4062b8455d9a1fcb18c8].
2018. 5. 17 오전 11시 42분 32초 UTC Completed creating service [demokafka] resources in domain [idcs-c69311285cdb4062b8455d9a1fcb18c8].
2018. 5. 17 오전 11시 42분 37초 UTC Started operation to establish security on VMs for component [connect].
2018. 5. 17 오전 11시 42분 38초 UTC Started operation to establish security on VMs for component [restprxy].
2018. 5. 17 오전 11시 42분 38초 UTC Completed operation to establish security on VMs for component [restprxy]
2018. 5. 17 오전 11시 42분 47초 UTC Started operation to establish security on VMs for component [kafka].
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [kafka]
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [restprxy]
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [connect]
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [restprxy] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [kafka] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [connect] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [restprxy].
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [kafka].
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [connect].
2018. 5. 17 오전 11시 44분 07초 UTC Successfully reached demokafka-connect-1 at port 8083. Successfully reached demokafka-connect-1 at port 1080.
2018. 5. 17 오전 11시 44분 07초 UTC Successfully completed service reachability check for [connect]
EHCS 인스턴스 생성 로그:
5 node, 11분 소요

21. 21

22. 22
Oracle Event Hub Service 아키텍처
OCI Classic
OCI

23. 23
OEHCS 클러스터| REST Proxy
REST Proxy
AVRO Registry
Monitoring App
REST Proxy
AVRO Registry
Monitoring App
REST Proxy
AVRO Registry
Monitoring App
User
1080
Kafka
brokers
Kafka
brokersKafka
brokers
• 포트 외부 오픈
• Zookeeper 2181 & JMX: 9098
• Kafka 6667 & JMX: 9097
Zookeeper
Zookeeper
Zookeeper
Nimbula Sec List - Kafka
6667

24. 24

25. 25
OEHCS 클러스터| Native API
Trusted
source IP
with in tenancy
6667Kafka
brokers
Kafka
brokers
Kafka
brokers
Zookeeper
ZookeeperZookeeper
OPC
Trusted
VM

26. 26
Step 02: Topic 관리

27. 27

28. 28

29. 29
Step 02-1: Topic 수정

30. 30
Step 02-1: Topic 수정

31. 31
Step 02-2: Topic 확장

32. 32
Step 02-2: Topic 확장

33. 33
Step 02-2: Topic 확장

34. 34
Step 02-3: Topic 모니터링

35. 35

36. 36
Step 03: Cluster 확장

37. 37
Step 03: Cluster 확장

38. 38
Oracle PaaS Service Manager
Oracle Cloud Infrastructure
REST
API
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
REST
API
OCI CLI
Compute Network Storage DNS Load Balancing Security
Web UI Chef Terraform Ansible
PaaS Service Manager(PSM)
Oracle PaaS Service Manager 소개 및 설치: http://taewan.kim/post/installing_pms/

39. 39
PaaS Service Manager
https://docs.oracle.com/en/cloud/paas/java-cloud/pscli/oehcs-create-service.html

40. 40
Oracle PaaS Service Manager
$ sudo -H psm oehpcs help
DESCRIPTION
Oracle Event Hub Cloud Service - Dedicated
SYNOPSIS
psm OEHPCS <command> [parameters]
AVAILABLE COMMANDS
o services o service
o create-serviceo delete-service
o restart-service o scale-out
o stop o start
o restart o add-storage
o scale o available-patches
o applied-patches
o patch o precheck-patch
o rollback
o check-health
o operation-status o activities
o access-rules o create-access-rule
o delete-access-rule o enable-access-rule
o disable-access-rule o add-ssh-public-key
o help

41. 41
Oracle Cloud SDK

42. 42
Flume
아파치 플룸(Apache Flume) : http://taewan.kim/post/flume_images/

43. 43
Flafka = Flume + Kafka

44. 44
https://www.confluent.io/product/connectors/
- Scalable Framework
- Open Source
- Only Configuration

45. 45
Step04: OEHCS의 Kafka Connector 지원

46. 46
OEHCS의 Kafka Connector 지원

47. 47
OEHCS의 Kafka Connector 지원 – OCI Object Storage

48. 48
OEHCS의 Kafka Connector 지원 – JDBC Sink

49. 49
OEHCS Roadmap
- Connector
- Sink
- Oracle DataHub: Cassandra
- Apache Hbase
- S3
- Elastic Search
- Source
- Twitter
- Management
- Auto Scale-out
- Multi Datacenter
- Audit
- Disk Storage 암호화
- PCI 인준

50. 50
Step05: Extension

51. 51
Step06: Patch
http://taewan.kim/post/oracloud_patch/

52. 52
Oracle Integrations
Confidential – Oracle 5
Oracle Application Container Cloud ServiceOracle Storage Cloud Service
Oracle Database Cloud Service Oracle Java Cloud Service
Oracle Golden Gate Cloud Service
Oracle Big Data Cloud - CE
Oracle IOT Cloud Service
Oracle Integration Cloud Service
Oracle Analytics Cloud Service

53. 53
Oracle Cloud – Data Lake
Storage Cloud Service (Data Lake)
Web
Crawler
Spark
Streaming
Event
hub
GPU
Deep
Learning
Identity Domain
• Upload CLI
• OSACS (비동기 복제)
BigData
Connector
SDK
REST API
Library
IaaS
Computing
Data Processing
Machine Learning
ADWC OAC
IaaS
Computing
On-Premise
Server
Oracle Software Appliance Cloud Service
Visualization

54. 54
Oracle Trial
http://taewan.kim/post/oracle_cloud_trial/

55. 55
실습 가이드
• Eventhub 소개
– http://taewan.kim/post/kafka_paas_eve
nthub_cs/
• Eventhub
– https://github.com/oracle/learning-
library/blob/master/workshops/journey
2-new-data-lake/LabGuide400.md
• Eventhub + Spark Streaming
– https://github.com/oracle/learning-
library/blob/master/workshops/journey
2-new-data-lake/LabGuide500.md

56. 56