This document discusses using Oracle Cloud Infrastructure (OCI) Object Storage. It provides examples of using the OCI CLI, Java SDK, and HDFS connector to upload, download, and access files in OCI Object Storage buckets. It also briefly mentions using Kafka connectors to stream data between Kafka and Object Storage.

1. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted
Oracle Cloud
Data Interface
김태완
@taewanme
https://taewan.kim

2. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
데이터 라이프 사이클
2
Ingestion
streaming
batch
replication
Data Store
Object store data
lake
Hadoop/HDFS
NOSQL/Cassandra
Transform
Data pipelines
ML pipelines
Data quality
Data Augmentation
Analyze
Python/R
Spark
Hive
Interactive BI
Learn
Machine learning
AI
Data science

3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Oracle Cloud
Data Lake
Oracle Object Storage
BigData Cloud
Event Hub Cloud
DIPC
Database Data Hub IaaS VM
NoSQL
OAC
ADW
*ADW: Autonomous Data warehouse
*DIPC: Data Integration Platform Cloud
* OAC: Oracle Analytics Cloud
Object Storage

4. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 4
OCI Object Storage
Case 01: Desktop & OCI Object Storage

5. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Object Storage 특징
5

6. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage
6
참조: http://bit.ly/2Z0lyA3

7. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage: bucket
7
참조: http://bit.ly/2Z0lyA3

8. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage: Connection Info
8
참조: http://bit.ly/2Z0lyA3

9. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage: Connection Info
9
참조: http://bit.ly/2Z0lyA3

10. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage: CloudBerry S3 Explorer
10
테넌트이름.compat.objectstorage.리전이름.oraclecloud.com
참조: http://bit.ly/2Z0lyA3

11. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OCI Object Storage: CloudBerry S3 Explorer
11
참조: http://bit.ly/2Z0lyA3

12. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 12
OCI Object Storage
현재 위치

13. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 13
OCI Object Storage
Case 02: OCI CLI & OCI Object Storage
oci cli

14. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설치
14
참조: http://bit.ly/2Z0lyA3
bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)"
Linux / Mac OS
powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object
System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/oracle/oci-
cli/master/scripts/install/install.ps1'))"
Window

15. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설치 확인
15
참조: http://bit.ly/2Z0lyA3

16. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli command
16

17. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: info
17
참조: http://bit.ly/2Z0lyA3
user OCID

18. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: info è Tenancy OCID, Region ID
18

19. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: oci setup config
19

20. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: key files
20

21. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: public key 등록
21

22. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: public key 등록
22

23. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: public key 등록

24. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli 설정: Connection Test

25. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: Bucket 생성
25
oci cli os bucket
--name
--compartment-id

26. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: Bucket 생성
26

27. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: 파일 업로드
27

28. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: 파일 업로드
28

29. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 29
OCI Object Storage
현재 위치
oci cli

30. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 30
OCI Object Storage
Case 03: 대용량 파일 업로드
oci cli

31. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: 데모 파일 생성
31

32. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: 10GB 파일 업로드
32

33. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
oci cli: 10GB 파일 업로드
33

34. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 34
OCI Object Storage
현재 위치
oci cli

35. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 35
OCI Object Storage
Case 04: Java를 이용한 파일 업로드 다운로드
OCI Java SDK

36. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK
36
https://search.maven.org/search?q=g:com.oracle.oci.sdkhttps://github.com/oracle/oci-java-sdk

37. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: Maven
37

38. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: Code
38

39. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: Code
39

40. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: Connection
40

41. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: Bucket 생성
41

42. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI Java SDK: 파일 추가
42

43. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 43
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK

44. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 44
OCI Object Storage
Case 05: OCI SDK & Object storage

45. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK & Rest API
45
https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type
=&language=

46. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK & Rest API
46
https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type
=&language=

47. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Python: 설치
47

48. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
48

49. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
49

50. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
50

51. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Python: Object 생성 결과
51

52. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle OCI SDK Go-lang: Install
52
> go get -u github.com/oracle/oci-go-sdk
import (
"github.com/oracle/oci-go-sdk/common"
"github.com/oracle/oci-go-sdk/identity”
)
https://github.com/oracle/oci-go-sdk

53. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 53
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK

54. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 54
OCI Object Storage
Case 06: Hadoop & Object Storage
HDFS Connector
for Object Storage

55. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 55
Hadoop & Object Storage
https://github.com/oracle/oci-hdfs-connector/releases

56. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 56
HDFS Connector for OCI: 설치
https://github.com/oracle/oci-hdfs-connector/releases
속성 타입 설명
fs.oci.client.auth.tenantId String tenancy의 OCID
fs.oci.client.auth.userId String 사용자 OCID
fs.oci.client.auth.fingerprint String 사용자 Fingerprint
fs.oci.client.auth.pemfilepath String PEM 파일 위치
fs.oci.client.auth.passphrase String 암호화 문자열
core-site.xml

57. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 57
HDFS Connector for OCI: 설치
scala> sc.wholeTextFiles("oci://PipedUploadTest@internalbriangustafson/")

58. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 58
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

59. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 59
OCI Object Storage
Case 07: Kafka & Object Storage

60. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 60
OCI Object Storage
Case 07: Kafka & Object Storage
Apps
NoSQL
Search
RDBMS
Monitoring
RT Analysis
DWH
Hadoop

61. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
+ =
Managed Kafka Service
Oracle Event Hub CS

62. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

63. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

64. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

65. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

66. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

67. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |

68. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
https://www.confluent.io/product/connectors/
- Scalable Framework
- Open Source
- Only Configuration

69. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
OEHCS의 Kafka Connector 지원

70. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Streaming Data + Database
Oracle
GoldenGate
추출 트랜잭션
추출 실시간 이관
JMS
Capture Trail Pump Route Deliver

71. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 71
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

72. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Case 10: ADW와 Object Storage
72

73. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Object Storage에 데이터 저장
73
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")

74. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Object Storage에 데이터 뷰
74
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
$ parquet-tools schema sales_extended.parquet
message hive_schema {
optional int32 prod_id;
optional int32 cust_id;
optional binary time_id (UTF8);
optional int32 channel_id;
optional int32 promo_id;
optional int32 quantity_sold;
optional fixed_len_byte_array(5)
amount_sold (DECIMAL(10,2));
optional binary gender (UTF8);
optional binary city (UTF8);
optional binary state_province (UTF8);
optional binary income_level (UTF8);
}

75. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Object Storage에 데이터 뷰
75
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
$ parquet-tools schema sales_extended.parquet
message hive_schema {
optional int32 prod_id;
optional int32 cust_id;
optional binary time_id (UTF8);
optional int32 channel_id;
optional int32 promo_id;
optional int32 quantity_sold;
optional fixed_len_byte_array(5)
amount_sold (DECIMAL(10,2));
optional binary gender (UTF8);
optional binary city (UTF8);
optional binary state_province (UTF8);
optional binary income_level (UTF8);
}
https://github.com/Guavus/parquet-mr-
1/tree/master/parquet-tools

76. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Object Storage에 데이터 뷰
76
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
https://github.com/Guavus/parquet-mr-
1/tree/master/parquet-tools

77. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
ADW의 External Table
77

78. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
ADW와 Parquet External Table
78
External Table

79. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 79
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

80. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Case 11: 다양한 인터페이스? è 데이터 가상화
80
Hadoop Security
ACL’s | Sentry | HDFS Encryption | Encryption in Motion
Big Data SQL Cloud Service

81. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Oracle Cloud
Data Lake
Oracle Object Storage
BigData Cloud
Event Hub Cloud
DIPC
Database Data Hub IaaS VM
NoSQL
OAC
ADW
*ADW: Autonomous Data warehouse
*DIPC: Data Integration Platform Cloud
* OAC: Oracle Analytics Cloud
Object Storage

82. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Oracle Data Platform
82

83. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 83