The TCP buffer size was too small for the SSL handshake. Increasing the TCP send/receive buffer sizes on the listeners resolved the issue.

1. Experience in implementing SSL
between Oracle DB and Oracle Clients

2. Agenda
● Project overview
● SSL session (1 way/2 ways)
● Certificate Authorities & Certificates
● 3 Different Oracle Listeners configurations (insecure/secure/ssl)
● SSL & Oracle Clients setup (tns configuration parameters)
● Issues faced & troubleshooting tools
● Conclusion
● Q & A

3. RAC Cluster
Project overview
SSL Listener
SEC Listener
SCAN Listeners
Developers & End
users / tools directly
connecting to DB
Main application
technology stack
OBIEE, WLS, Forms,
Reports
Integrations
Other DBs, Essbase,
Ora Net Clients
SSL Listener
SEC Listener
SSL Listener
SEC Listener

4. How does SSL/TLS work?
How does SSL/TLS work? from http://security.stackexchange.com/ => here
How does SSL work? What is an SSL handshake? from http://www.symantec.com/ => here
Transport Layer Security from https://en.wikipedia.org/ => here

5. SSL session overview (1 way)
ServerInitial request (to https://domainname.com)
Hello .... SSLv2, SSLv3, TLSv1, .... ciphers, hash function
● Server Certificate
● Server Private Key
Server Certificate (signed by Certificate Authority)
Public Key + Domain Name (subject) + Org name ....● List of Trusted
CA certificates
Establish trust ...
Agree on the symmetric encryption key ...
premaster secret
master secret & generate session keys
Use session key and agreed encryption cipher to
send data ...

6. SSL session overview (2 way)
Server
● Server Certificate
● Server Private Key● List of Trusted
CA certificates
Establish trust ...
Agree on the symmetric encryption key ...
master secret & generate session keys
Use session key and agreed encryption cipher to
send data ...
premaster secret
● Client Certificate
● Client Private Key
● List of Trusted
CA certificates

7. ● There are 3 options to obtain an SSL certificate
a. Generate your own self-signed certificate (from https://www.linux.com here)
b. Request certificate from your organisation CA
c. Request certificate from public CA
■ VeriSign
■ GeoTrust
■ ... https://en.wikipedia.org/wiki/Certificate_authority
● In A & B you or your organisation controls hashing and encryption protocols
● However no clients will recognize certificate by default
● In option C you will need to make sure your client versions supports protocols
dictated by external CA
a. Recent example => Many CAs announced that they will not support SHA-1 forcing SHA256/SHA-2
implementation
Certificate Authorities & Certificates

8. ● Intermediate certificate authorities
○ https://en.wikipedia.org/wiki/Intermediate_certificate_authorities
Certificate Authorities & Certificates

9. ●
Certificate Authorities & Certificates

10. ●
Certificate Authorities & Certificates

11. ●
Certificate Authorities & Certificates

12. Oracle

13. Listener Config

14. 3 Different Oracle Listeners configurations
● Default Listener (insecure)
○ Username/password are encrypted
● Secure Listener
○ Use encryption the same way as SSL Listener
○ No authentication phase (SSL handshake)
● SSL Listener
○ Use encryption
○ Could be configured with 1 way or 2 ways authentication

15. Default Listener (insecure)
listener.ora
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1521))
(ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER))
)

16. Secure Listener
listener.ora
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1522))
(ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER))
)
sqlnet.ora
sqlnet.crypto_checksum_server = REQUIRED
sqlnet.encryption_server = REQUIRED
sqlnet.crypto_checksum_types_server = (SHA1)
sqlnet.encryption_types_server = (AES256)

17. listener.ora
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL=TCPS)(HOST = db_host)(PORT = 1523))
(ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER))
)
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /u01/app/oracle/SSL/certs)
)
)
SSL_CLIENT_AUTHENTICATION = FALSE
SSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)
SSL Listener

18. oracle@host > openssl s_client -connect db_host:1523
CONNECTED(00000003)
depth=0 CN = PROD
...
verify return:1
Certificate chain
0 s:/CN=PROD
i:/CN=PROD
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBkDCB+gIBADANBgkqhkiG9w0BAQQFADARMQ8wDQYDVQQDEwZFQlNMTlAwHhcN
MTQwNzAyMDQzNTI4WhcNNDExMTE2MDUzNTI4WjARMQ8wDQYDVQQDEwZFQlNMTlAw
gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAJGbN1hxmtFKx1SpUU8JPvRyjMzW
...
-----END CERTIFICATE-----
subject=/CN=PROD
issuer=/CN=PROD
---
Acceptable client certificate CA names
/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority
/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root
/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification Authority
/C=US/O=VeriSign, Inc./OU=Class 2 Public Primary Certification Authority
/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
SSL handshake has read 1269 bytes and written 210 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
...
SSL & Oracle Clients setup

19. SSL & Oracle Clients setup
...
---
SSL handshake has read 1269 bytes and written 210 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID:
Session-ID-ctx:
Master-Key: 68E402B5CD161618522350EE...
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1447632207
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
---
oracle@host >

20. Client Config

21. == TCP ==========================================================================
tnsnames.ora
PROD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 1523))
)
(CONNECT_DATA =
(SERVICE_NAME = PROD_ETL01)
)
)
== TCPS =========================================================================
tnsnames.ora
PROD_SSL =
(DESCRIPTION =
(SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia"))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048))
)
(CONNECT_DATA =
(SERVICE_NAME = PROD_ETL01)
)
)
SSL & Oracle Clients setup (OCI)

22. == TCPS ==========================================================================
sqlnet.ora
SSL_SERVER_DN_MATCH = yes
SSL_CLIENT_AUTHENTICATION = FALSE
SSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /u01/app/oracle/SSL/cert)
)
)
SSL & Oracle Clients setup (OCI)

23. ● java … -Doracle.net.ssl_server_dn_match=true …
● jdbc:oracle:thin:@
(DESCRIPTION =(
SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia"))
(ADDRESS_LIST = (ADDRESS =
(PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)
))(CONNECT_DATA = (SERVICE_NAME = PROD_ETL01)))
● Java has it own certificate store
SSL & Oracle Clients setup (JDBC)

24. Certificates Config

25. SSL & Oracle Clients setup (OCI)
Using the orapki Utility to Manage PKI Elements
https://docs.oracle.com/database/121/DBSEG/asoappf.htm#DBSEG610

26. $ ls -l /u01/app/oracle/SSL/cert/*
total 16
-rw-r--r--@ 1 yvel 5000 1365 Aug 21 2014 cwallet.sso
-rw-r--r--@ 1 yvel 5000 1288 Aug 21 2014 ewallet.p12
oracle@host:/home/oracle> orapki wallet display -wallet /u01/app/oracle/SSL/cert ; date
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Subject: CN=PROD
Trusted Certificates:
Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign, Inc.,C=US
Subject: OU=Equifax Secure Certificate Authority,O=Equifax,C=US
Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign, Inc.,C=US
Subject: CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US
Subject: OU=Secure Server Certification Authority,O=RSA Data Security, Inc.,C=US
Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign, Inc.,C=US
Wed Sep 9 16:27:36 PDT 2015
oracle@kpfp2:/home/oracle>
orapki wallet create -wallet $WALLET -auto_login -pwd $v_pwd
orapki wallet remove -wallet $WALLET -trusted_cert_all -pwd $v_pwd
orapki wallet display -wallet $WALLET -pwd $v_pwd
orapki wallet jks_to_pkcs12 -wallet $WALLET -keystore /full/path/to/certificate.jks -jkspwd <pwd>
orapki wallet add -wallet $WALLET -trusted_cert -cert GIAG2.crt -pwd $v_pwd
SSL & Oracle Clients setup (OCI)

27. SSL & Oracle Clients setup (OCI)
$ openssl pkcs12 -info -in /u01/app/oracle/SSL/cert/ewallet.p12
Enter Import Password:
MAC Iteration 1024
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 1024
Certificate bag
Bag Attributes
localKeyID: E6 B6 52 DD 00 00 00 04 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 04
subject=/C=US/O=MyOrg Inc/CN=MyOrg Internet Authority G2
issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTUwNDA0MTUxNTU1WjBJMQswCQYDVQQG
EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
...
PKCS 12 => https://en.wikipedia.org/wiki/PKCS_12
It defines an archive file format for storing many cryptography objects as a single file. It is
commonly used to bundle a private key with its X.509 certificate.

28. ● JDBC Clients
○ Have it own default certificate store with preloaded trusted CA list
$ v_java_cert=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts
$ keytool -list -keystore $v_java_cert -storepass changeit
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 93 entries
digicertassuredidrootca, Apr 16, 2008, trustedCertEntry,
Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43
comodorsaca, May 12, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): AF:E5:D2:44:A8:D1:19:42:30:FF:47:9F:E2:F8:97:BB:CD:7A:8C:B4
thawtepremiumserverca, May 26, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): E0:AB:05:94:20:72:54:93:05:60:62:02:36:70:F7:CD:2E:FC:66:66
...
$ keytool -exportcert -alias digicertassuredidrootca -keystore $v_java_cert -storepass changeit -
file test.crt -rfc
$ keytool -printcert -file test.crt
Owner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Issuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Serial number: ce7e0e517d846fe8fe560fc1bf03039
Valid from: Thu Nov 09 16:00:00 PST 2006 until: Sun Nov 09 16:00:00 PST 2031
Certificate fingerprints:
MD5: 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72
SSL & Oracle Clients setup (JBDC)

29. SSL & Oracle Clients setup (JDBC)
● A JRE client must be updated with the JCE to enable
TLS_RSA_WITH_AES_256_CBC_SHA cipher.
○ Download appropriate JCE archive with 2 jar files
○ Copy the jar files to $JAVA_HOME/jre/lib/security/ directory
● The JCE could be obtained from the URL below depending on a JRE version
○ 1.6 JCE
○ 1.7 JCE
○ 1.8 JCE

30. Issues faced

31. RAC Cluster
Variety of clients and versions to cover ...
SSL Listener
SEC Listener
SCAN Listeners
Developers & End
users using tools
directly connecting to
DB
Main application
technology stack
OBIEE, WLS, Forms,
Reports
Integrations
Other DBs, Essbase,
Ora Net Clients
SSL Listener
SEC Listener
SSL Listener
SEC Listener
1. 2.3.

32. Some components used old Oracle Clients
● Essbase
○ Blend in 11.1.0.7 Oracle Client
○ Challenging to update
● OBIEE
○ Presentation layer use 11.1.0.7 Oracle Client

33. SSL Listener & TCP Buffer
Problem definition
Initial:
● DB Instances doesn't register DB Services with a remote SCAN
listeners, after a new certificate has been added to server wallet
Current:
● All SSL connections from one node hangs connecting to any SSL
enabled listeners that runs on a remote node if a listener is able to
send data out quickly enough

34. SSL Listener & TCP Buffer
SSL Listener works if ….
a) strace-ing the listener process
b) turning debug logging up to a high level in the listener
c) Running the listener on a non-bonded interface
d) Having the listener increase its send buffer size from 16k (default)
to 32k or 64k on the socket
d1) We can also do this at the OS level via `echo "4096 32768 4194304" >
/proc/sys/net/ipv4/tcp_wmem'

35. SSL Listener & TCP Buffer
Normal / Expected processing
sPORT:sIP - cIP:cPORT
TCP socketTCP socket buffer
Process
owner of the socket
O_NONBLOCK
W3 W2 W1
DATA
Data writes
O_NONBLOCK
sPORT:sIP - cIP:cPORTProcess
owner of the socket
DATA
Kernel waits a bit for more data and sends it out
Buffer is full. Kernel sends EAGAIN back
5k 1k 3k
3k
W3b
2k
W3bEAGAIN
sPORT:sIP - cIP:cPORTProcess
owner of the socket
The process code handles error and keep sending the rest of the data until successful
3k
W3b
3k
Kernel refuses writes until buffer is free

36. Oracle Listener - fails to process EAGAIN
TCP socketTCP socket buffer
O_NONBLOCK
Data writes
O_NONBLOCK
sPORT:sIP - cIP:cPORTSSL Listener DATA
Buffer is full. Kernel sends EAGAIN back
3k
W3b
2k
W3bEAGAIN
sPORT:sIP - cIP:cPORTSSL Listener
Listener never re-sends the W3b part Kernel ready to process writes after some time
Oracle Listener ssl handshake function fails to process EAGAIN errors
sPORT:sIP - cIP:cPORTSSL Listener DATA
Buffer is full. Kernel sends EAGAIN back
2k
W3b
3k
W3b
The W3b part is dropped
Where is client's
response?
W3b
The W3b part is dropped
SSL Listener & TCP Buffer

37. Conclusions

38. RAC Cluster
Variety of clients and versions to cover ...
SSL Listener
SEC Listener
SCAN Listeners
Developers & End
users using tools
directly connecting to
DB
Main application
technology stack
OBIEE, WLS, Forms,
Reports
Integrations
Other DBs, Essbase,
Ora Net Clients
SSL Listener
SEC Listener
SSL Listener
SEC Listener

39. Q & A