SlideShare a Scribd company logo
Who broke my crypto
Nikhil Prathapani
Enterprise Routing – SDWAN group
Cisco Systems Inc
Things you WON’T learn in this talk
• Crypto currency
• Bitcoin
• Blockchain
Agenda
• Chapter 0 – The Problem
• Chapter 1 – The Puzzle
• Chapter 2 – The Chase
• Chapter 3 – The Eureka
• Chapter 4 – The End
On a fine Monday morning:
I tried to SSH to my EC2 instance, but it kept bailing out on me.
Chapter 0 – The Problem
• Why am I unable to SSH to an instance which worked fine until Friday.
• I listed out the things changed from my end:
• EC2 Instance type: unchanged, not even touched since Friday
• Host machine : Same host machine - RHEL instance
Chapter 1 – The Puzzle
Oh I know how to debug this.
Its simple:
• Instead of SSH, just add –vvv for further debug.
• ssh -v will tell you what is happening mostly on your end
• ssh -vv will tell you low level on both ends
• ssh -vvv will tell you almost everything from both ends.
Chapter 2 – The Chase
Contacted AWS support.
A very patient support rep helped me debug the issue further
Step 0: SSH with "-vvv" flag for verbosity
I did that, didn’t help. Still lost connection.
Step 1: Create packet capture
$ sudo -i
# tcpdump -i any -w /tmp/$(hostname)_capturefile.cap -s256 port 22 &
# ssh -vvv user@<elastic_ip_ec2_instance>
# killall tcpdump ; pkill tcpdump
# zip -9 /tmp/$(hostname)_capturefile.cap.zip /tmp/$(hostname)_capturefile.cap
Step 2: Perform TCP traceroute over different ports, such as 22 and 443
$ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 443 <elastic_Ip>
$ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 22 <elastic_Ip>
$ tcptraceroute <elastic_Ip> 22
$ traceroute -T -p 22 –n <elastic_Ip>
To install tcptraceroute:
# yum -y install --enablerepo='*' tcptraceroute telnet
# apt install tcptraceroute # On Ubuntu
And that didn’t help either.
Okay, Let’s take a step back and check my email.
“IT has upgraded your VM from RHEL6 to RHEL8 over the weekend.
Please open a support case with us in case you are facing issues”.
Check the host machine:
Vm>lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description: Red Hat Enterprise Linux release 8.1 (Ootpa)
Release: 8.1
Codename: Ootpa
"ootpa" is IRC nick of Larry
Troan, who was a Red Hat
engineer and who died in
2016.
RHEL 8 "ootpa" codename
was chosen as a tribute to
Larry Troan.
Great, something has changed wrt host machine, but what exactly.
<Few days pass by>
How does SSH work behind the scenes?
<opens textbook>
Information Security: Principles and Practice, Mark Stamp
<Search google for Red Hat documentation>
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/securing_networks/using-secure-
communications-between-two-systems-with-openssh_securing-networks
• Two versions of SSH currently exist: version 1, and the newer version 2.
• The OpenSSH suite in Red Hat Enterprise Linux 8 supports only SSH version
2, which has an enhanced key-exchange algorithm not vulnerable to known
exploits in version 1.
• OpenSSH is a program depending on OpenSSL the library, specifically
OpenSSH uses the libcrypto part of OpenSSL.
Chapter 3 – The Eureka
man ssh_config: (on RHEL8)
The supported ciphers are:
• 3des-cbc
• aes128-cbc
• aes192-cbc
• aes256-cbc
• aes128-ctr
• aes192-ctr
• aes256-ctr
• aes128-gcm@openssh.com
• aes256-gcm@openssh.com
• chacha20-poly1305@openssh.com
<deep google search for redhat issues>
• “GCM ciphers are not available in SSH on RHEL 7.4 in FIPS mode”
https://github.com/ComplianceAsCode/content/issues/1613
• GCM ciphers used to be allowed in FIPS mode, but it seems that was a
bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1420910
• FIPS guide (Federal Information Processing Standards)
https://wiki.openssl.org/index.php/FIPS_mode_and_TLS
Go back to my host machine and look at logs:
• debug1: SSH2_MSG_KEXINIT sent
• debug1: SSH2_MSG_KEXINIT received
• debug1: kex: algorithm: ecdh-sha2-nistp256
• debug1: kex: host key algorithm: ecdsa-sha2-nistp256
• debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit>
compression: none
• debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit>
compression: none
• debug1: sending SSH2_MSG_KEX_ECDH_INIT
• debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
• Connection closed by <elastic-ip> port 22
https://www.cryptosys.net/pki/manpki/pki_aesgcmauthencryption.html
Go to my EC2 instance and take a look:
ec2:/etc/ssh# cat ssh_config
# Cipher 3des
# Port 22
# Protocol 2
# Cipher 3des
Ciphers aes256-gcm@openssh.com,aes128-
gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-
ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
Lessons learned:
1. Issue in EC2 instance code where its defaulting to GCM ciphers.
Real bug- filed and fixed
2. Genuine Red Hat bug which accidentally blocks GCM ciphers, which
kept me hanging (still not fixed yet)
3. Simple workaround:
1. Look for any common cipher in host and EC2 instance:
For example: “AES256-CTR” is there in both places
2. Use it to SSH to the instance:
Example usage: ssh - c “AES256-CTR” user@<elastic_ip_ec2_instance>
Chapter 4 – The End
Happy Ending after all.
Questions?

More Related Content

What's hot

Varnish Cache 4.0 / Redpill Linpro breakfast in Oslo
Varnish Cache 4.0 / Redpill Linpro breakfast in OsloVarnish Cache 4.0 / Redpill Linpro breakfast in Oslo
Varnish Cache 4.0 / Redpill Linpro breakfast in Oslo
Per Buer
 
Tuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish CacheTuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish Cache
Per Buer
 
What is new in Go 1.8
What is new in Go 1.8What is new in Go 1.8
What is new in Go 1.8
John Hua
 
Nmap not only a port scanner by ravi rajput comexpo security awareness meet
Nmap not only a port scanner by ravi rajput comexpo security awareness meet Nmap not only a port scanner by ravi rajput comexpo security awareness meet
Nmap not only a port scanner by ravi rajput comexpo security awareness meet
Ravi Rajput
 
Advanced Weapons Training for the Empire
Advanced Weapons Training for the EmpireAdvanced Weapons Training for the Empire
Advanced Weapons Training for the Empire
Jeremy Johnson
 
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
PROIDEA
 
Understanding kube proxy in ipvs mode
Understanding kube proxy in ipvs modeUnderstanding kube proxy in ipvs mode
Understanding kube proxy in ipvs mode
Victor Morales
 
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet
 
Debugging Network Issues
Debugging Network IssuesDebugging Network Issues
Debugging Network Issues
Apcera
 
Snaps on open suse
Snaps on open suseSnaps on open suse
Snaps on open suse
Zygmunt Krynicki
 
Loophole: Timing Attacks on Shared Event Loops in Chrome
Loophole: Timing Attacks on Shared Event Loops in ChromeLoophole: Timing Attacks on Shared Event Loops in Chrome
Loophole: Timing Attacks on Shared Event Loops in Chrome
cgvwzq
 
Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011
David Troy
 
FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018
Xavier Mertens
 
Building a network emulator with Docker and Open vSwitch
Building a network emulator with Docker and Open vSwitchBuilding a network emulator with Docker and Open vSwitch
Building a network emulator with Docker and Open vSwitch
Goran Cetusic
 
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David ShawBeginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
Redspin, Inc.
 
VCLをTDDで書いてデプロイする
VCLをTDDで書いてデプロイするVCLをTDDで書いてデプロイする
VCLをTDDで書いてデプロイする
Kengo HAMASAKI
 
FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)
Xavier Mertens
 
Time tested php with libtimemachine
Time tested php with libtimemachineTime tested php with libtimemachine
Time tested php with libtimemachine
Nick Galbreath
 
Nginx Internals
Nginx InternalsNginx Internals
Nginx Internals
Joshua Zhu
 
FPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpegFPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpeg
Chan Shik Lim
 

What's hot (20)

Varnish Cache 4.0 / Redpill Linpro breakfast in Oslo
Varnish Cache 4.0 / Redpill Linpro breakfast in OsloVarnish Cache 4.0 / Redpill Linpro breakfast in Oslo
Varnish Cache 4.0 / Redpill Linpro breakfast in Oslo
 
Tuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish CacheTuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish Cache
 
What is new in Go 1.8
What is new in Go 1.8What is new in Go 1.8
What is new in Go 1.8
 
Nmap not only a port scanner by ravi rajput comexpo security awareness meet
Nmap not only a port scanner by ravi rajput comexpo security awareness meet Nmap not only a port scanner by ravi rajput comexpo security awareness meet
Nmap not only a port scanner by ravi rajput comexpo security awareness meet
 
Advanced Weapons Training for the Empire
Advanced Weapons Training for the EmpireAdvanced Weapons Training for the Empire
Advanced Weapons Training for the Empire
 
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
DOD 2016 - Kamil Szczygieł - Patching 100 OpenStack Compute Nodes with Zero-d...
 
Understanding kube proxy in ipvs mode
Understanding kube proxy in ipvs modeUnderstanding kube proxy in ipvs mode
Understanding kube proxy in ipvs mode
 
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
Puppet Availability and Performance at 100K Nodes - PuppetConf 2014
 
Debugging Network Issues
Debugging Network IssuesDebugging Network Issues
Debugging Network Issues
 
Snaps on open suse
Snaps on open suseSnaps on open suse
Snaps on open suse
 
Loophole: Timing Attacks on Shared Event Loops in Chrome
Loophole: Timing Attacks on Shared Event Loops in ChromeLoophole: Timing Attacks on Shared Event Loops in Chrome
Loophole: Timing Attacks on Shared Event Loops in Chrome
 
Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011
 
FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018
 
Building a network emulator with Docker and Open vSwitch
Building a network emulator with Docker and Open vSwitchBuilding a network emulator with Docker and Open vSwitch
Building a network emulator with Docker and Open vSwitch
 
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David ShawBeginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
Beginner's Guide to the nmap Scripting Engine - Redspin Engineer, David Shaw
 
VCLをTDDで書いてデプロイする
VCLをTDDで書いてデプロイするVCLをTDDで書いてデプロイする
VCLをTDDで書いてデプロイする
 
FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)FPC for the Masses (SANSFire Edition)
FPC for the Masses (SANSFire Edition)
 
Time tested php with libtimemachine
Time tested php with libtimemachineTime tested php with libtimemachine
Time tested php with libtimemachine
 
Nginx Internals
Nginx InternalsNginx Internals
Nginx Internals
 
FPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpegFPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpeg
 

Similar to Who Broke My Crypto

Adventures in Femtoland: 350 Yuan for Invaluable Fun
Adventures in Femtoland: 350 Yuan for Invaluable FunAdventures in Femtoland: 350 Yuan for Invaluable Fun
Adventures in Femtoland: 350 Yuan for Invaluable Fun
arbitrarycode
 
Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Combell NV
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
Pierre Souchay
 
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
CODE BLUE
 
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in PragueDocker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
tomasbart
 
How to Troubleshoot OpenStack Without Losing Sleep
How to Troubleshoot OpenStack Without Losing SleepHow to Troubleshoot OpenStack Without Losing Sleep
How to Troubleshoot OpenStack Without Losing Sleep
Sadique Puthen
 
Bh usa-01-kaminsky
Bh usa-01-kaminskyBh usa-01-kaminsky
Bh usa-01-kaminskyDan Kaminsky
 
Reverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemReverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande Modem
Cyber Security Alliance
 
k8s practice 2023.pptx
k8s practice 2023.pptxk8s practice 2023.pptx
k8s practice 2023.pptx
wonyong hwang
 
FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
Evan McGee
 
introduction-infra-as-a-code using terraform
introduction-infra-as-a-code using terraformintroduction-infra-as-a-code using terraform
introduction-infra-as-a-code using terraform
niyof97
 
Shutdown that bastion host!
Shutdown that bastion host!Shutdown that bastion host!
Shutdown that bastion host!
MichaelLudvig
 
CEPH中的QOS技术
CEPH中的QOS技术CEPH中的QOS技术
CEPH中的QOS技术
suncbing1
 
SREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
SREcon Europe 2016 - Full-mesh IPsec network at Hosted GraphiteSREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
SREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
HostedGraphite
 
Php through the eyes of a hoster
Php through the eyes of a hosterPhp through the eyes of a hoster
Php through the eyes of a hoster
Combell NV
 
SNClient+ - General purpose monitoring agent
SNClient+ - General purpose monitoring agentSNClient+ - General purpose monitoring agent
SNClient+ - General purpose monitoring agent
Sven Nierlein
 
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven NieleinOSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
NETWAYS
 
Packaging perl (LPW2010)
Packaging perl (LPW2010)Packaging perl (LPW2010)
Packaging perl (LPW2010)
p3castro
 
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex LauDoing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Ceph Community
 
Hosting a Rails App
Hosting a Rails AppHosting a Rails App
Hosting a Rails App
Josh Schramm
 

Similar to Who Broke My Crypto (20)

Adventures in Femtoland: 350 Yuan for Invaluable Fun
Adventures in Femtoland: 350 Yuan for Invaluable FunAdventures in Femtoland: 350 Yuan for Invaluable Fun
Adventures in Femtoland: 350 Yuan for Invaluable Fun
 
Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11Php through the eyes of a hoster phpbnl11
Php through the eyes of a hoster phpbnl11
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
 
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
For the Greater Good: Leveraging VMware's RPC Interface for fun and profit by...
 
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in PragueDocker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
 
How to Troubleshoot OpenStack Without Losing Sleep
How to Troubleshoot OpenStack Without Losing SleepHow to Troubleshoot OpenStack Without Losing Sleep
How to Troubleshoot OpenStack Without Losing Sleep
 
Bh usa-01-kaminsky
Bh usa-01-kaminskyBh usa-01-kaminsky
Bh usa-01-kaminsky
 
Reverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande ModemReverse engineering Swisscom's Centro Grande Modem
Reverse engineering Swisscom's Centro Grande Modem
 
k8s practice 2023.pptx
k8s practice 2023.pptxk8s practice 2023.pptx
k8s practice 2023.pptx
 
FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
 
introduction-infra-as-a-code using terraform
introduction-infra-as-a-code using terraformintroduction-infra-as-a-code using terraform
introduction-infra-as-a-code using terraform
 
Shutdown that bastion host!
Shutdown that bastion host!Shutdown that bastion host!
Shutdown that bastion host!
 
CEPH中的QOS技术
CEPH中的QOS技术CEPH中的QOS技术
CEPH中的QOS技术
 
SREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
SREcon Europe 2016 - Full-mesh IPsec network at Hosted GraphiteSREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
SREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite
 
Php through the eyes of a hoster
Php through the eyes of a hosterPhp through the eyes of a hoster
Php through the eyes of a hoster
 
SNClient+ - General purpose monitoring agent
SNClient+ - General purpose monitoring agentSNClient+ - General purpose monitoring agent
SNClient+ - General purpose monitoring agent
 
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven NieleinOSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
OSMC 2023 | Replacing NSClient++ for Windows Monitoring by Sven Nielein
 
Packaging perl (LPW2010)
Packaging perl (LPW2010)Packaging perl (LPW2010)
Packaging perl (LPW2010)
 
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex LauDoing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
 
Hosting a Rails App
Hosting a Rails AppHosting a Rails App
Hosting a Rails App
 

More from John Varghese

Lessons Learned From Cloud Migrations: Planning is Everything
Lessons Learned From Cloud Migrations: Planning is EverythingLessons Learned From Cloud Migrations: Planning is Everything
Lessons Learned From Cloud Migrations: Planning is Everything
John Varghese
 
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPALeveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
John Varghese
 
AWS Transit Gateway-Benefits and Best Practices
AWS Transit Gateway-Benefits and Best PracticesAWS Transit Gateway-Benefits and Best Practices
AWS Transit Gateway-Benefits and Best Practices
John Varghese
 
Bridging Operations and Development With Observabilty
Bridging Operations and Development With ObservabiltyBridging Operations and Development With Observabilty
Bridging Operations and Development With Observabilty
John Varghese
 
Security Observability for Cloud Based Applications
Security Observability for Cloud Based ApplicationsSecurity Observability for Cloud Based Applications
Security Observability for Cloud Based Applications
John Varghese
 
Building an IoT System to Protect My Lunch
Building an IoT System to Protect My LunchBuilding an IoT System to Protect My Lunch
Building an IoT System to Protect My Lunch
John Varghese
 
Building a Highly Secure S3 Bucket
Building a Highly Secure S3 BucketBuilding a Highly Secure S3 Bucket
Building a Highly Secure S3 Bucket
John Varghese
 
Reduce Amazon RDS Costs up to 50% with Proxies
Reduce Amazon RDS Costs up to 50% with ProxiesReduce Amazon RDS Costs up to 50% with Proxies
Reduce Amazon RDS Costs up to 50% with Proxies
John Varghese
 
Keynote - Lead the change around you
Keynote - Lead the change around youKeynote - Lead the change around you
Keynote - Lead the change around you
John Varghese
 
AWS Systems manager 2019
AWS Systems manager 2019AWS Systems manager 2019
AWS Systems manager 2019
John Varghese
 
Acd19 kubertes cluster at scale on aws at intuit
Acd19 kubertes cluster at scale on aws at intuitAcd19 kubertes cluster at scale on aws at intuit
Acd19 kubertes cluster at scale on aws at intuit
John Varghese
 
Emerging job trends and best practices in the aws community
Emerging job trends and best practices in the aws communityEmerging job trends and best practices in the aws community
Emerging job trends and best practices in the aws community
John Varghese
 
Automating security in aws with divvy cloud
Automating security in aws with divvy cloudAutomating security in aws with divvy cloud
Automating security in aws with divvy cloud
John Varghese
 
AWS temporary credentials challenges in prevention detection mitigation
AWS temporary credentials   challenges in prevention detection mitigationAWS temporary credentials   challenges in prevention detection mitigation
AWS temporary credentials challenges in prevention detection mitigation
John Varghese
 
Securing aws workloads with embedded application security
Securing aws workloads with embedded application securitySecuring aws workloads with embedded application security
Securing aws workloads with embedded application security
John Varghese
 
Of CORS thats a thing how CORS in the cloud still kills security
Of CORS thats a thing how CORS in the cloud still kills securityOf CORS thats a thing how CORS in the cloud still kills security
Of CORS thats a thing how CORS in the cloud still kills security
John Varghese
 
Native cloud security monitoring
Native cloud security monitoringNative cloud security monitoring
Native cloud security monitoring
John Varghese
 
Last year in AWS - 2019
Last year in AWS - 2019Last year in AWS - 2019
Last year in AWS - 2019
John Varghese
 
Gpu accelerated BERT deployment on aws
Gpu accelerated BERT deployment on awsGpu accelerated BERT deployment on aws
Gpu accelerated BERT deployment on aws
John Varghese
 
EKS security best practices
EKS security best practicesEKS security best practices
EKS security best practices
John Varghese
 

More from John Varghese (20)

Lessons Learned From Cloud Migrations: Planning is Everything
Lessons Learned From Cloud Migrations: Planning is EverythingLessons Learned From Cloud Migrations: Planning is Everything
Lessons Learned From Cloud Migrations: Planning is Everything
 
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPALeveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
Leveraging AWS Cloudfront & S3 Services to Deliver Static Assets of a SPA
 
AWS Transit Gateway-Benefits and Best Practices
AWS Transit Gateway-Benefits and Best PracticesAWS Transit Gateway-Benefits and Best Practices
AWS Transit Gateway-Benefits and Best Practices
 
Bridging Operations and Development With Observabilty
Bridging Operations and Development With ObservabiltyBridging Operations and Development With Observabilty
Bridging Operations and Development With Observabilty
 
Security Observability for Cloud Based Applications
Security Observability for Cloud Based ApplicationsSecurity Observability for Cloud Based Applications
Security Observability for Cloud Based Applications
 
Building an IoT System to Protect My Lunch
Building an IoT System to Protect My LunchBuilding an IoT System to Protect My Lunch
Building an IoT System to Protect My Lunch
 
Building a Highly Secure S3 Bucket
Building a Highly Secure S3 BucketBuilding a Highly Secure S3 Bucket
Building a Highly Secure S3 Bucket
 
Reduce Amazon RDS Costs up to 50% with Proxies
Reduce Amazon RDS Costs up to 50% with ProxiesReduce Amazon RDS Costs up to 50% with Proxies
Reduce Amazon RDS Costs up to 50% with Proxies
 
Keynote - Lead the change around you
Keynote - Lead the change around youKeynote - Lead the change around you
Keynote - Lead the change around you
 
AWS Systems manager 2019
AWS Systems manager 2019AWS Systems manager 2019
AWS Systems manager 2019
 
Acd19 kubertes cluster at scale on aws at intuit
Acd19 kubertes cluster at scale on aws at intuitAcd19 kubertes cluster at scale on aws at intuit
Acd19 kubertes cluster at scale on aws at intuit
 
Emerging job trends and best practices in the aws community
Emerging job trends and best practices in the aws communityEmerging job trends and best practices in the aws community
Emerging job trends and best practices in the aws community
 
Automating security in aws with divvy cloud
Automating security in aws with divvy cloudAutomating security in aws with divvy cloud
Automating security in aws with divvy cloud
 
AWS temporary credentials challenges in prevention detection mitigation
AWS temporary credentials   challenges in prevention detection mitigationAWS temporary credentials   challenges in prevention detection mitigation
AWS temporary credentials challenges in prevention detection mitigation
 
Securing aws workloads with embedded application security
Securing aws workloads with embedded application securitySecuring aws workloads with embedded application security
Securing aws workloads with embedded application security
 
Of CORS thats a thing how CORS in the cloud still kills security
Of CORS thats a thing how CORS in the cloud still kills securityOf CORS thats a thing how CORS in the cloud still kills security
Of CORS thats a thing how CORS in the cloud still kills security
 
Native cloud security monitoring
Native cloud security monitoringNative cloud security monitoring
Native cloud security monitoring
 
Last year in AWS - 2019
Last year in AWS - 2019Last year in AWS - 2019
Last year in AWS - 2019
 
Gpu accelerated BERT deployment on aws
Gpu accelerated BERT deployment on awsGpu accelerated BERT deployment on aws
Gpu accelerated BERT deployment on aws
 
EKS security best practices
EKS security best practicesEKS security best practices
EKS security best practices
 

Recently uploaded

ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
CatarinaPereira64715
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 

Recently uploaded (20)

ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 

Who Broke My Crypto

  • 1. Who broke my crypto Nikhil Prathapani Enterprise Routing – SDWAN group Cisco Systems Inc
  • 2. Things you WON’T learn in this talk • Crypto currency • Bitcoin • Blockchain
  • 3. Agenda • Chapter 0 – The Problem • Chapter 1 – The Puzzle • Chapter 2 – The Chase • Chapter 3 – The Eureka • Chapter 4 – The End
  • 4. On a fine Monday morning: I tried to SSH to my EC2 instance, but it kept bailing out on me. Chapter 0 – The Problem
  • 5. • Why am I unable to SSH to an instance which worked fine until Friday. • I listed out the things changed from my end: • EC2 Instance type: unchanged, not even touched since Friday • Host machine : Same host machine - RHEL instance Chapter 1 – The Puzzle
  • 6. Oh I know how to debug this. Its simple: • Instead of SSH, just add –vvv for further debug. • ssh -v will tell you what is happening mostly on your end • ssh -vv will tell you low level on both ends • ssh -vvv will tell you almost everything from both ends. Chapter 2 – The Chase
  • 7. Contacted AWS support. A very patient support rep helped me debug the issue further Step 0: SSH with "-vvv" flag for verbosity I did that, didn’t help. Still lost connection.
  • 8. Step 1: Create packet capture $ sudo -i # tcpdump -i any -w /tmp/$(hostname)_capturefile.cap -s256 port 22 & # ssh -vvv user@<elastic_ip_ec2_instance> # killall tcpdump ; pkill tcpdump # zip -9 /tmp/$(hostname)_capturefile.cap.zip /tmp/$(hostname)_capturefile.cap
  • 9. Step 2: Perform TCP traceroute over different ports, such as 22 and 443 $ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 443 <elastic_Ip> $ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 22 <elastic_Ip> $ tcptraceroute <elastic_Ip> 22 $ traceroute -T -p 22 –n <elastic_Ip> To install tcptraceroute: # yum -y install --enablerepo='*' tcptraceroute telnet # apt install tcptraceroute # On Ubuntu
  • 10. And that didn’t help either. Okay, Let’s take a step back and check my email. “IT has upgraded your VM from RHEL6 to RHEL8 over the weekend. Please open a support case with us in case you are facing issues”. Check the host machine: Vm>lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: RedHatEnterprise Description: Red Hat Enterprise Linux release 8.1 (Ootpa) Release: 8.1 Codename: Ootpa "ootpa" is IRC nick of Larry Troan, who was a Red Hat engineer and who died in 2016. RHEL 8 "ootpa" codename was chosen as a tribute to Larry Troan.
  • 11. Great, something has changed wrt host machine, but what exactly. <Few days pass by> How does SSH work behind the scenes? <opens textbook> Information Security: Principles and Practice, Mark Stamp
  • 12.
  • 13. <Search google for Red Hat documentation> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/securing_networks/using-secure- communications-between-two-systems-with-openssh_securing-networks • Two versions of SSH currently exist: version 1, and the newer version 2. • The OpenSSH suite in Red Hat Enterprise Linux 8 supports only SSH version 2, which has an enhanced key-exchange algorithm not vulnerable to known exploits in version 1. • OpenSSH is a program depending on OpenSSL the library, specifically OpenSSH uses the libcrypto part of OpenSSL. Chapter 3 – The Eureka
  • 14. man ssh_config: (on RHEL8) The supported ciphers are: • 3des-cbc • aes128-cbc • aes192-cbc • aes256-cbc • aes128-ctr • aes192-ctr • aes256-ctr • aes128-gcm@openssh.com • aes256-gcm@openssh.com • chacha20-poly1305@openssh.com
  • 15. <deep google search for redhat issues> • “GCM ciphers are not available in SSH on RHEL 7.4 in FIPS mode” https://github.com/ComplianceAsCode/content/issues/1613 • GCM ciphers used to be allowed in FIPS mode, but it seems that was a bug. https://bugzilla.redhat.com/show_bug.cgi?id=1420910 • FIPS guide (Federal Information Processing Standards) https://wiki.openssl.org/index.php/FIPS_mode_and_TLS
  • 16. Go back to my host machine and look at logs: • debug1: SSH2_MSG_KEXINIT sent • debug1: SSH2_MSG_KEXINIT received • debug1: kex: algorithm: ecdh-sha2-nistp256 • debug1: kex: host key algorithm: ecdsa-sha2-nistp256 • debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none • debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none • debug1: sending SSH2_MSG_KEX_ECDH_INIT • debug1: expecting SSH2_MSG_KEX_ECDH_REPLY • Connection closed by <elastic-ip> port 22 https://www.cryptosys.net/pki/manpki/pki_aesgcmauthencryption.html
  • 17. Go to my EC2 instance and take a look: ec2:/etc/ssh# cat ssh_config # Cipher 3des # Port 22 # Protocol 2 # Cipher 3des Ciphers aes256-gcm@openssh.com,aes128- gcm@openssh.com,aes128-ctr,aes192-ctr,aes256- ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
  • 18. Lessons learned: 1. Issue in EC2 instance code where its defaulting to GCM ciphers. Real bug- filed and fixed 2. Genuine Red Hat bug which accidentally blocks GCM ciphers, which kept me hanging (still not fixed yet) 3. Simple workaround: 1. Look for any common cipher in host and EC2 instance: For example: “AES256-CTR” is there in both places 2. Use it to SSH to the instance: Example usage: ssh - c “AES256-CTR” user@<elastic_ip_ec2_instance> Chapter 4 – The End