Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elasticsearch on AWS - High Availability and Security best practices

19,384 views

Published on

I talk about Elasticsearch on AWS, and on those slides I cover the topics High Availability (HA) and Security best practices.

Published in: Technology

Elasticsearch on AWS - High Availability and Security best practices

  1. 1. ES on AWS Implementing ElasticSearch on AWS ~ High Availability and Best Security practices ~
  2. 2. Who Am I (log nerd AND DevOp AND Infrastructure Manager AND photographer AND .*) XOR (daddy); rafael@psafe.com for company and business purposes; dev@rafalop.es to have a beer and eat something; @bobeirasa; bobeirasa on Freenode IRC; ; You can also find me on Elasticsearch-PT on Google Groups; Tags: bobeirasa, rafalopes, rafael lopes, psafe, mpran, fotografia cotidiana;
  3. 3. About PSafe Tecnologia • Brazilian StartUp focused on security; • More than 12 MM Android Users; • #6 app on Top Free Google Play, Browser, Security Suite for Win/Mac and developing more products; • Infrastructure with more than 200 high-end servers, tons of memory and bandwidth; • 3 offices in Brazil, RJ [headquarters], SP and SC; • LatAm focus with products localized to portuguese and spanish; • We’ve been looking for you, join us! Tags: psafe, qihoo, startup, brazilian, psafe techchrunch, VC investment, series C;
  4. 4. What will be covered • High Availability (HA); • Security; Tags: rafael lopes, meetup, elasticsearch, segundo encontro;
  5. 5. High Availability ~ topics ~ • AWS Region and Availability Zone (AZ); • Multi-AZ environment; • Shard allocation awareness; • unicast or elasticsearch-cloud-aws plugin; • AZ label mismatch regarding your accounts; Tags: rafael lopes slides, meetup Elasticsearch rio, segundo encontro, slides;
  6. 6. High Availability AWS Region and Availability Zone (AZ) Tags: AWS region, AWS availability zone, difference;
  7. 7. High Availability Don’t run your cluster on the same AZ !!! Tags: Elasticsearch Single-AZ, failure, ec2 classic, failover, disaster, downtime, AWS AZ fail;
  8. 8. High Availability Run it on separate Availability Zones Tags: Elasticsearch Multi-AZ, VPC, AWS different Subnet, Multi-AZ advantages, HA, high availability;
  9. 9. High Availability Multi-AZ on Elasticsearch Cluster Tags: Elasticsearch shard allocation awareness, hadoop rack affinity, replica, safe shards;
  10. 10. High Availability Elasticsearch Shard Allocation Awareness Awareness is a keepalive concept, used to make sure primary shards and it’s corresponding replicas don't end up on same place, such as rack, environment or AWS Availability Zones! Tags: shard allocation awareness, hadoop rack affinity, replica, safe shards, raid10 comparison;
  11. 11. High Availability Shard Allocation Awareness configuration node.name = “ES01” node.rack_id: zona-A cluster.routing.allocation.awareness.attributes: rack_id ! node.name = “ES02” node.rack_id: zona-B cluster.routing.allocation.awareness.attributes: rack_id ! node.name = “ES03” node.rack_id: zona-A cluster.routing.allocation.awareness.attributes: rack_id ! node.name = “ES04” node.rack_id: zona-B cluster.routing.allocation.awareness.attributes: rack_id Tags: shard allocation syntax, Elasticsearch shard allocation configuration;
  12. 12. High Availability AWS doesn’t allow multicast/broadcast Tags: elasticsearch ec2 discovery, ec2 broadcast, ec2 ARP, VPC FAQ, VPC limitations, rackspace multicast;
  13. 13. High Availability AWS doesn’t allow multicast/broadcast 2 solutions for that ! Tags: elasticsearch ec2 discovery, ec2 broadcast, ec2 ARP, VPC FAQ, VPC limitations, rackspace multicast;
  14. 14. High Availability Disable multicast, use unicast to specify nodes cluster.name: StarTrek node.name: "Locutus" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.timeout: 5s discovery.zen.ping.unicast.hosts: [“Voyager”] ! cluster.name: StarTrek node.name: "Voyager" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.timeout: 5s discovery.zen.ping.unicast.hosts: [“Locutus”] ! cluster.name: StarTrek node.name: "Enterprise" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.timeout: 5s discovery.zen.ping.unicast.hosts: [“Locutus”, “Voyager” ] Tags: Elasticsearch unicast, Elasticsearch zen, Elasticsearch discovery ping, Elasticsearch manual hosts;
  15. 15. High Availability Use elasticsearch-cloud-aws plugin Tags: elasticsearch-cloud-aws, official plugin, github, Elasticsearch AWS credentials;
  16. 16. High Availability More about the elasticsearch-cloud-aws plugin • Easy installation (as any other plugin): ./plugin -install elasticsearch/elasticsearch-cloud-aws/2.3.0 • Official plugin provided by Elasticsearch; • Uses AWS API ec2-describe-instances to know stuff about your AWS ec2 instances. Smart, huh ? • Compatible with IAM roles (stay with me to see more details about this on Security session); • Uses ec2 tagging; • Check Elasticsearch version and Plugin version for mismatches; • It’s also used for s3 snapshots (which we won’t cover here); Tags: elasticsearch-cloud-aws install, elasticsearch-cloud instance metadata, ec2 discovery, fake multicast;
  17. 17. High Availability AZ label mismatch across different AWS accounts Tags: AWS availability zone mismatch, Elasticsearch on different AWS accounts;
  18. 18. High Availability ~ recap ~ AWS Region and Availability Zone (AZ); Shards are automatically replicated across the cluster, but don’t run your cluster on the same AZ; Use shard allocation awareness to tell ES to assign shards to different AZ’s (like Hadoop rack affinity); AWS doesn’t allow multicast at all, so use unicast instead of multicast, or elasticsearch-cloud-aws plugin; If you want (for any reason) to use the same cluster running different AWS accounts, open a support ticket to check AZ label mismatch regarding your accounts; Tags: rafael lopes slides, meetup Elasticsearch rio, segundo encontro, slides;
  19. 19. Security ~ topics ~ • built-in security credentials and web proxy; • IAM roles; • Kibana; • Bastion server; • Multicast protocol; • External scripting; Tags: rafael lopes slides, meetup elasticsearch rio, segundo encontro, slides;
  20. 20. Security MiTM (man in the middle) • Elasticsearch does not provide native connection strings, passwords or any kind of authentication: traditional DBA’s would say that Elasticsearch is insecure because of this; they say that because they are traditionalists, not DevOps ;) • Change Elasticsearch http port and use a local firewall to close this port to everybody unless localhost; • It must reside on a private subnet with no communication to the public, as any database server; • Use 3 firewalls. In addition to security groups and NACL’s, create local firewall rules to allow traffic just to the consumers (frontend, middletier or internal elb) on the ‘Elasticsearch port’; • Install a web server acting as a proxy, such as nginx or mitmproxy to control requests and passwords; • Jetty plugin also used to implement authentication and encription; Tags: Elasticsearch proxy, Elasticsearch nginx, protecting Elasticsearch, Elasticsearch firewall, NACL,AWS security group, AWS private subnet, iptables, ipfw, linux firewall;
  21. 21. Security nginx example: limit everything but not GET on /_settings, except if the host is 127.0.0.1 location ~ /_settings { proxy_pass http://elasticsearch; limit_except GET { allow 127.0.0.1; deny all; } } Tags: nginx Elasticsearch protect
  22. 22. Security nginx - implement basic password authentication, if the host is 127.0.0.1, this auth can be skipped location ~ /_plugin { satisfy any; allow 127.0.0.1; deny all; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/pwds/file.pwd; proxy_pass http://elasticsearch; } Tags: nginx Elasticsearch protect
  23. 23. Security IAM Roles Tags: IAM roles, codespaces.com, identity steal, access key
  24. 24. Security IAM Roles • Mandatory to security best practices; • Keys expires in 6 hours (James Bond would like to use that); • No access keys in the code or config files; • Uses the ec2 metadata (http layer between ec2 and hypervisor) to get the provisory key; • It just works magically with the elasticsearch-cloud-aws plugin; Tags: AWS security best practices, AWS IAM roles, ec2 instance metadata, AWS STS, MFA
  25. 25. Security Kibana is a frontend tool meant to be used in internal network (VPN). Kibana, like ES should not be exposed to public internet Tags: kibana, protect, node.js, wrapper, kibana frontend danger, close kibana
  26. 26. Security Use a Bastion Server Tags: bastion server, security best practice;
  27. 27. Security Disable Multicast in production environment • Reduces noise on environment; • Safer due to eavesdroppers/sniffers in shared environments; Tags: multicast, production environment, security best practice;
  28. 28. Security ~ recap ~ Elasticsearch does not provide built-in security credentials, connection strings or passwords, so protect yourself using a web proxy such as nginx or mitmproxy to prevent deletes or updates; IAM roles with AWS STS (simple token service) on ec2 instances (also works with cloud plugin), stop copy-pasting access key and secret over there; Kibana is for insiders: use a wrapper to protect if you want to show to the world, or set it up on an private subnet with VPN access; Use a bastion server with 2 factor auth on ssh layer to make the DevOps works; On a shared environment such as a cloud, disable multicast at all; Disable external scripting if you don't use it; Tags: rafael lopes slides, meetup elasticsearch rio, segundo encontro, slides;
  29. 29. Questions ?
  30. 30. Thank you! Coffee Break ? (sorry for make you wait so long) Tags: free food, free bier, elasticsearch rio second meetup, ideais tecnologia, sponsor, happiness;

×