Successfully reported this slideshow.
Your SlideShare is downloading. ×

Exploring a simpler, more portable, less overhead solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 44 Ad

Exploring a simpler, more portable, less overhead solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections

Download to read offline

After the last release of Component Pack for IBM Connections, some time has been spent reflecting and discussing the solution for supplying IBM Connections services to customers to enhance collaboration and boost productivity.
Come join us as one of our advisory engineers walks through a simpler, more cost effective, less overhead proof of concept solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections – a totally flexible solution that can be deployed anywhere

After the last release of Component Pack for IBM Connections, some time has been spent reflecting and discussing the solution for supplying IBM Connections services to customers to enhance collaboration and boost productivity.
Come join us as one of our advisory engineers walks through a simpler, more cost effective, less overhead proof of concept solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections – a totally flexible solution that can be deployed anywhere

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Exploring a simpler, more portable, less overhead solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections (20)

Advertisement

More from LetsConnect (20)

Recently uploaded (20)

Advertisement

Exploring a simpler, more portable, less overhead solution to deploy Elasticsearch Pink Metrics and Customizer for IBM Connections

  1. 1. Philadelphia, April 26-27 2018 13 Exploring a simpler, more portable, less overhead solution to deploy Elasticsearch Metrics and Customizer Conall Ó Cofaigh, IBM ocofaigh@ie.ibm.com
  2. 2. Social Connections 13 Philadelphia, April 26-27 2018 PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR
  3. 3. Social Connections 13 Philadelphia, April 26-27 2018 Agenda • About Me • What is Customizer and Elasticsearch? • Reflection & Feedback • IBM Cloud Private • Proof of Concept • Demo • Next Steps • Summary • Q&A
  4. 4. Social Connections 13 Philadelphia, April 26-27 2018 About Me • Working in IBM Collaboration Solutions (ICS) since 2008. • Based out of the IBM Technology Campus, Mulhuddart, Dublin, Ireland • Background in System Test on the IBM Sametime product. • Working on Connections Pink for the last year. • Current role: • DevOps for IBM Connections private cloud offering • Responsible for the “Component Pack” installer (the big zip!) • Installer/bootstrap scripts • Packaging up the zip so all of our microservices will work together • Installation & configuration documentation
  5. 5. Social Connections 13 Philadelphia, April 26-27 2018 Questions For You • Who has experience with Docker? • Who is using a Docker registry? • Who has experience with Kubernetes? • Who has worked with Helm? • Who is using Helm repo? • Who has services hosted in a managed cloud (e.g. Bluemix, AWS, Azure etc) • Who has deployed Component Pack?
  6. 6. Social Connections 13 Philadelphia, April 26-27 2018 What is Customizer? • IBM Connections Customizer is a proxy service that lets you modify the IBM Connections user experience. • Customizer intercepts and modifies requests from clients and responses from servers, so it can customize anything that flows through it, such as the behaviour of APIs or the look- and-feel of the user interface. • More info:
  7. 7. Social Connections 13 Philadelphia, April 26-27 2018 What is Elasticsearch? • IBM's Elasticsearch Metrics lets you visualize your data more easily and get meaningful metrics. NB: To continue to receive IBM support for the IBM Connections Metrics feature after April 30, customers are required to install Elasticsearch to replace IBM Cognos • Both Elasticsearch and Customizer are available in the Component Pack 6.0.0.5 release.
  8. 8. Social Connections 13 Philadelphia, April 26-27 2018 Reflection • Component Pack 6.0.0.5 shipped in Feb 2018. • Since then, IBM have spent some time discussing the current solution for supplying IBM Connections services to customers. • We have gathered customer and business partner feedback, as well as explored the current industry standard for supplying services. • General feedback was: • Even though deployment is a lot easier than the WebSphere Connections solution, some people still find it too complicated. • Love the “Pink” functionality, but requires a lot of resources for what it does. • Not really flexible for hosted clouds such as Bluemix /AWS / Azure etc.
  9. 9. Social Connections 13 Philadelphia, April 26-27 2018 Feedback Even though deployment is a lot easier than the WebSphere Connections solution, some people still find it too complicated. • Trying to make the installer work for every kind of possible customer architecture led to a lot of optional flags…
  10. 10. Social Connections 13 Philadelphia, April 26-27 2018 Feedback Love the “Pink” functionality, but requires a lot of resources for what it does. • For example, to deploy a full Component Pack stack, production ready environment, with full high-availability support, one would require: • 3 Master machines (4 CPU, 16GB RAM, 100GB disk) + a floating IP to load balance • 3 Generic Worker machines – co-located proxy on each worker node (6 CPU, 16GB RAM, 100GB disk) • 3 Infrastructure Worker machines (4 CPU, 16GB RAM, 100GB disk) • 1 Storage server for persistent volumes and ICp image repositories (150GB disk) • 9 very high specked machines (+ Storage) on top of the current Connections WebSphere / DB2 stack was just ringing alarm bells! • The bulk of these specifications come from IBM Cloud Private.
  11. 11. Social Connections 13 Philadelphia, April 26-27 2018 Feedback Not really flexible for hosted clouds such as AWS / Azure etc. • Many of Connections customers have deployed (or plan to deploy) third party services on hosted clouds such as AWS, Azure or Bluemix. • Such cloud platforms provide very simple startup scripts to deploy Kubernetes clusters. For example: • AWS – Kops, Kube-AWS • Azure - Azure Container Service (Using CLI or online web browser portal) • Bluemix - IBM Cloud Container Service CLI or API • The Component Pack installer installs and configures the underlying Kubernetes cluster using the IBM Cloud Private product.
  12. 12. Social Connections 13 Philadelphia, April 26-27 2018 IBM Cloud Private • Formerly IBM Spectrum Conductor for Containers (CfC). • Server platform for developing and managing on-premises, containerized applications using Docker and Kubernetes. • Provides services such as authentication, storage, networking, logging, and monitoring. • A cluster management console is also provided, which allows for centralized management of these services.
  13. 13. Social Connections 13 Philadelphia, April 26-27 2018 IBM Cloud Private + Component Pack • When “Pink” development started more than 2 years ago, Docker and Kubernetes were technologies new to many IBM Connections admins. • We choose IBM Cloud Private (or CfC as it was called then) as a base platform to set the Kubernetes environment up as part of our installer. • CfC could be pulled from the web with a docker pull command which fit nicely into our bash installer scripts. • Our installer also installed Docker and configured Kubernetes for our microservices to run on. • This meant admins did not have to have an in-depth knowledge about Docker or Kubernetes, and by following the online documentation, could deploy a working Orient Me solution out of the box.
  14. 14. Social Connections 13 Philadelphia, April 26-27 2018 IBM Cloud Private Evolution • CfC became ICp. • ICp was split into CE and EE installers: • Community Edition (CE) – no HA support • Enterprise Edition (EE) – HA support but only available from Passport Advantage (1.7GB in size) • Component Pack 6.0.0.5 is shipped with IBM Cloud Private 1.2.1. • ICp 1.2.1 deploys Kubernetes v1.6 – no longer supported by Kubernetes (current latest is v1.10).
  15. 15. Social Connections 13 Philadelphia, April 26-27 2018 IBM Cloud Private 2.1.0.2 • The latest ICp version is 2.1.0.2. • This comes with Kubernetes v1.9. • Installer package is now 8GB in size. • Contains more than is needed by IBM Connections. EG: • IBM Cloud Private Cloud Foundry • Metering dashboard • VMware integration • ICp 2.1.0.2 EE is also still only available from Passport Advantage (no docker pull). • Conclusion: ICp is a fantastic product, but it is over featured for Connections services.
  16. 16. Social Connections 13 Philadelphia, April 26-27 2018 Action Items • Does our solution work without ICp? • Does it work without Kubernetes? • Can we simplify our deployment process? • Can we make it more flexible? • Can we make it less resource heavy?
  17. 17. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Removing ICp • Think back to why we choose to use ICp… • It installed and configured the Kubernetes environment – no in- depth Kubernetes knowledge required • Lightweight and easily automated – docker pull • It did everything we needed to at the time
  18. 18. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Removing ICp • Reality check: • ICp EE only available on Passport Advantage – download can’t be automated • ICp 2.1.0.2 is 8GB in size, too big to fit inside our installer zip • ICp is over featured for Connections • Kubernetes has become a well known technology and can now be deployed very easily using a wide range of different tools • Many IBM Connections customers are using their own Kubernetes platforms for other services • It does not make sense to deploy ICp on hosted clouds such as AWS, Azure or Bluemix when similar solutions are already provided • Most service providers pre-req Docker and an orchestration tool (such as Kubernetes, Swarm, Apache Marathon etc) – more of an industry standard these days
  19. 19. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Removing ICp • As a proof of concept, we decided to “hack” the Component Pack 6.0.0.5 installer scripts to see if everything would work in a world without ICp. • We decided to focus on our 2 smaller stacks, Elasticsearch and Customizer. • To do this, we had to manually install Docker and Kubernetes as a pre-req. • We used the Kubeadm tool to install a Kubernetes cluster, but there are tons of tools out there to do this for you depending on what your hosting platform is. • Result: • We successfully deployed all pods required for both stacks – great!
  20. 20. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Removing K8s? • With ICp now gone in our POC, we decided to investigate if we could remove the Kubernetes dependency. • We looked at using Docker Compose (tool for defining and running multi-container Docker applications). • With a single command and a YAML file, you can create and start the services. • Disadvantages: • No orchestration • No container management • Not recommended for production • No HA • Hard to manage
  21. 21. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Removing K8s? • Conclusion: • Docker Compose is great for standing up a service with a single command, but without orchestration to manage your containers, launch them on the underlying machines, make sure they are distributed, and keeping them healthy, its not going to be feasible for production • For this reason, we decided to continue using Kubernetes for this proof of concept …
  22. 22. Social Connections 13 Philadelphia, April 26-27 2018 Helm • Helm is a package manager for Kubernetes. • Think of it like apt/yum/homebrew for Kubernetes. • A Helm Chart is a package that contains information that is required to stand up a service. • Charts can be stored on disk, or fetched from remote chart repos. • Component Pack 6.0.0.5 is using Helm v2.7.2 to install / upgrade services.
  23. 23. Social Connections 13 Philadelphia, April 26-27 2018 Installing Helm • Installing Helm is extremely simple.. Prereq: A Kubernetes cluster 1. Download the Helm binary from https://github.com/kubernetes/helm 2. Unpack the helm binary and add it to you PATH
  24. 24. Social Connections 13 Philadelphia, April 26-27 2018 Helm commands • Once you have Helm ready, you can initialize the local CLI and also install Tiller into your Kubernetes cluster in one step: helm init • This will install Tiller into the Kubernetes cluster. • To install a chart, you can run the helm install command: $ helm install alpine ---> No chart named "alpine" in your workspace. Fetching now. ---> Running `kubectl create -f` ... pods/alpine ---> Done ======================================== # Alpine This Chart provides a Pod running Alpine Linux. ========================================
  25. 25. Social Connections 13 Philadelphia, April 26-27 2018 Helm commands • Helm handles the packages up through initial installation, but kubectl is still the way to go for working with your cluster: $ kubectl get pods NAME READY REASON RESTARTS AGE alpine 1/1 Running 0 22s • The helm list function will show you a list of all deployed releases. $ helm list NAME VERSION UPDATED STATUS CHART Alpine 1 Wed Sep 28 12:59:46 2016 DEPLOYED alpine-0.1.0
  26. 26. Social Connections 13 Philadelphia, April 26-27 2018 Helm commands • To uninstall a release, use the helm delete command: $ helm delete alpine --purge release “alpine” deleted Use --purge to remove the release from the release list, freeing up its name for later use (removes release history)
  27. 27. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – The Overview • Based on the investigation done, a POC without ICp using pure Helm commands seemed to be the way forward to a simpler, more portable, less overhead solution. • This would open the door for: • Deploying in hosted clouds • Using more lightweight Linux Operating Systems such as Alpine or CoreOS • Configuring a Docker registry to pull Docker images • Configuring a Helm repository to pull Helm charts
  28. 28. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Pre-reqs • Install Docker • For this POC, we choose to stick with Docker 17.03 as this is the recommended version to use with Kubeadm • Install Kubernetes • For this POC, we choose Kubernetes v1.8, as this currently the latest version that our service YAML files will work with • Install Helm • For this POC, we choose Helm v2.7.2, as it is the version applicable for Kubernetes v1.8
  29. 29. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Pre-reqs • Create Kubernetes namespace • A namespace must be created for our services to live in • Command: kubectl create namespace <namespace> • Create Storage • Both Customizer and Elasticsearch require Kubernetes persistent storage • Docker Registry • For this POC we have removed the images from the zip, and in the values.yaml of all the Helm packages we have added our internal pipeline JFrog server • This can be updated to use a Docker Registry (using global variable replacement) when running the helm install command
  30. 30. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – The Breakdown • setup.sh • Sets up environment for Helm install • Supports choosing a pre-exisitng namespace (-n connections) • Supports multiple Starter Stack set up (-st elasticsearch,customizer) • Calls validate.sh and configure.sh • validate.sh • Validate Docker • Validate Kubernetes • Validate Helm • Validate Persistent Volumes • configure.sh • Creates required secrets / certs by prompting for passwords • All prompts can be avoided (for automation) by using flags with setup.sh
  31. 31. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install with Helm • In Component Pack, every service has its own Helm Chart. • For this POC, in order to use Helm to install Elasticsearch and Customizer, we require just 4 Helm charts (using sub- charts): 1. connections-env – creates a configmap of values used by the services and some generic secrets 2. infrastructure – mongo, redis, redis-sentinel & haproxy 3. elasticsearch - elasticsearch 4. customizer – mw-proxy, appreg-client, & appreg-service
  32. 32. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install with Helm • Before installation, run the setup.sh • To install Customizer, the following Helm charts would need to be installed: 1. connections-env 2. infrastructure 3. customizer • To install Elasticsearch, the following Helm charts would need to be installed: 1. connections-env 2. elasticsearch
  33. 33. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install Customizer • Install connections-env: helm install --name=connections-env helmbuilds/connections-env/connections-env- 0.1.40-20180413-180651.tgz --set createSecret=false, ic.host=ic-entry.ibm.com, ic.internal=http-server.ibm.com This can be Helm repo instead IBM Connections front door address (e.g. load balancer) IBM Connections HTTP Server
  34. 34. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install Customizer • Install infrastructure: helm install --name=infrastructure helmbuilds/infrastructure/infrastructure- 0.1.0.tgz --set mongodb.createSecret=false, mongodb.onPrem=true This can be Helm repo instead This is where you would pass in your Docker registry that would contain the images , image_repo=https://mydockerrepo.com
  35. 35. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install Customizer • Install customizer: helm install --name=customizer helmbuilds/customizer/customizer-0.1.0.tgz This can be Helm repo instead
  36. 36. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install Elasticsearch • Install connections-env: • NOTE: If you installed customizer, this will already be installed helm install --name=connections-env helmbuilds/connections-env/connections-env- 0.1.40-20180413-180651.tgz --set createSecret=false, ic.host=ic-entry.ibm.com, ic.internal=http-server.ibm.com This can be Helm repo instead IBM Connections front door address (e.g. load balancer) IBM Connections HTTP Server
  37. 37. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Install Elasticsearch • Install elasticsearch: helm install --name=elasticsearch helmbuilds/elasticsearch/elasticsearch-0.1.0- 20180307-024239.tgz This can be Helm repo instead
  38. 38. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Upgrading • Releases can be upgraded using the helm upgrade command Example: helm upgrade customizer helmbuilds/customizer/customizer-0.2.0.tgz • You can check the history of a release using the helm history command: $ helm history customizer REVISION UPDATED STATUS CHART DESCRIPTION 1 Mon Apr 23 11:01:49 2018 SUPERSEDED customizer-0.1.0 Install complete 2 Mon Apr 23 11:02:17 2018 DEPLOYED customizer-0.2.0 Upgrade complete Example:
  39. 39. Social Connections 13 Philadelphia, April 26-27 2018 13 Demo
  40. 40. Social Connections 13 Philadelphia, April 26-27 2018 Proof of Concept – Next Steps • All of this is still a work in progress! • The following items are currently being investigated: • Can we remove the bootstrapping bash scripts that set up the Kubernetes secrets and certificates? (Vault, Docker container ??) • Can we reduce the sub-charts required to run our services? • Can we implement some kind of monitoring dashboard (Prometheus??)
  41. 41. Social Connections 13 Philadelphia, April 26-27 2018 Summary • POC takes minutes to deploy • Reduced complexity and cost • Flexible deployment location • Leverages Helm to its full potential • Opens the door for Docker Registry and Helm repo release model
  42. 42. Social Connections 13 Philadelphia, April 26-27 2018 13
  43. 43. Social Connections 13 Philadelphia, April 26-27 2018 PLATINUM SPONSOR GOLD SPONSORS BRONZE SPONSORS GOLD PLUS SPONSOR SILVER SPONSORS SPEEDSPONSORING BEER SPONSOR

Editor's Notes

  • - PersistentVolume
    - StorageClass (dynamic provisioning)

×