FICO Red Hat Summit Presentation on OpenStack, OpenShift, Foreman, Pupet,container usage and generaloverview of hardware and implementation

1. © 2012 Fair Isaac Corporation. Confidential.1
Confidential. This presentation is provided for the recipient only and cannot
be reproduced or shared without Fair Isaac Corporation's express consent.
© 2012 Fair Isaac Corporation.
Openstack | Openshift| FICO
Lesson learned over the past 12 months
Chris Ferraro | Andy Quang | Oscar Sandoval | Nick Gerasimatos
Cloud Engineeringrraro | Kiet Quang | Oscar Sandoval-Partain | Nick
Gerasimatos
Platform Engineering
FICO
May 22, 2015

2. © 2012 Fair Isaac Corporation. Confidential.2
Agenda
»Who is FICO?
»Why Openstack
»FAC (FICO Analytic Cloud)
»Why Openshift
»Architecture
»Automation
»Questions

3. © 2012 Fair Isaac Corporation. Confidential.3
Who is FICO?
»FICO Score, MyFICO
»Analytics software and tools for businesses
»Debt management
»Fraud & security analytics
»Customer engagement
»Big Data Analytics
»Designs custom algorithms for many specific use cases
» 90% of all lending decisions in the U.S. rely on the FICO Score
»2.5B credit cards globally are protected by FICO Fraud Systems
»Founded 1956
»50+ year history of data and analytics excellence

4. © 2012 Fair Isaac Corporation. Confidential.4 © 2012 Fair Isaac Corporation. Confidential.4 © 2012 Fair Isaac Corporation. Confidential.4
“The FICO Analytic Cloud provides
Platform-as-a-Service (PaaS) access to
the FICO Decision Management
Platform, allowing customers to use
FICO tools and technology to create
and deploy applications and services.”
– Tony McGivern, FICO CIO

5. © 2012 Fair Isaac Corporation. Confidential.5
Why move from VMware to Open Source?

6. © 2012 Fair Isaac Corporation. Confidential.6
Why move to the Cloud?
»Need to expand beyond top tier financial services
companies and get into middle market
»Traditional on premise technologies moving to a
self service or XaaS/SPI model
»Open Source nature allows for participation in
driving change
»Need to lower cost
»Thousands of virtual and container instances
»Faster time to market
»Templatize apps deployment regardless of
public cloud, private cloud, or underlying
infrastructure

7. © 2012 Fair Isaac Corporation. Confidential.7 © 2012 Fair Isaac Corporation. Confidential.7
OpenStack Infrastructure

8. © 2012 Fair Isaac Corporation. Confidential.8
FICO Cloud Architecture v1.0
» Cisco UCS Converge Infrastructure

9. © 2012 Fair Isaac Corporation. Confidential.9
FICO Cloud Architecture v1.0
Lessons Learned
» Complex Design
» Too many technologies and distributions to select from
» Difficult to troubleshoot when things go wrong
» Lack of mature tools for capacity planning
» Poor “end to end” integration between IaaS, PaaS, XaaS
» No true documented architecture designs and few white papers
made public for customers/users
» FICO Cloud v1.0
» Red Hat Enterprise Virtualization and Red Hat OpenStack
» RHEL 6.x
» Optimize design for compute workloads
» Gluster used for primary and only shared storage solution
» OpenShift used for PaaS

10. © 2012 Fair Isaac Corporation. Confidential.10
FICO Cloud Architecture v2.0
HAProxy Load
Balancer
Openstack
Controller/CEPH
Node
OpenStack
Compute/CEPH
Node
»Cisco UCS C240M3 & C220M3
C220 C240 C240

11. © 2012 Fair Isaac Corporation. Confidential.11
FICO Cloud Architecture v2.0
Lessons Learned
» Simplify design
» OpenStack (KVM)
» CEPH for primary and only distributed shared storage
» RHEL 7.x
» Simplify design for easier handoff for Operations teams
» Targeted baseline optimized for compute and storage performance
» Haproxy / Keepalived
» Cloudforms Deployment/Proof of Concept
» Neutron SDN
» Layer 2 only
» Performance issues and realizations
» Does Ceph work for all workloads?
» Does the HW selected fit the application and product requirements?

12. © 2012 Fair Isaac Corporation. Confidential.12
FICO Cloud Architecture v3.0
Openstack
Controller Node
C220
OpenStack
Compute/CEPH
Node
C240
» Cisco UCS C240M3 & C220M3
SolidFire SF4805
F5 Load balancer

13. © 2012 Fair Isaac Corporation. Confidential.13
FICO Cloud Architecture v3.0
Lessons Learned
» F5 to load balance between Openstack APIs
» Selected the Cisco C240 rack servers as the standard hardware
“workhorse” to provide compute and storage
» Moved the OpenStack controller layer to Cisco C220 rack servers
» Implemented tiered Storage
» Ceph to provide non SLA driven storage
» SolidFire distributed flash storage to provide high performance storage
» Revised the sizing of our OpenShift Node
» Cloudforms single pane of glass for private and public cloud
» Swift object oriented storage

14. © 2012 Fair Isaac Corporation. Confidential.14 © 2012 Fair Isaac Corporation. Confidential.14
OpenShift

15. © 2012 Fair Isaac Corporation. Confidential.15
Why Openshift?
»On-premise
»Business needs to reduce the Time to Value and Time to Market
»Templatize apps deployment regardless of public cloud, private cloud, or
underlying infrastructure
»Ability to build custom Cartridges
»Decrease development costs & simplified tools used for development and
deployment
»FICO has reduced time to value for developing analytic solutions by up to
70% versus legacy on premise software

16. © 2012 Fair Isaac Corporation. Confidential.16
FICO Analytic Cloud | Openshift
»Launched FAC in 2013 before containers were cool
»Business needs required rapid deployment and continuous integration while
simplifying the current design
»FICO Analytic Cloud provides IaaS and PaaS access to the FICO Decision
Management Platform for multiple tenants and customers
»FICO Decision Management Analytic Platform allows application developers,
business users and FICO partners around the world “one-stop” access to use
FICO technology to create, customize, and deploy analytics-driven
applications and services empowering them to make educated decisions and
lowering risk
»FAC/DMP was built using Openshift as the backend engine

17. © 2012 Fair Isaac Corporation. Confidential.17
Decision Management Suites
»Platform as a server that allows non-developers(e.g. data scientists) to
construct and analytic solution through the use of a rapid application
development and catalog of FICO component
»User “stiches” together FICO components with appropriate data flow and
business rules RAD environment
»Solution is then deployed via Openshift Enterprise for consumption
»Full lifecycle management allows user to develop promote to a separate test
environment and ultimately promote to run time / production
»OSE not used for internal DevOps, but instead is a productization for
customer consumption

18. © 2012 Fair Isaac Corporation. Confidential.18
FICO Analytic Cloud Architecture | Openshift v 1.0

19. © 2012 Fair Isaac Corporation. Confidential.19
FICO Analytic Cloud Architecture | Openshift v 1.0

20. © 2012 Fair Isaac Corporation. Confidential.20
Openshift | 1.0
»Installation was manual (bind, activemq, mongo,OSE)
»Deployed on existing VMware infrastructure & Netapp
»Limited online documentation
»Single Point of Failure (1 broker/dns/activemq/mongodb)
»Break fix / Bugs
»Tips:
»Documentation
»Operationalize Openshift

21. © 2012 Fair Isaac Corporation. Confidential.21
Openshift | 1.1
»Node settings (base on your environment Application/node size
»Gear Idling (2hrs)
»Node gear size/profiles
»Over commitment: number of active gears per node
»Identify & eliminate SPOF
»Openshift (broker, bind, activemq, mongo)
»Application ==> scaled apps
»Tips:
»Monitoring
»DNS
»Platform (internal/external name resolution)
»Offload name resolution Infoblox DNS Appliance
»Dynamic DNS on Bind

22. © 2012 Fair Isaac Corporation. Confidential.22
Openshift | 1.2
»Openshift Upgrade
»OSE Deployment via PXE
»HA design (3 brokers, 3 mongo, 3 activeMQ)
»F5 (brokers)
»Tuning Openshift
»Load Test (jmeter / custom scripts)
»ActiveMQ JVM OOM (activemq.xml)
»MongoDB(set primary)
»Tips:
»Monitoring / Trending (Keynote, Zabbix, Cacti)
»Total active gears
»Total Capacity per environment
»Total idled gear in the past 90 days

23. © 2012 Fair Isaac Corporation. Confidential.23
Openshift | Monitoring | Trending Stats

24. © 2012 Fair Isaac Corporation. Confidential.24
Openshift | Current Architecture

25. © 2012 Fair Isaac Corporation. Confidential.25
FICO Analytic Cloud Architecture | Openshift v 2.0

26. © 2012 Fair Isaac Corporation. Confidential.26
Openshift | 2.0
»New deployment
»Parallel 1.2 / 2.0
»Openstack / CEPH
»Foreman / Heat OSE Deployment
»Implemented active/passive DR/HA across 2 datacenter to meet SLAs
»OSE 2.2 , 3.0 origin

27. © 2012 Fair Isaac Corporation. Confidential.27 © 2012 Fair Isaac Corporation. Confidential.27
Automation

28. © 2012 Fair Isaac Corporation. Confidential.28
Our Tools Foreman, Puppet, Cisco UCS
Provision
Configure
Manage

29. © 2012 Fair Isaac Corporation. Confidential.29
Why FICO Automated Using Puppet and Foreman
» Save Time and Manpower.
» Ensured consistency of configurations across
all nodes.
» Repeatability of deployments.
» Source controlled configurations.

30. © 2012 Fair Isaac Corporation. Confidential.30
Foreman
» Foreman can deploy to both virtual and bare metal machines.
» Foreman Provisions RHEL on Cisco UCS C240s and C220s
» Foreman Host groups with Parameter overrides preps the Cisco UCS
240s and 220s for puppet configuration.
» Foreman Proxies in every Geo-location allows ease of deployment.

31. © 2012 Fair Isaac Corporation. Confidential.31
Puppet
» FICO Custom Puppet Module to install OpenStack Icehouse and Juno.
» Custom Puppet module was built using many different Puppet Forge
Modules.
» Wrote custom facts to tie everything together.
» Reduce build time from days to minutes.
» Average build time per server is 3 minutes.

32. © 2012 Fair Isaac Corporation. Confidential.32
Satellite 6
» Consistency of Packages across OpenStack, Openshift and VMware
» Repeatability of deployments
» Controlled package repositories.
» Different views per release, allowing testing without affecting current
environments.
» Reduce the amount of tools within our environment
» Already using components of Satellite 6
» Foreman
» Puppet
» Better integration with CloudForms than Spacewalk

33. © 2012 Fair Isaac Corporation. Confidential.33
CloudForms
» Manage OpenStack and legacy VMware from one tool
» Resource usage and optimization
» Self-service portal and catalog
» Single API and GUI for both development and production
environments
» Dashboards and reporting
» Ahead of the curve on integration
» Professional services needs
» Needed functionality 1 – 2 releases out

34. © 2012 Fair Isaac Corporation. Confidential.34
Red Hat Ceph Storage
» Why Ceph?
» Scalable
» Open Source
» Software-defined storage
» Provides both block and object oriented storage
» Optimize small footprint deployments with OpenStack/CEPH
converged hardware
» Tight integration with OpenStack
» SolidFire for high performance SLA driven workloads
» Scalable, clustered all-flash storage array provides inline compression,
deduplication and replication

35. © 2012 Fair Isaac Corporation. Confidential.35
Red Hat Enterprise Linux
» RHEL 6 & 7
» OSP runs on RHEL 7
» Provides solid platform understood by Operations
» Common and supported implementation across multiple products
» Cisco UCS
» Oracle RAC

36. © 2012 Fair Isaac Corporation. Confidential.36
What’s Next
» Use UCS Central automation for
quick stand up of private cloud
environments regardless of geo
location or availability zone
» OpenStack Kilo Automation
» Puppet OpenStack module?
» Cisco UCS M4 series
» Eliminate Ceph and use only
Solidfire distributed flash
storage?
» Migrate away from NFS/NAS to
object oriented storage such as
Swift?
» Satellite 6 replacement for
Spacewalk, etc
» CloudForms
» Single pane of glass
» Ability to manage both Private
and Public Clouds
» Capacity planning
» Reporting
» Integration with OpenStack and
OpenShift
» Single API regardless of the
endpoint

37. © 2012 Fair Isaac Corporation. Confidential.37
Questions ?

38. © 2012 Fair Isaac Corporation. Confidential.38
Confidential. This presentation is provided for the recipient only and cannot
be reproduced or shared without Fair Isaac Corporation's express consent.
© 2012 Fair Isaac Corporation.
Thank You
May 18, 2015
Nick Gerasimatos
NickGerasimatos@fico.com
Chris Ferraro
ChrisFerraro@fico.com
Kiet Quang
kietquang@fico.com