Developer Toolchain
at UC Riverside
Mike Kennedy
Enterprise Architect
Developer Toolchain
at UC Riverside
Mike Kennedy
Enterprise Architect
Application Deployment
About Me
About Me
• Mike Kennedy
About Me
• Mike Kennedy
• At UC Riverside since 1995 (Student then Staff)
About Me
• Mike Kennedy
• At UC Riverside since 1995 (Student then Staff)
• Enterprise Architect
About Me
• Mike Kennedy
• At UC Riverside since 1995 (Student then Staff)
• Enterprise Architect
• Interests lie in web services, application architecture,
integration, middleware
About Me
• Mike Kennedy
• At UC Riverside since 1995 (Student then Staff)
• Enterprise Architect
• Interests lie in web services, application architecture,
integration, middleware
• Major projects - New SIS (Banner), UCPath,
ServiceNow, New development environment, ESB
Talk Background
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
• Began effort to modernize: .NET vs. Java
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
• Began effort to modernize: .NET vs. Java
• Ultimately settled on Grails (Groovy/JVM based
framework)
Talk Background
• Up until recently, applications developed using
PL/SQL Web Toolkit and Oracle IAS
• Began effort to modernize: .NET vs. Java
• Ultimately settled on Grails (Groovy/JVM based
framework)
• Opportunity: Improve on application delivery
processes and architecture (my focus today)
Grails
Grails
Grails Application
Tomcat App Server
Java Virtual Machine (JVM)
Operating System
F5
app-prd-1.cis.ucr.edu app-prd-2.cis.ucr.edu
APPPROD
User 1
Tomcat Session Replication
User 2
DatabaseConnection
DatabaseConnection
HTTP
HTTP
ProxyHTTP
ProxyHTTP
Puppet Server
Deployment Server
Configure
Configure
Configure
Source Control
Management
Server (Git)
ESB
WebServiceCall
Public Network
Private Network
Deploy
Pull
Nagios
(Monitoring)
Graphite
(Performance
Stats)
Dev
CIS
CommitsCode
Starts
Deployments
W
rites
Scripts
Writes Modules
W
rites M
onitoring
Checks
Monitor
Collaboration
Grails Application and
Support Architecture
https://app.ucr.edu/
Process Improvement
Process Improvement
• What are the strengths/weaknesses of current
process?
Process Improvement
• What are the strengths/weaknesses of current
process?
• What is the target state?
Process Improvement
• What are the strengths/weaknesses of current
process?
• What is the target state?
• Identify technologies/tools that could help
Legacy Deployment Process
Legacy Deployment Process
• Developer writes code and deploys
Legacy Deployment Process
• Developer writes code and deploys
• Developer has complete control
Legacy Deployment Process
• Developer writes code and deploys
• Developer has complete control
• Little to no logging
Legacy Deployment Process
• Developer writes code and deploys
• Developer has complete control
• Little to no logging
• Potential issues with coordination
Legacy Deployment Process
• Developer writes code and deploys
• Developer has complete control
• Little to no logging
• Potential issues with coordination
• Infrastructure (Dev Ops) not in the loop
New Deployment Process
Requirements
New Deployment Process
Requirements
• Developers must be empowered to work
asynchronously from Dev Ops - BUY IN!
New Deployment Process
Requirements
• Developers must be empowered to work
asynchronously from Dev Ops - BUY IN!
• Must be logged and auditable
New Deployment Process
Requirements
• Developers must be empowered to work
asynchronously from Dev Ops - BUY IN!
• Must be logged and auditable
• Automation! Automation! Automation!
New Deployment Process
Requirements
• Developers must be empowered to work
asynchronously from Dev Ops - BUY IN!
• Must be logged and auditable
• Automation! Automation! Automation!
• Implicit notification of events
New Deployment Process
Requirements
• Developers must be empowered to work
asynchronously from Dev Ops - BUY IN!
• Must be logged and auditable
• Automation! Automation! Automation!
• Implicit notification of events
• Close collaboration between developers and dev
ops teams
Application Delivery Cycle
Application Delivery Cycle
Develop
Build
Deploy
Monitor
Application Delivery Cycle
Develop
Build
Deploy
Monitor
Graphite
Develop
Develop
• Gitlab
Develop
• Gitlab
• Git server (Source Control Management)
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
• Code reviews
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
• Code reviews
• Merge requests
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
• Code reviews
• Merge requests
• Open source
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
• Code reviews
• Merge requests
• Open source
• On premise or cloud
Develop
• Gitlab
• Git server (Source Control Management)
• Very “Github-like”
• Web Front End
• Code reviews
• Merge requests
• Open source
• On premise or cloud
Develop
Develop
• Slack
Develop
• Slack
• Collaboration tool
Develop
• Slack
• Collaboration tool
• Chat
Develop
• Slack
• Collaboration tool
• Chat
• Integrations galore!
Develop
• Slack
• Collaboration tool
• Chat
• Integrations galore!
• Free w/ special EDU pricing for advanced
features
Develop
• Slack
• Collaboration tool
• Chat
• Integrations galore!
• Free w/ special EDU pricing for advanced
features
Develop
Developer Git Server (SCM)
Collaboration
Developer
Developer
Develop
Developer Git Server (SCM)
Collaboration
Push
Developer
Developer
Develop
Developer Git Server (SCM)
Collaboration
Push
Notify
Developer
Developer
Receive
Develop
Developer Git Server (SCM)
Collaboration
Push
Developer
Developer
Develop
Developer Git Server (SCM)
Collaboration
Push
Developer
Developer
Pull
Build
Build
• Gitlab, Slack
Build
• Gitlab, Slack
• Jenkins
Build
• Gitlab, Slack
• Jenkins
• Continuous Integration Server (CI) - Builds
Build
• Gitlab, Slack
• Jenkins
• Continuous Integration Server (CI) - Builds
• Can be triggered by branch push
Build
• Gitlab, Slack
• Jenkins
• Continuous Integration Server (CI) - Builds
• Can be triggered by branch push
• We use special branch names - release/x.y.z
Build
Build
• Gitlab, Slack
Build
• Gitlab, Slack
• Nexus
Build
• Gitlab, Slack
• Nexus
• Maven artifact repository
Build
• Gitlab, Slack
• Nexus
• Maven artifact repository
• Jenkins publishes built wars here
Build
• Gitlab, Slack
• Nexus
• Maven artifact repository
• Jenkins publishes built wars here
• Can contain other artifacts (jars, zips)
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
file.groovy
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
file.groovyPush
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
file.groovyPush
NotifyReceive
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
file.groovyPush
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
file.groovyPush
Build Webhook
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration file.groovy
Push
Pull
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Push
Pull
Build WAR
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Push
Pull
Notify
Build WAR
Receive
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Push
Pull
Build WAR
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Push
Pull
Publish
Build WAR
Build
Developer
Git Server (SCM)
CI (Build) Server
Artifact Repository (Maven)
Collaboration
Push
Pull
Publish
Build WAR
Deploy
Deploy
• Slack, Nexus
Deploy
• Slack, Nexus
• Rundeck
Deploy
• Slack, Nexus
• Rundeck
• Job scheduler and runbook automation
Deploy
• Slack, Nexus
• Rundeck
• Job scheduler and runbook automation
• Jobs can be prebuilt and shared
Deploy
• Slack, Nexus
• Rundeck
• Job scheduler and runbook automation
• Jobs can be prebuilt and shared
• Auditable
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
Run
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
Run Execute
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
Run Execute
Download WAR
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
Run Execute
App Server 2
App Server 1
Deploy
Developer
Artifact Repository (Maven)
Collaboration
Job Scheduler/Deployment Server
App Server N
Run Execute
Notify
Receive
Monitor
Monitor
• Slack
Monitor
• Slack
• Nagios
Monitor
• Slack
• Nagios
• Ping test
Monitor
• Slack
• Nagios
• Ping test
• Application availability
Monitor
• Slack
• Nagios
• Ping test
• Application availability
Monitor
• Slack
• Nagios
• Ping test
• Application availability
• Graphite
Monitor
• Slack
• Nagios
• Ping test
• Application availability
• Graphite
• Application Metrics
Monitor
• Slack
• Nagios
• Ping test
• Application availability
• Graphite
• Application Metrics
• JVM Performance
Monitor
Monitor
• Logging
Monitor
• Logging
• ELK stack
Monitor
• Logging
• ELK stack
• Log Stash (Log Collection)
Monitor
• Logging
• ELK stack
• Log Stash (Log Collection)
• Elastic Search (Log Indexing, Searching)
Monitor
• Logging
• ELK stack
• Log Stash (Log Collection)
• Elastic Search (Log Indexing, Searching)
• Kibana (Visualization)
Monitor
• Logging
• ELK stack
• Log Stash (Log Collection)
• Elastic Search (Log Indexing, Searching)
• Kibana (Visualization)
• Splunk Killer?
Monitor
• Logging
• ELK stack
• Log Stash (Log Collection)
• Elastic Search (Log Indexing, Searching)
• Kibana (Visualization)
• Splunk Killer?
• Check out Nick Turley’s talk:
“Security Data Analytics Platform” tomorrow
at 9am!
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Graphite
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Log Messages
Graphite
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Log Messages
View
Graphite
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
View
Graphite
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
View
Graphite
Monitor
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Notify
Receive
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Receive
View
Graphite
Monitor
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Notify
Receive
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Receive
View
Graphite
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Notify
Receive
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Receive
View
Graphite
Metrics
Monitor
App Server 2
App Server 1
Developer
Collaboration
App Server N
Notify
Receive
Dev Ops
Log Collection and
Visualization (ELK)
Monitoring
Performance Metrics
Receive
View
GraphiteView
Metrics
Demo Time
Outcomes
Outcomes
• Still early
Outcomes
• Still early
• Buy-in was quick
Outcomes
• Still early
• Buy-in was quick
• Good feedback from developers and dev ops
teams
Outcomes
• Still early
• Buy-in was quick
• Good feedback from developers and dev ops
teams
• Many of the tools deployed used for non-
development projects (New SIS)
Outcomes
• Still early
• Buy-in was quick
• Good feedback from developers and dev ops
teams
• Many of the tools deployed used for non-
development projects (New SIS)
• All tools deployed are free and open source!
Questions
Mike Kennedy
michael.kennedy@ucr.edu
Twitter: @mike_kennedy

Application Deployment at UC Riverside