Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Full	Stack	Java	Microservices	with	
Mesos/Marathon	and	Docker	
How	to	take	an	app	server	and	blow	it	up	into	a	lots	of	pieces…and	s5ll	be	
able	to	call	it	a	“win”.	
Edward	Burns	@edburns	
ConsulKng	Member	of	Technical	Staff	
April	2017
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Safe	Harbor	Statement	
The	following	is	intended	to	outline	our	general	product	direcKon.	It	is	intended	for	
informaKon	purposes	only,	and	may	not	be	incorporated	into	any	contract.	It	is	not	a	
commitment	to	deliver	any	material,	code,	or	funcKonality,	and	should	not	be	relied	upon	
in	making	purchasing	decisions.	The	development,	release,	and	Kming	of	any	features	or	
funcKonality	described	for	Oracle’s	products	remains	at	the	sole	discreKon	of	Oracle.
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Speaker	QualificaKons	–	Ed	Burns	
•  25	Years	SoXware	Development	Experience	
– NCSA	Mosaic	è	Marathon	and	much	in	between	
– Spec	Lead	for	Java	EE	Servlet	and	JSF	
•  Informal	spare	Kme	trainer	
•  Author	of	four	IT-related	books	for	McGraw-Hill
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Speaker	QualificaKons	–	Ed	Burns	
•  The	PerspecKve	I	offer	
– On	the	front	lines	of	the	disrupKon	caused	by	the	shiX	to	cloud	naKve	development	
– I	work	on	things	that	have	historically	enabled	“The	Monolith”	
•  Java	EE:	JSF,	Servlet	
•  App	Servers:	GlassFish	and	WebLogic	
– I	understand	the	need	to	adapt	to	changing	developer	requirements	
•  I	also	understand	the	perils	of	chasing	the	shiny	new	thing	
•  Blog	hdp://ridingthecrest.com/blog/
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	New	Stack	
•  Economic	forces	are	driving	the	move	to	the	cloud	
– More	efficiency	
•  The	commodiKzaKon	of	runKmes	is	nearly	complete	
– CPU/Storage/Network:	all	virtual,	all	billable	per-use	
– Several	choices	of	vendors	
•  Much	of	the	funcKon	of	an	IT	department	can	be	codified	into	a	Service	Level	Agreement	and	
purchased	from	a	cloud	vendor	
– “SoXware	defined	data	center”
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	New	Stack	
•  Economic	forces	are	driving	the	move	to	the	cloud	
– Beder	environment	for	conKnuous	improvement	
•  At	runKme:	horizontal	scalability	
•  At	innovaKon	Kme:	conKnuous	delivery	in	pracKce
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Architecture	QuesKons
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Source	
hdp://www.artofadventure.net/two-paths-
and-the-road-not-taken/	
	
Two	Paths	for	Cloud	NaKve	Enterprises
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Two	Paths	for	Cloud	NaKve	Enterprises	
•  Structured	Plagorms	
– You	buy	into	their	parKcular	concept	layout	and	soluKon	assignment	for	each	of	the	boxes	
•  Unstructured	Plagorms	
– You	make	up	your	own	boxes	and	decide	how	to	fill	them	out,	usually	with	off-the-shelf	open	
source	offerings	
– You	have	to	decide	how	to	provision/host/run/maintain	it
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Two	Paths	for	Cloud	NaKve	Enterprises	
•  You’re	sKll	going	to	need	a	lot	of	support	technology	in	either	approach	
•  Build	System	
•  CI/CD	Pipeline	
– TesKng	
•  ArKfact	management	
•  Version	Control	
– Source	code	
– Binary	code
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	ExisKng	Java	EE	Stack	is	SKll	Useful	
•  One	of	the	only	two	games	in	town	before	cloud	disrupKon,	the	other	being	.NET	
•  Structured	when	Structured	wasn’t	cool	
•  What	is	a	monolithic	app?		That	usually	means	“app	server”.	
–  You	get	a	lot	of	benefits	with	the	app	server	
•  Simplified	deployment	model:	some	collecKon	of	WAR/EAR	files	
•  Easy	ORM:	JPA	database	connecKon	
•  Easy	redundancy:	clustering	support	
•  Easy	load	balancing	
•  Easy	TransacKonality:	JTA	
•  Statefulness	
•  Reliability	
•  Debugability	and	logging
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	ExisKng	Java	EE	Stack	is	SKll	Useful	
•  But	there	are	also	problems	
– Ops	departments	may	not	support	WAR/EAR	deployment	model	and	prefer	GNU/
Linux	naKve	packages	such	as	RPMs	
– Best	pracKce	for	app	servers	seems	to	have	converged	on	the	“one	app	per	app	
server”	model.		This	is	a	lot	of	wasted	resources.	
– Tend	to	be	vulnerable	to	environment	specific	problems:	not	repeatable.	
– Not	as	easy	to	do	massive	horizontal	scaling
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	ExisKng	Java	EE	Stack	is	SKll	Useful	
• The	good	stuff	from	Java	EE	
– Easy	programming	model	
• Java	is	sKll	predy	great	
• Stable,	Kme	proven	technologies	
• Rich	tooling	
• Root	cause	analysis	is	well	understood	and	manageable	
• Leverage	eventual	consistency	where	beneficial
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
New	Stack	Risks	
• It	is	new	
– Best	pracKces	sKll	emerging	
– Hard	to	stay	up	to	date	with	them	as	they	emerge	
– Lots	of	soluKons	are	offered	as	side	effects	that	started	out	as	enabling	other	
businesses	
• LyX	Confidant	
• Neglix	Hystrix	
• AWS	itself	
• Google	cloud	
• …	your	needs	are	not	their	needs	
– Open	Source	as	employee	retenKon	
• Employees	benefit	by	building	their	reputaKon
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
New	Stack	Risks	
• It’s	a	much	more	“roll	your	own”	approach	
– RBAC	
– Security	and	AuthorizaKon	
• Everyone	has	to	be	more	of	a	sysadmin	
• No	more	single	neck	to	strangle	
• Subject	to	the	“flavor	of	the	day”	syndrome	
• With	containers	you	need	to	know	exactly	what	is	in	your	containers	
– What	versions	of	all	the	components?
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
New	Stack	Rewards	
•  ContainerizaKon	
– Package	dependencies	
•  Resiliency	
– Beder	upKme	
– Beder	response	Kme	
•  Enable	things	such	as	eventual	consistency	and	CQRS	
•  Beder	embrace	iteraKon	lifestyle
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
How	to	Get	Ready	
•  You	have	to	be	more	of	a	generalist	
•  There	is	no	more	“throwing	it	over	the	wall”,	everyone	has	to	think	like	a	
sysadmin	
•  Study	up	on	Linux	Sysadmin	skills	
– Systemd	iniKalizaKon	tool	
– Package	management	and	proxy	navigaKon	
– Networking,	including	the	OSI	protocol	stack	
– Get	some	familiarity	with	Ansible	
•  Study	up	on	build	systems	
•  Study	up	on	arKfact	management	systems	
•  Study	up	on	JVM	tuning	arguments
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
No	Catchy	Name	
•  Lots	of	stacks	have	catchy	names	
– LAMP	
– SMACK	
– MEAN	
– MERN	
– hdps://en.wikipedia.org/wiki/SoluKon_stack	
•  I	don’t	have	a	catchy	name	for	this	stack	
– Plagorm	as	a	Service	on	bare	metal
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Conceptual	Layout	
Foundation Layer
Support Layer
Business Layer
Your	code	goes	here	
But	with	DevOps,	you	must	
understand	here…	
…	and	here
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Quick	Tour	of	the	Stack:	
FoundaKon	Layer	
•  OperaKng	System	
– networking	
•  Node	management	
•  ContainerizaKon	
•  Scheduling	of	high	level	processes	
•  Load	balancing	
•  Binary	arKfact	management	
•  GNU/Linux	
– SDN	
•  Apache	Mesos	
•  Docker	
•  Marathon	
•  NGINX	
•  ArKfactory	
Conceptual	Layout	
One	Possible	Concrete	
Implementa5on	Layout
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Quick	Tour	of	the	Stack:	Support	Layer	
•  Logging	
•  Telemetry	
•  Lightweight	Persistence	
•  Heavyweight	Persistence	
•  IdenKty	and	Secret	Management	
•  ELK	
•  Prometheus	
•  Etcd	
•  NoSQL	DB	(MongoDB)	
•  Hashicorp	Vault	
Conceptual	Layout	
One	Possible	Implementa5on	
Layout
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Quick	Tour	of	the	Stack:	Business	Layer	
•  ContainerizaKon	technology	
•  Interchange	technology	
•  Programming	Language	
•  Docker	
•  REST	(JAX-RS/Jersey)	
•  Java	8	
Conceptual	Layout	
One	Possible	Implementa5on	
Layout
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Aside:	Which	Container	Technology?	
•  Docker	vs	Rkt	(pronounced	rock-it)	
– hdps://coreos.com/blog/rkt-accepted-into-the-cncf.html	
– hdp://containerops.org/2014/12/19/docker-vs-rocket-gimme-a-break/	
– hdps://medium.com/@adriaandejonge/moving-from-docker-to-
rkt-310dc9aec938#.szvh6atkt	
– Rkt	can	run	Docker	images	
•  Converts	container	to	appc	format	
•  CoreOS	(maker	of	rkt)	and	Docker	
– Open	Container	IniKaKve	hdps://www.opencontainers.org/	
•  Image	SpecificaKon		
•  RunKme	SpecificaKon
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Installing	and	Running	the	Stack	
•  The	challenge	is	to	have	the	developer	environment	be	as	similar	as	
possible	to	the	producKon	environment.	
– How	much	resides	on	the	local	developer	machine?	
– How	to	integrate	with	ConKnuous	IntegraKon	and	ConKnuous	Delivery?
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	FoundaKon	Layer	
•  Apache	Mesos	hdp://mesos.apache.org/	
– Allows	abstracKng	a	computer	as		
•  CPU/Memory/IO/Storage	
– Allows	starKng	and	stopping	containers:	containerizer	
•  Support	for	Docker,	rkt	and	appc	containers	
– Runs	as	a	“mesos	cluster”	
•  Three	kinds	of	nodes	
– Master	(at	least	one)	
– Slave	(at	least	one)	
– Zookeeper	(cluster)	(at	least	one,	but	it	can	run	as	a	cluster)
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	FoundaKon	Layer	
•  Marathon	hdps://mesosphere.github.io/marathon/	
•  Is	a	Mesos	“Framework”	
– Handles	scale	up/scale	down	
– Provides	a	deployment	descriptor	concept	
– Introduces	the	concept	of	an	“ApplicaKon”	
– Storage	accounKng	
– Health	checks	
– Metrics	
– Constraints	(mapping	applicaKons	to	racks/nodes,	etc)
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	FoundaKon	Layer	
•  Marathon	hdps://mesosphere.github.io/marathon/	
•  Has	a	REST	API	for	gezng	its	work	done	
– hdps://mesosphere.github.io/marathon/docs/rest-api.html	
– There	is	a	not	maintained	Java	API	to	this	rest	client	
hdps://github.com/mohitsoni/marathon-client	
•  Can	build	your	own	orchestrator	on	top	of	it
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	FoundaKon	Layer	
•  Demo:	Show	foundaKon	layer	running	using	docker	maven	plugin	
Thanks	Manfred	Riem	<hdp://www.manorrock.com/blog/>	
–  ReplicaKon	of	some	part	of	producKon	env	
–  It	is	also	runnable	under	CI	
–  Review	pom.xml	
•  address	uid/gid	permissions	issues	
•  address	docker	network	issues	
–  Review	deployment	json	
–  Need	to	access	logs	in	a	special	way	
•  Can’t	simply	docker	logs	–f	<hash>	
•  docker	exec	into	mesos	slave	
•  cd	/var/tmp/mesos/slaves/*/frameworks/*/executors/mycontainer8g*/runs/latest	and	tail	-f	stderr|stdout
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	Support	Layer	
•  A	wide	collecKon	of	off-the-shelf	support	services,	running	on	top	of	the	
foundaKon	layer	
•  Look	at	Three	Specific	Examples	
– NGINX	
– ELK	+	FileBeat	
– Etcd		
•  Other	Services	
– Services	to	enable	security	in	the	Business	Layer
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
The	Moving	Parts	of	the	Stack:	Business	Layer	
•  Docker	container	with	Java	fat	jar	inside	
•  Must	keep	much	in	mind	
– hdps://developers.redhat.com/blog/2017/03/14/java-inside-docker/	
– hdp://ridingthecrest.com/blog/2017/02/23/ray-tsang-docker-Kps-notes.html
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Running	the	Stack:	Maven	
•  Demo:	Show	foundaKon	layer	running	using	docker	maven	plugin	
Thanks	Manfred	Riem	hdp://www.manorrock.com/blog/	
•  Plugins	with	InteresKng	Uses	
– maven-antrun-plugin	
•  Convey	current	UID/GID	all	the	way	through	to	the	Docker	running	the	workloads	
•  Assemble	the	individual	service	Marathon	deployment	descriptors	into	one	json	file	
•  Use	curl	to	install	the	Business	Layer	app	before	tesKng	and	remove	it	aXer	tesKng	
– maven-resources-plugin	
•  Template	properKes	into	json	files	for	ports,	variables,	configuraKon	
– Fabric8	docker-maven-plugin	
•  Stand	up	foundaKon	layer,	note	hard-coded	network	values
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Running	the	Stack:	Maven	
•  Demo:	Show	foundaKon	layer	running	using	docker	maven	plugin	
Thanks	Manfred	Riem	hdp://www.manorrock.com/blog/	
•  Plugins	with	the	Usual	Uses	
– maven-compiler-plugin	
– maven-deploy-plugin	
– maven-failsafe-plugin	
•  Able	to	use	standard	Junit	with	Jersey	Client	to	assert	correctness	of	Business	layer	app	
•  Need	to	pass	system	properKes	for	endpoint	locaKon	
– maven-javadoc-plugin	
– maven-source-plugin
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Java	EE	8	Will	Be	Available	by	JavaOne	2017!
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	 34	
Java	EE	7	
Connector	 JAXB	JSP	Debugging	
Managed	Beans	JSP	Concurrency	EE	 Interceptors	 JAX-WS	 WebSocket	
Bean	ValidaKon	 JASPIC	 Servlet	JMS	 JTA	Deployment	
Batch	 JACC	
Dependency	
InjecKon	 JAXR	 JSTL	 Management	
CDI	 EJB	 JAX-RPC	 Web	Services	JSF	 Java	Persistence	
JSON-P	
Common	
AnnotaKons	 EL	 JAX-RS	
Web	Services	
Metadata	JavaMail	
CDI	
JSON-B	
Security	
Bean	ValidaKon	
JSF	
JAX-RS	
JSON-P	
Servlet	
Java	EE	8	Will	Be	Available	by	JavaOne	2017!	
JSP
Copyright	©	2017,	Oracle	and/or	its	affiliates.	All	rights	reserved.		|	
Safe	Harbor	Statement	
The	preceding	is	intended	to	outline	our	general	product	direcKon.	It	is	intended	for	
informaKon	purposes	only,	and	may	not	be	incorporated	into	any	contract.	It	is	not	a	
commitment	to	deliver	any	material,	code,	or	funcKonality,	and	should	not	be	relied	upon	
in	making	purchasing	decisions.	The	development,	release,	and	Kming	of	any	features	or	
funcKonality	described	for	Oracle’s	products	remains	at	the	sole	discreKon	of	Oracle.	
35

Chicago JUG / GOTO Meetup