SpringOne 2020 Replatforming Legacy Packaged Applications: Block-by-Block with Minecraft Michael Villiger, Sr. Technical Partner Manager at Dynatrace

1. Michael Villiger
Sr.Technical PartnerManager,Dynatrace
Replatforming Legacy
Packaged Applications: Block by
Block with Minecraft

2. Agenda
• Why Minecraftas an example?
• Earlystepstowardsmodernization:
• Minecraftona VM
• MinecraftonDocker
• Assessingcurrentstate
• Migrating to Tanzu KubernetesGrid(TKG)
• Initialperformancegoals
• SolvingforI/O
• vSphere CPI / CSI
• StorageClasses, PersistentVolumes, and PVCs, OH MY
• AccessingMinecraftonK8sfromthe outsideworld
• K8srequests/ limits
• Final thoughts

3. WhyMinecraft as anexample?
• Java based, but closed source
• Effectivelysingle-threaded
• Mostactionsinworldtake placewithina single“tick”
• Breaking blocks
• Placing blocks
• What the various blocks aredoing (SIGNIFICANT in modded Minecraft)
• What monsters and creatures aredoing
• Ticksaredesignedtorunwithin50ms
• If actions in a tick run longer than 50ms, ticks will be skipped!
• Skipped ticks perceived as lag
• Brokenblocksre-appear,placedblocksdisappear,playerslocationcan shiftbackwards
• Significantpersistentdiskrequirements
• The world itself
• The all-important backups!

4. Earlysteps towards “modernization”

5. Minecraft on aVM(KVM)
• Initialsteps at isolating Minecraft workload:
• QEMU+KVMon Ubuntu
• QCOW on ZFS +Spinning Rust
• This worked for a couple of years… But too slow ona Xeon E5-2620v2:
[minecraft/MinecraftServer]: Can't keep up! Did the system time change, or is the server overloaded? Running 23690ms behind, skipping 473 tick(s)

6. Minecraft on Docker
• Isolated(ish) from underlyingOS
• Allowed nearbare-metal performance
• Allowed easier access to both fast (SSD) and slow (ZFS) storage
• Fast storage for the world
• Slow storage forbackups
• Still using bindmounts because this predates volumes in Docker…
# dockerfile
FROM openjdk:8-jdk-slim
ENV MOTD="Minecraft Server: Enigmatica2"
ENV MAP=world
ENV DIFFICULTY=2
ENV GAMEMODE=0
ENV MAXPLAYERS=20
ENV PVP=true
ENV VIEWDISTANCE=10
ENV HARDCORE=false
RUN apt-get update && apt-get install -y wget unzip gosu
RUN addgroup --gid 1234 minecraft
RUN adduser --disabled-password --home=/home/minecraft/enigmatica2 --uid 1234
--gid 1234 --gecos "minecraft user" minecraft
RUN mkdir /tmp/enigmatica2 && cd /tmp/enigmatica2 &&
wget -q -c <snipped>
unzip Enigmatica2Server.zip &&
rm Enigmatica2Server.zip &&
echo "eula=true" > eula.txt
ADD entrypoint.sh /entrypoint.sh
RUN chown minecraft:minecraft /entrypoint.sh
COPY mc-monitor /usr/local/bin/mc-monitor
COPY . /tmp/enigmatica2
RUN chown -R minecraft:minecraft /tmp/enigmatica2 &&
chmod +x /tmp/enigmatica2/ServerStartLinux.sh &&
chmod +x /entrypoint.sh &&
chmod +x /usr/local/bin/mc-monitor
WORKDIR /home/minecraft/enigmatica2
EXPOSE 25565
ENTRYPOINT ["/entrypoint.sh"]
docker run -d -it --name minecraft-enig-06
--mount type=bind,source=/var/minecraft/data/enigmatica2_pregen_bop_world/world,target=/home/minecraft/enigmatica2/world
--mount type=bind,source=/tank/isos/minecraft/worlds/enigmatica2_pregen_bop_world/backups,target=/home/minecraft/enigmatica2/backups
-p 0.0.0.0:25569:25565 -e DEFAULT_OP=Akirasoft -e MINECRAFT_EULA=true mvilliger/enigmatica2:1.77

7. Assessing current state
• Host and all containers on it automatically
monitored by Dynatrace
• CPU mostly ~6-8%(pegging a bit morethan a core
24/7)
• Modded Minecraft is a memoryhog

8. Assessing current state
• Need to tracktransaction response time and
throughput(tick time):
• CPUanalysis shows significant CPUin func_71217p
• Normally,we’d just decompile the class, but Minecraft
is obfuscated. Luckilythe Mod Coder Pack (MCP)*
exists, and can tell us about this method:
• Name: tick
• Description: Mainfunction called by run() every loop.
• So wecreate a custom service:
*http://export.mcpbot.bspk.rs/

9. Migratingto Tanzu Kubernetes Grid

10. Initialperformance goals–Quarantine homelabupgrades!
• Needed to moveto an efficient yetaffordable modern CPU with decent
thread countyet highclockspeed
• AMD Epyc7302p (16c/32c, 3.3ghz)
• Needed relatively low latencystorage and HCIbenefits
• 3 node cluster
• All-flash vSAN
• IntelP4610forcache tier
• CapacitytierisnotontheHCLsoIwon’tmentionithere…
• 10gb networking all around!

11. Solvingfor I/O–vSphereCPI/CSI–In anutshell…
• Cloud Provider Interface(CPI)
• Interface between K8sandthe underlying IaaS (providing config infotonodes, andmore)
• In-treeCPI
• Part of Kubernetescore
• Subjectto Kubernetesreleasecycle
• Out-of-treeCPI(https://cloud-provider-vsphere.sigs.k8s.io/concepts/cpi_overview.html)
• Managedby cloudprovider,in thiscaseVMware
• ContainerStorage Interface(CSI) (https://cloud-provider-vsphere.sigs.k8s.io/concepts/csi_overview.html)
• Abstraction layerbetween Container Orchestrator (K8s) andStorage
• Works with CPItoattach storage tothe rightnodes
• Requires out-of-tree CPI
• TKG deploys and configures the out-of-treevSphere CPIand CSI out of the box, it just works!

12. • StorageClasses allow the CSIto provision storage for us:
• Fast vSAN flash:
• Slow NFS via vSAN:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
provisioner: csi.vsphere.vmware.com
parameters:
datastoreurl: "ds:///vmfs/volumes/476fefe3-22c24a52/"
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
datastoreurl: "ds:///vmfs/volumes/vsan:5284746fcc615ee8-051620842ced0820/"
Solving forI/O –
StorageClasses, PersistentVolumes, and PersistentVolumeClaims, OH MY!

13. • StorageClasses:
• When using the out-of-tree vSphere CSIbe sure touse the rightprovisioner!!!
USE:
provisioner: csi.vsphere.vmware.com
NOT:
provisioner: kubernetes.io/vsphere-volume
Solving forI/O –
StorageClasses, PersistentVolumes, and PersistentVolumeClaims, OH MY!

14. • PersistentVolumeClaim:
• Used todefine information aroundthe persistent storage our appneeds
• PersistentVolume:
• Either created in advance ordynamically when a PersistentVolumeClaim is created
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mc-world-claim
annotations:
volume.beta.kubernetes.io/storage-class: fast
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: fast
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mc-backup-claim
annotations:
volume.beta.kubernetes.io/storage-class: slow
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: slow
$ kubectl get persistentvolumes
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-a70792ee-353c-45ce-9ccf-f56ecd863161 20Gi RWO Delete Bound minecraft/mc-world-claim fast 5d4h
pvc-bb6eeb96-ca94-423d-a7e1-c42c7d94d55a 100Gi RWO Delete Bound minecraft/mc-backup-claim slow 5d4h
Solving forI/O –
StorageClasses, PersistentVolumes, and PersistentVolumeClaims, OH MY!

15. • PersistentVolumes get mounted as volumesin yourdeployment:
• Volumematches up to PersistentVolumeClaim
• volumeMount defines wherein the containerthe filesystem appears
spec:
volumes:
- name: minecraft-data
persistentVolumeClaim:
claimName: mc-world-claim
- name: minecraft-backups
persistentVolumeClaim:
claimName: mc-backup-claim
containers:
- name: mc-enigmatica-server
image: mvilliger/enigmatica2:1.77
env:
- name: DEFAULT_OP
value: 'Akirasoft'
- name: MINECRAFT_EULA
value: 'true'
ports:
- containerPort: 25565
name: minecraft
volumeMounts:
- name: minecraft-data
mountPath: /home/minecraft/enigmatica2/world
- name: minecraft-backups
mountPath: /home/minecraft/enigmatica2/backups
Solving forI/O –
StorageClasses, PersistentVolumes, and PersistentVolumeClaims, OH MY!

16. Deploymentinfull
• Onlya single instance since wecan’t “scale-out” this monolith
• Usingsimple definitions of environmentvariables, could beimproved with config-
map but KISSprinciple applies
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mc-enigmatica-server
namespace: minecraft
labels:
app: mc-enigmatica-server
spec:
replicas: 1
selector:
matchLabels:
app: mc-enigmatica-server
template:
metadata:
labels:
app: mc-enigmatica-server
spec:
volumes:
- name: minecraft-data
persistentVolumeClaim:
claimName: mc-world-claim
- name: minecraft-backups
persistentVolumeClaim:
claimName: mc-backup-claim
containers:
- name: mc-enigmatica-server
image: mvilliger/enigmatica2:1.77
env:
- name: DEFAULT_OP
value: 'Akirasoft'
- name: MINECRAFT_EULA
value: 'true'
ports:
- containerPort: 25565
name: minecraft
volumeMounts:
- name: minecraft-data
mountPath: /home/minecraft/enigmatica2/world
- name: minecraft-backups
mountPath: /home/minecraft/enigmatica2/backups

17. AccessingMinecraft onK8s from theoutside world
• Ideallywe’d use NSX-Tbut I don’t havethat deployed yet, so MetalLB to the rescue!
• MetalLB requires configuration before it does anything,hereI just give it a pool of unusedIPs on myLAN
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.18.10-192.168.18.250
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

18. AccessingMinecraft onK8s from theoutside world
• Now that we havea load balanceravailable, wecan create a service of type LoadBalancer:
• This provides access to LANclients, for myuse-case Iuse NAT/port-forwardingto provide access to the outside world
---
apiVersion: v1
kind: Service
metadata:
name: mc-enigmatica-server-service
labels:
app: mc-enigmatica-server
namespace: minecraft
spec:
type: LoadBalancer
ports:
- port: 25565
protocol: TCP
targetPort: 25565
selector:
app: mc-enigmatica-server

19. K8srequests andlimits
• Requests:
• Used by scheduler toplace pods on nodes
• If set to higher than node size, pods will never get scheduled (we need
XL worker nodes here!)
• We know Minecraft will use over acore, andquite a bit ofmemory
• Limits
• Exceeding memory limit will result in podbeing killed
• Exceeding CPUlimit willresult in pod being throttled
• butwecanmonitorforthatoccurring!
• Setting these tosane limits just over what our requests are
resources:
limits:
memory: 12Gi
cpu: 2000m
requests:
memory: 11Gi
cpu: 2000m

20. Current state:Runsgreat!!!

21. Finalthoughts

22. Final thoughts
• Thereare more complicated Minecraft Docker images out there,the most advanced is probably itzg/minecraft-server
(https://hub.docker.com/r/itzg/minecraft-server/)
• Sourcing hardware during a pandemic is hard. Some thoughts:
• From placedorder toworking environment took over 3months
• Hardware failures abounded:
• OneDOACPU thattook6weekstoreplaceduetonone existingin theUS
• Onefailed (melted)CPU socket,takinganotherCPU withit
• Bad CAT6cables causing sporadic vSANoutages
• If anyone wants tohop on the server, reach out tome on Twitter: @mikevilliger
• The new server is really fast!

23. Simply smarter clouds