The document discusses making a stateless service-oriented application highly available using GlusterFS. It describes setting up a GlusterFS cluster with replicated volumes to provide a centralized data store. The application is configured to mount the GlusterFS volume and an update mechanism is built to notify the application when data changes by monitoring the volume for modifications. This allows making the application redundant and aware of data changes for high availability.

1. High Availability SOA APP with GlusterFS
Vladimir Vitkov
2017.10.07 / TarnovoConf
High Availability SOA APP with GlusterFS Vladimir Vitkov

3. Who am I
• Vladimir
• Sysadmin / DevOPS
• FOSS proponent and addopter
• Experimentor
• Relatively nice guy for a chat*
High Availability SOA APP with GlusterFS Vladimir Vitkov

4. What is SOA
High Availability SOA APP with GlusterFS Vladimir Vitkov

5. Let’s put the basis on what SOA is and why it is usefull for the
world. Service Oriented Architecture is a way of thinking and
coincidentaly separates the application into multiple services (not
necessarily micro ones).

6. The monolith
High Availability SOA APP with GlusterFS Vladimir Vitkov

7. The monolith approach is usually the first one. It is easier to
construct as everything is in the same place. There is no need to
account for delays, communication failures and similar. Of course it
has it’s drawbacks. The piece of code is huge, hard to maintain and
scale. Sometimes ugly, but always hard on supporting.

8. Services
High Availability SOA APP with GlusterFS Vladimir Vitkov

9. The services approach usually evolves from the monolith one. The
big APPLICATION (yes all capital) gets split at some point into
smaller more contained chunks that do few things (initially). The
final incarnation is one service does one and only one thing. The
benefits are easier maintenance, scaling and supporting the code.
Usually this is done with a total rewrite but sometimes it is done
with a state of mind as refactoring code.

10. SOA
High Availability SOA APP with GlusterFS Vladimir Vitkov

11. SOA is mostly a pattern in software design in which the components
of the system are separated from one another and communicate via
a protocol (usually network one). The principles are neutral and
independent of vendor, product or technology. Most often this is
achieved with the help of the protocols. Most common ones are
SOAP and REST. As such it is in fact microservices architecture but
on a conceptual level. Currently large-scale applications are based on
the SOA principles and microservices model.

12. The Application
High Availability SOA APP with GlusterFS Vladimir Vitkov

13. Well one of the services only

14. What is the system
• Large scale Ecommerce product
• Quite a few services that interact
• Lots of users
• Lots of traffic
High Availability SOA APP with GlusterFS Vladimir Vitkov

15. I am running a fairly complex and large application, we have quite
some services, lots of interactions, lots of traffic and users and this
needs to always be available and run.

16. What is the service
• Brand New service
• Replacing custom hacks
• To be in service in 3 weeks
• After several total redesigns
High Availability SOA APP with GlusterFS Vladimir Vitkov

17. The service i am going to tell you today was brand new. It was born
in 3 weeks. Main purpose of it was to replace some custom hacks in
our processess and more fluidly control access to resources. It has
been totally redesigned several times. It has the core of it changed at
least 3 times. And finally introduced without anyone noticing (except
directly involved parties).

18. What is the service (2)
• Speaks REST
• Stores some local data
• Transforms requests
• Pulls data from 3rd party service
• Is stateless
High Availability SOA APP with GlusterFS Vladimir Vitkov

19. This service speaks rest, stores some configuration in local storage.
When someone asks it the right question, respoonds after pulling
some 3rd party data, applying transforms and generally humming
along. The best thing: it was stateless.

20. Wha the service was not
• Highly available
• Redundant
• Without central data store
• But needs to be
• Does not know of changes
High Availability SOA APP with GlusterFS Vladimir Vitkov

21. Unfortunately not everything is hunky-dory in the SOA land. The
service was not highly available, not redundant, had no idea of
central storage and generally was not ready for prime time. But it
needed to be all this things. So OPS had to rescue it and give it
wings.

22. The Solution
High Availability SOA APP with GlusterFS Vladimir Vitkov

23. Leave the devs to develop and ops to operate. But join them for a
synergy.

24. Highly Available
• Bring up more instances
• Spread them around
• Load balance them
• Monitor them for health
• Recover/replace them
High Availability SOA APP with GlusterFS Vladimir Vitkov

25. To make the service more highly available do the sane thing. The
service is stateless so apply the regular tricks. Add more
instances/server. Put them in a load balancer (either DNS or some
other like HAProxy). Of course do not forget to monitor the stuff.
Use your best judgement. Monitoring is moot if there are no
procedures to detect, recover or replace failed instances. Doing this
in an automated fashion helps a lot.

26. Redundant
• High Availability
• Monitoring
• Recover/replace
High Availability SOA APP with GlusterFS Vladimir Vitkov

27. Keep that service stateless and Highly available. Redundancy is
necessary for high availability anyways.

28. No central data store
• Core issue
• Create GlusterFS cluster for replica
• Reconfigure app to use clustered data store
• Monitor it
• Recover/replace
High Availability SOA APP with GlusterFS Vladimir Vitkov

29. The missing central data store become the core issue that prompted
developing this approach. The application was aware only of local
storage and porting it to support remote/central storage would have
missed deadlines. So ops had to rescue the situation. The solution
we thought of included creating a replicated GlusterFS cluster. It
took care to synchronise and replicate the needed local data to all
nodes. After that it was just necessary to reconfigure the application
to use this local (but in reality replicated) data storage. As a
recurring theme do not forget the monitoring. Monitor all
components available and monitor them well. Again do not forget to
automate recovery in case of troubles with the nodes.

30. Become aware of changes
• App does not know about new data
• Build update mechanism
• Inotify + curl
• NFS is not a friend
• FUSE is not a friend too
High Availability SOA APP with GlusterFS Vladimir Vitkov

31. Another problem with this approach was that the application is not
monitoring local data store for changes. To solve this add a
mechanism to inform it that the data store has changed and should
be reread/reconsidered. Devs built a mechanism to do this and
exposed it. The detect the changes on the filesystem level and
trigger this mechanism. Inotify and curl fit nicelu into this. But
unfortunately NFS and FUSE do not support inotify events. FUSE
may support them but requires special codding in the FS to have it.

32. How to do it
High Availability SOA APP with GlusterFS Vladimir Vitkov

33. Show how it is built

34. Components
• GlusterFS
• nfs-common
• inotify-hookable
• nice utils around
High Availability SOA APP with GlusterFS Vladimir Vitkov

35. First step is to install everything necessary. This includes the
GlusterFS itself, nfs client, inotify monitoring tool and any utilities
you may need

36. Create GlusterFS cluster
• Start gluster daemon
• Assemble cluster
gluster peer probe remote.peer.host
• /var/lib/glusterd/peers/*
High Availability SOA APP with GlusterFS Vladimir Vitkov

37. Next step is to build the cluster. Building the cluster is basically
probing the peers. Do the probing from one node only. No need to
do it from the other nodes. Make sure your DNS works properly and
can resolve the names. If you don’t want to mix names/ip addressess
you may have to manually edit gluster peers to use names.

38. Create volume
• Create replicated volume
gluster volume create vol1 replica 2
transport tcp peer1:/vol peer2:/vol
High Availability SOA APP with GlusterFS Vladimir Vitkov

39. Creating the volume is a simple command. This will initialize the so
called bricks that form the GlusterFS fabric. As we are aiming for
redundancy and replication we create the volume with replica count
2 and add only 2 bricks. If bricks are created on the root partition
you will need to use "force"option. Take some time to review the
GlusterFS tunning options (Resources slide).

40. Reconfigure
• mount
mount -t nfs localhost:/vol1 /app
• Start the app
High Availability SOA APP with GlusterFS Vladimir Vitkov

41. We chose to use local nfs mount to enable functioning of the service
in a split brain/down situation. The data is replicated everywhere so
it is not important from where we serve. GlusterFS has a
reconciliation mechanism that can handle split brain situations. Of
course chosing NFS (FUSE also won’t help you) leads us to the next
problem. No inotify events

42. Update info
• Data added on other node
inotify-hookable
--watch-directories /vol
--on-modify-command curl
High Availability SOA APP with GlusterFS Vladimir Vitkov

43. It is important to update the applications view of the data store. We
do this by detecting changes (to the local brick) and triggering the
update cycle with curl. monitoring the local brick is necessary due to
inotify events not being emitted for nfs.

44. DEMO Time
High Availability SOA APP with GlusterFS Vladimir Vitkov

45. In this demo we’ll show you the final result. We’ll play with disabling
the update mechanism, so we can demonstrate that content is
properly replicated but the application is not aware of it.

46. Closing
High Availability SOA APP with GlusterFS Vladimir Vitkov

48. Q/A?
High Availability SOA APP with GlusterFS Vladimir Vitkov

49. Contact
• Demo: http://is.gd/gluster_demo
• Slides: https://is.gd/ha_soa_tarnovoconf
• mail: vvitkov@linux-bg.org
• GPG: A162 1211 8ACB 4CC5
High Availability SOA APP with GlusterFS Vladimir Vitkov

50. Resources
• http://is.gd/gluster_tune
• http://is.gd/gluster_quickstart
• http://flask.pocoo.org/
High Availability SOA APP with GlusterFS Vladimir Vitkov