Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin


Published on

Our web hosting company is using many different services for managing e-mail, spam-filters, DNS, domain registrations, SSL registrations, ticket systems and more. Some of these services have well defined Web API’s, others can only be managed by simple command line scripts.
In this session we will explain how we tried to automate the various workflows by using a messaging system such as RabbitMQ for communication between our cfml based customer control panel and these services.

Published in: Technology
  • Be the first to comment

ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin

  1. 1. Multi-language / multi-OS commmunication using RabbitMQ Wil de Bruin
  2. 2. Who am I • Wil de Bruin – Software / system engineer • Wageningen,The Netherlands • Graduated in Environmental Sciences • Reseach microbiologist (1987) • CTO – Founder Site4u (1994) • CFML since Allaire ColdFusion 1.5 • Coldfusion training, consultancy, software development and hosting • Coldbox user since 2008
  3. 3. Agenda • Use case:Web Hosting Panel • RPC vs messaging • RabbitMQ Concepts explained • Our RabbitMQ messaging setup • Demo / code • Conclusions
  4. 4. Use case: Web hosting panel
  5. 5. Web hosting panel • Dns, Email functionality: just like Cpanel, plesk etc • Customers demanding SLA reports: integration with Monitoring • Resource usage:VMWare integration • Management of servers and networks: complex networks require more automation
  6. 6. Server 700… Server 2 VueJS frontend Portal DB MS SQL CF Rest API backend Redirects mySQL Accounting MS SQL Domain registrar XML API Domain registrar (no CFML) API Antispam service ( API ) Nagios monitoring API ?? Mail Server Mail DB /var/vmail/domain DNS Server powerDNS Commandline Commandline Zabbix trend monitoring API ?? Server 1 Commandline
  7. 7. Some limitations: Mailserver • Mail Server is OpenSource (postfix / Dovecot (IMAP) / webmail (Roundcube) • User config is defined in a database • User mailboxes as files and directories.Will NOT be deleted when removed from database. • Roundcube in separate database • Report sizing with commandline tools, noAPI available. • Environment: Linux, bash and Python
  8. 8. More limitations: DNS and DNSsec • PowerDNS has mySQL backend, but no management of DNSsec • Workflow • Create DNSsec in powerDNS (commandline) • Send key to registrar • Registrar 1: XML API, several languages • Registrar 2: API for CFML is hard, several other language libs available (e.g PHP) • Change status in control panel • Environment: Linux, bash, commandline,
  9. 9. Still more limitations: SSL and HTTP configs • SSL installation is manual, customers need overview (commandline) • Mixture of • Custom certificates from several providers • Let’s encrypt: automated management, but how to notify our control panel? • Environment: • Linux, bash, commandline, python • Windows, powershell
  10. 10. RPC vs messaging
  11. 11. RPC and REST limitations • Only one receiver • Sender knows about receiver • Receiver knows about sender • Blocks request usage
  12. 12. RPC
  13. 13. RPC
  14. 14.
  15. 15. Messaging advantages • Cross platform, multiple technologies: Flexibility • Decoupling Producers and Consumers • Event driven programming: scalability • Queuing • Load balancing
  16. 16. Rabbit MQ concepts explained
  17. 17. AMQP Advanced Message Queuing Protocol ( • Binary protocol • Diverse programming languages can communicate • Queuing • Routing • Exchanges • Secure • Reliable
  18. 18. RabbitMQ • Multiple messaging protocols (AMQP, STOMP, MQTT) • Open source, well known • Lots of libraries in different languages (Java, .NET, Python,Ruby, Javascript) • Http management • Simple concept: just accepts and forwards messages • Developer friendly: deploy with Docker, Puppet and more
  19. 19. RabbitMQ messaging concepts (1/5)
  20. 20. RabbitMQ messaging concepts (2/5)
  21. 21. RabbitMQ messaging concepts (3/5)
  22. 22. RabbitMQ messaging concepts (4/5)
  23. 23. RabbitMQ messaging concepts (5/5)
  24. 24. Our RabbitMQ messaging setup
  25. 25. Sending commands in our hosting environment CF control panel Mail server Delete Domain Get Mailbox Size Return Result CF control panel Dns server Enable DnsSec Return Result CF control panel ANY server List SSL certificates Return Result
  26. 26. Communication between hosts
  27. 27. Messaging
  28. 28. Our messaging setup • Topic exchange • Routing keys: • mysite4u.servicename.cmd for commands to a service • mysite4u.servicename.response for responses of a service • Queues: servicename-commands and mysite4u-responses • Exchange: mySite4u-topic • Queues and messages are DURABLE i.e persistent.
  29. 29. Commands and responses COMMAND Headers: action = string (for example: archive of getmailboxsize) Properties: correlation_id: string content_type: string (application/json of text/plain) Body: [string,json] { "params": { "domainname": "", "param2": "Second parameter" } } Response Properties: correlation_id: string content_type: string (application/json) Body: json { data: [string, json] status: [ok,warning,error] messages: string }
  30. 30. Setup • CF Rabbit Producer (hosting control panel) • RabbitMQ consumer on hosting servers • RabbitMQ producer on hosting servers • CF Rabbit Consumer • Polling, the easy way • Polling at higher frequency, still easy • Pushing messages (a.k.a the hard way)
  31. 31. CFML RabbitMQ producer • Let’s start with • • RabbitMQ tutorials • Java API Guide • Create cfproject, download jar and dump in /lib folder • Create mappings in wirebox • Add rabbitsettings in coldbox.conf • Create onAppInit to establish RabbitMQ connection • Create a Service to send messages
  32. 32. Flexible messaging daemon for hosting servers • RabbitMQ Consumer written in Python • Executes any command based on configuration, e.g: [mysite4u.sharedmail.cmd:archive] script: /usr/local/scripts/archiveMail/archiveMailDomain.php $domainname run-as: vmail reply-to: mysite4u.sharedmail.response • Output of script is piped into RabbitMQ producer response.
  33. 33. RabbitMQ CF consumer
  34. 34. CF RabbitMQ consumer • Pulling vs pushing • Pushing should be easy (just like other languages….) • Pushing more flexible • Missing correct examples! • Pulling is easy • 1 minute limitation • CFConcurrent to the rescue..
  35. 35. What’s missing • Security: • Use SSL in rabbit connections • Make sure your reply is a valid one: use correlationID for command and match this in handling the reply • Monitoring • Make sure your commands are processed • Monitor your queue lengths
  36. 36. Conclusions • Messaging is very flexible, new ways of communication • No more coding barriers, many languages can be used • CF rabbitMQ producers easy to achieve • CF rabbitMQ consumers are hard to configure, due to language limitations • Is CF the best tool for a rabbit Consumer in a micro services architecture? •
  37. 37. Thank you! Questions? More questions? E-mail: Slack: @wil-site4u