This document discusses using queues to improve the scalability of PHP applications. Queues allow asynchronous and distributed processing of tasks to improve performance and allow applications to handle more traffic. The document recommends using Zend Server's job queue to process long-running or asynchronous tasks outside the main request. It provides an example of using the job queue to asynchronously process payments in Magento to improve the customer experience.

1. Do you queue? Characteristics of scalabilityKevin SchroederTechnology EvangelistZend Technologies

2. About me Past: Programming/Sys Admin Current: Technology Evangelist/Author/Composer @kpschrade

3. I blog at eschrade.comFollow us!(good things will happen to you)Zend Technologieshttp://twitter.com/zendhttp://twitter.com/kpschrade (me!)

4. Twtpoll resultsCould your PHP apps benefit from being able to process data or execute asynchronously?

5. Why would you want to queue?PerformanceExecute logic apart from the main request (asynchronicity)ScalabilityThe ability to handle non-immediate logic as resources are available

6. Typical anatomy of a PHP ApplicationBad for scala-bility!PresentationApplication ControlDatabase AccessBusiness LogicPresentationApplication ControlBusiness LogicPresentation| 6

7. Good for ScalabilityPresentationSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlSomething.phtmlApplication ControlSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpBusiness LogicSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpDatabase AccessSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.phpSomething.php| 7 | 2-Mar-11

8. What helps make software scalable?Defined tasksLoose couplingResource discovery

9. The Golden Rule of Scalability“It can probably wait”

10. Asynchronous execution usesPre-caching dataData analysisData processingPre-calculating (preparing data for the next request)Data is “out of date” once it leaves the web serverImmediacy is seldom necessary

11. ConsiderationsWaste disk spaceControl usage (don’t let users do anything they want)Pre-calculate as much as possibleCalculate and cache/storeKeep data processing off the front end serversDon’t just cacheDon’t let it substitute for thoughtCache hit rates can be meaningless if you have hundreds of cache hits for a request

12. ConsiderationsBuild a deployment mechanism with NO hardcoded values like directory or resource locations

13. Make as much as possible configurable/discoverableDecouple/PartitionDon’t tie everything (relationships and such) into the databaseUse queues/messagingStomp interfaces are really good for PHP – Can also use Java BridgeZend_Queue has several interfacesTry to use stateless interfaces(polling is more scalable than idle connections)

14. OptionsUse Cron /w PHP CLI (not the best idea)People mostly use cron simply due to availabilityIt’s an available option, not necessarily the best option.Use GearmanUse home-grown (don’t do this)Use pcntl_fork() (NEVER do this)Use Zend Server Job Queue

15. Your only real optionsVery cloud friendly* I am not an expert on Gearman. Corrections will be taken in the spirit that they are given.For obvious reasons, I will focus on Zend Server

16. Zend FrameworkZend ServerPHPApplicationPerformanceReliability &ManagementScale-OutBusiness-gradePHPAccelerationMonitoringClusteringHot FixesOptimizationRoot-CauseJob QueueSupportCachingDownloadsJava BridgeConfigurationLinux(rpm/web repositories)IBM i(PTF)Windows(MSI) What the heck is Zend Server?15Zend Server

17. Job Queue Architecture – Elastic BackendWeb Server /w JQWeb ServerUsers!Web Server /w JQWeb ServerLoad BalancerWeb Server /w JQWeb ServerProsScale the backend as necessaryDefault (easy) mechanismConsGetting the job status requires using a DB

18. Job Queue Architecture – Elastic FrontendWeb ServerWeb Server /w JQUsers!Web ServerWeb Server /w JQLoad BalancerWeb ServerWeb Server /w JQProsEasy to communicate with the Job Queue server handling the jobConsRequires you to build your own interface (or just use mine!)

19. Kevin’s favorite way to implement itCreate a task-handling controllerCreate an abstract task classUnderstands the Job QueueSelf containedIf Elastic Backend: connects to localhostIf Elastic Frontend: connects to load balancer (my preferred), load balanced JQ servers manage themselvesExecute the task, have it serialize itself and send it to send to the task handler

20. Classes involved in the demoManagerHandles connecting to the queue and passing results back and forthJobAbstractAbstract class that a job would be based off ofResponseThe response from the manager when a job is queued. Contains the server name and job numberGetRemoteLinksScans a given web page and reports back a list of all the links on the page

21. Execution FlowCreate job and set dataExecute jobJob passes itself to the queue managerManager serializes jobManager uses HTTP call through a load balancer to queue the jobThe queue on the other end returns the job id and server nameJob ID and server name is passed to the clientClient polls the manager to get a completed jobWhen the job is returned pass the serialized version of the executed job

22. Demo

23. Neato, but how in the world does this pertain to Magento?

24. How?Long running front-end requests kill front-end responsiveness which kills the user experience40% will wait no more than 4 seconds for a website (Forrester)Calculating cache data on-the-fly can lead to bad user experiences (abruptly slow page load times) and consistency problemsLong running page requests can push up against max_execution_timeFor example, handling payments from a slow CC APISimply setting it to run longer is a band-aid on a broken bone

25. Solution?Build Zend Server integration extensionsZendserver_JobqueueZendserver_MonitorUtilize the event system to intercept actionsAsync_Payment (for payments, in this case)Profit! (scalability AND performance, actually)

26. Building a jobInstall the Zend Server Magento extensionCreate a class that extends Zendserver_Jobqueue_JobAbstractCall $job->execute();

27. Payment as an exampleHooks an observer into controller_action_predispatch_checkout_onepage_saveOrder, core_block_abstract_to_html_afterChecks if a defined template is being renderedInjects JavaScript that changes the review.save() methodInjects a new URL for submitting the transaction which creates a jobPasses POSTed data to a jobJob executes by submitting the payment URL on behalf of the browser, storing the resultBrowser pings the server to see if the job has completed executing yet

28. Scaling the solutionWebserverJob Queue/ WebserverWebserverLoad BalancerJob Queue/ WebserverWebserverJob Queue/ Webserver* Duplicate deployments on ALL machinesDatabase or Session Clustering

29. DownloadsZend Serverhttp://www.zend.com/serverZendserver_Jobqueuehttps://github.com/kschroeder/Magento-ZendServer-JobQueueZendserver_Monitorhttps://github.com/kschroeder/Magento-ZendServer-MonitorAsync_Paymenthttps://github.com/kschroeder/Magento-Async-PaymentJob Queue library & demohttps://github.com/kschroeder/ZendServer-JobQueue-Job-APIhttp://bit.ly/hWJBYw

30. Magento/Zend Case Study - http://bit.ly/horpFFFollow us!Zend Technologieshttp://twitter.com/zendhttp://twitter.com/kpschrade (me!)

31. Get this information and all the examples at eschrade.com…