Design Guidelines for Large Java Message-based EAI SystemsAn EAI Case StudyIntertechMarch 2006

This TalkPresents an EAI case studyA very large EAI system for a retail chain.Identify issues and challenges encountered in the projectIdentifies lessons learned and recommendations for your EAI projects.Let’s you know others do have it as bad as you.The story does have a happy ending – maybe providing hope to the hopeless.

Large Java EAIMessaging/EAI development  Web or other distributed app developmentEspecially when very largeMany new or significantly altered considerationsRequirement differencesTime and space needsProcess control/orchestrationFailure handlingMonitoringProprietary nature of vendor solutionsSupport turnoverStaffing needs

The Case Study SituationA major retail chain has dozens of distribution centersEach distribution center or warehouse services hundreds of stores (>1200 total stores).Each distribution center is moving thousands of “cartons” (i.e. boxes) around the warehouse each dayReceiving them from trucks through dock doors.Moving them with fork lifts to storage areas in the warehouseConveying them to “break down” areas for distribution to stores.Conveying them down belts to storage areas or outbound trucks.Moving them onto trucks that depart the warehouse.

The Case Study SituationA box is tracked via labels and bar code readers.Some “reads” are manual and some are automated.Generating literally hundreds of “events” per second per warehouse.RFID was about to make create more events.More “reads” from more points in the warehouse.Potentially adding store “reads” to the event list.

“The System”The retail chain wanted all the data on events regarding the movement of cartons sent to HQProviding them with unparalleled real time information on inventory levels and product status. Providing more accurate information for merchandise analyst and productivity monitoring for warehouse managers.Providing a Java Web application to nearly 10,000 users to access the data company wide.Reports galore.Some limited ad hoc query reporting.

Let’s do the math…>25 warehouses Each generating ~15-20 carton events per secondAveraging 400 messages a second incoming at HQPeak around 1300 messages a second incoming at HQData around an event ~200bytes/msg24x7x52 (31,449,600 seconds for those not counting)= ~ 4-7GB a day

…and the math wasn’t getting any betterDuring Christmas time things were worse – much worse.The organization wants to double its current size by 2010!Oh ya…did I mention RFID was comingTripling or quadrupling the number of events

My ChallengeDesign and implement a system to get the data from the warehouses to HQ In near real time to support the reporting needsUse whatever makes sense (to some degree – more later)With a good size team (20-25 people in various roles)

My Background15 year “grizzled” veteran of software development.6 years of Java experience.Author of a Java book.Experienced architect, manager, mentor, trainer.Eager to take on any software system challenge.No experience in EAI!An organization with limited EAI experience.

“The Perfect Storm”The size of the EAI project + abilities of the development team =

The SolutionSignificant company resources and investment in SeeBeyond EAI product.Put SeeBeyond at all the endpoints (warehouses and HQ).All data would move through SeeBeyond.SeeBeyond is Java based (also a company technology direction).Write routing/minor processing code in Java in SeeBeyond.Significant company resources and investment in Oracle RDBMS.Oracle already at the warehousesObtain a “honking” big Oracle DB at HQ.Use Oracle stored procedures for heavy lifting (data processing – report data preparation).

Solution diagram

Problem 1 – We weren’t readyAs an architect, I was not aware how different an EAI/Java messaging system is.Asynchronous-everywhere natureHad no patterns to follow (No – I had not read Hohpe/Woolf EAI book)Did not have an awareness of the vendor landscapeWas easily talked into solutions by others.My organization didn’t see how big it wasHad only implemented smaller EAI solutionsFinding good help was hard – and a critical stepInternally – lots of support but no experienceContractors – lots of desire, but little implementation experience to the scale/level of effort

Getting Yourself ReadyGet yourself readyUnderstand your options – all the three letter E’s (EAI, ETL, EII, etc.)Read EAI patternsKnow the products (WBI, Vitria, Tibco, WebMethods, SeeBeyond, etc.)Find people with real EAI experienceExperienced with systems matching the size of your appFind people with product expertiseFind people with design/pattern expertise

EAI PatternsEnterprise Integration Patterns: Hohpe/WoolfNext Generation Application Integration: LinthicumIT Architectures and Middleware: Britton

Getting Resources ReadyLet the network engineers know of your plansYou are going to be using a significant amount of pipe.Have you considered failover/load balancing? (comm lines around warehouses get cut on occasion)Let the database engineers know of your plansTerabytes of data to be stored and processed – where will it go?Consider backup/recovery systemsDatabase logs/archivingPerformance tuning

Getting Support ReadySupport staffs will be lost at turnoverHow many of your support shops really know …How to manage application servers?How to manage web applications effectively?Can you expect them to be able to operate, maintain and support component based messaging systems?Do they know what a message server or bus is? Across a very distributed environment?Get them trained early (in messaging infrastructure).Have them help you design the monitoring tools and alert systems.Work together to develop proactive systems checks and troubleshooting procedures.

Get Others ReadyIf your development team isn’t ready, what about…Testing/QA teams?Analyst?Managers?For example, finding experienced testers for asynchronous messaging systems is difficult.They usually need intricate knowledge of the messaging subsystem monitors and admin capabilities.

Problem 2 – Proprietary EAIEAI Products/Solutions are many.EAI Standards are few.EAI/ETL/EII/… market place is tumultuousSun has purchases SeeBeyondIBM bought AscentialEveryone calling their product an ESBProducts/Solutions have scale limitsSome they know aboutOthers they do notJava alone does not make you platform independent.

Examine your solution optionsSee if what you have would work.There is a reason MQ has been around a long time.Where possible consider tried, true and already deployed platforms (but again do the math and see if they can support the extra load).In house support is probably equipped (more in a bit)Not everything has to travel by message.Consider multiple/alternate technologies for parts of your solution.ETL is great for certain parts of a large solutionThere is a reason why products like Oracle are expensive (technologies like Oracle Replication – more in a bit).Does, however, create more issues of timing.

Not everything has to travel by messageConsider multiple/alternate technologies for parts of your solution.Replication of reference dataBulk/batch transfersNon-real time needsETL is great for certain parts of a large solutionExamine features in your DB/App ServersThere is a reason why products like Oracle are expensive (technologies like Oracle Replication – more in a bit).How about those Message Beans in the app server?This can, however, create more issues of timing.

Reference DataIn many applications, you need reference data on both ends of the messaging systems.You can build a “replicating” message engine to treat this like other message data (not recommended).Referential integrity becomes a real problem.Consider issues of message timing (PR becomes the 51st state but messages with PR references start to arrive before the new state data does)Use simple replication technologies where possibleETL tools if reference data changes only happen at certain times.Technologies like Oracle Replication for real time (it can operate over a WAN).

Java = interoperability (not always)Even when you use Java, how is it being applied?Java running inside of proprietary components (like SeeBeyond eWays) does not make you portable. Write component code that can be used by or incorporated by proprietary systems.Under the covers, is the vendor using JMSJMXJAX-RPCEtc…

Process Outside the BusProcess outside the message bus/subsystem if you canLet the bus focus on delivering the goods.Too much processing time in the bus will createScalability problemsMonitoring problemsPossibly interoperability problems (especially when using proprietary technology/components)Process with components that are Flexibleeasy to get at (and change)interoperable (if possible)and contain reusable business logic (if possible)

Problem 3 – We didn’t figure or figured poorlyWe didn’t do enough “math” up front.We didn’t plan for failure/growth.The messages moved slower than anticipated.The message processing took more time than expected.The amount of data was larger than expected.

Do the math…How much time its going to take to get a message from A to BTest that estimate early.Work with the business analysts to figure out how many messages need to be moved.Make volume estimates part of the non-functional requirements gathering process.Check that against the existing databases if possible.How much data needs to be packaged, shipped, processed, stored?Design the messages and calculate the size of the overall message (XML and all).Calculate the rate and add up the total volume.

…and pad your answerDo you have room to spare??Can the messaging system handle that (on both ends)?Can the consuming database handle that?Can the hardware and network handle that?Anticipate failureWhat happens if something/anything goes down for an hour?What happens if you go down for a day? What happens if you have unexpected growth?

Problem 4 – Exception Handling Wasn’tMore considerations for failover and redundancyVersus Web applicationWe did not plan on downtimeUnplanned system issuesPlanned outagesWe didn’t build in enough redundancyLoad balancing andFailover were both after thoughtsAll messages always correct all the time (NOT)At first, we had no proper dead letter queuingNo proper exception processingNo means to properly see and react to issuesMany more points of failure and potential issuesMore widely distributed

Design Load Balancing and Failover UpfrontLoad balancing and failover must be accommodatedLike security, you need a multi-layered approachHardware (like Big IP)Redundant message bus/message serversProcessing componentsDatabaseEAI system throttlingHow are you going to kick over to the failover systems (and return to regular systems)?Without losing messagesWithout causing timing problems in message deliver/receipt

Space, space and more spacePlan on extra space for failureA place for queued messages to sit if something goes downSpace in the DB or space in the message channels – or bothPlan on extra space for logsYou are going to want to keep log files around for a while.Some problems take time to manifest to a point of awareness.Devise an automated archive/clean up for logs.No…not all EAI systems provide log clean up utilities.

Anticipate Bad MessagesBuild a Dead Letter Queue (see EAI Patterns book).Unless you have a simple system, you will have messages the system can’t handleImproper format, wrong data, etc…Build a means to capture and handle theseLess they clog your process.Where do you put them? DB, other queue?Who checks them (do you have a “one’s” issue or systemic problem?)

Message RepairIf possible, build a message triage mechanism to inspect, fix, resend DLQ’ed messagesThis can be built/improved over timeMore manual at firstAutomated as you learn more.ConsiderationsHow are you going to clean up the error “droppings” (messages that are truly dead)Consider a “retry” queue with varied strategies to retry messages that have failed.Failure may be due to row locks or reference updates that are just microseconds away from completion.Be cautious of when/why messages end up in the “dead letter” queue. You don’t want it flooded because the DB is down.

Dead Letter Queue (DLQ)

Managing it/monitor itThe multiple points of failures and issues of your systems make them complicated to manage and support.Build in automated monitoring facilities and system health dashboards.You need a one stop shop for what’s up, what’s down, what’s queuing properly, what’s queuing too much, etc.Consider the use of JMX (it is probably already built into some of your infrastructure components.Calculate system thresholds and provide automated alerts to the dashboard and email/page/etc. systems when they start to get close (not once they have been achieved).

Problem 5 – Change is inevitableThe size and shape of our messages changed over time.We had no way to deal effectively with change.Consequently, new system versions/updates causedShutdownReplace (sometimes transforming data to a new structure)RestartThe real world was the only time we saw some situationsWe had no effective test harness Typically leading to ugly back outs

Version StrategyEAI system stability/life span depends on it message structure.Message structure is the hardest part to get exactly right up front.When message formats need to change, this creates a real problem. The entire must be down, queues emptied, etc.Consider version information in the message and routing/processing instructions in the bus.More complicated systemCan also affect performanceAllows for dual operation (old and new systems) without failure and major down time.Its going to happen – especially early – plan for it.

Version Routing

Testing EAI is a b*&%#Consider collecting days worth of messages or message generating data and using it for replay scenarios.Problem - even if you have all the data, you don’t have the same timing issues you will see in the real world.Testing all the potential message scenarios is impossible with any significant sized system.Consider developing a message “replicator” subsystem.Send replicated messages to a test harness.A “live” test faucet of messages ready whenever you need them.Critical to be able test new/updated processes, performance, etc.Requires a fair amount of hardware and some switch to turn it on/off.Will impact performanceConsider putting the “faucet” on just one of the servers in a farm

Test “Faucet”

Wrap upDespite the issues – the system is up and running today.Extremely useful to the business – providing unparalleled distribution information.Like most things in software system development, the lessons learned are more about organization, architecture and design than implementation.Thank you – for your time and attention.