This is the problem solving philosophy I tell our consultantsConfiguration over Coding: Out of the box WLS is not setup for production needs many changes for production not usually done by developers“Out of the Box” set up for developersWLS is likely to have fewer bugs than custom code. Proven throughout my career. YES there are bugs but I’ve probably only reported a handful compared to customer bugsWLS if fundamentally quick a few ms for a simple JSP page to render. Everything else is your overhead!If you do something once and get a certain result do it again you should get the same resultScientific Method!!
These are things we’ve seen people do a lot which you shouldn’t do.
We see many examples of lots and lots of datasourcesShow a real example
Developer writes a bunch of components in separate EARSEach EAR has it’s own DatasourceAll going to the same databaseC1 calls C2 call C3 all in the same TransactionSimilar antipattern with JMS Connection Factories
Strange XA exceptions are seen by developers relating to LLR optimisationUntick Honour Global Transactions to make Exception “Go Away”Some Datasources then in “autocommit” mode as they are not enlisted in the Global TXRequires full XA configuration for Oracle!Avoid XA especially when you only have 1 database!!!!!
This picture of sexual activityCover a scenario we see in quite complex EJB environments with a number of EJB applications
WE HAVE two APPLICATIONS developedindependantlyLater application 1 needs to retrieve data from 2Initially possibly get a security issueSet Domain credential to be the sameReceive XA Exception again related to LLTX’s inadvertently coupled as WLS will enlist second app in first TX
Exception will be failure on client side to lookup JDBC Datasource on server side when it is actually there!Often also get security errors relating to anonymous user trying to commit TX depending on WLS versionCan also get difficulties with NATREQUIRES NEW ON SIDE 2Seen a developer try to fix by placing a wrapper bean on side 1 still couples at least removes Oracle 1.
Typical Architect commentAsk question anyone here think clustering too heavyweight!!!Fallacy why is clustering heavyPerhaps your Session is way too heavySo you don’t need failover???Your customers probably do!!
Logging is VERY Heavily SynchronisedGotta Fit all those messages from Multiple Threads into a single FileIn Memory Buffer Logging can Kill youExample next slide
Tell story 1 about Matt’s case studyTell Story 2 Atos case studySystem Crash every day near 4pm – slow down to unrepsonsiveObserved 100% cpu utilisationTypical of constant GCConsole run out of memoryUse JMS for Logging is more performant
Things that cause problems but server is doing the right thing as per spec or configuration
See a couple of other typical session issues a lotSerialization -> Write code to Serialize to a file all your session attributesWATCH OUT FOR 3rd Party web frameworks know what they put in the session (Form Beans)
System Exception is runtime or RemoteExceptionWe see this in many applications where the behaviour is not what people are expectingInsufficient knowledge of the details of the EJB spec.One company policy wrap all Exceptions into their applications so as not to expose 3rd party apis.Wrapped remotes with application exception -> CAUSING COMMITS
A couple of typical JMS issues
Poisoned Message on JMS -> constant spinning CPU (100%)Other messages can never be deliveredSeen this on customer killing all order processingThis slide buildsTalk through the BuildExplain each PointRedeliver Delay allows other messages through watch out if require strict ordering
Not such a big issue since later versions of WLS 9.2 onwards – still requires tuningMessage headers still in memoryConsider setting quotas and messages can arrive quicker than they are pagedAlso will use 1/3 JVM memory
3 issues we have seen take out servers and cause system outages
Especially on Sun JDKNot as common on JrockitOnly for 32bit platformsSeen one production system unable to restart because of this and numerous system hot spot dump.Large number of JMS messages (see paging from before). Hot Spot crashed couldn’t restart.Reducing JVM heap size fixed the problem
Node Manager should be set up correctly.Powerful and important – Amazingly some customers don’t use it - NEVER been to a customer where this has been properly configuredBy default will attempt a number of quick restarts and then give up.Configure to try a number of times with a delay for a long period.In 9 and 10 Set Panic action to exit
Major Government WLI System taken out by this.Regular restart of WLS carried out same time upgrade of Oracle. Oracle outage happened before but never seen a problem (Connection Testing etc. all enabled).WLS restarted while Oracle outSupport team Arrived morning to see that Name Not Found JNDI Exceptions in the log files.ALL ORDER PROCESSING FAILEDDefault setting is to switch this off!! Again suitable for development as quick errors!!
“Out of the Box” WLS supports Ease of development.Last preproduction domain review I did for a large dual cluster WLS and WLI domain had 80+ recommendations with over 10 Classified as HIGH which means could cause a system outage, significant down time or major security breach.
Tales from the WebLogic Frontline
Tales from the WebLogic Front Line(What NOT to do with WebLogic)<br />Steve Millidge<br />C2B2 Consulting Limited<br />
Agenda<br />Health Warning<br />Setting the Scene<br />Common Anti-patterns<br />Ain’t misbehaving<br />This could kill<br />
C2B2 Consulting<br />Fast<br />Performance and Scalability<br />Reliable<br />High Availability and Recoverability<br />Manageable<br />Monitoring and Supporting<br />Secure<br />Security<br />
Philosophy<br />Don’t Panic you didn’t #@#@ it up.<br />Suspect Configuration over Coding<br />Suspect bespoke code over WebLogic code<br />WebLogic is FAST<br />Computers are Logical and Rational<br />
Too Many Datasources<br />Data Source per Component<br />All to the Same Database<br />JNDI Name per Component<br />C1<br />C1<br />C1<br />DS1<br />DS1<br />DS1<br />Oracle<br />
Why It’s Bad<br />Requires XA for all Datasources<br />Developers receive strange “XA” exception<br />Untick Honour Global Transactions<br />Makes Exception “go away”<br />Datasources now in autocommit<br />DATA CORRUPTION!!<br />Requires more DB connections<br />BIND same Datasource to multiple JNDI locations<br />
Why It’s BAD<br />Results in “Strange Exceptions” if not All XA<br />Is the REALLY what you want<br />XA is very heavyweight<br />And not guaranteed<br />XA requires particular database configuration<br />You have 4 minimum XA resources to coordinate<br />Fix using REQUIRES NEW<br />
Too Much Logging<br />Too Synchronized<br />New Application 30% Slower<br />Tx Timeouts<br />High CPU<br />Many Threads waiting on Logging Monitor<br />Debug turned on, on one node<br />In Memory Buffer<br />Each Thread Logged to in memory queue<br />Writer thread writes messages to file<br />Under heavy load memory queues grew unbounded<br />1.5Gb heap in log messages<br />
Session Shenanigans<br />Serialization<br />Session Attributes must be Serializable<br />See this one all the time<br />No Failover!!!<br />Working as per Spec<br />No Put<br />You must Put dirtied objects back in the Session<br />Don’t Get modify then no Put<br />No Failover!!!<br />Working as per Spec<br />
Exceptions and TX<br />What happens to the TX when you throw an Exception from an EJB method?<br />Application ExceptionCommit<br />System ExceptionRollback<br />
Too Much RAM<br />-XMS2G –XMX2G<br />32 bit Can be Too Much<br />O/S Reserves RAM<br />VM Starts up<br />When you need it!<br />HOTSPOT DUMP<br />
Node Manager Configuration<br />Not Good Out of the Box<br />Auto Kill If Failed = TRUE<br />Auto Restart = TRUE<br />Restart Delay should be set<br />Restart Max should be set<br />Set Panic Action to System.exit<br />
Disappearing Datasource<br />Start WebLogic when DB not Available<br />Datasource is not created<br />Connection Creation Retry Frequency Seconds should be non-zero<br />
Summary<br />Production Configuration is Complex<br />Call in the Professionals<br />