Blackboard DevCon 2011 - Developing B2 for Performance and Scalability


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Blackboard DevCon 2011 - Developing B2 for Performance and Scalability

  1. 1. Developing B2 forPerformance and Scalability Noriaki Tatsumi Kevin Weisser
  2. 2. Why Performance Matters?• User experience – Your users don’t like slow and unresponsive apps – Slow applications lose users in frustrations – Users tend to wander off beyond captive response time of 7 – 10 seconds• User adoption – Google saw 25% decline in searches within 6 weeks after introducing 0.5s delay (~$2.5B loss) – Wikia observed up to 20% user loss as the page load time reached 5 seconds
  3. 3. Wikia – Exit rate (abandon)
  4. 4. “With great power comes great responsibility” – Uncle Ben, Spiderman• Reputations at stake• System resources are shared and limited• B2s with scalability problem impacts system performance – System-wide performance degradation – Server downtime
  5. 5. What is an anti-pattern?• Some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results• A refactored solution exists that is clearly documented, proven in actual practice and repeatable. - Definition from Wikipedia
  6. 6. How can anti-patterns help?• Prevents performance as an afterthought• Helps identify performance issues that typically don’t arise until deployed in production• Prevents pre-mature optimizations
  7. 7. How do I use anti-patterns?• Identify and correct the occurrences of anti- patterns during: – Design review – Implementation/code review – Execution of experiments and profiling – Static code analysis
  8. 8. Top 10 violated high-level performanceanti-patterns fixed in Blackboard Learn™ Inefficient Filtering Unnecessary Processing The Ramp Data Overreach Frequency (%)Unlimited Resource Usage Functional Bug Excessive Granularity More is Less Usage of Outdated Tools and APIs Mishandled Concurrency 0 5 10 15 20 25 30
  9. 9. Top 10 violated low-level performanceanti-patterns fixed in Blackboard Learn™ Missing or non-optimal index [db] Inefficient or erroneous criteria [db] Unnecessary data retrieval [app} Too many objects in memory [app} Presentation of unlimited results [ui] Frequency (%) Inefficient access path [ui] No data caching [app}Not releasing resource (immediately) after use [app} Incompatiblity with HA deployment [app} Excessive DB round trips [app} 0 5 10 15 20 25
  10. 10. Most violated high-level anti-pattern #1: Inefficient Filtering• Description – Querying data using inefficient logic, algorithm, or technique• Problem – Consumes resources that can be used by other processes – Increases execution/response time• Solution – Store data in both centralized and organized manner – Search against smaller data set first, fail fast, and don’t look at unnecessary data – Leverage indexing and other efficient techniques/algorithms
  11. 11. Most violated high-level anti-pattern #1: Inefficient FilteringMissing or non-optimal index (DB)• Most violated low-level anti-pattern• Problem – Missing appropriate index on search columns leads to inefficient data retrieval operation by SQL query optimizer – Having index can inversely add unnecessary overhead if the column is never used for searched criteria• Solution – Analyze SQL execution plan to verify usage of proper index – Create index on foreign keys – Create unique index on unique columns – Use an index to eliminate unnecessary full-table or index scans to access small percentage of data on a large table – Use covering index for key lookup
  12. 12. Creating Database Objects with B2• Release 9.1 SP1 and above – B2s authored by independent developers can now define and install schema elements• Not enabled by default – Administrators must enable this functionality from the B2 management section of the admin panel• Documentation – Database+Objects+with+Building+Blocks
  13. 13. Most violated high-level anti-pattern #1: Inefficient FilteringInefficient or erroneous criteria (DB)• The second most violated low-level anti-pattern• Problem – Duplicate, unnecessary, incorrect, or missing predicate checks and joins exist in SQL – Increases response time, system resource consumption, and locking• Solution – Be specific on what data you need – Keep it simple and avoid unnecessary checks – Use JOIN criteria that aligns with index – Analyze execution plan to derive efficient and balanced query
  14. 14. Most violated high-level anti-pattern #1: Inefficient FilteringWrapping filtering column in a function (DB)• Problem – The query optimizer does not see the column – Available index likely does not get used• Solution – Rewrite query to avoid using function if possible – Format bind variables in application tier
  15. 15. Example of bad T-SQL function usage - Without -
  16. 16. Example of bad T-SQL function usage - With LOWER-
  17. 17. Example of bad T-SQL function usage - Comparison - Without T-SQL function With T-SQL functionLogical I/O reads 3 3238Estimated CPU cost 0.396307 15.454153
  18. 18. Most violated high-level anti-pattern #1: Inefficient FilteringOther low-level anti-pattern examples• Filtering larger query first• Implicit data type conversion• Inefficient CSS selectors• Inefficient DOM element lookup• Inefficient regex• Unnecessary query on VIEW instead of table• Usage of "NOT IN" comparison rather than JOIN• Usage of wrong index• Using UNION when returned records always unique
  19. 19. Most violated high-level anti-pattern #2: Unnecessary Processing• Description – Processing when not needed or not needed at the time – Excessive or duplicate executions• Problem – Consumes resources that can be used by important work – Increases execution/response time• Solution – Remove, postpone (asynchronous), prioritize, or reorder the processing steps – Leverage caching to reuse loaded or calculated data
  20. 20. Most violated high-level anti-pattern #2: Unnecessary ProcessingPresentation of Unlimited Results (UI Design)• 5th most violated low-level anti-pattern• Problem – Users typically do not consume all data when presented with large amount so the used resources are wasted – Severe system performance issue can arise depending on the volume of prepared results• Solution – Paginate results – Guide users to narrow down search criteria – Limit amount of data that can be returned
  21. 21. Example usage of pagination and limiting the number of returned results
  22. 22. Most violated high-level anti-pattern #2: Unnecessary ProcessingInefficient access path (UI Design)• 6th most violated low-level anti-pattern• Problem – Room for optimization exists in actions/clicks users must take to complete his/her task – Results to creating additional work for the system (also impacts user’s time to task)• Solution – Refactor UI or click path to let users accomplish their tasks more efficiently
  23. 23. Example of enhancing access path
  24. 24. Most violated high-level anti-pattern #2: Unnecessary ProcessingOther low-level anti-pattern examples• Excessive executions of maintenance task• HTTP 404 error on cacheable static content• Missing short-circuit logic• No data caching• No http caching• Unnecessary cache reload• Unnecessary conditional check• Unnecessary object creation• Updating unchanged data
  25. 25. Most violated high-level anti-pattern #3: The Ramp• Description – Processing time increases or the system crashes as the application gets used over a period of time• Problem – The application cannot handle the increased volume of data/traffic – Resource consumption increases or exhausts• Solution – Select algorithms or data structures based on maximum size expected – Use algorithms/designs that adapt to the increasing size or doesn’t get effected by it (ex. flyweight pattern, streaming technique)
  26. 26. Most violated high-level anti-pattern #3: The RampToo many objects in memory• #4 most violated low-level anti-pattern• Problem – Loading large amount of data or objects at once for processing can result to out-of-memory error and other system-level performance impact – Examples include unmarshalling entire DB resultset in heap and generation of large XML documents using DOM object• Solution – Flyweight and lazy loading patterns – Streaming and chunking techniques
  27. 27. Example of streaming database result set - Overview -• DataList/DataListIterator – Interface for a list/iterator data structure materialized from a data source, such as a database or a file – Extends the List/Iterator interface to leverage polymorphism• ResultSetDataList – Implementation of DataList that’s materialized from a SQL resultset – Unmarshalls the resultset rows one by one as the getter method is called• ResultSetDataListIterator – Iterator that’s materialized from ResultSetDataList
  28. 28. Example of streaming database result set - Usage considerations -Avoid out-of-memory and SQL timeout• Use ResultSet.TYPE_FORWARD_ONLY• Set Statement fetch size – Default for Oracle JDBC = 10 – Default for i-net MERLIA = 0 • ‘0’ uses client cursor that reads all rows into memory • Specify fetch size to use cursor on server side• Use flyweight pattern in the unmarshaller if possible
  29. 29. Example of streaming database result set - Code sample -• See Appendix A
  30. 30. Most violated high-level anti-pattern #3: The RampNot releasing resource (immediately) after use• #8 most violated low-level anti-pattern• Problem – Cause of connection leaks, deadlocks, performance degradation and other unexpected behavior – Threads, sockets, database connection, file handler, and other resources can be a victim• Solution – Use finally block to ensure release – Release resource as quickly as possible
  31. 31. Example of resource management - CSContext -CSContext csCtxt=null;try { csCtxt = CSContext.getContext(); // your code} catch (Exception e) { csCtxt.rollback(); csCtxt=null;} finally { if (csCtxt!=null) { csCtxt.commit();}
  32. 32. Most violated high-level anti-pattern #3: The RampOther low-level anti-pattern examples• Exceeding max number of expressions in a list• No data caching• No priority logic• Not releasing resource (immediately) after use• Presentation of unlimited results• Transaction isolation unnecessarily wide• Unlimited cache storage
  33. 33. Low-level performance anti-patterns fixed in Blackboard Learn™ by tierApplication Database Frequency (%) UI Design Browser Network 0 10 20 30 40 50 60
  34. 34. Tools to identify anti-patterns: First line of defense• Peer code review – Atlassian Crucible – Review Board• Static code analyzer – FindBugs – PMD – Sonar – Custom rules• Performance unit testing – JUnitPerf
  35. 35. Tools to identify anti-patterns: Deeper analysis• Java – VisualVM – Memory Analyzer (MAT) – YourKit• Browser – Firebug – Google Page Speed – dynaTrace AJAX Edition – Fiddler – WebPagetest
  36. 36. Tools to identify anti-patterns: Deeper analysis• Database – MS SQL Server • SQL Server Management Studio • SQL Server Profiler • Performance Dashboard – Oracle • SQL*Plus • Automatic Workload Repository (AWR) • Active Session History (ASH)
  37. 37. Takeaways• The importance of performance and scalability for your B2• Top violated performance anti-patterns in Blackboard Learn™• Forensics Tools
  38. 38. Please provide feedback for this session by emailing The title of this session is: Developing B2 for Performance and Scalability
  39. 39. Appendix A:Example of streaming database result set
  40. 40. Example of streaming database result set - -public interface DataList<E> extends List<E>{ /** Closes the list and releases all of the resources that back the list */ void close();}
  41. 41. Example of streaming database result set - -public <X> ResultSetDataList( DataListSelectQuery<X> query ) { // make sure the query object and resultset object in the query object are not null … try { if ( query.getRst().isClosed() ) { throw new IllegalArgumentException( "Result set must not be closed" ); } // Use this information to restrict the usage of methods that require scrolling _resultSetType = query.getRst().getType(); } catch ( SQLException e ) { throw new IllegalArgumentException( "Unable to validate input result set", e ); } _query = query; _rst = query.getRst(); _unmarshaller = query.getUnmarshaller();…
  42. 42. Example of streaming database result set - -public E get( int index ) { if ( _resultSetType == ResultSet.TYPE_FORWARD_ONLY ) { String msg = "You must use the Iterator because your ResultSet is set to TYPE_FORWARD_ONLY."; throw new IllegalArgumentException( msg ); } if ( index < 0 ) { throw new IndexOutOfBoundsException(); } try { if ( _rst.absolute( index + 1 ) ) { if ( _rst.getRow() != index + 1 ) { throw new IndexOutOfBoundsException(); } return (E) _unmarshaller.unmarshall(); } else { throw new IndexOutOfBoundsException(); }…
  43. 43. Example of streaming database result set - -/** The close() method closes the prepared statement and result set wrapped in this object. This is called by the finalize method that garbage collector executes when appropriate. However, it should still be called as soon as this object is no longer necessary by the programmer. Please do not rely on the garbage collector.*/ @Override protected void finalize() throws Throwable { try { close(); } finally { super.finalize(); } }
  44. 44. Example of streaming database result set - -@Overridepublic E next(){ if ( !hasNext() ) { throw new NoSuchElementException(); } try { E element = (E) _list._unmarshaller.unmarshall(); _elementRetrieved = true; return element; } //handle exceptions …