Successfully reported this slideshow.
Taking Blackboard to the next    level: Apache 2.2, Crypto    Acceleration, Shibboleth, and    BbMobile for 80k users in fi...
Contact information	José Manuel López Luján	 •  Email: jm.lopez@utoronto.ca	 •  Twitter: @jmanuel_ll	 •  Blog: http://jose...
University of Toronto	        Where are we?	                           79,085 Students	                           3,229 Ac...
University of Toronto	                                                                      Statistics	demographics	      ...
Infrastructure: Hardware	                         Bb 9.1SP5	                                   App1!  Data1!              ...
Infrastructure: Hardware	                            Bb	  9.1SP5	      SPARC T3-4!4 CPU @ 1.65GHz!                        ...
Infrastructure: Software	                              Bb 9.1SP5	                                                SSL	Pubco...
Looking for service and performance improvements	                                                      8
The Plan	                      •  SSL	  and	  Compression	  working	  together	  Apache	  2.2.x	                        • ...
1 out of 4:	                                  Blackboard Mobile	PROS	             Using	  PubCookie	             Force	  W...
1 out of 4:	                                     Blackboard Mobile	        mobile.lms.utoronto.ca	          F5	           ...
Looking for service and performance improvements	               Apache	  2.2	                 •  Feasible	  on	  SP5	  yet...
Looking forward to		9.1SP8	 OCHO	                          13
The Plan	Apache 2.2.x	                  •  SSL and Compression working together	Shibboleth	                  •  New Authen...
Target version: 2.2.2	•    Modules                                          15
Apache2	                              Compilation	Shared Modules (DSO)	                   64bit Binary for SPARC	         ...
Apache2	                                 Configuration	Performance.conf	                          Proxy_ajp.conf	IfModule m...
Apache2	                               Shibboleth Configuration	# Blackboard secure area !# This will ensure that mod_shib ...
Apache2	               Web Compression + SSL	Total Weight	                   Compressed Weight	                         VS...
Blackboard Mobile Learn	                       Authentication Type	         vS	  force to	web	                            ...
Shibboleth and LDAP	                           Implementing a New Authentication Framework	        mobile.lms.utoronto.ca	...
Shibboleth and LDAP	             Implementing a New Authentication Framework	      LDAP 	                                 ...
Minification                                                     Savings	  on	  payload	                                  ...
Minification	                                                      Blackboard	  	  JS	  Grouping	  Tool	  	          •  Pre...
Minification	                      Blackboard	  Grouping	  Tool	  	                      Firebug console output sample	    ...
26
Solaris Cryptographic Framework (SCF)	       ©	  Sun	  Microsystems:	  Using	  The	  Cryptographic	  Accelerators	  in	  t...
Solaris Cryptographic Framework (SCF)	                                       Linking	  Apache2	  binary	      64bit Binary...
Current environment		9.1SP8	 OCHO	                           29
Infrastructure: Hardware	                               Bb 9.1SP8	                                              Collab1!  ...
Infrastructure: Hardware	                 SPARC T4-4 and Oracle VM Server for SPARC v2.2	                                 ...
Infrastructure: Hardware	                              SPARC T4-4 and Oracle VM Server for SPARC v2.2	                    ...
Infrastructure: Hardware	                                  Live Migration	                     Source	                    ...
Infrastructure: Software                                                   Bb	  9.1SP8	                                   ...
Performance	                                      Benchmark	50k	    requests were sent sequentially with          differen...
Performance                                     Load	  Times	                          2%	       1%	        0%	   0%	     ...
Thank you.	Jose Manuel Lopez Lujan	Jm.lopez@utoronto.ca
Upcoming SlideShare
Loading in …5
×

Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU

2,585 views

Published on

The University of Toronto wanted a faster Blackboard Learn environment and needed to squeeze the most out of its hardware. Apache 1.3 that Blackboard bundles with its BB Learn was not up to the job. By upgrading to Apache 2.2, a new world of options become available. Now, Shibboleth 2 as a Single Sign-on, LDAP as a native authentication scheme for Mobile Learn and OpenSSL PKCS#11 Cryptographic support are all realities for Blackboard.

The PKCS#11 support was particularly important, as is leverages the Solaris Cryptographic Framework (SCF) and transparently offloads cryptographic operations to available hardware providers included in the new Oracle T4 chip. UofT migrated all of its Blackboard VMs to a single Oracle SPARC T4-4 after upgrading to BB9.1SP8. The results were remarkable. A single box, 5 rack-U in height, delivers Blackboard to 80k users. Response times are down by 60% and we have no need for SSL-offload at the load-balancer.

Published in: Technology
  • Be the first to comment

Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU

  1. 1. Taking Blackboard to the next level: Apache 2.2, Crypto Acceleration, Shibboleth, and BbMobile for 80k users in five RU Jose Manuel Lopez Lujan Senior LMS CoordinatorUniversity of Toronto
  2. 2. Contact information José Manuel López Luján •  Email: jm.lopez@utoronto.ca •  Twitter: @jmanuel_ll •  Blog: http://jose-manuel.me •  G+: http://gplus.to/josemanuel John Calvin •  Manager, Data Centres •  Email: john.calvin@utoronto.ca 2
  3. 3. University of Toronto Where are we? 79,085 Students 3,229 Academic 5,224 Non-Academic 3 Campuses 7 Colleges 205 Undergraduate programs 79 Graduate programs 13,313,110 Visits (Dec 11 Jun 12) 4,241,247 Sessions per month 2,153,536 Unique Visitors 1.5 M Hits/hr peak 5.0 TB/hr peak 3
  4. 4. University of Toronto Statistics demographics Canada  98.0100%   United  States    0.7100%   China    0.23%   (not  set)  0.1200%   Hong  Kong    0.0900%   United  Kingdom  0.0700%   South  Korea  0.0600%   United  Arab  Emirates  0.0600%    Chrome     technology  Firefox      Internet  Explorer    Safari      Android  Browser      Opera      IE  with  Chrome  Frame      Mozilla  CompaNble  Agent      Opera  Mini      RockMelt     4
  5. 5. Infrastructure: Hardware Bb 9.1SP5 App1! Data1! App2! Collab1! F5! App3! Load Balancer!Hitachi 9985! 5
  6. 6. Infrastructure: Hardware Bb  9.1SP5   SPARC T3-4!4 CPU @ 1.65GHz! Collab1! 16 cores/CPU! 8 threads/core! 512 threads! App1! 512GB of RAM! LDOMs! 32 VCPUs! App2! 64G of RAM! App3! LDOMs! Hitachi 9985! 2 Pools 2 RA! Data1! 80 VCPUs! 300G FC ! 64G of RAM! 15k RPM! ! 6
  7. 7. Infrastructure: Software Bb 9.1SP5 SSL Pubcookie (DSO) Apache 1.3 Blackboard Learn 9.1 SP5 Apache 1.3 PubCookie •  No Compression with SSL •  SSO Solution •  No Blackboard Mobile •  Hard to maintain •  Custom Authentication Module 7
  8. 8. Looking for service and performance improvements 8
  9. 9. The Plan •  SSL  and  Compression  working  together  Apache  2.2.x   •  Custom  AuthenNcaNon  Module  for  Bb   Shibboleth   •  Possible  with  Apache  1.3  and  PubCookie?   Bb  Mobile   •  Possible  with  Apache  2.2.x  and  Shibboleth?     •  Worthwhile  without  compression?  MinificaNon   9
  10. 10. 1 out of 4: Blackboard Mobile PROS   Using  PubCookie   Force  Web  AuthenNcaNon       Simple  to  administer   SSO  Page  not  mobile   capable   CONS   Location /webapps/Bb-mobile-bb_bb60! !satisfy any! !AuthType none! !order deny,allow! !allow from all! /Location! 10
  11. 11. 1 out of 4: Blackboard Mobile mobile.lms.utoronto.ca   F5   portal.utoronto.ca   App1!App4! App2! Web Login ! App3! (pubookie)! Enterprise! LDAP Server! ! ! bbconfig.auth.type=ldap! bbconfig.auth.type=toronto! ! ! 11
  12. 12. Looking for service and performance improvements Apache  2.2   •  Feasible  on  SP5  yet  hard  to  administer   Shibboleth   •  Possible  with  Apache  2.2  yet  hard  to   administer   MinificaNon   •  Not  worthwhile  without  compression   12
  13. 13. Looking forward to 9.1SP8 OCHO   13
  14. 14. The Plan Apache 2.2.x •  SSL and Compression working together Shibboleth •  New Authentication Framework Bb Mobile •  Possible with Apache 2.2.x and Shibboleth? Minification •  Worthwhile without compression? •  Consolidation and Cryptographic Acceleration T4-4 14
  15. 15. Target version: 2.2.2 •  Modules 15
  16. 16. Apache2 Compilation Shared Modules (DSO) 64bit Binary for SPARC !! CC=cc -m64“ !--enable-mem-cache=shared! CXX=CC -m64“ !--enable-file-cache=shared! CFLAGS=-m64 -xO2 -DSSL_ENGINE“ !--enable-headers=shared! CXXFLAGS=-m64 -xO2“ !--enable-usertrack=shared! LDFLAGS=-L/usr/sfw/lib/sparcv9 !--enable-expires=shared! ! ! -R/usr/sfw/lib/sparcv9“ ! CCFLAGS=-m64“ ! 16 Read  More  
  17. 17. Apache2 Configuration Performance.conf Proxy_ajp.conf IfModule mpm_worker_module! IfModule proxy_module!        ServerLimit 1024!         ProxyRequests Off!        StartServers 341!         ProxyTimeout 3600!        MinSpareThreads 64!         # Shibboleth !        MaxSpareThreads 128!         ProxyPassMatch ^(/shib.*)$ !!        ThreadLimit 128!         ProxyPass /Shibboleth.sso !!        MaxClients 1280!         ProxyPass /shibboleth-sp !!        ThreadsPerChild 128!         ProxyPass /Shibboleth.sso/Status !!        MaxRequestsPerChild 0! /IfModule!/IfModule! !! ! 17
  18. 18. Apache2 Shibboleth Configuration # Blackboard secure area !# This will ensure that mod_shib ignore all!# requests except those sent to !# .../execute/shibbolethLogin.!Location /webapps/bb-auth-provider-shibboleth-bb_bb60/execute/shibbolethLogin!        AuthType shibboleth!        Require shibboleth!        ShibRequestSetting requireSession 1!        Require affiliation ~ ^member@.+$!        Require user ~ ^.+$!        Require affiliation isstaff! # Blackboard Mobile Learn B2 Configuration! # In older installations BBLEARN should be !        Require affiliation isstudent! # changed by bb_bb60!/Location! Location /webapps/Bb-mobile-bb_bb60!     AuthType shibboleth!     ShibRequestSetting requireSession 0!     Require shibboleth!     Require user ~ ^.+$!     Require affiliation ~ ^member@.+$!     Require affiliation isstaff! /Location! ! 18
  19. 19. Apache2 Web Compression + SSL Total Weight Compressed Weight VS   1036.9K     265.7K   74.3%   19
  20. 20. Blackboard Mobile Learn Authentication Type vS  force to web native 20 ©  Blackboard  Mobile:  h`p://help.blackboardmobile.com  
  21. 21. Shibboleth and LDAP Implementing a New Authentication Framework mobile.lms.utoronto.ca   F5   portal.utoronto.ca   App1!App4! App2! Shibboleth ! App3! 2.4.3! LDAP Server! Hostname  RestricNon  Provided  by  the  New  AuthenNcaNon  Framework   Provider:  Toronto  LDAP  Auth   Provider:  Toronto  Shibb  Auth   21
  22. 22. Shibboleth and LDAP Implementing a New Authentication Framework LDAP Shibboleth mobile.lms.utoronto.ca   portal.utoronto.ca   22
  23. 23. Minification Savings  on  payload   Real  path   MinificaNon   MinificaNon  CR   MinificaNon  t  $BBHOME/docs   835,860.00   50.93%   3.1886  $BBHOME/webapps/blackboard   183,477.00   58.46%   0.6999  $BBHOME/webapps/assessment   51,225.00   58.01%   0.1954  $BBHOME/webapps/discussionboard   30,919.00   35.52%   0.1179  $BBHOME/webapps/gradebook   277,527.00   54.81%   1.0587  $BBHOME/webapps/caliper   119,764.00   48.56%   0.4569  $BBHOME/webapps/portal   27,595.00   54.81%   1.0600  $BBHOME/webapps/cms+xy   49,532.00   52.59%   0.1889  $BBHOME/webapps/wysiwyg   99,681.00   52.43%   0.3803  $BBHOME/webapps/webeq-­‐plugin       15,354.00   52.43%   0.3800  $BBHOME/webapps/taglibs   44,054.00   52.43%   0.3800  $BBHOME/webapps/*       4,936.00   52.43%   0.3800       Grand  Total   1734988.00     8.11   1694.32K     MR  ~  52.0%   23
  24. 24. Minification Blackboard    JS  Grouping  Tool     •  Prematurely released on SP5 •  Released on SP8 as certified. •  Implementing YUI Compressor Library •  Grouping and minifying on-the-fly (inside JVM) script  type=text/javascript  src=/branding/__js__/C131DA0400D29916A81632A83B91BAD2.js?v=9.1.50119.0/script   •  Enabled by default on SP8 ##  Whether  related  JavaScript  files  should  be  grouped  together  ##   ##  for  be`er  HTTP  performance  ##   bbconfig.javascript.group.files=true   24Read  More.  
  25. 25. Minification Blackboard  Grouping  Tool     Firebug console output sample 25Read  More.  
  26. 26. 26
  27. 27. Solaris Cryptographic Framework (SCF) ©  Sun  Microsystems:  Using  The  Cryptographic  Accelerators  in  the  ULTRASPARC  T1  and  T2  Processors.   27
  28. 28. Solaris Cryptographic Framework (SCF) Linking  Apache2  binary   64bit Binary for SPARC conf/pkcs11.conf ! SSLCryptoDevice pkcs11! CC=cc -m64“ ! CXX=CC -m64“ ! CFLAGS=-m64 -xO2 -DSSL_ENGINE“ ! CXXFLAGS=-m64 -xO2“ ! LDFLAGS=-L/usr/sfw/lib/sparcv9 ! ! ! -R/usr/sfw/lib/sparcv9“ ! CCFLAGS=-m64“ ! 28Read  more.  
  29. 29. Current environment 9.1SP8 OCHO   29
  30. 30. Infrastructure: Hardware Bb 9.1SP8 Collab1! SPARC T4-4!4 CPU @ 3.0 GHz! App1! 8 cores/CPU! 4 x LDOMs! 256 threads! 24 vCPUs! 512GB of RAM! App2! 64G of RAM! App3! 1 x LDOM! Data1! 56 vCPUs! 120G of RAM! Hitachi 9985! 2 RAID 6 Arrays! App4! 2 x LDOMs! 2 TB x 7200 RPM SATA!Carved into 192 GB Ldev! 24 vCPUs! ! 32G of RAM! App5! 30
  31. 31. Infrastructure: Hardware SPARC T4-4 and Oracle VM Server for SPARC v2.2 Memory 64   64   64   32   32   120   64   8  Collab1   App1   Ap2   App3   App4   App5   Data1   IO/Controller   Free   31
  32. 32. Infrastructure: Hardware SPARC T4-4 and Oracle VM Server for SPARC v2.2 vCPUs 32   32   32   16   16   56   32   24  Collab1   App1   Ap2   App3   App4   App5   Data1   IO/Controller   Free   32
  33. 33. Infrastructure: Hardware Live Migration Source   Target   33Read  More.  
  34. 34. Infrastructure: Software Bb  9.1SP8   SSL   Shibboleth     (DSO)   Apache  2.2.2  64  bit  SPARC     AuthenNcaNon   Providers:   Blackboard  Learn  9.1  SP8   LDAP  +  Shibbholeth  Apache  2.2.2   Blackboard  Mobile     Shibboleth  •  Compression  with  SSL   •  NaNve  AuthenNcaNon   •  LDAP  •  SSL  Offloading  –  PKCS11   34
  35. 35. Performance Benchmark 50k requests were sent sequentially with different concurrency levels AJP Proxy https://server/webapps/portal/healthCheck SSL deflate Concurrency   Apache  2.2.2   Apache  2.2.2   SSL-­‐H,  AJP   SSL-­‐H,  AJP,  COM   10   1230.59   1143.12   100   1962.52   1704.3   200   1699.73   1625.22   500   1870.60   1075.2   1000   1214.95   1173.457   2000   1129.87   1234.44   @  1k  request/sec    process  ~1.2K  req   35
  36. 36. Performance Load  Times   2%   1%   0%   0%   0%   0  -­‐  1     17%   24%   1  -­‐  3     3  -­‐  7     7  -­‐  13     Avg.  Page     13  -­‐  21    Load  Time:   21  -­‐  35     2   .44   SEC   56%   35  -­‐  60     60+   36
  37. 37. Thank you. Jose Manuel Lopez Lujan Jm.lopez@utoronto.ca

×