Quality of ExperiencePerformance Benchmark on Messaging solutionsMicrosoft Office 365, IBM LotusLive and Google Apps      ...
Synergie Informatique Cloud Services Provider, France Social Business Specialist Expert in messaging and  collaboration...
ip-label.newtestAround the world, ip-label.newtest solutions measure the quality of applicationsfrom the user’s point of v...
Cloud goes Mainstream  “ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research“10% of Enterprise Ema...
Messaging in the line of sightCloud Usage Scenarios                          CIO Expected Advantages1.   Prototyping      ...
The CIO Evolution   Guarantees, disponibility (SLA)   Warranty and « Quality » improvement   Security, externalisation,...
Adapted Needs Governance, « Cloud » Best Practices Adapted procedures flowchart More straight to the point (support, sp...
Editor’s Focus / Collaborative Suite                            1911                             1975                     ...
E-Mail Systems Focus                         “ Inbox = Tetris getting zero lines doesn’t win                              ...
Methodology         Data Solution Pilot for all web applications , rich web or client (  Office 365, Outlook…) run like a...
Solution Architecture
Tests Description 3 platforms tested 3 types of tests   – Access to the portal from 5 spots in France      •   https://p...
Tests Description   Scenario SENDING     –   Connexion to the portal     –   Authentication     –   Access to the e-mail ...
Tests Description   When the portal we are trying to access is considered unavailable?     – Non loading of the portal in...
Portal Access                                Availibility and average access time to each portal                      Resu...
Portal Access                                                     Access to portal availibility                    Access ...
Authentication                                 Authentication availibility and online messaging access                    ...
Authentication       Authentication Incident for Lotus Live on 18/09                                                     T...
AuthenticationAuthentification errors examples              Authentification timeout on Office 365            Authentifica...
Authentication                     Examples : Authentication errors on Google Apps portalErreur du serveur (HTTP 500) sur ...
Sending/Receiving mail                              Performance                          Average length of time to send an...
Sending/Receiving mail                                           Stability                                        Average ...
2                                      1                                                3   Google Apps Gmail can be cons...
Conclusions                                                   Results Details   Gmail Google     –   Best Portal Access P...
ConclusionsEach editor commits himself to 99.9% availibility     –   « Microsoft provides a financially-backed 99.9% uptim...
Q/A   Were mails send with attachments ?     – No , only text content; the add on of attachments of different sizes could...
Thank You            Synergie-informatique.fr                       @SynergieInfo
Upcoming SlideShare
Loading in …5
×

20111006 synergie informatique-label-qualityof-experience_en

706 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
706
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

20111006 synergie informatique-label-qualityof-experience_en

  1. 1. Quality of ExperiencePerformance Benchmark on Messaging solutionsMicrosoft Office 365, IBM LotusLive and Google Apps Webinar October 6 th 2011 Alexis Blandin, Cloud & Social Business Leader | Synergie Informatique Arnaud Bécart, International Presales Engineer| ip-label David Wais, Business Development Leader| ip-label
  2. 2. Synergie Informatique Cloud Services Provider, France Social Business Specialist Expert in messaging and collaboration solutions integration IBM, Microsoft and Google partner
  3. 3. ip-label.newtestAround the world, ip-label.newtest solutions measure the quality of applicationsfrom the user’s point of view: business applications as well as web, voice andeven TV over IP platforms.Over 1500 direct and indirect customers rely on ip-label.newtest to validate theend-to-end reliability of their websites and critical applications worldwide
  4. 4. Cloud goes Mainstream “ Cloud Computing Market Will Top $241 Billion in 2020 ” – Forrester Research“10% of Enterprise Email Will Be Based on a Cloud/SaaS Model in 2014” – Gartner Innovation = Public versus Enterprise Social Network Mobility, highly connected, convergence Increasing Data SaaS, a serious consideration
  5. 5. Messaging in the line of sightCloud Usage Scenarios CIO Expected Advantages1. Prototyping  Commodities delegation2. Test & Development  Industrialisation  Fast deployment3. Messaging  Savings, ROI4. Collaboration  Self-service, flexibility / PAYG  Innovation (« beta »)5. Web Applications  Standardization, integration  New challenges (mobility, …)6. Backup, Archiving
  6. 6. The CIO Evolution Guarantees, disponibility (SLA) Warranty and « Quality » improvement Security, externalisation, reversibility Sales relationships (B2B) Contracting and legal management, (Patriot Act, CNIL) Going from CAPEX to OPEX (subscription) Operational and organisational restrains
  7. 7. Adapted Needs Governance, « Cloud » Best Practices Adapted procedures flowchart More straight to the point (support, specifics, infrastructures) Dependant to the service/ network Metrology Tools User experience
  8. 8. Editor’s Focus / Collaborative Suite 1911 1975 1998 Identity, Services / Softwares Windows / Office Search Strategy Social Business « All In » Pure Player Product LotusLive Office 365 Google Apps For BusinessAcquisition By Yes Progressive (options) No modules €/U/Y 34 to 113 € 43 to 306 € 40 €(public prices) • Social and Collaborative • Pack Office Integrated • Web Experience Advantages Network • Sharepoint Framework • Google ecosystem • External Opening • Some missing features • SharePoint Out-of-the-Box • No web conference Weaknesses (online office suite, (ergonomy, features) • Very basic integrated social and blogs/wikis, call audio/video) • Document-Centric collaborative tools
  9. 9. E-Mail Systems Focus “ Inbox = Tetris getting zero lines doesn’t win it just makes the next move easy “ Product • LotusLive iNotes/Notes • Exchange Online • GMail • Codes different than a classic • Classic codes • Classic codes messaging suite Features • Native Lotus Notes included • Native Outlook integrated • GMail offline (Google Chrome) • Two offers: basic(web mail) + Technical advanced (Domino) • Hybride Mode • Full Web • Hybrid Mode • Leader On-Premise • Lack of visibility, can look strong • In position of conquest • Late Coming-Out Cloud Analysis, enough just for Domino clients • Big efforts made in some • Difficult move from BPOS to trend • Domino/LotusLive future vision points (sécurity / Office 365 : which available administration) consequeces ? Space 25 Go 25 Go 25 Go SLA,Credit according 99,9 % 99,9 % 99,9 %to the available 2 à 10% monthly cost 25 à 100% monthly cost 3 to 15 days %
  10. 10. Methodology Data Solution Pilot for all web applications , rich web or client ( Office 365, Outlook…) run like a user: mouse click, filling application forms, text control, click on image etc. Fast launch on one of the 70 measuring points in the world (5 operators are tested in France, Orange, Neuf-SFR, Iliad-Free, Colt, Cogent) Mode « Saas » on demand, from set up tests to operational robot and servers maintenance Access to all measures via an extranet, reports sent via mail and warnings on incidents Access to back up for all set up , expertise and training requests Scenari recording tools for clients wanted to be autonomous
  11. 11. Solution Architecture
  12. 12. Tests Description 3 platforms tested 3 types of tests – Access to the portal from 5 spots in France • https://portal.microsoftonline.com • https://apps.lotuslive.com • https://www.google.com/a/synergie-informatique.fr/ • Tests origins: France Telecom, Neuf-SFR, Iliad-Free, Colt, Cogent • Tests frequency: 15 minutes • Tests duration: 21/07/11 au 30/09/11 (71 jours) – Scenario SENDING from 1 spot in France • Sending a mail from the Cloud platform to a external messging address • Delivery mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours) – Scenario RECEIVING from 1 spot inFrance • Sending a mail from an external messaging address rto the Cloud platform • Receiving mail authentication in less than 60 seconds • Tests frequency: 30 minutes • Tests duration: 1/09/11 au 30/09/11 (30 jours)
  13. 13. Tests Description Scenario SENDING – Connexion to the portal – Authentication – Access to the e-mail – New online message to mailXX@ip-label.net – Send the message and wait for POP from ip-label (mail.ip-label.com) – waiting time max 60 seconds Scénario RECEIVING – Connexion to the portal – Authentication – Access to the e-mail – Send a mail via SMTP from a server from ip-label (mail.ip-label.com) – Wait for mail receipt on the tested platform – waiting time max 60 seconds Each sent mail is automatically deleted after each test Each portal account uses a distinctive external messaging account
  14. 14. Tests Description When the portal we are trying to access is considered unavailable? – Non loading of the portal in less than 60 seconds – Network incidents detected ( DNS faulty resolution , failed connexion) and server(loading slowness) When a scenario is considered unavailable ? – Non complete loading of a page in less than 30 seconds linked to network or server incidents – Keywords not found • Username to validate the loading of the portal • Username/Password, connexion button, then button ‘new mail to validate the authentication step • SEND : button new mail and send • RECEIVING : refresh button to verify the mail had been received – Non transmission of the e-mail ( external SMTP  online messaging or online messaging SMTP) in less than 60 seconds
  15. 15. Portal Access Availibility and average access time to each portal Results from 21/07/11 to 30/09/11 – Around 34000 tests per portal 1,541 seconds 1,097 seconds 3,552 seconds 99.980 % 99.977 % 99.968 %google apps Résolution DNS lotus live Temps de connexion Temps de réponse du serveur Office 365 Chargement du contenu 0 500 1000 1500 2000 2500 3000 3500 4000– Google has the best portal loading time– Lotus Live is quite fast (1,5 seconds) but the page is longer to start (600ms), with longer DNS resolution, connexion and server time– Office 365 is the longest to load it is also the heaviest (1 Mo, versus 17 ko for google and 32 ko for Lotus Live)– Lotus Live shows the best availibility with 7 failures, Google (8 échecs) and Office 365 (11 failures)
  16. 16. Portal Access Access to portal availibility Access to portal performance Evolution over a 3 months period 4000 100 100 99,99 99,98 99,98 99,98 99,98 99,98 3500 3750 99,95 99,96 3474 3432 3000 99,9 2500 1982 Office 365 2000 99,85 1357 lotus live 1500 1286 99,83 google apps 99,8 1000 1119 1113 1061 500 99,75 0 Juillet Aout Septembre 99,7 Juillet Aout Septembre Conclusion- For the LotusLive portal , the answering time network and server has increased since 31/08/11 ( for an identical content)- Main errors encountered- Google : 5 DNS resolution errors and 3 time outs Lotus Live : 4 interrupted connections and 2 unreachable servers Office 365 : 4 DNS resolution errors, 5 unreachable servers and 2 timeouts- A maintenance intervention on the Office 365 portal on 9/9/11 from 5am to 8;15am was not taken into account
  17. 17. Authentication Authentication availibility and online messaging access Results from 01/09/11 to 30/09/11 –About 2800 tests per portal Performance Authentication and online 99,76 messaging access Disponibilité 99,86 24h/24 99,17 11438 Disponibilité 99,89 Office 365Heures Ouvrées 99,89 google apps 8h-23h 99,56 15238 lotus Live Disponibilité 99,54 19054 HNO 99,81 23h-8h 98,52 0 5000 10000 15000 20000 97,50 98,00 98,50 99,00 99,50 100,00 – Google is the strongest plaftform in terms of authentication 99.86 % in 24/24 – Office 365 99.76% in 24/24 on authentication (mainly linked to some timeouts and DNS errors); those incidents usually occur at night and from 8h to 23h the availibility is the same as Google – LotusLive has the most errors at the authentication (99.17%); even if those errors occur mainly at night time, (99.56% availibility between 8h and 23h), one incident last September 18th between 8h and 10h penalise their position on this scale – Google is the fastest platform to authentify yourself and access to messaging : 11,5 seconds, LotusLive 15 seconds ( in one page load in opposition to the others) – Office 365 is the slowest to reach online Outlook : 19 seconds
  18. 18. Authentication Authentication Incident for Lotus Live on 18/09 Timeout : incomplete page after login (88k out of 212ko expected) Page error after loginl 18
  19. 19. AuthenticationAuthentification errors examples Authentification timeout on Office 365 Authentification timeout on Google Apps
  20. 20. Authentication Examples : Authentication errors on Google Apps portalErreur du serveur (HTTP 500) sur l’objet Dashboard qui devrait rediriger sur le portail Erreur Cas normal Impact utilisateur : le champ login absent de la page et l’utilisateur ne peut pas s’authentifier
  21. 21. Sending/Receiving mail Performance Average length of time to send and receive e-mails 12271Temps de réception 23213 18895 google apps lotus live 7558 Office 365 Temps denvoi 10049 15095 0 5000 10000 15000 20000 25000– On average, to send a mail from a Cloud platform is faster than receiving a mail from an external platform– Google is the platform which transmits mails the fastest, as well as sending to an external account (12,2 seconds on average) and receiving from an external account(7.5 seconds)– Office 365 is faster than Lotus Live for receiving mail, but slower than LotusLive to send mails
  22. 22. Sending/Receiving mail Stability Average availibility for sending and receiving mails 86,17Disponibilité de réception (<60s) 99,18 71,66 google apps lotus live 85,17 Office 365 Disponibilité denvoi (<60s) 97,62 74,81 0 10 20 30 40 50 60 70 80 90 100 – Lotus Live is the most available platform in terms of sending mails, if we consider as unavailable a length of time superior to 60 seconds ,to send a mail CloudExterne or ExterneCloud – Office 365 (~75%) and Google Apps (~85%) : This means even with a fast sending and receiving mail process (<15 secondes), engorgements regularly slow down the e-mail sending or receiving process – Sending and receiving mail process is more unstable for Google and Office 365
  23. 23. 2 1 3 Google Apps Gmail can be considered as a reference in terms of online Scores Google Office messaging access speed and you are guaranteed fast sending and Apps Lotus Live 365 receiving mail Portal performance 3 2 1 Portal availibility 2 3 1 IBM Lotus Live is a stable platform in terms of sending and receiving Authentication availability and online messaging access 3 2 1 mails but can be a disappointment for users because of access Authentifcation availability unavailibility and average access time (15 sec) (working hours) 3 1 3 Receiving Time 3 1 2 Microsoft Office 365 has a good messaging availibility (99.89% during Sending Time 3 2 1 Receiving Stability 2 3 1 working hours) but some progress need to be made on page loading Sending Stability 2 3 1 time ( particularly on authentication and access to Outlook) and on Total 21 17 11 sending stability
  24. 24. Conclusions Results Details Gmail Google – Best Portal Access Performance – Best authentication performance and availibity – Best sending time to an external e-mail address and receiving from an external e-mail address – Relative unstability in transfert time frames • 15 mails out of 100 are sent in more than 60 seconds • 14 mails out of 100 are received in more than 60 seconds IBM Lotus Live – Look out on the highest portal loading time since the beginning of September – Best portal access availibility – Best transfert stability 1 mails out of 100 are sent in more than 60 seconds 3 mails out of 100 are received in more than 60 seconds – Authentication performance and availibility is not good Microsoft Office 365 – Very good authentication disponibility during working hours. – Portal access performance and access to messaging to be improved (19 seconds) – Stabilité des durées de transmission des mails à améliorer • 25 mails out of 100 are sent in more than 60 seconds • 28 mails out of 100 are sent in more than 60 seconds
  25. 25. ConclusionsEach editor commits himself to 99.9% availibility – « Microsoft provides a financially-backed 99.9% uptime guarantee » • http://www.microsoft.com/fr-fr/office365/what-is-office365.aspx – « LotusLive Notes delivers the capabilities your business users need to increase efficiency while reducing IT complexity and associated bottom line costs - all backed by a dedicated IBM team with 24x7 support and a 99.9% up-time service level objective » • https://www.lotuslive.com/styles/tours/transcripts/LLNotes_datasheet.pdf – « Google Apps includes a 99.9% uptime SLA and synchronous replication » • http://www.google.com/apps/intl/en/business/messaging.htmlAs a reminder data for September 2011… According to you, what is the quality really perceived by the online messaging users ?
  26. 26. Q/A Were mails send with attachments ? – No , only text content; the add on of attachments of different sizes could be a criteria to do more tests Which internet navigator was used for the tests? – Internet Explorer 8 Can those tests be run on a client (Lotus Notes, outlook) ? Yesthe robots from ip-label can simulate any user running windows Can this benchmark be testing with files /document sharing? – Sharing files tools can also be tested by the robots ip-label What about other navigators , especially if Microsoft Exchange is losing out on fonctionnality ? – The robots can test Explorer or Firefox; the multi navigators cans be given through other solutions like Cloud Observer for instance Can we audit audio / video flow? – Ip-label can measure those flows using different measuring solutions on a user standpoint (streaming, voice quality analysis, video tests /audio conference, etc.)
  27. 27. Thank You Synergie-informatique.fr @SynergieInfo

×