SlideShare a Scribd company logo
BP106 Worst Practices in IBM Lotus Domino   Paul Mooney – Blue Wave Technology Bill Buchan - HADSL
Some wise words…. Text slide  with multiple  quotes 2 quotes on page:  24pt Arial Italic, yellow R255 | G204 | B0 Reference: 18pt Arial Regular, white “   Learn all you can from the mistakes of others. You won't have time to make them all yourself ” Alfred Sheinwold “  Experience is learning from your mistakes. Competence is learning from other peoples mistakes Professionalism is knowing that you can make mistakes, and accept that they will happen. ”
“PHB” Says…. Text slide  with multiple  quotes 2 quotes on page:  24pt Arial Italic, yellow R255 | G204 | B0 Reference: 18pt Arial Regular, white “ Perhaps I can find new ways to motivate you . ” Darth Vadar, long long ago “   There is no mistake, there has been no mistake; and there shall be no mistake. ” Wellingtoniana, 1852
Agenda For each case study, we shall Review some of the more “unusual” topology/application configuration errors that have occurred Diagnose the problem Determine the problem How was it resolved What lessons can be learned We have 12 case studies, and some short cases near the end. We cover both infrastructure and development… Remember: We only have time for limited questions at the end – so if you have more, we shall be in the Speakers room after this session Agenda slide
Case Studies Directory Designs Roaming, Roaming…  Greenfield grief  A Very Secure Environment  Hell’s Agent Application Migration Spaghetti Junction  The Long Way Round Time travel for beginners  Lucifers' Lotusscript The eMail from hell The Domino Virus Contents can be interactive: highlight on mouse over and hyperlink to section on mouse click.
Story Number 1: Lets play with the Directory…. The Story… A support call comes in  - no mail routing Many errors appearing on console Server document not found / path errors The Investigation Lets see the log file  Access the console remotely Assess the errors Ask administrator to open directory “what do you see there”… Gotcha! Inbox, Drafts, Sent, All Documents etc……… Basic text slide
1: Lets play with the Directory…. The Cause Administrator replaced design of directory with mail file template by accident Resolution Replace design back manually on all servers or restore from backup…  Restart Servers Force Replication  Lessons Learned… Limit Manager rights to the Domino directory Break down access based on experience and knowledge/Rank within department ND6 has excellent granular access levels within the directory LocalDomainServers does NOT require manager access to the Nab Hub and spoke topology…. Grant manager access rights to the servers from which you manage the domain Spokes need minimum editor access  Prevents breakout of directory design problem Basic text slide
2: Roaming, roaming The Story Many companies use “network home drives” for users’ notes data directories, so that they can “roam” between workstations. Whilst not supported, it does work.. New users get a “default” notes data directory – an image copy… Lotus notes “Roaming” was switched on. Everyone started to see each others personal name and address book entries.. The Investigation Lotus Notes roaming places replica copies of the users personal address books on the server.  The Domino server started replicating. Gotcha! Since all the personal address books had the same replica ID, the notes server was quite rightly replicating... Basic text slide
2: Roaming, roaming Resolution Change the replica ID's in all personal address books Lessons Learnt Unlike Domino v5, Domino v6 will quite happily replicate with multiple databases on the same server with the same replica ID Understand, think through and test your processes and data for any “discovered check” style problems such as this. Basic text slide
3: Greenfield grief….. The Story Brand new domain Fresh rollout of hardware/network/domino Administrator had no previous knowledge of Domino Domino server communication not working Certificate errors on all servers/clients The Investigation The Domino directory seemed to be replicating fine at the first look All data was synchronised  Certificate errors and cross certificate requests appearing constantly Multiple certificate documents in directory with same name – different keys Gotcha! Asked the administrator a couple of questions… Basic text slide
3: Greenfield grief….. The Cause The administrator installed each new server as “First server in the domain” He created each server with same domain/certificate names He “replicated” all data by copying in all server documents/certificate documents manually every evening Resolution Reinstall environment. Lessons learned Education! Education! Education! Understand at least the basics of a product before installing it Use the resources at hand Help files Lotus Support LDD forums Blogs Basic text slide
4: A very secure site The Story A huge environment had just been constructed – lots of new servers, in many countries 2 hours before handing over to the customer, everyone was locked out! The project team decided to switch on “Public Key Checking” in the hours before handover. Gotcha! It had locked  everyone  out of the environment. Resolution One person had left a laptop logged in, and could edit the Directory. The first server came back up, just as the customer entered their building…. Basic text slide
4: A very secure site Lessons learned Don’t lose Certificate passwords half way through deployment, then just quietly create a new certificate key of the same name and hope no-one spots it.... As any project nears completion, there should be more and more resistance to making any changes to the plan. Change Control! Wide, sweeping changes to infrastructure should be tested in labs in advance. Well in advance! Basic text slide
5: Hell’s agent! The Story A large Multinational A critical application sits on all servers 3GB Database / 65,000 documents Replicates from three global hub clusters to all spokes hourly All server communication grinds to a halt No Mail routing/replication Application grows to 28GB Masses of replication conflicts The Investigation Check application for design changes Check replication history and schedule Check server tasks Sniff the bandwidth Gotcha! New scheduled agent Basic text slide
5: Hell’s agent! The cause Developer wanted to modify all documents Built an all documents view Wrote and agent to modify a field Agent set as scheduled “every hour” Set agent  to run on …. ALL SERVERS Ran on Hub first…  Hub replicated with all spokes on 1-hour replication schedule Then ran on all servers Then continued to run and replicate for the weekend 4.8 Million documents per hour! Basic text slide
5: Hell’s agent! Lessons Learned Developers must never change design on production systems  Even basic agents Have separate development domain/UAT/Production domains Developers should NOT have designer access on UAT/Production domains Domino is very powerful, and WILL do whatever you tell it to do – no matter how stupid.. Never leave new code unsupervised Basic text slide
6: Application Migration The Story Application migration project upgrading from v4.5.x to v6.5.x Migrating platform from W32 to iSeries 18,000 + Applications Applications refuse to work on new servers. Investigation Approximately 18% of all applications used hard-coded server and database names in LotusScript and formula language. Windows-specific OLE style calls executed on the server Gotcha! Hard-coded server and database names in applications should be kept to a complete minimum Resolution Analyse every single application for code weaknesses and Fix those applications with issues. Basic text slide
6: Application Migration Lessons learned Keep an application register of all applications showing: Who “owns” the application Any dependences outside the application – such as OLE, platform, other lotus notes databases, file transfer directories. Any business critical processes around this application. Monitor applications for usage, and retire or archive the ones no longer used. Activity analysis Use Profile documents for application configuration Use separate Development/UAT/Production environments, where the developers have NO access to UAT/production databases.  This will FORCE all hard coded dependencies to the surface. Basic text slide
7: Spaghetti Junction The Story Server performance issues even after hardware upgrade Six domino servers All in one single Domino Named Network Busy site 2 x 2way Domino clusters Over 1000 concurrent users Very high specification hardware We were called in to have a peek…. The Investigation Checked server performance Win2k “perfmon” Log files Server analyser Trend analysis Gotcha! Checked directory Basic text slide
7: Spaghetti Junction Basic text slide CLA SMTP HUB CLA CLB CLB Scheduled Replication Mail Braces Belt
7: Spaghetti Junction Basic text slide CLA SMTP HUB CLA CLB CLB Scheduled Replication Mail Braces Belt 96 Connection Documents Servers replicate with all servers every 5 minutes Maintenance tasks running during office hours Transactional logs on same Raid array as Domino data No data quotas whatsoever All servers running multiple DEBUG parameters
7: Spaghetti Junction Resolution No mail routing connections were required It’s a Domino Named Network! Maintenance tasks changed and schedules modified Transaction logging moved to separate Raid Arrays Replication connection documents drastically reduced Logging reduced Multiple CPU’s utilised on servers Lessons learned Never be afraid of the manual Education! Education! Education! Understand the basics of Domino and it is an easy environment to manage Never implement a feature without researching it thoroughly For example Transactional logging and disk arrays Basic text slide
8: The Long Way Round The Story A new custom application. Failed to execute scheduled agent within the 15-minute limit. The Investigation Very large data collection exercise – thousands of documents to be scanned and reported on. A nightly task. Gotcha! The Developer was NOT using ANY views! The Cause The developer was unaware of views and was parsing ALL documents in ALL databases Re-reading documents from previous days. Basic text slide
8: The Long Way Round Resolution Application, when ran on workstation, against live data – took 9 HOURS. When re-written to actually use views (but still open each document in turn) – 2 hours. When re-written to use view information – executed within 15 minutes.  Lessons learned “But it worked in testing” is not a valid excuse Testing data has to be representative at UAT (User Acceptance Testing) stage A proper Development/UAT/Production environment should be set up. Developer should learn more about Notes before being given task of this complexity. Basic text slide
9: Time Travel for Beginners The Story SMB Domino site Fully intranet based Domino Web  Access/Custom applications Three-way Domino cluster W32 based OS on all servers Time zone issues on system default profile Common for non-US customers DMY comes up as MDY Local Admin runs some testing over weekend Data stops replicating to other servers The Investigation Replcation tasks and logs checked  - Seems to be fine ACL’s checked - All correct Basic text slide
9: Time Travel for Beginners Gotcha! Replication history checked… Last successfully replicated at 10.48 am,  11/8/2057  on applications The Cause Administrator was baffled by Time/Date issue on server consoles Reset date to 2057 as a test Brought domino server back up on cluster LAN Went to lunch…… Server replicated When the date is returned to the present, Domino sees no need to replicate! Resolution Replication history has to be purged from all databases on server Deletion stubs must be purged from databases Basic text slide
9: Time Travel for Beginners Lessons learned Domino servers rely on the OS date configuration/settings Domino servers and replication architecture rely on replication history Never go forward in time on a domino server. In fact, this is true of all database systems where timestamps are used. This is not restricted to Domino! All time/date work on a Domino server must be done on a simulated environment Basic text slide
10: Lucifers' Lotusscript The Story A large application migration project from v4.5 to v6.5 Application server had just been migrated,  Developer is informed that his agent has failed Developer contacts administrators and tells them to fix problem, without examining code. Gotcha! Agent used OLE calls to Oracle database, and no documentation existed to show external linkage. Mix of back-end and front-end code in scheduled agent No error handling Hardcoded Data link names, server names, database path names. Domino v6 “agent security” was set to “1 – do not allow restricted...” Resolution Recode agent. Basic text slide
10: Lucifers' Lotusscript Lessons learned Separate Development/UAT/Production environments Applications that depend on anything outside their own database should have installation documentation. The following should be mandatory in your development best practices: “Option Declare”  “Errorhandling” Logging of critical errors to some monitored Log. Documentation DONT have 300-lines of code in “initialize”. Try and structure ? DONT mix front-end and back-end code in a scheduled agent. Time-sensitive events such as the web events should not have heavy duty code such as OLE links to Oracle embedded in them.  You can write the best code in the world, but if your administrators cannot manage it, its wasted effort. Basic text slide
11: The email from hell The Story Domino server becomes very slow/almost unresponsive Users complaining All users eventually fail over  to cluster server and continue working The Investigation Go to Server “Nrouter.exe” claiming 97% CPU constantly Show Tasks on console Router in “Dispatching messages” mode Gotcha! Open mail.box (or mail1.box, mail2.box, etc) on server Basic text slide
11: The email from hell Basic text slide , 995 Bytes , 397 , 145 2 !
11: The email from hell The Cause A sales rep had mailed the contents of his H:ser directory to HIMSELF Was moving desk Resolution Stop router task, Delete the mail from “mail.box” Reload router task, Compact “mail.box” -  “tell router compact” Monitor mail routing and cluster task Small Point. Router would have delivered this email… Cluster would have then clustered this email - Inbox and sent view – almost 5GB Credit to a fantastic mail routing/cluster engine Lessons learned Limit the internal mail attachments Configuration document – set mail priority Mail Rules Customise mail template? Basic text slide
12: The Domino Virus The Story All servers in this large environment stopped mail routing and replication User access and administration access was fine. Investigation Mail was building up on main routing hubs, and on all servers. Gotcha! A Junior administrator had dropped “LocalDomainServers” into the “Terminations” group. As the directory replicated, it would successfully update other servers, then disallow server to server access. Resolution Administrators had to edit the “Terminations” group in each and every server in order to allow server to server communication. Basic text slide
12: The Domino Virus Lessons learned Inexperienced administrators should NOT be allowed open access to directories in large environments. Education! Education! Education! PHB’s think that ANYONE can be trained to do ANY job.  Sometimes, someone with NO computer experience whatsoever cannot be trained to be a Lotus Domino administrator in 4 weeks… Basic text slide
Shorties When you have more than one Notes domain, try and make sure the domain names are unique… Dont allow developers to sign agents in production environments.  Especially if they are contractors on short term assignments. Lotus notes Name fields used in messaging ONLY support ASCII characters. Really! Use “Alternate names” for non-ASCII names, and using Ampersands (“&”) in names causes pain.  Notes domain names shouldn’t really have dots in them.  Certificate/Server names should not have spaces Beta software in a live domain is not always a good idea Never leave the client up on a server.  Don’t switch id on a server. Ever. Be careful what ports you stop on the server -Default- = Manager = Bad Idea! Allow Anonymous Notes connections  = Bad idea Enable the cleanup agent on the Domlog.nsf file Dont cluster over 64k/bit links...
What have we done here? We’ve passed on a few experiences Learned a few lessons on what  not  to do Felt a bit better about any mistakes you may have made Remember. Mistakes  will  happen. No-one is perfect. Its how you deal with them that defines your professionalism. Never make sweeping changes under time pressure. You'll just dig a deeper hole... “Measure Twice, Cut Once” Time for some questions ? Remember – we will listen to your confessions in the speaker room… Basic text slide
Thank you Paul Mooney  www.pmooney.net Blue Wave Technology  www.bluewave.ie Bill Buchan  www.billbuchan.com Hadsl www.hadsl.com Basic text slide

More Related Content

What's hot

DevOps For Solo Developers
DevOps For Solo DevelopersDevOps For Solo Developers
DevOps For Solo Developers
Jure Cuhalev
 
P2P with Flash Player 10.1
P2P with Flash Player 10.1P2P with Flash Player 10.1
P2P with Flash Player 10.1
Peter Elst
 
DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.
Vlad Fedosov
 
PowerShell Remoting
PowerShell RemotingPowerShell Remoting
PowerShell Remoting
Concentrated Technology
 
Defcon 22-philip-young-from-root-to-special-hacking-ibm-main
Defcon 22-philip-young-from-root-to-special-hacking-ibm-mainDefcon 22-philip-young-from-root-to-special-hacking-ibm-main
Defcon 22-philip-young-from-root-to-special-hacking-ibm-main
Priyanka Aash
 
PS scripting and modularization
PS scripting and modularizationPS scripting and modularization
PS scripting and modularization
Concentrated Technology
 
From VB Script to PowerShell
From VB Script to PowerShellFrom VB Script to PowerShell
From VB Script to PowerShell
Concentrated Technology
 
TeelTech - Advancing Mobile Device Forensics (online version)
TeelTech - Advancing Mobile Device Forensics (online version)TeelTech - Advancing Mobile Device Forensics (online version)
TeelTech - Advancing Mobile Device Forensics (online version)
Mike Felch
 
Real-World WebAppSec Flaws - Examples and Countermeasues
Real-World WebAppSec Flaws - Examples and CountermeasuesReal-World WebAppSec Flaws - Examples and Countermeasues
Real-World WebAppSec Flaws - Examples and Countermeasues
volvent
 

What's hot (9)

DevOps For Solo Developers
DevOps For Solo DevelopersDevOps For Solo Developers
DevOps For Solo Developers
 
P2P with Flash Player 10.1
P2P with Flash Player 10.1P2P with Flash Player 10.1
P2P with Flash Player 10.1
 
DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.
 
PowerShell Remoting
PowerShell RemotingPowerShell Remoting
PowerShell Remoting
 
Defcon 22-philip-young-from-root-to-special-hacking-ibm-main
Defcon 22-philip-young-from-root-to-special-hacking-ibm-mainDefcon 22-philip-young-from-root-to-special-hacking-ibm-main
Defcon 22-philip-young-from-root-to-special-hacking-ibm-main
 
PS scripting and modularization
PS scripting and modularizationPS scripting and modularization
PS scripting and modularization
 
From VB Script to PowerShell
From VB Script to PowerShellFrom VB Script to PowerShell
From VB Script to PowerShell
 
TeelTech - Advancing Mobile Device Forensics (online version)
TeelTech - Advancing Mobile Device Forensics (online version)TeelTech - Advancing Mobile Device Forensics (online version)
TeelTech - Advancing Mobile Device Forensics (online version)
 
Real-World WebAppSec Flaws - Examples and Countermeasues
Real-World WebAppSec Flaws - Examples and CountermeasuesReal-World WebAppSec Flaws - Examples and Countermeasues
Real-World WebAppSec Flaws - Examples and Countermeasues
 

Viewers also liked

Speedgeeking 30 Tips
Speedgeeking   30 TipsSpeedgeeking   30 Tips
Speedgeeking 30 Tips
Bill Buchan
 
Supertition Scissor
Supertition ScissorSupertition Scissor
Supertition Scissor
guest9bbe3e
 
Teorema De Pitágoras4
Teorema De Pitágoras4Teorema De Pitágoras4
Teorema De Pitágoras4
guest276e62
 
Hello Look After Yourself
Hello  Look After YourselfHello  Look After Yourself
Hello Look After Yourself
guest5ab9fe
 
Cadena, Jauría de Perros Atacando Cocodrilo
Cadena, Jauría de Perros Atacando CocodriloCadena, Jauría de Perros Atacando Cocodrilo
Cadena, Jauría de Perros Atacando Cocodrilo
Eduardo Ortiz
 
Amazonian Flora At Marajo Island
Amazonian Flora At Marajo IslandAmazonian Flora At Marajo Island
Amazonian Flora At Marajo Island
ivan anzuategui
 
Caracteristicas De LiderançA
Caracteristicas De LiderançACaracteristicas De LiderançA
Caracteristicas De LiderançA
oleinik
 
Huallpa Etica 4
Huallpa Etica 4Huallpa Etica 4
Huallpa Etica 4
huallpa_odo
 
La Paloma
La PalomaLa Paloma
Provérbios2
Provérbios2Provérbios2
Provérbios2
juliana_bohn
 
.Carpinteria
.Carpinteria.Carpinteria
.Carpinteria
nogueherre
 
Macbook Laptop Orinentation 2008
Macbook Laptop Orinentation 2008Macbook Laptop Orinentation 2008
Macbook Laptop Orinentation 2008
guest7b19f3
 
What Do I Do Elevator Talk
What Do I Do Elevator TalkWhat Do I Do Elevator Talk
What Do I Do Elevator Talk
Jerome Connolly
 
Ppt17a0 [Recuperado]
Ppt17a0 [Recuperado]Ppt17a0 [Recuperado]
Ppt17a0 [Recuperado]
guest99abfb
 
Because You'Re Beautiful2
Because You'Re Beautiful2Because You'Re Beautiful2
Because You'Re Beautiful2
guest1667ec
 
Ciencias Antropológicas
Ciencias AntropológicasCiencias Antropológicas
Ciencias Antropológicas
Joselinne_0985
 
Kingston,Ny
Kingston,NyKingston,Ny
Kingston,Ny
guest00d7ef
 
Bryce Canyon
Bryce CanyonBryce Canyon
Bryce Canyon
guestae29dc
 

Viewers also liked (20)

Speedgeeking 30 Tips
Speedgeeking   30 TipsSpeedgeeking   30 Tips
Speedgeeking 30 Tips
 
Supertition Scissor
Supertition ScissorSupertition Scissor
Supertition Scissor
 
Teorema De Pitágoras4
Teorema De Pitágoras4Teorema De Pitágoras4
Teorema De Pitágoras4
 
Hello Look After Yourself
Hello  Look After YourselfHello  Look After Yourself
Hello Look After Yourself
 
Cadena, Jauría de Perros Atacando Cocodrilo
Cadena, Jauría de Perros Atacando CocodriloCadena, Jauría de Perros Atacando Cocodrilo
Cadena, Jauría de Perros Atacando Cocodrilo
 
Amazonian Flora At Marajo Island
Amazonian Flora At Marajo IslandAmazonian Flora At Marajo Island
Amazonian Flora At Marajo Island
 
Caracteristicas De LiderançA
Caracteristicas De LiderançACaracteristicas De LiderançA
Caracteristicas De LiderançA
 
Huallpa Etica 4
Huallpa Etica 4Huallpa Etica 4
Huallpa Etica 4
 
Ouvir Camoes
Ouvir CamoesOuvir Camoes
Ouvir Camoes
 
La Paloma
La PalomaLa Paloma
La Paloma
 
Iris Alvial
Iris AlvialIris Alvial
Iris Alvial
 
Provérbios2
Provérbios2Provérbios2
Provérbios2
 
.Carpinteria
.Carpinteria.Carpinteria
.Carpinteria
 
Macbook Laptop Orinentation 2008
Macbook Laptop Orinentation 2008Macbook Laptop Orinentation 2008
Macbook Laptop Orinentation 2008
 
What Do I Do Elevator Talk
What Do I Do Elevator TalkWhat Do I Do Elevator Talk
What Do I Do Elevator Talk
 
Ppt17a0 [Recuperado]
Ppt17a0 [Recuperado]Ppt17a0 [Recuperado]
Ppt17a0 [Recuperado]
 
Because You'Re Beautiful2
Because You'Re Beautiful2Because You'Re Beautiful2
Because You'Re Beautiful2
 
Ciencias Antropológicas
Ciencias AntropológicasCiencias Antropológicas
Ciencias Antropológicas
 
Kingston,Ny
Kingston,NyKingston,Ny
Kingston,Ny
 
Bryce Canyon
Bryce CanyonBryce Canyon
Bryce Canyon
 

Similar to Bp106 Worst Practices Final

Lotusphere 2009 The 11 Commandments
Lotusphere 2009 The 11 CommandmentsLotusphere 2009 The 11 Commandments
Lotusphere 2009 The 11 Commandments
Bill Buchan
 
Lug
LugLug
Webinar: From Frustration to Fascination: Dissecting Replication
Webinar: From Frustration to Fascination: Dissecting ReplicationWebinar: From Frustration to Fascination: Dissecting Replication
Webinar: From Frustration to Fascination: Dissecting Replication
Howard Greenberg
 
From frustration to fascination: dissecting Replication
From frustration to fascination: dissecting ReplicationFrom frustration to fascination: dissecting Replication
From frustration to fascination: dissecting Replication
Benedek Menesi
 
Cs121 Unit Test
Cs121 Unit TestCs121 Unit Test
Cs121 Unit Test
Jill Bell
 
Top 10 Scalability Mistakes
Top 10 Scalability MistakesTop 10 Scalability Mistakes
Top 10 Scalability Mistakes
John Coggeshall
 
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
Martijn de Jong
 
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
Codium
 
Intro To Linux
Intro To LinuxIntro To Linux
Intro To Linux
techlug
 
PHP Performance: Principles and tools
PHP Performance: Principles and toolsPHP Performance: Principles and tools
PHP Performance: Principles and tools
10n Software, LLC
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goals
kamaelian
 
Top 30 Scalability Mistakes
Top 30 Scalability MistakesTop 30 Scalability Mistakes
Top 30 Scalability Mistakes
John Coggeshall
 
Open Source in the Enterprise
Open Source in the EnterpriseOpen Source in the Enterprise
Open Source in the Enterprise
Social Media Performance Group
 
Wcl303 russinovich
Wcl303 russinovichWcl303 russinovich
Wcl303 russinovich
conleyc
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
IXIASOFT
 
Performance Analysis of Idle Programs
Performance Analysis of Idle ProgramsPerformance Analysis of Idle Programs
Performance Analysis of Idle Programs
greenwop
 
System design for Web Application
System design for Web ApplicationSystem design for Web Application
System design for Web Application
Michael Choi
 
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Bhupesh Bansal
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Hadoop User Group
 
2019 PHP Serbia - Boosting your performance with Blackfire
2019 PHP Serbia - Boosting your performance with Blackfire2019 PHP Serbia - Boosting your performance with Blackfire
2019 PHP Serbia - Boosting your performance with Blackfire
Marko Mitranić
 

Similar to Bp106 Worst Practices Final (20)

Lotusphere 2009 The 11 Commandments
Lotusphere 2009 The 11 CommandmentsLotusphere 2009 The 11 Commandments
Lotusphere 2009 The 11 Commandments
 
Lug
LugLug
Lug
 
Webinar: From Frustration to Fascination: Dissecting Replication
Webinar: From Frustration to Fascination: Dissecting ReplicationWebinar: From Frustration to Fascination: Dissecting Replication
Webinar: From Frustration to Fascination: Dissecting Replication
 
From frustration to fascination: dissecting Replication
From frustration to fascination: dissecting ReplicationFrom frustration to fascination: dissecting Replication
From frustration to fascination: dissecting Replication
 
Cs121 Unit Test
Cs121 Unit TestCs121 Unit Test
Cs121 Unit Test
 
Top 10 Scalability Mistakes
Top 10 Scalability MistakesTop 10 Scalability Mistakes
Top 10 Scalability Mistakes
 
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
BP101 - 10 Things to Consider when Developing & Deploying Applications in Lar...
 
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
Learned lessons in real world projects by Jordi Anguela at Mallorca Software ...
 
Intro To Linux
Intro To LinuxIntro To Linux
Intro To Linux
 
PHP Performance: Principles and tools
PHP Performance: Principles and toolsPHP Performance: Principles and tools
PHP Performance: Principles and tools
 
Scaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, GoalsScaling Streaming - Concepts, Research, Goals
Scaling Streaming - Concepts, Research, Goals
 
Top 30 Scalability Mistakes
Top 30 Scalability MistakesTop 30 Scalability Mistakes
Top 30 Scalability Mistakes
 
Open Source in the Enterprise
Open Source in the EnterpriseOpen Source in the Enterprise
Open Source in the Enterprise
 
Wcl303 russinovich
Wcl303 russinovichWcl303 russinovich
Wcl303 russinovich
 
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
Troubleshooting: The Two Laws - IXIASOFT User Conference 2016
 
Performance Analysis of Idle Programs
Performance Analysis of Idle ProgramsPerformance Analysis of Idle Programs
Performance Analysis of Idle Programs
 
System design for Web Application
System design for Web ApplicationSystem design for Web Application
System design for Web Application
 
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
 
2019 PHP Serbia - Boosting your performance with Blackfire
2019 PHP Serbia - Boosting your performance with Blackfire2019 PHP Serbia - Boosting your performance with Blackfire
2019 PHP Serbia - Boosting your performance with Blackfire
 

More from Bill Buchan

Dummies guide to WISPS
Dummies guide to WISPSDummies guide to WISPS
Dummies guide to WISPS
Bill Buchan
 
WISP for Dummies
WISP for DummiesWISP for Dummies
WISP for Dummies
Bill Buchan
 
WISP Worst Practices
WISP Worst PracticesWISP Worst Practices
WISP Worst Practices
Bill Buchan
 
Marykirk raft race presentation night 2014
Marykirk raft race presentation night 2014Marykirk raft race presentation night 2014
Marykirk raft race presentation night 2014
Bill Buchan
 
Dev buchan best practices
Dev buchan best practicesDev buchan best practices
Dev buchan best practices
Bill Buchan
 
Dev buchan leveraging
Dev buchan leveragingDev buchan leveraging
Dev buchan leveraging
Bill Buchan
 
Dev buchan leveraging the notes c api
Dev buchan leveraging the notes c apiDev buchan leveraging the notes c api
Dev buchan leveraging the notes c api
Bill Buchan
 
Dev buchan everything you need to know about agent design
Dev buchan everything you need to know about agent designDev buchan everything you need to know about agent design
Dev buchan everything you need to know about agent design
Bill Buchan
 
Dev buchan 30 proven tips
Dev buchan 30 proven tipsDev buchan 30 proven tips
Dev buchan 30 proven tips
Bill Buchan
 
Entwicker camp2007 calling-the-c-api-from-lotusscript
Entwicker camp2007 calling-the-c-api-from-lotusscriptEntwicker camp2007 calling-the-c-api-from-lotusscript
Entwicker camp2007 calling-the-c-api-from-lotusscript
Bill Buchan
 
Entwicker camp2007 blackberry-workshop
Entwicker camp2007 blackberry-workshopEntwicker camp2007 blackberry-workshop
Entwicker camp2007 blackberry-workshop
Bill Buchan
 
Bp301
Bp301Bp301
Ad507
Ad507Ad507
Ad505 dev blast
Ad505 dev blastAd505 dev blast
Ad505 dev blast
Bill Buchan
 
Admin2012 buchan web_services-v101
Admin2012 buchan web_services-v101Admin2012 buchan web_services-v101
Admin2012 buchan web_services-v101
Bill Buchan
 
Reporting on your domino environment v1
Reporting on your domino environment v1Reporting on your domino environment v1
Reporting on your domino environment v1
Bill Buchan
 
12 Step Guide to Lotuscript
12 Step Guide to Lotuscript12 Step Guide to Lotuscript
12 Step Guide to Lotuscript
Bill Buchan
 
Everything you ever wanted to know about lotus script
Everything you ever wanted to know about lotus scriptEverything you ever wanted to know about lotus script
Everything you ever wanted to know about lotus script
Bill Buchan
 
Admin camp 2011-domino-sso-with-ad
Admin camp 2011-domino-sso-with-adAdmin camp 2011-domino-sso-with-ad
Admin camp 2011-domino-sso-with-ad
Bill Buchan
 
Softsphere 08 web services bootcamp
Softsphere 08 web services bootcampSoftsphere 08 web services bootcamp
Softsphere 08 web services bootcamp
Bill Buchan
 

More from Bill Buchan (20)

Dummies guide to WISPS
Dummies guide to WISPSDummies guide to WISPS
Dummies guide to WISPS
 
WISP for Dummies
WISP for DummiesWISP for Dummies
WISP for Dummies
 
WISP Worst Practices
WISP Worst PracticesWISP Worst Practices
WISP Worst Practices
 
Marykirk raft race presentation night 2014
Marykirk raft race presentation night 2014Marykirk raft race presentation night 2014
Marykirk raft race presentation night 2014
 
Dev buchan best practices
Dev buchan best practicesDev buchan best practices
Dev buchan best practices
 
Dev buchan leveraging
Dev buchan leveragingDev buchan leveraging
Dev buchan leveraging
 
Dev buchan leveraging the notes c api
Dev buchan leveraging the notes c apiDev buchan leveraging the notes c api
Dev buchan leveraging the notes c api
 
Dev buchan everything you need to know about agent design
Dev buchan everything you need to know about agent designDev buchan everything you need to know about agent design
Dev buchan everything you need to know about agent design
 
Dev buchan 30 proven tips
Dev buchan 30 proven tipsDev buchan 30 proven tips
Dev buchan 30 proven tips
 
Entwicker camp2007 calling-the-c-api-from-lotusscript
Entwicker camp2007 calling-the-c-api-from-lotusscriptEntwicker camp2007 calling-the-c-api-from-lotusscript
Entwicker camp2007 calling-the-c-api-from-lotusscript
 
Entwicker camp2007 blackberry-workshop
Entwicker camp2007 blackberry-workshopEntwicker camp2007 blackberry-workshop
Entwicker camp2007 blackberry-workshop
 
Bp301
Bp301Bp301
Bp301
 
Ad507
Ad507Ad507
Ad507
 
Ad505 dev blast
Ad505 dev blastAd505 dev blast
Ad505 dev blast
 
Admin2012 buchan web_services-v101
Admin2012 buchan web_services-v101Admin2012 buchan web_services-v101
Admin2012 buchan web_services-v101
 
Reporting on your domino environment v1
Reporting on your domino environment v1Reporting on your domino environment v1
Reporting on your domino environment v1
 
12 Step Guide to Lotuscript
12 Step Guide to Lotuscript12 Step Guide to Lotuscript
12 Step Guide to Lotuscript
 
Everything you ever wanted to know about lotus script
Everything you ever wanted to know about lotus scriptEverything you ever wanted to know about lotus script
Everything you ever wanted to know about lotus script
 
Admin camp 2011-domino-sso-with-ad
Admin camp 2011-domino-sso-with-adAdmin camp 2011-domino-sso-with-ad
Admin camp 2011-domino-sso-with-ad
 
Softsphere 08 web services bootcamp
Softsphere 08 web services bootcampSoftsphere 08 web services bootcamp
Softsphere 08 web services bootcamp
 

Recently uploaded

(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
Priyanka Aash
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
Priyanka Aash
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
Ivanti
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 

Recently uploaded (20)

(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 

Bp106 Worst Practices Final

  • 1. BP106 Worst Practices in IBM Lotus Domino Paul Mooney – Blue Wave Technology Bill Buchan - HADSL
  • 2. Some wise words…. Text slide with multiple quotes 2 quotes on page: 24pt Arial Italic, yellow R255 | G204 | B0 Reference: 18pt Arial Regular, white “ Learn all you can from the mistakes of others. You won't have time to make them all yourself ” Alfred Sheinwold “ Experience is learning from your mistakes. Competence is learning from other peoples mistakes Professionalism is knowing that you can make mistakes, and accept that they will happen. ”
  • 3. “PHB” Says…. Text slide with multiple quotes 2 quotes on page: 24pt Arial Italic, yellow R255 | G204 | B0 Reference: 18pt Arial Regular, white “ Perhaps I can find new ways to motivate you . ” Darth Vadar, long long ago “ There is no mistake, there has been no mistake; and there shall be no mistake. ” Wellingtoniana, 1852
  • 4. Agenda For each case study, we shall Review some of the more “unusual” topology/application configuration errors that have occurred Diagnose the problem Determine the problem How was it resolved What lessons can be learned We have 12 case studies, and some short cases near the end. We cover both infrastructure and development… Remember: We only have time for limited questions at the end – so if you have more, we shall be in the Speakers room after this session Agenda slide
  • 5. Case Studies Directory Designs Roaming, Roaming… Greenfield grief A Very Secure Environment Hell’s Agent Application Migration Spaghetti Junction The Long Way Round Time travel for beginners Lucifers' Lotusscript The eMail from hell The Domino Virus Contents can be interactive: highlight on mouse over and hyperlink to section on mouse click.
  • 6. Story Number 1: Lets play with the Directory…. The Story… A support call comes in - no mail routing Many errors appearing on console Server document not found / path errors The Investigation Lets see the log file Access the console remotely Assess the errors Ask administrator to open directory “what do you see there”… Gotcha! Inbox, Drafts, Sent, All Documents etc……… Basic text slide
  • 7. 1: Lets play with the Directory…. The Cause Administrator replaced design of directory with mail file template by accident Resolution Replace design back manually on all servers or restore from backup… Restart Servers Force Replication Lessons Learned… Limit Manager rights to the Domino directory Break down access based on experience and knowledge/Rank within department ND6 has excellent granular access levels within the directory LocalDomainServers does NOT require manager access to the Nab Hub and spoke topology…. Grant manager access rights to the servers from which you manage the domain Spokes need minimum editor access Prevents breakout of directory design problem Basic text slide
  • 8. 2: Roaming, roaming The Story Many companies use “network home drives” for users’ notes data directories, so that they can “roam” between workstations. Whilst not supported, it does work.. New users get a “default” notes data directory – an image copy… Lotus notes “Roaming” was switched on. Everyone started to see each others personal name and address book entries.. The Investigation Lotus Notes roaming places replica copies of the users personal address books on the server. The Domino server started replicating. Gotcha! Since all the personal address books had the same replica ID, the notes server was quite rightly replicating... Basic text slide
  • 9. 2: Roaming, roaming Resolution Change the replica ID's in all personal address books Lessons Learnt Unlike Domino v5, Domino v6 will quite happily replicate with multiple databases on the same server with the same replica ID Understand, think through and test your processes and data for any “discovered check” style problems such as this. Basic text slide
  • 10. 3: Greenfield grief….. The Story Brand new domain Fresh rollout of hardware/network/domino Administrator had no previous knowledge of Domino Domino server communication not working Certificate errors on all servers/clients The Investigation The Domino directory seemed to be replicating fine at the first look All data was synchronised Certificate errors and cross certificate requests appearing constantly Multiple certificate documents in directory with same name – different keys Gotcha! Asked the administrator a couple of questions… Basic text slide
  • 11. 3: Greenfield grief….. The Cause The administrator installed each new server as “First server in the domain” He created each server with same domain/certificate names He “replicated” all data by copying in all server documents/certificate documents manually every evening Resolution Reinstall environment. Lessons learned Education! Education! Education! Understand at least the basics of a product before installing it Use the resources at hand Help files Lotus Support LDD forums Blogs Basic text slide
  • 12. 4: A very secure site The Story A huge environment had just been constructed – lots of new servers, in many countries 2 hours before handing over to the customer, everyone was locked out! The project team decided to switch on “Public Key Checking” in the hours before handover. Gotcha! It had locked everyone out of the environment. Resolution One person had left a laptop logged in, and could edit the Directory. The first server came back up, just as the customer entered their building…. Basic text slide
  • 13. 4: A very secure site Lessons learned Don’t lose Certificate passwords half way through deployment, then just quietly create a new certificate key of the same name and hope no-one spots it.... As any project nears completion, there should be more and more resistance to making any changes to the plan. Change Control! Wide, sweeping changes to infrastructure should be tested in labs in advance. Well in advance! Basic text slide
  • 14. 5: Hell’s agent! The Story A large Multinational A critical application sits on all servers 3GB Database / 65,000 documents Replicates from three global hub clusters to all spokes hourly All server communication grinds to a halt No Mail routing/replication Application grows to 28GB Masses of replication conflicts The Investigation Check application for design changes Check replication history and schedule Check server tasks Sniff the bandwidth Gotcha! New scheduled agent Basic text slide
  • 15. 5: Hell’s agent! The cause Developer wanted to modify all documents Built an all documents view Wrote and agent to modify a field Agent set as scheduled “every hour” Set agent to run on …. ALL SERVERS Ran on Hub first… Hub replicated with all spokes on 1-hour replication schedule Then ran on all servers Then continued to run and replicate for the weekend 4.8 Million documents per hour! Basic text slide
  • 16. 5: Hell’s agent! Lessons Learned Developers must never change design on production systems Even basic agents Have separate development domain/UAT/Production domains Developers should NOT have designer access on UAT/Production domains Domino is very powerful, and WILL do whatever you tell it to do – no matter how stupid.. Never leave new code unsupervised Basic text slide
  • 17. 6: Application Migration The Story Application migration project upgrading from v4.5.x to v6.5.x Migrating platform from W32 to iSeries 18,000 + Applications Applications refuse to work on new servers. Investigation Approximately 18% of all applications used hard-coded server and database names in LotusScript and formula language. Windows-specific OLE style calls executed on the server Gotcha! Hard-coded server and database names in applications should be kept to a complete minimum Resolution Analyse every single application for code weaknesses and Fix those applications with issues. Basic text slide
  • 18. 6: Application Migration Lessons learned Keep an application register of all applications showing: Who “owns” the application Any dependences outside the application – such as OLE, platform, other lotus notes databases, file transfer directories. Any business critical processes around this application. Monitor applications for usage, and retire or archive the ones no longer used. Activity analysis Use Profile documents for application configuration Use separate Development/UAT/Production environments, where the developers have NO access to UAT/production databases. This will FORCE all hard coded dependencies to the surface. Basic text slide
  • 19. 7: Spaghetti Junction The Story Server performance issues even after hardware upgrade Six domino servers All in one single Domino Named Network Busy site 2 x 2way Domino clusters Over 1000 concurrent users Very high specification hardware We were called in to have a peek…. The Investigation Checked server performance Win2k “perfmon” Log files Server analyser Trend analysis Gotcha! Checked directory Basic text slide
  • 20. 7: Spaghetti Junction Basic text slide CLA SMTP HUB CLA CLB CLB Scheduled Replication Mail Braces Belt
  • 21. 7: Spaghetti Junction Basic text slide CLA SMTP HUB CLA CLB CLB Scheduled Replication Mail Braces Belt 96 Connection Documents Servers replicate with all servers every 5 minutes Maintenance tasks running during office hours Transactional logs on same Raid array as Domino data No data quotas whatsoever All servers running multiple DEBUG parameters
  • 22. 7: Spaghetti Junction Resolution No mail routing connections were required It’s a Domino Named Network! Maintenance tasks changed and schedules modified Transaction logging moved to separate Raid Arrays Replication connection documents drastically reduced Logging reduced Multiple CPU’s utilised on servers Lessons learned Never be afraid of the manual Education! Education! Education! Understand the basics of Domino and it is an easy environment to manage Never implement a feature without researching it thoroughly For example Transactional logging and disk arrays Basic text slide
  • 23. 8: The Long Way Round The Story A new custom application. Failed to execute scheduled agent within the 15-minute limit. The Investigation Very large data collection exercise – thousands of documents to be scanned and reported on. A nightly task. Gotcha! The Developer was NOT using ANY views! The Cause The developer was unaware of views and was parsing ALL documents in ALL databases Re-reading documents from previous days. Basic text slide
  • 24. 8: The Long Way Round Resolution Application, when ran on workstation, against live data – took 9 HOURS. When re-written to actually use views (but still open each document in turn) – 2 hours. When re-written to use view information – executed within 15 minutes. Lessons learned “But it worked in testing” is not a valid excuse Testing data has to be representative at UAT (User Acceptance Testing) stage A proper Development/UAT/Production environment should be set up. Developer should learn more about Notes before being given task of this complexity. Basic text slide
  • 25. 9: Time Travel for Beginners The Story SMB Domino site Fully intranet based Domino Web Access/Custom applications Three-way Domino cluster W32 based OS on all servers Time zone issues on system default profile Common for non-US customers DMY comes up as MDY Local Admin runs some testing over weekend Data stops replicating to other servers The Investigation Replcation tasks and logs checked - Seems to be fine ACL’s checked - All correct Basic text slide
  • 26. 9: Time Travel for Beginners Gotcha! Replication history checked… Last successfully replicated at 10.48 am, 11/8/2057 on applications The Cause Administrator was baffled by Time/Date issue on server consoles Reset date to 2057 as a test Brought domino server back up on cluster LAN Went to lunch…… Server replicated When the date is returned to the present, Domino sees no need to replicate! Resolution Replication history has to be purged from all databases on server Deletion stubs must be purged from databases Basic text slide
  • 27. 9: Time Travel for Beginners Lessons learned Domino servers rely on the OS date configuration/settings Domino servers and replication architecture rely on replication history Never go forward in time on a domino server. In fact, this is true of all database systems where timestamps are used. This is not restricted to Domino! All time/date work on a Domino server must be done on a simulated environment Basic text slide
  • 28. 10: Lucifers' Lotusscript The Story A large application migration project from v4.5 to v6.5 Application server had just been migrated, Developer is informed that his agent has failed Developer contacts administrators and tells them to fix problem, without examining code. Gotcha! Agent used OLE calls to Oracle database, and no documentation existed to show external linkage. Mix of back-end and front-end code in scheduled agent No error handling Hardcoded Data link names, server names, database path names. Domino v6 “agent security” was set to “1 – do not allow restricted...” Resolution Recode agent. Basic text slide
  • 29. 10: Lucifers' Lotusscript Lessons learned Separate Development/UAT/Production environments Applications that depend on anything outside their own database should have installation documentation. The following should be mandatory in your development best practices: “Option Declare” “Errorhandling” Logging of critical errors to some monitored Log. Documentation DONT have 300-lines of code in “initialize”. Try and structure ? DONT mix front-end and back-end code in a scheduled agent. Time-sensitive events such as the web events should not have heavy duty code such as OLE links to Oracle embedded in them. You can write the best code in the world, but if your administrators cannot manage it, its wasted effort. Basic text slide
  • 30. 11: The email from hell The Story Domino server becomes very slow/almost unresponsive Users complaining All users eventually fail over to cluster server and continue working The Investigation Go to Server “Nrouter.exe” claiming 97% CPU constantly Show Tasks on console Router in “Dispatching messages” mode Gotcha! Open mail.box (or mail1.box, mail2.box, etc) on server Basic text slide
  • 31. 11: The email from hell Basic text slide , 995 Bytes , 397 , 145 2 !
  • 32. 11: The email from hell The Cause A sales rep had mailed the contents of his H:ser directory to HIMSELF Was moving desk Resolution Stop router task, Delete the mail from “mail.box” Reload router task, Compact “mail.box” - “tell router compact” Monitor mail routing and cluster task Small Point. Router would have delivered this email… Cluster would have then clustered this email - Inbox and sent view – almost 5GB Credit to a fantastic mail routing/cluster engine Lessons learned Limit the internal mail attachments Configuration document – set mail priority Mail Rules Customise mail template? Basic text slide
  • 33. 12: The Domino Virus The Story All servers in this large environment stopped mail routing and replication User access and administration access was fine. Investigation Mail was building up on main routing hubs, and on all servers. Gotcha! A Junior administrator had dropped “LocalDomainServers” into the “Terminations” group. As the directory replicated, it would successfully update other servers, then disallow server to server access. Resolution Administrators had to edit the “Terminations” group in each and every server in order to allow server to server communication. Basic text slide
  • 34. 12: The Domino Virus Lessons learned Inexperienced administrators should NOT be allowed open access to directories in large environments. Education! Education! Education! PHB’s think that ANYONE can be trained to do ANY job. Sometimes, someone with NO computer experience whatsoever cannot be trained to be a Lotus Domino administrator in 4 weeks… Basic text slide
  • 35. Shorties When you have more than one Notes domain, try and make sure the domain names are unique… Dont allow developers to sign agents in production environments. Especially if they are contractors on short term assignments. Lotus notes Name fields used in messaging ONLY support ASCII characters. Really! Use “Alternate names” for non-ASCII names, and using Ampersands (“&”) in names causes pain. Notes domain names shouldn’t really have dots in them. Certificate/Server names should not have spaces Beta software in a live domain is not always a good idea Never leave the client up on a server. Don’t switch id on a server. Ever. Be careful what ports you stop on the server -Default- = Manager = Bad Idea! Allow Anonymous Notes connections = Bad idea Enable the cleanup agent on the Domlog.nsf file Dont cluster over 64k/bit links...
  • 36. What have we done here? We’ve passed on a few experiences Learned a few lessons on what not to do Felt a bit better about any mistakes you may have made Remember. Mistakes will happen. No-one is perfect. Its how you deal with them that defines your professionalism. Never make sweeping changes under time pressure. You'll just dig a deeper hole... “Measure Twice, Cut Once” Time for some questions ? Remember – we will listen to your confessions in the speaker room… Basic text slide
  • 37. Thank you Paul Mooney www.pmooney.net Blue Wave Technology www.bluewave.ie Bill Buchan www.billbuchan.com Hadsl www.hadsl.com Basic text slide

Editor's Notes

  1. Welcome by Bill and Paul Good morning and welcome to the graveyard shift presentation… My name is Bill Buchan – I am a Notes developer from Scotland And my name is Paul Mooney – a Notes administrator from Ireland. As you can see we are both quite short, so we hope you can see us Also where we are from, we tend to speak quite quickly, so some volunteers at the front will let us know if we speak too fast.