So the story is that we upgraded hardware over the weekend. Prominic as usual had done a fantastic job, but I had tried something previously unheard of.
Now. Most of the mail files use the Domino Web Access (the new version of iNotes, if you like) template, based on a v7.0.2 build. DWA/iNotes is a rather curious design, in that most of the actual code is held in a set of shared templates buried in the "iNotes" directory. So your mail template makes lots of calls to this underlying infrasructure. Benefits all round, as only one copy of the code is cached in memory, making it faster and of course the design code in the mailfile smaller.
Obviously there's a high degree of interconnectedness between these two things - they're tightly bound, if you like. The mailfile code calls the iNotes code.
Now, given Domino's fantastic BACKWARD compatibility, you can easily have the server running a later version of the code. You can have the server at 7.0.2, for instance, and still be running DWA code from 7.0. And people do this all the time. It works.
However, if you move hardware, and install an OLDER version of Domino - as I did, then you have DWA code (v7.0.2) calling iNotes code from v7.0.0. Guess what. It doesnt work. Its unsupported. Its very stupid. And its a fantastic way of causing server instability.
Yup. I installed an older version of the domino code. And so the server would crash every time someone opened a mail file via the web client. Its easy to diagnose - just type in your web password, and watch the server die. The HTTP process is highlighted in the NSD as the crashing process.
So why did this happen ? When I do this for customers, I of course construct huge checklists, and have two people checking all the time. For my own stuff - well. You get the picture. The phrase "The cobblers children always has the worst shoes", right.
Of course, Murphy's law dictates that this is in a very intense period of time - huge code deliverables, testing, and the run up for Lotusphere. It wouldnt otherwise happen, would it ?
Whats your worst story ? Submit it to TheWorstPractices.com...