Universal Agent Profiling in Domino

[Note: I've had this blog entry in draft for ages - sorry to take so long to get this out. Yes, this is the one I mentioned I'd do from the Worst Practices stage. And yes, confusingly I published it without the code. D'oh!]

We've all been there. We've all had a server suddenly and unexpectedly get very slow, odd things happening and so forth. And we've all wanted to know - which agent is causing issues ? Which agents - out of all these databases - are working ? 

Scary. And its a question that can be easily answered.

Firstly, if you have money and want to buy product - go see TeamStudio Profiler. Its a fantastic tool that actually injects code into your lotusscript agents on the fly, and then uses these markers to measure exactly how your agents work. You can actually see how often any particular function is called, and for how long. 

Lets assume that you dont have this, as you have already solved this issue if you have Profiler.

Lets assume that your servers and apps are running Domino 6.x 7.x or above. In v6 v7, we have a fantastic new 'profile this agent' utility. And of course, like all good developers, you've never read the release notes. And like all good admins, you never read developer release notes.

Stick with me folks, this is good.

So - open up an agent, and open agent properties. Open the second tab - you've never done that, have you - and click on 'Profile this agent'. Easy, wasnt it ?

Or if your a DDE chappie, then: 


What does this do? Well, this tells agent manager that every time this agent runs, create a profile document in this database specific to this agent and then list all the calls that this agent made to the notes infrastructure. So you can see how long your agent took, and how long all the calls to the Notes infrastructure took. For instance (and this is a particularly complex agent)

Go off and play with this *right now*. Seriously. 

Back again? Now. You can see how cool this is, right. Hows about being able to switch this on for *all* agents ? Well, thats easy. Its just a flag. That is, a simple notes field on the agent design document itself. You can just open up all the agents in Lotuscscript, and set the flag, right ? In particular, you pick up the field $FlagsExt and add or remove a capital-F to the text string (if it exists).

And some code that might switch this on and off looks like this:

Sub Initialize
 Dim sSession As New NotesSession
 Dim ui As New NotesUIWorkspace
 Dim dbthis As notesDatabase
 Set dbThis = sSession.currentDatabase
 ' prompt the user for a target database
 Dim dbPath As Variant
 dbPath = ui.Prompt(13, "Target Database", "Select a target Database")
 Dim dbTarget As notesDatabase
 Set dbTarget = sSession.getDatabase(dbPath(0), dbPath(1))
 If (dbTarget Is Nothing) Then Exit Sub
 If Not dbTarget.IsOpen Then Exit Sub
 Dim ret As Integer
 ret = ui.prompt(2, "Continue", "Do you want to enable Agent Profiling on all agents in database: " + dbTarget.Title)
 If (ret <> 1) Then
  Print "Cancelled"
  Exit Sub
 End If
 Dim nn As notesNoteCollection
 Set nn = dbTarget.CreateNoteCollection(False)
 nn.SelectAgents = True
 Call nn.BuildCollection()
 Dim noteiD As String
 noteID = nn.getFirstNoteID
 While noteID <> "" 
  Dim docAgent As NotesDocument
  Set docAgent = dbTarget.GetDocumentByID(noteID)
  If Not docAgent Is Nothing Then
   Dim itm As NotesItem, dirty As Boolean
   dirty = False
   Set itm =docAgent.GetFirstItem("$FlagsExt")
   If (itm Is Nothing) Then
    dirty = True
    Set itm = docAgent.replaceItemValue("$FlagsExt", "F")
   End If
   If (Instr(itm.Text, "F") < 1) Then
    dirty = True
    Set itm = docAgent.replaceItemValue("$FlagsExt", itm.text + "F") 
   End If
   If (dirty) Then
    itm.IsSigned = True
    Print "Saving agent: " + docAgent.getitemvalue("$Title")(0)
    Call docAgent.Sign()
    Call docAgent.Save(False, False)
   End If
  End If
  noteID = nn.GetNextNoteId(noteID)
End Sub

Okay. If this was a stretch for you, seriously consider not using this stuff.  Secondly, remember, that this will open, update and sign all your agents using the current user ID. This may not be a good thing in your environment. Again, if you don't understand what I'm blathering on about, DONT DO IT. You are on your own.

We now have one or more agents enabled for profiling. All is well. And we can see the status of the last run of the agent itself. And get a nice profile report. So far, so very-out-of-the-box. Hows about collecting ALL the agent runs in another database so you can see how it works over time ? 

Simple. A rather smart chap called Damien Katz developed something called Trigger Happy a while ago. Damien, as you recall, was the genius who managed to completely refactor the @formula engine in v6. This chap doesn't stick his tounge out when he types, if you know what I mean. 

Trigger Happy is an extension manager. That is, it hooks in part of a DLL (on windows servers) so that anytime anything exciting happens on a domino server, Agent Trigger can run. And you can tell it to look for particular events. In this case, I'm asking it to run anytime ANY document is saved on the server. This is very dangerous - my code could easily cause the server to go bananas. Tread carefully here. We're literally standing on a office chair, watering the plants on our 40th storey balcony.

This code would trap every single document save on the server, and run this lotusscript agent, passing it the open document. Can you see how dangerous this is ?

What would this code look like?

Sub Initialize
 Dim session As New NotesSession
 On Error Goto errorhandler
 If (session.DocumentContext.Form(0) <> "$BEProfileR7") Then Exit Sub
 Dim d As NotesDocument, dbSrc As NotesDatabase
 Set d = session.documentContext
 If (d Is Nothing) Then Exit Sub
 Dim db As NotesDatabase
 Set db = session.CurrentDatabase
 Dim doc As NotesDocument
 Set doc = db.CreateDocument
 If (doc Is Nothing) Then Exit Sub
 Call session.DocumentContext.CopyAllItems(doc, True)
 Call doc.ReplaceItemValue("Form", "Agent Performance Profile")
 Dim rt As notesRichTextItem
 Set rt = doc.GetFirstItem("Body")
 Dim T As Variant
 T = Split(rt.GetFormattedText(True, 132), Chr(10))
 If (Ubound(T) >0) Then Call doc.replaceitemValue("Time.Elapsed",    Clng(Strleft(Trim(Strrightback(T(1), ":")), " ")))
 If (Ubound(T) >1) Then Call doc.replaceitemValue("Methods",     Clng(Strrightback(T(2), ":")))
 If (Ubound(T) >2) Then Call doc.replaceitemValue("Time.Measured",   Clng(Strleft(Trim(Strrightback(T(3), ":")), " ")))
 Call doc.replaceitemValue("Text",  T)
 Dim ts As New TriggerSession
 Call doc.ReplaceItemValue("UserName", ts.username)
 Set dbSrc = d.ParentDatabase
 If Not (dbSrc Is Nothing) Then 
  If (dbSrc.IsOpen) Then 
   Call doc.replaceitemValue("db.Server",  dbSrc.server)
   Call doc.replaceitemValue("db.Title",  dbSrc.Title)
   Call doc.replaceitemValue("db.Path",  dbSrc.FilePath)
  End If
 End If
 Call doc.save(False, False)
 Print |Saving Agent Performance Profile for user: | & ts.UserName & |, agent: | & d.subject(0)
 Exit Sub
 Print  "Copy Agent Performance Logs: Run-time error: "+ Error$ + " at line: "+ Cstr(Erl)
 Resume exitFunction
End Sub

Note that we're hinging this on the fact that the document profile document that is being saved, is being saved with a form-name of "$BEProfileR7". If we find a document (and remember, it could be ANY document in ANY database on this server), we're saving a copy in this current database (where this agent is lodged).

Now for the bad news. I'm not going to tell you how to set this up in a step-by-step fashion. You have enough clues. Its important that you understand both the risks and rewards for all the steps. If you don't understand this, or if you cant get it working - I cant help you. And honestly, this is running-with-scissors++. I'd rather you cursed me for not spoon feeding, than curse me for killing your servers.

So there you have it. Less than 40 lines of code, and all of a sudden you have a very important, free, somewhat dangerous,  testing and measurement tool. 

Many thanks to Damien for doing the difficult stuff.