The LotusScript to C-API Testing Harness
Category None
Bookmark :
After Tuesday's spectacular success in the USA, I have decided (as CodeWeavers did last week) to give something back to the community.
If you have ever tried to call Notes C-API functions from LotusScript, or had to do an 'Option Declare', then read on. This might just make your day!
I have (in collusion with some others) have developed a methodology which makes it very simple to define and use Notes C-API calls without recourse to Option Declare and the lottery of defining the C-API function signatures. It makes it very easy to actually support new platforms (such as the new Windows 64 platform) without having to bother with all that mucking around in option declare.
You can download this database from here, and follows the about database document...
More after the break...
Bookmark :
After Tuesday's spectacular success in the USA, I have decided (as CodeWeavers did last week) to give something back to the community.
If you have ever tried to call Notes C-API functions from LotusScript, or had to do an 'Option Declare', then read on. This might just make your day!
I have (in collusion with some others) have developed a methodology which makes it very simple to define and use Notes C-API calls without recourse to Option Declare and the lottery of defining the C-API function signatures. It makes it very easy to actually support new platforms (such as the new Windows 64 platform) without having to bother with all that mucking around in option declare.
You can download this database from here, and follows the about database document...
More after the break...
The LotusScript to C-API Testing Harness.
Introduction
The basic problem is taking a definition such as:
Database Profile
64 bit platforms
Whats all these views for?
Can I use this ?
No really, whats the licensing ?
Current Issues.
- Welcome to the LotusScript to C-API test harness.
- Daniel Nashed - a small-ish, perfectly formed German with a brain far larger than his skull. He's forgotten more C-API stuff than most people will ever know. Including, I suspect, folks within IRIS. Without Daniels patient work with Lotus, this library would not exist. Thank you Daniel
- Julian Robichaux. Julian is an unassuming, soft spoken southern gentleman from the United States, and always seems to come up with really cool stuff. I dont knw how he does it. He kinda beta tested this, improved it greatly, and said nice things about it. Which, as we know as developers, is what we live for. Julian, name your Tapas bar. The finger food is on me!
This database is for people familar with the pain and misery of trying to make Notes C-API calls from LotusScript. In multiple platforms, languages. In 32 bit and in 64 bit. People who enjoy misery. (See the explicit note on 64-bit below)
Why ? In just about no other type of programming, can a simple mistake just blow up your workstation. Or server. Or both. Randomly. Its the ultimate white knuckle ride.
What on earth are we doing ? We're using the 'declare function' or 'declare sub' calls to map a lotusscript call directly into a DLL, or a Unix Library. Or an abyss.
Lotusscript itself does a fantastic job of hiding us from really horrible platform differences. Character set, little or big endian, integer widths and so forth. Notes C-API strips away all that nanny-state cotton wool and replaces it with a cocktail of razor blades and napalm.
Till now, we relied on the excellent http://www.ls2capi.com web site, home on the web of the book: 'LotusScript to Lotus C API Programming Guide', by the exceptionally talented and patient Normunds Kalnberzins. He literally wrote the book on how to achieve this goal (and if this is your first exposure to this book - buy it now!).
Remember - we all stand on the shoulders of giants - its how we progress in a collaborative universe. And its good to acknowlege some other amazing people who have helped me over the years.
Introduction
The basic problem is taking a definition such as:
- STATUS LNPUBLIC NSFGetServerLatency(
char far *ServerName,
DWORD Timeout,
DWORD far *retClientToServerMS,
DWORD far *retServerToClientMS,
WORD far *ServerVersion);
- and then writing code to deal with the plaform differences.
- Declare Function W32_NSFGetServerLatency Lib LIB_W32 Alias {NSFGetServerLatency} (_
Byval ServerName As Lmbcs String, _
Byval Timeout As Long, _
retClientToServerMS As Long, _
retServerToClientMS As Long, _
ServerVersion As Integer) As Integer
Declare Function AIX_NSFGetServerLatency Lib LIB_AIX Alias {NSFGetServerLatency} (_
Byval ServerName As Lmbcs String, _
Byval Timeout As Long, _
retClientToServerMS As Long, _
retServerToClientMS As Long, _
ServerVersion As Integer) As Integer
Declare Function TUX_NSFGetServerLatency Lib LIB_TUX Alias {NSFGetServerLatency} (_
Byval ServerName As Lmbcs String, _
Byval Timeout As Long, _
retClientToServerMS As Long, _
retServerToClientMS As Long, _
ServerVersion As Integer) As Integer
Declare Function SUN_NSFGetServerLatency Lib LIB_SUN Alias {NSFGetServerLatency} (_
Byval ServerName As Lmbcs String, _
Byval Timeout As Long, _
retClientToServerMS As Long, _
retServerToClientMS As Long, _
ServerVersion As Integer) As Integer
I have decided to write some code (and give it out as freeware) to help solve this issue. Instead of having to write the option declare code for every function you wish to use (in the 700+ available in the Notes C-API reference guide) and then write different versions of this for different platforms, I thought - 'why not write some code to figure out the platform differences'.
And so the ls2capiWrapper class was born.
What we can now do is create a class - in this database, I created one called APICalls:
- Option Public
Option Declare
Use "class:ls2capiWrapperClass"
' ----------- This is where you put your API code -----------------
Class APICalls As ls2capiWrapperClass
' Get the server latency to the listed server in milliseconds. Note that
' we add the outgoing and return latencies together.
Public Function getServerLatency(Byval serverName As String) As Long
On Error Goto errorhandler
%REM
' This is how its declared in the C-API reference
STATUS LNPUBLIC NSFGetServerLatency(
char far *ServerName,
DWORD Timeout,
DWORD far *retClientToServerMS,
DWORD far *retServerToClientMS,
WORD far *ServerVersion);
%END REM
' how the caller passed in the server name.
Dim nnServer As New NotesName(serverName)
' Now build up each of our parameters as an API Parameter instance
' type, ByVal, variable name, value
Dim p1 As New ls2capiParameterClass(cParent, type_string, True, "serverName", nnServer.Canonical, "")
Dim p2 As New ls2capiParameterClass(cParent, type_DWORD, True, "TimeOut", 1000, "")
Dim p3 As New ls2capiParameterClass(cParent, type_DWORD, False, "retClientToServerMS", 0, "")
Dim p4 As New ls2capiParameterClass(cParent, type_DWORD, False, "retServerToClientMS", 0, "")
Dim p5 As New ls2capiParameterClass(cParent, type_WORD, False, "serverVersion", 0, "")
' Now bind the parameters into an array
Redim params(4) As ls2capiParameterClass
Set params(0) = p1
Set params(1) = p2
Set params(2) = p3
Set params(3) = p4
Set params(4) = p5
' Define our return type. As its going to be a STATUS value, then it'll
' autmatically convert our run-time error (if any) based on errorvalue.
Dim returnVal As New ls2capiParameterClass(cParent, type_status, True, "return", 0, "")
' Now define the call itself, passing in the name (as defined in C-API, the parameter collection,
' and the return value.
Dim A As New ls2capiCallClass(cParent, "NSFGetServerLatency", params, returnval)
' Now. Scary stuff. Actually call the API.
If Me.callAPI(A) Then
' Now get our results
If (returnVal.getValue() = 0) Then
Call Me.logVerboseMsg("APICallClass::getServerLatency: Got latency of: " + Cstr(p3.getValue() + p4.getValue()) )
getServerLatency = p3.getValue() + p4.getValue()
End If
Else
' Optimistic Basically if the call fails, then the process crashes.
Call Me.logErrorMsg("APICallClass::getServerLatency: Failed to return from the CallAPI function" )
End If
- exitFunction:
- errorhandler:
Resume exitFunction
End Function
end class
- As you can see, I've provided a very simple way of passing the type of parameter (WORD, DWORD, DHANDLE, STRING, etc) and the return value. In that manner, the underlying code can then write code which will deal with platform differences, and call it for you.
- Firstly on your local workstation, possibly with a debugger. Once your happy (and it doesnt blow up your workstation!),
- On the server that this database is hosted on, via a 'Tell Amgr Run' command. Beware, if this fails, it WILL blow up this server. So make sure no-one else is using it, as once your finished, no-one else WILL be using it. Once your happy with that. then
- Run on all servers. This replicates this instance of the database with others lised in the 'servers' field on the database profile, then executes the function, and replicates back with the primary instance.
Here's another example where were using a 'Type' - in this case, TIMEDATE:
- ' This is an example of using a complex type-based function. This function calls OSCurrentTIMEDATE
' which returns a TIMEDATE structure.
Public Function OSCurrentTIMEDATE (thistd As TimeDate) As Boolean
On Error Goto errorhandler
- %REM
void LNPUBLIC OSCurrentTIMEDATE(
TIMEDATE far *retTimeDate);
%END REM
' Remember that our API can only 'see' globally scoped items declared in WrapperClass - so copy my type
' into that object
global_TIMEDATE.innards(0) = thistd.innards(0)
global_TIMEDATE.innards(1) = thistd.innards(1)
' define our parameter list for this type declaration. In this case, I'm copying the values from a local version
' of this type called 'td'.
' thisType, isByVal, thisName, thisVal As Variant)
Dim p1 As New ls2capiParameterClass(cParent, type_DWORD, False, "innards1", 0, "global_TIMEDATE.innards(0)")
Dim p2 As New ls2capiParameterClass(cParent, type_DWORD, False, "innards2", 0, "global_TIMEDATE.innards(1)")
' Bundle them into an array
Dim params(1) As ls2capiParameterClass
Set params(0) = p1
Set params(1) = p2
' define our type from the parameter list, and call the type TIMEDATE
Dim t1 As New ls2capiTypeClass("TIMEDATE", params)
' Now define our parameter list for our function call. In this context I'm going to call the generated
' copy of the type (in our generated code..) 'myTimeDate'. I then pass it my type declaration:
' thisType, isByVal, thisName, thisVal
Dim p3 As New ls2capiParameterClass(cParent, type_CUSTOM, False, "myTimeDate", t1, "")
' Bundle the parameters into an array
Dim params2(0) As ls2capiParameterClass
Set params2(0) = p3
' define our actual API call, using the function parameters
Dim A As New ls2capiCallClass(cParent, "OSCurrentTIMEDATE", params2, Nothing)
' REMEMBER TO ADD THIS TYPE TO THE API CALL, else it wont initialse, declare or copy back information.
Call A.addType(t1)
If Me.callAPI(A) Then
OSCurrentTIMEDATE = True
thistd.innards(0) = global_TIMEDATE.innards(0)
thistd.innards(1) = global_TIMEDATE.innards(1)
End If
exitFunction:
Exit Function
errorhandler:
Call Me.raiseError()
Resume exitFunction
End Function
Types are not pretty. You have to add global variable versions of the type to the ls2capiWrapper class (so that the EXECUTed code can find it). But it does work, and we do copy back and forth the values correctly.
This database encapsulates a number of already-written calls in the APICalls Script library. And as an aid to testing, an Agent: 'Test All API Calls' calls a class 'APICallsTest' which then exercises each function.
And because we're in control of the function, it means we can log and debug to our hearts content, till we get the API calls we need up and running on the platforms we need. As you can see, there are a number of agents which make it simple to test:
This keeps track of which calls are being made in which platform.
Finally, once your happy, just copy the ls2capi.. libraries and the APICalls library to your targetDB and integrate the code.
- Windows/32 (Notes Client and Domino Server)
- Windows/64 with Hotfix 710
- Solaris (Domino Server)
- AIX (Domino Server 32 bit)
- Linux (Domino Server 32 bit)
- Macintosh Client (Notes v8 Intel based)
- Linux 64
- Linux (Notes Client 32-bit)
- AIX 64
- iSeries
- zSeries (thought I believe its the same as AIX/32)
Supported Platforms:
Platforms which I'd like to get supported
If you can help - if you have test hardware in any of these platforms (especially iSeries and zSeries), I'd like to hear from you. If you need support from a platform which is not listed here - get in touch! By "Supported" I mean - I've got this running on the platform, and it doesnt crash out.
- As you can see from above, most of the interesting stuff happens around the ls2capiParameterClass. We use it to declare our type members, and our function/sub parameters.
Here's an example:
- ' thisType, isByVal, thisName, thisVal global
Dim p1 As New ls2capiParameterClass(cParent, type_DWORD, False, "innards1", 0, "global_TIMEDATE.innards(0)")
Dim p2 As New ls2capiParameterClass(cParent, type_DWORD, False, "innards2", 0, "global_TIMEDATE.innards(1)")
I'm using the Factory pattern in my OO code, and the first parameter - cParent in this example, is a pointer to the factory object. Note that this code doesnt actually use this (my production code which I've developed this for DOES) cParent/Factory thing at all. So you can pass in 'nothing' if you like. Or you can use it to pass around your factory object.
The second parameter is the type, one of:
- Public Const type_Status = 0 Status is our 'STATUS' value. Most API functions use this
If I spot you declaring your return type as this, I'll automatically decode
your return value and give you the error string in the return parameter.
Public Const type_dHandle = 2 A domino handle, such as a database, note, or view handle.
Public Const type_String = 3 A String
Public Const type_WORD = 4 A WORD - usually an integer
Public Const type_DWORD = 5 A Double-Word - usually a lONG
Public Const type_BOOL = 6 A Boolean.
Public Const type_custom = 10 A type defintion, such as the second example above
The third parameter is whether this is passed as a reference or a value. A rough rule of thumb is unless the C-API documentation (in the Notes C-API reference database) says that the value is returned from Notes, then this should be 'true' for 'pass it as a value'. False means that we'll get something back (perhaps).
The fourth parameter is the name that you want to know it by in the computed function. I usually put the parameter name from the C-API documentation.
The fifth parameter is the value you wish to assign to this variable when its first created. This is a variant field so pass anything you like.
And the last is the 'passthrough global variable name'. For types, as we have in the second example, linking the function parameter with the name of the type we've defined. If you have defined this as anything other than a blank string, then the value parameter is ignored.
Now step through an example with the debugger and look at some log output.
Database Profile
- Choose your logging in the database profile. And select a list of servers that already have replica copies of this database, which you want to test on. On very buggy functions, I wholeheartedly recommend turning on the file-based logging, as that will pretty much survive a nuclear-winter of server crashes.
64 bit platforms
- A word of warning on 64 bit platforms. They are very very new. In testing Domino 8.0.1 64-bit, we found that ls2capi calls crashed the server. Dead. Every time. No kidding. Lotus Westford has analysed the problem and produced a hotfix - hotfix 701 - that deals with this issue.
At the time of writing, we successfully ran all tests on Windows 64 bit with Domino 801 and hotfix 710. We have not yet got a hotfix for domino 802 (aside from confirming that we can crash it! and Domino 8.5 (beta 12).
In other words:
Be very very careful of using this code on 64-bit plaforms. Its up to YOU to test this, and up to YOU to inform your customers of any hotfix requirements. Until we have a release of Domino 8.5, we do not yet know if it requires a hotfix.
If you crash your server, dont say we didnt warn you.
Whats all these views for?
This database is a test harness.
Critical Area by Status: Every time I run a piece of C-API, I create a document. If the C-API doesnt complete, the document has status 'CRASHED' which helps you narrow down which functions stlil need work.
Log: Each time the agent runs, I keep all log documents. This helps you debug crashes and figure out whats going on.
Test Documents: One of the tests means I have to create a new test document each run. They're kept in this view so the test code can find them.
All of these documents can be regularly deleted. I suggest you clear em all at the start of a major testing run. If you find anything really strange, sending me the relevant log document saves a lot of your time.
- A word of warning. This is difficult, horrible, tedious frustrating error prone work that WILL take far more time than you think, and WILL come back and bite you in the .. leg .. at some point. I reserve the right to say 'Look, this is beyond you' if you contact me. Please dont take this personally - this was beyond me for the first 12 YEARS of my Lotus programming career. This is black-belt stuff, and unless you've been sufficiently beaten up in the past, your not going to enjoy this experience. Seriously. Dont feel I'm trying to put you off here. I *AM* putting you off.
If you dont understand C-API programming, or didnt understand the ls2capi book, or dont understand whats going on when you step through with the debugger - theres no shame in that. Its horrible stuff. Just dont ask me to fix your issues with it unless you're waving money at me.
I'm Bill Buchan, of HADSL. We develop enterprise strength Lotus Notes, AD and Blackberry User provisioning management tools. And this is why I need this functionality. I blog at http://www.billbuchan.com.
Can I use this ?
- Sure. Dont try and pass it off as your own work however, as I'll find out and laugh at you. (I'm quite a horrible person and enjoy this sort of thing) Feel free to tell me where I've broken things.
I dont really have time for 'How does this work', 'Whats Notes C-API' or 'My function doesnt work'. I'll have pretty much assumed by the time you've contacted me that you've read the source, tried a bunch of things, and paid Normunds for a copy of his book. And read it. Sorry if this sounds really hard, but its how things are.
If you do contact me, dont expect an immediate reply, unless your paying me for that service. Because I'm incredibly busy at about a million other things. Sorry. I'm based in Scotland (which is gods country, really), so pinging me a mail outside normal office hours probably means next day. I'm not superhuman. Hell, some days, I'm barely human.
If you try and get in contact with me over this and have clearly not read this or the code, then I reserve the right to not reply for a very very long time.
If any of the language in this note offends you, for Gods sake, dont look at the code. You'll probably explode. Swearing well is the sign of an old programmer, and right now I feel about a thousand years old.
No really, whats the licensing ?
- At this point, its fluid.
- If your going to build it into a product, then you need to talk to me. This doesnt mean that its going to be incredibly expensive, but I do reserve the right to say 'No', or charge you money. If you need me to support you, then its going to have to cost something. If you need me to do something with this for you, get in touch.
- If your building it into a internal corporate database, then ping me to say 'hi' and buy me a beer at Lotusphere. Or two. ESPN on Saturday night. Seriously.
Current Issues.
- v1.0 - First release.
- Works on W64 with hotfix 710. Many thanks, Daniel!
- Lots and Lots of functions added. 3,000 lines in APICalls. The ID password recovery stuff works.
- v0.91. Still very much pre-release
- Mac platform for client is working. I suspect this will ONLY ever work on intel-macs, but lets leave it at that. (endian-mapping).
- v 0.90 - pre-release
- It vaguely works, but does NOT work on any 64 bit platform yet. Windows, AIX, Linux. We dont know why, and we're expending a lot of time trying to figure it out. It looks like ALL 64 bit function calls fail.
- I've tested this on:
- Windows/32 (server and client)
- Solaris
- AIX (32 bit)
- Linux (32 bit)
- Its failed on
- Windows/64 - windows 2003 / 64 bit edition, domino 8.5 beta 11 (64 bit), as well as Windows 2008/64 and Domino 8.5 beta 11 (64 bit).
Comments
Posted by Ben Langhinrichs At 15:25:03 On 06/11/2008 | - Website - |
---* Bill
Posted by Bill At 15:26:38 On 06/11/2008 | - Website - |
Amazig job on the tool and contributing it BTW!
Posted by Ben Poole At 16:09:57 On 06/11/2008 | - Website - |
Posted by Ben Langhinrichs At 16:40:03 On 06/11/2008 | - Website - |
Posted by Finn Knudsen At 22:00:22 On 06/11/2008 | - Website - |
---* Bill
Posted by Bill At 08:39:17 On 07/11/2008 | - Website - |
This is fried gold - thanks for sharing it
Posted by Matt Buchanan At 14:02:21 On 10/11/2008 | - Website - |
--* Bill
Posted by Bill At 10:25:29 On 11/11/2008 | - Website - |
WRT to wanting to know more about referencing the API on iSeries my basic advice is 'Don't bother'. Due to memory alignment issues even simple API calls such as opening a Db and closing a Db don't work as you would expect (if they work at all).
Normunds did some really cool work on a set of wrapper classes for iSeries but even they are an incomplete set and most companies would think twice before allowing critical apps to be reliant upon unsupported software (or you'd hope they would anyway).
Perhaps you brainboxes at HADSL can provide a full API wrapper set for iSeries and sell it. Not being a big 'C' programmer I'll leave it to the people who are (and who probably care more)
Again - great piece of work this and thanks for sharing it. 'Big Up' to yourself and the other folks who worked on it.
TD.
Posted by Tom At 12:08:36 On 14/11/2008 | - Website - |
---* Bill
Posted by Bill At 09:19:57 On 01/12/2008 | - Website - |
Everytime when I Call
OSPathNetConstruct(0, db.Server, db.FilePath, pathName)
and pass pathName to
NSFDbOpen(pathName, hDb)
It always told me that I can't open the database ... I have no idea about how to debug or solve it. Can you give me some advise or suggestion ?
Posted by Phosphor At 14:21:13 On 30/12/2008 | - Website - |
Usually all osPathNetConstruct does is concatonate server and filepath with "!!" as a separator and pass it back.
Check out the C-API progamming guide reference, and check out the section on platform differences.
Or cheat. The 'handle' is an undocumented property from notesDatabase. So use lotusscript to open the database, get a handle... ?
---* Bill
Posted by Bill At 17:16:37 On 26/01/2009 | - Website - |