Usability - Productivity - Business - The web - Singapore & Twins

By Date: January 2014

GMail2Notes in less than 300 lines

I admit, the headline is an attention grabber
The total number of lines is, of cause, longer. For one, I used previous code, as well e.printStackTrace() isn't error handling.
Moving data between two places is a favorite IT pasttime (believe me, there are worse ways to kill time). Since I made friends with MIME Messages before, I was wondering how that would fit together with gMail. Every eMail system has its little "specialties". To connect to gMail I used the IMAP with username and password route (which is the only one I know of for IMAP).
Most of the Googles point to " reading the inbox" which is a distinct different task from: " pulling out all your eMail". When connecting to gMail using the enhanced IMAP classes you can pull all "Folders" using store.getDefaultFolder().list(). There a particular folder is interesting: [Gmail]. In Notes speak I would say: it contains the system views, most notably "All documents", "Sent" and "Spam".
To capture all documents going after that one view is sufficient. Google abandoned the idea of "folders" and replaced them with "labels". The idea there was: a document can be in one folder only, but can have many different labels. For other email systems that might hold true, but the way folders a implemented in Notes nicely fits with the idea of labels.
Each message comes with a getLabels() function that directly maps (with a little attention) to a NotesDocument.putInFolder(label) call. The code isn't perfect, but can be a good starting point for your own adventure.

Read more

Posted by on 14 January 2014 | Comments (1) | categories: IBM Notes

Using a web service to send an eMail

In an integration project with other platforms the need arose to send messages that are triggered by a web service. In Domino this is just a few lines of code. Added bonus: the message is fully "embedded experiences" enabled. This is the LotusScript class:
Public Class SendADocumentWithPDFAttachment

    Public sendTo As String
    Public subject As String
    Public textBody As String
    Public htmlBody As String
    Public callBackURL As String
    Public attachmentAsMime As String

    Public Function sendMessage(message As SendADocumentWithPDFAttachment) As String
        Dim s As New NotesSession
        Dim db As NotesDatabase
        Dim doc As NotesDocument
        Dim header As NotesMIMEHeader
        Dim body As NotesMIMEEntity

        On Error Goto Err_sendMessage

        'We are dealing with a MINME message here!
        s.Convertmime = False
        Set db = s.Currentdatabase
        Set doc = db.Createdocument()
        Call doc.Replaceitemvalue("Form", "Memo")
        Call doc.Replaceitemvalue("From", s.Username)
        Call doc.Replaceitemvalue("Subject", message.subject)
        Call doc.Replaceitemvalue("SendTo", message.sendTo)
        'Add other fields as you deem fit

        'Prepare the message body
        Set body = doc.Createmimeentity("Body")

        'Adding the content
        If message.textBody <> "" Then
            Call Me.addContent(s, body, message.textBody,"text/plain;charset=UTF-8")
        End If  
        If message.htmlBody <> "" Then
            Call Me.addContent(s, body, message.htmlBody,"text/html;charset=UTF-8")
        End If
        If message.callBackURL <> "" Then
            Call Me.addContent(s, body,|{ "url"  : "|+message.callBackURL+|" }|,"application/embed+json;charset=UTF-8")
        End If
        Call doc.Save(True, True)

        'Sending has 2 options: use the send() function or copy the document
        'into the mail.box. We use send here
        Call doc.send(False)
        s.Convertmime = True

        sendMessage = "Message submitted"
        Exit Function

        sendMessage = Error$
        Resume Exit_sendMessage

    End Function

    Private Sub addContent(session As NotesSession, body As NotesMIMEEntity, content As String, contentType As String)
        Dim mime As NotesMIMEEntity
        Dim contentStream As NotesStream
        On Error Goto Err_addContent
        Set contentStream = session.Createstream()
        Set mime = body.Createchildentity()
        Call contentStream.Writetext(content)
        contentStream.Position = 0
        Call mime.setcontentFromText(contentStream,contentType,ENC_BASE64)
        Exit Sub

        Print Error$
        Resume Exit_addContent
    End Sub
End Class
Once you add that to a Domino web service you get the following web service definition:
<?xml version="1.0" encoding="UTF-8"?>
<definitions targetNamespace="urn:DefaultNamespace"
  <schema targetNamespace="urn:DefaultNamespace" xmlns="http://www.w3.org/2001/XMLSchema">
   <complexType name="SendADocumentWithPDFAttachment">
   <element name="MESSAGE" type="impl:SendADocumentWithPDFAttachment"/>
   <element name="SENDMESSAGEReturn" type="xsd:string"/>
 <message name="SENDMESSAGERequest">
  <part element="impl:MESSAGE" name="MESSAGE"/>
 <message name="SENDMESSAGEResponse">
  <part element="impl:SENDMESSAGEReturn" name="SENDMESSAGEReturn"/>
 <portType name="SendADocument">
  <operation name="SENDMESSAGE">
   <input message="impl:SENDMESSAGERequest" name="SENDMESSAGERequest"/>
   <output message="impl:SENDMESSAGEResponse" name="SENDMESSAGEResponse"/>
 <binding name="DominoSoapBinding" type="impl:SendADocument">
  <wsdlsoap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
  <operation name="SENDMESSAGE">
   <wsdlsoap:operation soapAction="SENDMESSAGE"/>
   <input name="SENDMESSAGERequest">
    <wsdlsoap:body use="literal"/>
   <output name="SENDMESSAGEResponse">
    <wsdlsoap:body use="literal"/>
 <service name="SendADocumentService">
  <port binding="impl:DominoSoapBinding" name="Domino">
   <wsdlsoap:address location="http://localhost"/>

To use that service the web service client needs to Base64 encode the PDF attachment
As usual YMMV

Posted by on 06 January 2014 | Comments (0) | categories: IBM Notes

A short history of directory trees

I would like, if I may, to take you on a strange journey: Where did directory trees come from?
This isn't about flowers and bees or our green friends, who missed the evolutionary advantage to emit WIFI signals, but the constructs we rely on for authentication and keeping the network in order, in other words: your directory.

Banyan VINES

In the beginning there was Banyan VINES and StreetTalk. While they are gone, they are fondly remembered. Other than the Wikipedia article I recall, that it needed almost 300k of memory (probably with all services loaded) which wasn't good when 640k was the limit.
At that time it was the only system offering directory services as a tree. Its demise was triggered by Banyan's late awakening to the fact, that requiring a propriety OS to run a directory service will lead to a collapse of hardware support.

Novell eDirectory

Along came Novell's eDirectory (ca. 20 years ago). Originally called NDS (Novell Directory Services) it suffered from teething problems giving room for other entrants. Initially it required a Novell Netware 4.x server and almost missed the boat in TCP/IP support.
Similar to VINES, the propriety OS became an issue and only in 2003 eDirectory became multi-platform. Novell had quite some ideas: HTTP based file sharing ( webDAV), access via multiple protocols (via LDAP, DSML, SOAP, ODBC, JDBC, JNDI, and ADSI) and storing any information about any network object, relations and access rights.

Embrace, Extend and Extinguish

However the rise of Windows NT made way for the latest entrant: Active Directory from Microsoft. Using its marketing muscles and aptitude for easy to use interfaces, Microsoft swept the directory market and most followed. While loosely based on Open Standards, Microsoft messed with the Kerberos implementation using their successful embrace, extend and extinguish pattern.

Can you see the forest through the trees?

The biggest innovation in Active Directory is the concept of a Directory Forest. Vines and eDirectory only use a single tree. When you ask around for opinions about the feature, you will find, that any consultant paid by the hour will love it. It introduces complexity and endless hours of configuration. IT managers who know their stuff loath it. So what happened?

The leaky abstraction

IMHO this is a case of Leaky abstraction. VINES stored its data in some ISAM file, Netware eDirectory uses FLAIM (I thought they used Btrieve, but that's a minor detail here), while AD uses the JET engine. Originally shared with MS Access, it got forked into JET Blue instead of the more robust (?) SQL server.
Despite the impressive numbers in the specs JET didn't scale as well as expected, so instead of fixing it, large trees were broken down into smaller trees and regrouped into a forest (Wearing my asbestos underwear for this claim).
So the storage model leaked into the product capabilities. Anyway storing anything extra (which is the whole idea of a directory service) is a perilous undertaking, once added to the schema, it never will get out. This led to fun and entertainment and a whole cottage industry of tool vendors.

Buying a product is not a strategy

So you can follow mainstream (and don't argue, I heard them all before) or give it a really hard thought:
  • What do I want to accomplish?
  • Am I comfortable with Jet Blue?
  • How easy can I take the dependency on a single vendor or do I want to be able to select from a rich heritage?
  • Is my long term strategy better served with open standards?
For user management and authentication LDAP (for your own users) or OAuth and/or SAML (for others) might do a better job. AD doesn't do much for end-point management without additional software, so your are not bound by that. The list goes on.
Happy musing!

Posted by on 05 January 2014 | Comments (1) | categories: Software

Domino Development - Back to Basics - Part 5: Finding data - Collections and Search

Continuing from Part 4, this installment will clarify finding data. It, again, is different from other database concepts. Read on:
In a RDBMS your data retrieval queries are formulated in SQL, while XML, RDF or Graph databases might use sparQL, Gremlin, Cypher or JavaScript. There is no shortage of Query languages (one you should master in any case is XPath, get the full reference from the inventor himself).
On a first look a Query Language seems absent from IBM Notes. On a second look, there are two.

The IBM Notes Query Languages

Access to Notes data mostly happens using a view, so the pre-selection happens using the SELECT statement and the @Formula language. We will see how to narrow results down, just below. But you are not limited to creating a view first, a database can be queried directly using @Formula when executing Database.search("some @Formula expression"). This works like any other query language around. This flexibility comes with a price: speed. Database.search is the slowest way to retrieve data from a NSF and you shouldn't use that for your default access path (that is: the actions most users will use in a regular manner).
The second query language is the composition of a fulltext query. In its simplest form it is just some text you want to find, but it can be extended to specify the fields to look into and construct additional constraints. It takes a little to get used to it, but it is quite powerful, just one example: fish SENTENCE chips finds documents where fish & chips are mentioned in the same sentence. Important note here: make sure the FullText index is configured, so it isn't crawling slow.

Finding data

There are several ways data retrieval happens in IBM Notes:
  1. Direct access
    Notes documents can be addressed using a notes:// address. The session class has a resolve(notesurl) method (not the fastest) and the database class has Database.getDocumentByUNID(id). Don't confuse it with Database.getDocumentByURL(...) which created a document based on HTML content at the specified URL. Internally the byUNID is used whenever you click on a Notes document link. It is one of the fastest access methods. There is also byID, which you should avoid, since it isn't save across replicas or copy style compaction
  2. View/Folder access
    The view selection formula (for folders: action taked by users or code) defines a set of documents shown in a view. Using one of the collections (see below) you can process all those using View.getAllEntries, View.createViewNav() or by directly starting to loop through the documents using View.getFirstDocument() (or getFirstEntry()). When looping through them, any saving back of a document might remove it from the view index, so you need to fetch the next document/entry before that
  3. Subsets of View/Folder
    This is the most typical query access to Notes data. To make this work the view needs to be sorted or categorized (depending on the method). A typical example in a workflow application would be a view with the selection formula SELECT Status="Pending Approval" and the first column categorized with CurrentApprover. You would retrieve documents using View.getAllDocumentsByKey(...). This method works already with sorted columns (don't need to be categorized) and can take a Vector as search term. It would then match the Vector elements to the sorted columns for search.
    However using categorization has the advantage, that you can directly address that view in an URL using ?OpenView&RestrictToCategory=search term and the view control has a matching property as well. An alternate approach is to use View.getDocumentByKey(...) to jump to the first Document/Entry and then loop until the desired end. This approach is useful when you might not process all entries or process beyond the current key (e.g. process all requests from R-Z)
  4. FullText search
    Make sure you actually have a FTIndex on the database, otherwise that will be slow. The syntax takes a while to get used to. There is one FTIndex per database, even while you can limit the search to one view
  5. Database.Search(...)
    The most flexible search, with the price of speed and sort order: slow and you have to sort it yourself. Don't use unless unavoidable
Depending on your needs pick one or the other


Depending on your access route choosen above, you might get one of the 4 result collections. Those are no proper Java Collections, but come in their own little API implementation, unless you use a well designed Notes API. You will find getFirst and getNext(currentOne) methods
  • DocumentCollection
    The oldest and most flexible collection. Can originate from a database, a view, a search. Since you have full access to all documents in it, it is the slowest collection. It was also the first collection available in the API and is a good candidate for being replaced by a ViewEntryCollection when tuning your database. A DocumentCollection contains only data documents. Is has fancy set operations
  • NoteCollection
    The NoteCollection can contain data and design Note elements and is slightly faster to build than the DocumentCollection, since it initializes a Note class only on access. So the total time "creation + access" is similar to the DocumentCollection. The main use case is access to design elements and some fancy set operations
  • ViewEntryCollection
    This collection provides all (or selected) documents from a view. Instead of opening each document it gives access to the column values in the ViewEntry object, as well as the same fancy set operations you learned to love from the DocumentCollection and the NoteCollection. The ViewEntryCollection performs faster, given you have all values you need in a view column
  • ViewNavigator
    Technically it isn't a collection, but the ability to point around in the view index. So you won't get any set operation to refine the result, but you get speed. You can jump from category to category (great for reading the reduce() values) and run through an entire view very very quickly. The ViewNavigator is a main reason to plan your views well

Next up: Better save than sorry - Security

Posted by on 02 January 2014 | Comments (0) | categories: IBM Notes XPages

The 5 Stages

In all areas of live things grown, mature and decline. In Buddhist scripture that is called Samsara, the wheel of life. IT is no exception to it. When something new, a violation of the natural order of things, comes along and displaces beloved technology, every fanboy has to go through the 5 stages of grief:
The 5 stages of grief

  1. Denial
    There is no question OLDTECH is the best in the market, there is nothing that comes close, especially NEWTECH doesn't live up to its expectations. Look at OLDTECH's installed base, the capabilities and the compatibility
  2. Anger
    How on earth anybody can deploy or use NEWTECH? Are they out of their minds? What siren songs does NEWTECH sing, so they are blindly following that new crap? NEWTECH needs to be eliminated from the face of the earth. Why does nobody see the superiority of OLDTECH? It is so obvious!
  3. Bargaining
    OK, how about coexistence of OLDTECH and NEWTECH? Lets add OTHERTECH to OLDTECH, so it is more attractive. If there is a discount, will you stick with OLDTECH? Look at your code base, migration is too expensive! Don't fall prey to the "the grass is greener on the other side" syndrom. Look at all the experts still around, be part of that community
  4. Depression
    How could it happen that OLDTECH fell into decline? It was my everything, my lifeblood (optional: return to step 2). How can IT ever work without OLDTECH? What should I do, now that my heart is lost?
  5. Acceptance
    Nobody can take the memories of the good times with OLDTECH, still there is a decent living to be made with it and demand can sustain me. Actually NEWTECH does look really interesting and exiting now, I'll give it a shot
It has has happened before and it will happen again (so hold back your reaction).
Like in eMail, the solution here is non attachment and letting go

Posted by on 02 January 2014 | Comments (8) | categories: Software