wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

Importing EML files into Notes (lots of them)


My friend and Lotus Champion Weerapong Panpai from Zenith in Thailand asked me " How difficult is it to do a bulk import of eml files into Lotus Notes"? MIME as a format is plain ASCII, so on first look it seems simple.
However on a closer look you might guess where the author of an outstanding movie got his inspiration from. RFC 2045 nicely states:" NOTE: The previous four definitions are clearly circular. This is unavoidable, since the overall structure of a MIME message is indeed recursive." So trying to parse that messages on my own was clearly out of the question. Luckily there is ready code available. On one side there is JavaEE and the the MimeMessage and on the other Apache James. An implementation of the javax.mail.internet classes is hidden somewhere in the Notes client, so that would favour MimeMessage. However I found it easier to work with the mime4j classes. Here is what I came up with:
package com.notessensei.mimeimport ;

import java.io.IOException ;
import java.io.InputStream ;

import lotus.domino.Document ;
import lotus.domino.NotesException ;
import lotus.domino.Session ;

import org.apache.james.mime4j.MimeException ;
import org.apache.james.mime4j.parser.ContentHandler ;
import org.apache.james.mime4j.parser.MimeStreamParser ;


public class Mime2Doc {
   
    public void importMail (Session s, InputStream in, Document doc ) throws NotesException, MimeException, IOException {
        doc. replaceItemValue ( "Form", "Memo" ) ;
        MimeStreamParser parser = new MimeStreamParser ( ) ;
        ContentHandler h = new DocContentHandler (s, doc ) ;
        parser. setContentHandler (h ) ;
        parser. parse (in ) ;
    }
}
The bulk of the work happens inside the ContentHandler. Using a Stack (that works LIFO) I can track Mime entries inside mime entries inside mime entries. Still the code is just below 100 lines.
The final piece in the puzzle is the class that helps to track the MIME Parts that are put onto the Stack and returns MIME-Type and encoding derived from the header fields. You are ready to test.I used a directory full of eml files to test:
package com.notessensei.mimeimport ;

import java.io.File ;
import java.io.FileInputStream ;
import java.io.FileNotFoundException ;
import java.io.IOException ;

import org.apache.james.mime4j.MimeException ;

import lotus.domino.Document ;
import lotus.domino.NotesException ;
import lotus.domino.NotesFactory ;
import lotus.domino.NotesThread ;
import lotus.domino.Session ;
import lotus.domino.Database ;

public class Tester {

    public static void main ( String [ ] args ) {

        // Change these for your own test
        final String WORKDIR = "/home/user/testingmime/" ;
        final String EXTENSION = ".eml" ;
        final String DBNAME = "testmail.nsf" ;
        final String SERVER = "" ;

        NotesThread. sinitThread ( ) ;
        try {
            Session s = NotesFactory. createSession ( ) ;
            Database db = s. getDatabase (SERVER, DBNAME ) ;
            File folder = new File (WORKDIR ) ;
            File [ ] tobeImported = folder. listFiles ( ) ;
            Mime2Doc md = new Mime2Doc ( ) ;
            for ( int i = 0 ; i < tobeImported. length ; i ++ ) {
                File f = tobeImported [i ] ;
                if (f. isFile ( ) && f. getName ( ). endsWith (EXTENSION ) ) {
                    System. out. println ( "File: " +f. getName ( ) ) ;
                    Document doc = db. createDocument ( ) ;
                    FileInputStream in = new FileInputStream (f ) ;
                    md. importMail (s,in, doc ) ;
                    doc. recycle ( ) ;
                }
            }
            System. out. println ( "--- DONE ---" ) ;
            // Cleanup
            db. recycle ( ) ;
        } catch (NotesException e ) {
            e. printStackTrace ( ) ;
        } catch ( FileNotFoundException e ) {
            e. printStackTrace ( ) ;
        } catch (MimeException e ) {
            e. printStackTrace ( ) ;
        } catch ( IOException e ) {
            e. printStackTrace ( ) ;
        } finally {
            NotesThread. stermThread ( ) ;
        }
    }
}
As usual: YMMV!

Posted by on 23 April 2012 | Comments (12) | categories: Show-N-Tell Thursday

Comments

  1. posted by Martin Durao on Wednesday 29 August 2012 AD:
    Hi Stephan

    Great solution!!!
    The only drawback is in my case all EML docs are imported in Sent folder...
    I tried setting db view but without success.
    Do you know if could be some way to select the view/folder where import docs?

    Thanks for your post!
    Martin
  2. posted by Stephan H. Wissel on Thursday 30 August 2012 AD:
    @Martin,

    there is no such thing as a "sent folder". In Lotus Notes "Sent" is a view that has a fixed formula and should show the messages you have sent out, but not the messages you have received (they would be in the "all documents" view - with all the others).

    To have a message showing up in a different folder (you can show them in as many folders as you want) you add an document.putInFolder call. SO when you import a whole message set that might be in a directory structure resembling your original folder structure you would use the directory name. If the folder doesn't exist it is automatically created. When you execute the command multiple times with different folder names, the document is shown in all those folders, but exists only once.
  3. posted by Vikram on Tuesday 02 October 2012 AD:
    In my case the issue is its giving me error message in "MimeException", the error message is "No exception of type MimeException can be thrown; an exceptipn type must be a subclass of Throwable". Please let me know what should I do?
  4. posted by Stephan H. Wissel on Tuesday 02 October 2012 AD:
    @Vikram - short answer: debug!
    long answer: (this is a wild guess) It looks like something is not right with the MIME you try to read and the Mime4J classes try to throw an error and that error seems buggy. Head over to the mailing list and ask for help there. Make sure to provide an isolated code snippet and the message that throws that error.
  5. posted by Vikram on Wednesday 03 October 2012 AD:
    Hi, There is no error now, but the issue is the documents gets created and saved in draft view. I dont have issues in there too. But the issue is it keeps To, From, BCC, Subject fields blank. And whatever it gets in EML file it copies that into body(RichTextField) field. Also second big issue is it keeps the message content in base64 encoded format.

    Please help me if I am missing anything. my id is viky_rsolanki@yahoo.com.. if possible you can copy this agent into a sample database and send it across to me it would be great help.

    I apreciate the coding partner. And even the code. It hardly takes some time to execute 75 eml files of size around 1MB. Highly apreciate the performance. :)
  6. posted by Stephan H. Wissel on Wednesday 03 October 2012 AD:
    @Vikram: I run thisas standalone Java application (pointing to the Domino JVM as runtime. Never tested it as agent and have no intention to do so. Too much headache with and permission.&;br /&gt;This might be your problem, so run it standalone.&lt;br /&gt;
  7. posted by Vikram on Wednesday 03 October 2012 AD:
    Thanks for your response.

    How did u do that? Let me know the steps, even I will do the same way. Please let me know..
  8. posted by Vikram on Wednesday 03 October 2012 AD:
    And were u able to create proper email documents?
  9. posted by Vikram on Wednesday 03 October 2012 AD:
    In my case, the agent is able to create the documents. Now the issue is it creates the emails with complete EML content in body. And the mail body content as base64 encrypted. Can you help me to understand base 64 encoding format. And how can we get the eml file in normal format. This is only the big issue, rest the code works absolutely as per our expectation.
  10. posted by Stephan H. Wissel on Wednesday 03 October 2012 AD:
    /opt/ibm/lotus/notes/jvm/java com.notessensei.mimeport.Tester

    eventually a -cp and memory parameters
  11. posted by Vikram on Wednesday 03 October 2012 AD:
    cool will try this out
  12. posted by Vikram on Friday 05 October 2012 AD:
    Now I have copied all teh jar files,apache-mime4j-0.3.jar,apache-mime4j-0.4.jar,commons-logging-api-1.0.4.jar and mail.jar in [Notes]\jvm\lib\ext directory.

    And did not made any changes to your code, jst copied as it is into an agent. Now it works fine, only issue is it keeps it decoded. And only the body content is been displayed the rest of the fields like from, To and Subject does not appears.. I would like you to help me to get this message decoded.

    Awaiting for your positive response.
  13. posted by Vikram on Friday 05 October 2012 AD:
    Hi Stephen,

    The code works fine with the agent also.. Only thing is it saves the memo document as draft. I am fine with it. The major issue which I am facing now, its only getting the body content and not the other fields like to, copy, bcc and subject which is also required.

    Please suggest me.
  14. posted by Stephan H. Wissel on Friday 05 October 2012 AD:
    @Vikram: which part of "I will not debug your agent, use standalone Java first" was it you didn't understand?
    As stated before: debug!
    This is not a site for copy-paste development. When your code loads a mime part what does it discover? Is the mime type in your source wrong?
    Emoticon angry.gif stw
  15. posted by Anders Naslund on Wednesday 17 October 2012 AD:
    @Martin,
    Why all EML Docs shows up in the sent folder is because they lack a field called "DeliveredDate" after being imported. Create a formula agent that runs on all documents that are not supposed to shown as "sent":

    FIELD DeliveredDate := PostedDate;
    SELECT @All

    Emoticon cool.gif
  16. posted by Michelangelo on Monday 18 November 2013 AD:
    Hi Stephan, thanks for your post.
    I am trying to do an agent to import from an IMAP account into Notes / Domino. It works on most of the email messages. Two glitches.
    1) A null pointer exception was thrown on certain eml files in DocContentHandler.field so I placed the code inside a try catch like this
    try {
    this.mimeParts.peek().createHeader(rawField.getName(), rawField.getBody());
    } catch (NullPointerException e) {
    System.out.println("Null pointer exception, skipping field");
    }
    No problem with this fix.
    2) I get a NotesException: Item Body already exists when a message has another message inside it (Content-Type: message/rfc822). When this happens, you see on Java Console a "startMessage" so the line
    MIMEEntity body = doc.createMIMEEntity(DocContentHandler.RTFIELDNAME);
    throws an exception.
    I do not know how to fix this.
    What came to my mind is cumbersome: create a new doc (let's say doc2), then render to doc2 (recursion), then render doc2 into original body rich text . But ... I do not think I can do so until I finish importing MIME.

    So before starting this I wonder if you had had such a problem.

    If you think it might help, I have a sample eml file that causes this to happen.
    Thanks for your great post and if you are at Lotusphere ... errrr ... Connect 2014, I hope to see you there.


  17. posted by Filip Navratil on Sunday 14 December 2014 AD:
    I wrote the agent to import from IMAP account into Notes. In order to handle embedded messages (Content-Type: message/rfc822) you have to modify the startMessage method as follows:
    if (doc.getMIMEEntity()==null) {
    //create mimeEntity
    System.out.println("mimeEntity Creating");
    MIMEEntity body = doc.createMIMEEntity(DocContentHandler.RTFIELDNAME);
    this.mimeParts.push(new MimePartInfo(body)); // First element on the stack
    } else {
    System.out.println("Root mimeEntity created already");
    MIMEEntity child=doc.getMIMEEntity().createChildEntity();
    this.mimeParts.push(new MimePartInfo(child)); // First element on the stack
    }

    You also have to modify endMessage method as follows:
    public void endMessage() throws MimeException {
    this.mimeParts.pop();
    System.out.println(" -- Import complete");
    return;
    }

    As you cannot save within the endMessage method, you have to save the notes Document after md.importMail(s,in, doc) in the "Tester.main".
  18. posted by Don Smith on Wednesday 22 August 2018 AD:

    This switch is possible with GainTools EML to NSF Conversion Tool. This software will help you to move mailboxes database from EML to NSF file format. Download here: https://www.gaintools.com/eml/nsf/