Usability - Productivity - Business - The web - Singapore & Twins

By Date: January 2017

SAML and the Command Line

One of the best kept secrets of Connections Cloud S1 is the Traveler API. The API allows interactions that are missing from the Admin UI, like deleting a specific device or implementing an approval workflow.
Unfortunately the API only offers authentication via SAML, OAuth or BasicAuth are missing. So any application interacting with the API needs to do The SAML Dance. That's annoying when you have an UI to use, and a formidable challenge when you have a command line application, like a cron Job running unsupervised at interval.
One lovely step in the process: the IBM IdP returns a HTML page with a hidden form containing the SAML assertion result to be posted back to the application provider. Quite interesting, when your application provider is a command line app. Let's get to work.
The script is written in node.js and uses request and fast-html-parser npm package. The first step is to load the login form (which comes with a first set of cookies)

var requestOptionsTemplate = {
    headers: {
        'Origin': 'https://api.notes.ap.collabserv.com/api/traveler/',
        'User-Agent': 'ancy CommandLine Script',
        'Connection': 'keep-alive',
        'Cache-Control': 'max-age=0',
        'Upgrade-Insecure-Requests': 1
    'method': 'GET'

function scLoginPart1() {
    console.log('Authenticating to SmartCloud ...');
    var requestOptions = Object.assign({}, requestOptionsTemplate);
    requestOptions.url = 'https://apps.na.collabserv.com/manage/account/dashboardHandler/input';
    request(requestOptions, scLoginPart2);

The function calls the URL where the login form can be found. The result gets delivered to the function scLoginPart2. That function makes use of a global configuration variable config that was created through const config = require("./config.json") and contains all the credentials we need. Step2 submits the form and hands over to Step3.

function scLoginPart2(err, httpResponse, body) {
    if (err) {
        return console.error(err);
    // Capture cookies
    var outgoingCookies = captureCookies(httpResponse);
    var requestOptions = Object.assign({}, requestOptionsTemplate);
    requestOptions.headers.Cookie = outgoingCookies.join('; ');
    requestOptions.headers['Content-Type'] = 'application/x-www-form-urlencoded';
    requestOptions.method = 'POST';
    requestOptions.url = 'https://apps.ap.collabserv.com/pkmslogin.form';
    requestOptions.form = {
        'login-form-type': 'pwd',
        'error-code': '',
        'username': config.smartcloud.user,
        'password': config.smartcloud.password,
        'show_login': 'showLoginAgain'
    request(requestOptions, scLoginPart3);

function captureCookies(response) {
    var incomingCookies = response.headers['set-cookie'];
    var outgoingCookies = [];
    if (incomingCookies) {
        incomingCookies.forEach(function(cookie) {
    // Array, allows for duplicate coolie names
    return outgoingCookies;

Part 3 / 4 finally collect all the cookies we need, so to turn attention to getting the API token in step 5

function scLoginPart3(err, httpResponse, body) {
    if (err) {
        console.error('Login failed miserably');
        return console.error(err);
    // Login returns not 200 but 302
    // see https://developer.ibm.com/social/2015/06/23/slight-changes-to-the-form-based-login/
    if (httpResponse.statusCode !== 302) {
        return console.error('Wrong status code received: ' + httpResponse.statusCode);

    var outgoingCookies = captureCookies(httpResponse);
    var redirect = httpResponse.headers.location;

    // This is the 3rd request we need to make to get finally all cookies for app.na
    var requestOptions = Object.assign({}, requestOptionsTemplate);
    requestOptions.headers.Cookie = outgoingCookies.join('; ');
    requestOptions.url = redirect;
    request(requestOptions, scLoginPart4);

function scLoginPart4(err, httpResponse, body) {
    if (err) {
        console.error('Login redirect failed miserably');
        return console.error(err);
    var cookieHarvest = captureCookies(httpResponse);
    // Now we have some cookies in app, we need the SAML dance for api.notes

In Part 5 we first request the URL with actual data (devices in our case), but get another SAML dance step, since we have apps.na vs api.notes in the URL

Read more

Posted by on 30 January 2017 | Comments (1) | categories: JavaScript NodeJS

GIT deploy your static sites - Part 1

When you, in principal, like the idea to serve SPA from the http server, you will encounter the pressing question: where do babies come from how to get your application deployed onto the http server? This applies to nodeJS applications too, but that is part of another story for another time.
On Bluemix that's easy: just use a Pipeline.
For mere mortal environments there are several options:
  • Just FTP them - insecure unless you use sftp/scp. Big pain here: deleting obsolete files
  • Setup rsync. When done with a ssh certificate can be reasonably automated. Same pain applies: deleting obsolete files
  • Use a GIT based deployment. This is what I will discuss further
I like a repository based deployment since it fits nicely into a development based workflow. The various git gui tools provide insight what has changed between releases and if things go wrong, you can roll back to a previous version or you can wipe data and reestablish them from the repository. Designing the flow, I considered the following constraints:
  • The repositories would sit on the web server
  • Typically a repository would sit in .git inside the site directory. While you could protect that with access control, I decided I don't want to have it in separate directories
  • When pushing to the master branch, the site should get updated, not on any other branch. You can extend my approach to push other branches to other sites - so you get a test/demo/staging capability
  • Setting up a new site should be fast and reliable (including https - but that's part 2)
The "secret" ingredients here are git-hooks, in specific the post-receive. Hooks, in a nutshell are shell scripts that are triggered by events that happen to a git environment. I got inspired by this entry but wanted to automate the setup.

Read more

Posted by on 12 January 2017 | Comments (0) | categories: nginx WebDevelopment

Serving Single Page Applications with Domino

Single Page Applications (SPA) are all the rage. They get developed with AngularJS, ReactJS or {insert-your-framework-of-choice}. Those share a few communialities:
  • the application is served by a static web server
  • data is provided via an API, typically reading/writing JSON via REST or graph
  • authentication is often long lasting (remember me...) based on JWT
  • authentication is highly flexible: login with {facebook|google|linkedin|twitter} or a corporate account. Increasingly 2 factor authentication is used (especially in Europe)
How does Domino fit into the picture with its integrated http stack, authentication and database? The answer isn't very straight forward. Bundling components creates ease of administration, but carries the risk that new technologies are implemented late (or not at all). For anything internet facing that's quite some risk. So here is what I would do:
Red/Green Zone layout for Domino

Read more

Posted by on 11 January 2017 | Comments (5) | categories: IBM Notes XPages

Lessons from Project OrangeBox

Project OrangeBox, the Solr free search component, was launched after the experiments with Java8, Vert.x and RxJava in TPTSNBN concluded. With a certain promise we were working on a tight dead line and burned way more midnight oil than I would have wished for.

Anyway, I had the opportunity to work with great engineers and we shipped as promised. There are quite some lesson to be learned, here we go (in no specific order):

  • Co-locate
    The Verse team is spread over the globe: USA, Ireland, Belarus, China, Singapore and The Philippines. While this allows for 24x7 development, it also poses a substantial communications overhead. We made the largest jumps in both features and quality during and after co-location periods. So any sizable project needs to start and be interluded with co-location time. Team velocity will greatly benefit
  • No holy cows
    For VoP we slaughtered the "Verse is Solr" cow. That saved the Domino installed base a lot of investments in time and resources. Each project has its "holy cows": Interfaces, tool sets, "invaluable, immutable code", development pattern, processes. You have to be ready to challenge them by keeping a razor sharp focus on customer success. Watch out for Prima donnas (see next item)
  • No Prima Donnas
    As software engineers we are very prone to perceive our view of the world as the (only) correct one. After all we create some of it. In a team setting that's deadly. Self reflection and empathy are as critical to the success as technical skills and perseverance.
    Robert Sutton, one of my favourite Harvard authors, expresses that a little bolder.
    In short: A team can only be bigger than the sum of its members, when the individuals see themselves as members and are not hovering above it
  • Unit test are overrated
    I hear howling, read on. Like "A journey of a thousand miles begins with a single step" you can say: "Great software starts with a Unit Test". Begins, not: "Great software consists of Unit Tests". A great journey that only has steps ends tragically in death by starvation, thirst or evil events.
    Same applies to your test regime: You start with Unit tests, write code, pass it on to the next level of tests (module, integration, UI) etc. So unit tests are a "conditio sine qua non" in your test regime, but in no way sufficient
  • Test pyramid and good test data
    Starting with unit tests (we used JUnit and EasyMock), you move up to module tests. There, still written in JUnit, you check the correctness of higher combinations. Then you have API test for your REST API. Here we used Postman and its node.js integration Newman.
    Finally you need to test end-to-end including the UI. For that Selenium rules supreme. Why not e.g. PhantomJS? Selenium drives real browsers, so you can (automate) test against all rendering engines, which, as a fact of the matter, behave unsurprisingly different.
    One super critical insight: You need a good set of diverse test data, both expected and unexpected inputs in conjunction with the expected outputs. A good set of fringe data makes sure you catch challenges and border conditions early.
    Last not least: Have performance tests from the very beginning. We used both Rational Performance Tester (RPT) and Apache JMeter. RPT gives you a head start in creating tests, while JMeter's XML file based test cases were easier to share and manipulate. When you are short of test infrastructure (quite often the client running tests is the limiting factor) you can offload JMeter tests to Blazemeter or Flood.io
  • Measure, measure, measure
    You need to know where your code is spending its time in. We employed a number of tools to get good metrics. You want to look at averages, min, max and standard deviations of your calls. David even wrote a specific plugin to see the native calls (note open, design note open) or Java code would produce (This will result in future Java API improvements). The two main tools (besides watching the network tab in the browser) were New Relic with deep instrumentation into our Domino server's JVM and JAMon collecting live statistics (which you can query on the Domino console using show stats vop. Making measurements a default practise during code development makes your life much easier later on
  • No Work without ticket
    That might be the hardest part to implement. Any code item needs to be related to a ticket. For the search component we used Github Enterprise, pimped up with Zenhub.
    A very typical flow is: someone (analyst, scrum master, offering manager, project architect, etc.) "owns" the ticket system and tickets flow down. Sounds awfully like waterfall (and it is). Breaking free from this and turn to "the tickets are created by the developers and are the actual standup" greatly improves team velocity. This doesn't preclude creation of tickets by others, to fill a backlog or create and extend user stories. Look for the middle ground.
    We managed to get Github tickets to work with Eclipse which made it easy to create tickets on the fly. Once you are there you can visualize progress using Burn charts
  • Agile
    "Standup meeting every morning 9:30, no exception" - isn't agile. That's process strangling velocity. Spend some time to rediscover the heart of Agile and implement that.
    Typical traps to avoid:
    • use ticket (closings) as (sole) metric. It only discourages the us of the ticket system as ongoing documentation
    • insist on process over collaboration. A "standup meeting" could be just a Slack channel for most of the time. No need to spend time every day in a call or meeting, especially when the team is large
    • Code is final - it's not. Refactoring is part of the package - including refactoring the various tests
    • Isolate teams. If there isn't a lively exchange of progress, you end up with silo code. Requires mutual team respect
    • Track "percent complete". This lives on the fallacy of 100% being a fixed value. Track units of work left to do (and expect that to eventually rise during the project)
    • One way flow. If the people actually writing code can't participate in shaping user stories or create tickets, you have waterfall in disguise
    • Narrow user definitions and stories: I always cringe at the Scrum template for user stories: "As a ... I want ... because/in order to ...". There are two fallacies: first it presumes a linear, single actor flow, secondly it only describes what happens if it works. While it's a good start, adopting more complete use cases (the big brother of user stories) helps to keep the stories consistent. Go learn about Writing Effective Use Cases. The agile twist: A use case doesn't have to be complete to get started. Adjust and complete it as it evolves. Another little trap: The "users" in the user stories need to include: infrastructure managers, db admins, code maintainer, software testers etc. Basically anybody touching the app, not just final (business) users
    • No code reviews: looking at each other's code increases coherence in code style and accellerates bug squashing. Don't fall for the trap: productivity drops by 50% if 2 people stare at one screen - just the opposite happens
  • Big screens
    While co-located we squatted in booked conference rooms with whiteboard, postit walls and projectors. Some of the most efficient working hours were two or three pairs of eyes walking through code, both in source and debug mode. During quiet time (developers need ample of that. The Bose solution isn't enough), 27" or more inches of screen real estate boost productivity. At my home office I run a dual screen setup with more than one machine running (However, I have to admit: some of the code was written perched into a cattle class seat travelling between Singapore and the US)
  • Automate
    We used both Jenkins and Travis as our automation platform. The project used Maven to keep the project together. While Maven is a harsh mistress spending time to provide all automation targets proved invaluable.
    You have to configure your test regime carefully. Unit test should not only run on the CI environment, but on a developers workstation - for the code (s)he touches. A full integration test for VoP on the other hand, runs for a couple of hours. That's the task better left to the CI environment. Our Maven tasks included generating the (internal) website and the JavaDoc.
    Lesson learned: setting up a full CI environment is quite a task. Getting the repeatable datasets in place (especially when you have time sensitive tests like "provide emails from the last hour") can be tricky. Lesson 2: you will need more compute than expected, plan for parallel testing
  • Ownership
    David owned performance, Michael the build process, Raj the Query Parser, Christopher the test automation and myself the query strategy and core classes. It didn't mean: being the (l)only coder, but feeling responsible and taking the lead in the specific module. With the sense of ownership at the code level, we experienced a number of refactoring exercises, to the benefit of the result, that would never have happened if we followed Code Monkey style an analyst's or architect's blueprint.
As usual YMMV

Posted by on 03 January 2017 | Comments (3) | categories: Continuous integration IBM Notes Software