wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

Handle HTTP chunked responses - Java edition


The Domino REST API delivers collections using chunked transfer encoding. This has the advantage, that you can process results as they arrive. It produces the challenge that the usual client side code is designed to first wait for completion of the request. I wrote about the JavaScript solution a while ago, this is the Java edition.

Client choices

In JavaScript land the choice of client is simple: the Fetch API. In Java we have some choices:

There are probably more around. This article uses the JDK HttpClient. I'll skip the parts with Authentication and TLS handling, check the full example for details.

How it works

First we create an java.net.http.HttpClient. It takes care of the http version and the TLS context.

HttpClient getClient(SSLContext sslContext) {
  return HttpClient.newBuilder()
           .sslContext(sslContext)
           .build();
}

Then we build and execute the request. The magic is the BodySubscriber (more on that below).

Integer runGetRequest(HttpClient client, String url, String authHeader, BodySubscriber subscriber) throws Exception {
  HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .header("Authorization", authHeader)
            .GET()
            .build();

  CompletableFuture<Integer> response =
          client.sendAsync(request, responseInfo -> subscriber)
          .whenComplete((r, t) -> System.out.println("Response: " + r.statusCode()))
          .thenApply(HttpResponse::body);

  return response.get();
}

Read more

Posted by on 09 October 2024 | Comments (0) | categories: Java WebDevelopment

Quarkus in Multi-Module projects


You are developing a web application using Quarkus that consists of multiple (micro)services and a bunch of supporting libraries. Since maven modules provide sufficient isolation, you decide to use a parent project to keep dependency versions and parameters in sync and a Maven Reactor to build them together.

This blog post is for you. A special thanks to Alexey for helping out.

Moving parts

Our objective is to have a development setup where we can edit any of the services or libraries and then run them individually or all together. Ideally without the need to alter configurations between runs and the ability to deploy the setup using devcontainers (note: that's about the development setup, not about deploying the finished application). There are some moving parts:

That's a lot, let's dig in.


Read more

Posted by on 28 August 2024 | Comments (0) | categories: Java Maven Quarkus WebDevelopment

Simplify JUnit tests with custom annotations


In the beginning was a test, that provided to be vital, but not sufficient. In modern application development we encounter:

  • Unit Tests: testing a single function or a tuple of related functions
  • Module Tests: testing bigger parts of your application, without actual external dependencies
  • Integration or End-to-End tests: a.k.a life firing exercise

This entry isn't a discussion about the merits of how much and when test, but making tests easy to setup and distinguish

The manual way

We typically use Mockito, vert.x and REST-assured in our tests, so a typical test class would look like this:

@ExtendWith(VertxExtension.class, MockitoExtension.class, MyCustomExtension.class})
@Tag("UnitTest")
class SomethingTest {

  @Test
  does_it_blend() {
    // Test goes here
  }
}

It is just two lines, but everywhere. You can simplify it by creating your own annotation.

The custom annotation

@Target({TYPE, METHOD, ANNOTATION_TYPE})
@Retention(RUNTIME)
@ExtendWith({VertxExtension.class, MockitoExtension.class, MyCustomExtension.class})
@Tag("UnitTest")
public @interface UnitTest {
  // no action needed here, JUnit use only!
}

Now you simply use:

@UnitTest
class SomethingTest {

  @Test
  does_it_blend() {
    // Test goes here
  }
}

While this looks like minor cosmetic, it allows to control test extensions from a single place, your annotation source. Repeat that process for the other test types (ModuleTests, IntegrationTests, PerformanceTests etc.) you want to use.

In your pom.xml, in the build-plugins section you can use the tag to ensure all your unit test, but only them execute on mvn test and the others on mvn verify

<plugin>
  <!-- Run UNIT and MODULE tests, no backend calls -->
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <groups>UnitTest,ModuleTest</groups>
  </configuration>
</plugin>

As usual YMMV


Posted by on 05 August 2024 | Comments (0) | categories: Java jUnit Maven TDD

Hexagonal Architecture meets Maven


Know your patterns, so you don't re-invent the wheel. Patterns are like LEGO: You can combine them to larger patterns. One of those is the Hexagonal Achitecture officially called:

Ports and Adapters

proposed by Alistair Cockburn, one of the co-signers of the Agile Manifesto.

I've written about his work before and can highly recommend his latest book "Hexagonal Architecture Explained" (or for that matter any of his writings).

In a nushell, Ports and Adapters proposes an approach that leads to loosely coupled systems, that separates business logic from the software environment (think I/O, databses, API and UI) to facilitate clarity and ease of testing.

This post doesn't aim to explain it, for that you can refer to the Wikipedia article or, better, read the book but trying to describe how to apply it to Java in a Maven project.


Read more

Posted by on 24 July 2024 | Comments (0) | categories: Java Maven WebDevelopment

How deep do you authenticate?


Accessing applications usually entails some kind of identity. Some part(s) of your application provide identity (called IdP), while other's consume it (paraphrased from Captain Obvious). Identity could be provided from a record or document in your or another database, an LDAP directory, an OICD or a 3d party like your eMail provider or social account, or with some hoops and loops Webauthn (a.k.a passkey).

The question is: how deep does it go?

A typical archtecture

For simplicity I'm skipping load balancers and cache facilities

ApplicationTiers 0-4

Each tier might or might not have its own identity, lets have a closer look

(0) User tier

When is authentication information stored at the user tier (think Browser, app or OS) and how immediate is it? Like a username/password or a passkey private key. Those can be exchanged for a token. Or is it a session cookie or an access_token allowing immediate access? How are those credentials protected and/or synced? How vulnerable are they on physical access?

(1) Access tier

A.k.a the firewall or VPN Does it assert a valid user? Are the credentials the same or different from your application. Does your access tier include IP ranges or geo location as part of identity? I've seen identity requirements 100% on VPN, but rarely on firewalls

(2) Web tier

TYpically you find a static web server like nginx, Apache http, the Kubernetes Ingress or a service by a cloud provider.

Even when all your static resources are served by your application tier, you can identify your web tier where requests flow through. When you can establish identity there (and reject invalid ones), you have one more protective layer. nginx+ can do that with JWT

(3) Application tier

YOur application could be a monolith, microlith, microservice, follow a layered architecture, be message driven, event driven or be contemporary with the hexagonal architecture. In any case your user facing access will require identity.

It becomes blurry when your user facing services then call out to other services (via http or message/event), what identity are they using to communicate: user, service or both? You might start looking at RBAC. In any case, this needs to be planned carefully

(4) Persistence / database tier

The prevalent examples you find online , especially in the realm of DBaaS, use just one service identity to access your persistence (file system, database , imp, etc.). So database logs won't tell you who accessed data (RW/RO) and you won't have a chance to implement row level security.

Interestingly this isn't a limitation of databases, they all come with user management, but rather the headache maintaining it or setup another auth

When we designed the Domino REST API, we decided to stick to the JWT based user identity all the way to the database. Apache CouchDB also allows for JWT based authentication. It was lacking the ability to point to an IdP's jwks, so I contributed the CouchDB IdP updater, go check it out.

How do you use identity?


Posted by on 23 June 2024 | Comments (0) | categories: Development Identity Management JWT WebDevelopment

NoSQL schema design


A question that pops up frequently in developer discussions is "how to structure your data in a NoSQL way?". To shed a light on this, we have a look at the approach invented 50 years ago and still an all time favorite

Normalization

In a simple order example, we are looking at four tables:

  • Customer
  • Product
  • Order
  • OrderEntry

CLassic SQL schema

In this design, there are no duplicates and some simple SQL can list out all I need, for example the order value for a given order:

SELECT oi.order_id,
       SUM(oi.quantity * p.price) AS order_total
FROM OrderItem oi
JOIN Product p ON oi.product_id = p.id
GROUP BY oi.order_id
WHERE oi.order_id = 67111;

or the revenue per customer:

SELECT c.id AS customer_id,
       c.name AS customer_name,
       SUM(oi.quantity * p.price) AS revenue
FROM Customer c
JOIN Order o ON c.id = o.customer_id
JOIN OrderItem oi ON o.id = oi.order_id
JOIN Product p ON oi.product_id = p.id
GROUP BY c.id, c.name;

Read more

Posted by on 06 June 2024 | Comments (0) | categories: NoSQL WebDevelopment

Development Containers - the fine print


Development Containers are supposed to liberate your development environment from a specific local installation, like container technology liberated your runtimes (a.k.a YAMLed them into Docker or Kubernetes).

Helmchart Pun

Development != Runtime

Containerization for development has some overlap and quite some difference to containerization for production:

Topic Development Production
Mutability You alter container content Container is static
Network Use internal network Use internal network
Access Developer tools++ Browser / App
Containers multiple multiple
Volumes primary container binds projectdir
all others mount only
all: bind or mount
Configuration devcontainer.json, docker-compose.yml docker-compose.yml,Helm Chart
Scope Runtime & Tooling Runtime
Dockerfile Setup development environment Configure production

Insights

  • There are many getting started resources available: here, here, here, here and here. They are working examples, commonly strong on what and how, but light on why
  • There are plenty of templates to learn by example
  • There seem to be substantial differences how it works on different platforms, subtle and annoying
  • On macOS (14.4.1) with with the devcontainer plugin 0.364.0 mount binds would not work in auxiliary containers, only in the main app
  • I couldn't find any descrption which subset of docker-compose.yml is supported
  • The most interesting templates, for now, live on the Microsoft Artifact Registry, when you know how to look. Starting with those saves you tons of time and frustration
  • You will love code --list-extensions to figure out the extensions you have in your vscode (unless you are a code n00b and don't have any)

Read more

Posted by on 12 May 2024 | Comments (0) | categories: Container Docker Java NodeJS WebDevelopment

Maven build with multiple Java versions


Imagine, you are tasked with maintaining a Java application that needs to run on more than one Java version. You want to ensure that it compiles, tests and builds on all of them.
This is our story, buckle up, there are a few moving parts

The big picture

  • We use Apache Maven to drive the project using the pom.xml
  • The Maven Toolchains plugin controls the Java versions
  • Using <properties> ... </properties> and Build Profiles to adjust conditions for processing
  • Annotatiosn like @Only8 and @Only17 help to qualify tests
  • Our build tool (Jenkins or Github Actions) will use a container provided (in our case based on Redhat UBI)

Read more

Posted by on 16 April 2024 | Comments (0) | categories: Java Maven WebDevelopment

nginx as ingress for Docker compose


In June I wrote about how to use Docker & nginx to deliver statically rendered brotli files for your web (frontend) application. It improves delivery quite a bid, but left me wonder: isn't there too much static WebServer involved?

Double hop to deliver static files

A typical web application using micro/mini/midi services looks like this:

A typical Docker configuration

It is common, easy and concerns quite separated. However it comes with a set of challenges:

  • nginx doesn't do http/2 on proxy_pass, so you miss the ability to serve static files directly with http/2
  • For static file we have two nginx involved
  • Each service needs to be exposed to the host at some port
  • The service architecture leaks to the host based nginx. SO any change in service needs an update to the docker-compose.yml AND the host based nginx configuration
  • the containers depend on that, external to them, configuration

So I tried to design a better way to handle this without going all K-in:

Docker configuration with ingress as component

This looed like a more promising approach:

  • Services could be addressed with their internal network name
  • Only Ports 80 and 443 of one container need exposure on the host
  • The nginx configuration inside the container is immutable and can't accidentially be reconfigured in production (your image comes from a pipeline isn't it)

Challenges

  • When trying to configure certbot, I initially tried using the --nginx parameter with an http challenge and shared data mounts. None of the attempts worked satisfactory, so at the end I settled on aDNS-01 using CloudFlare.
  • Since I wanted the nginx configuration to be inside the container image (and not on a data mount), a good understandig of nginx's configuration is necessary. The only persisted information was /etc/letsencrypt for the certificate and a secret for CloudFlare credentials
  • When the nginx configuration is statically configured for TLS, on initial load it will fail since the certs don't exist yet. Auntie Google suggested a manual run of certbot, but I favour docker compose up to handle everything
  • I ended up creating my own docker images, which was an epiphany: it absolutely makes sense to build a container image for single use instead of trying hard to make it configurable and vulnerable to mis-configuration

Read more

Posted by on 15 November 2023 | Comments (0) | categories: Docker nginx WebDevelopment

Quarkus and GraalVM starter


When Java is one of the languanges in your portfolio, you might have heard of Quarkus, an alternative to Spring Boot build on top of vert.x and GraalVM, a polyglot runtime for Java and other languages.

This article describes the getting started that worked for me.

Moving parts

The fist hurdle to overcome is to install all the bits and pieces. There are plenty of versions (Java 11 - Java 21) and methods (maven, CLI, packet managers), so it con be confusing what to pick. I tried most of them and created a mess and a deep appreciation for the "reset to snapshot" feature afforded by virtual machines. Instructions work for macOS, Linux or Linux on Windows.Here we go:

  1. install SdkMan.

The tool helps to keep your software development kits under control. From their website:

"Meet SDKMAN! – your reliable companion for effortlessly managing multiple Software Development Kits on Unix systems. Imagine having different versions of SDKs and needing a stress-free way to switch between them. SDKMAN! steps in with its easy-to-use Command Line Interface (CLI) and API."

curl -s "https://get.sdkman.io" | bash

You can thank me later. Side note: there are 16 different JDK offerings that can be installed, we are spoiled for choices

  1. install GraalVM

Currently, as of time of writing, there are three GraalVM distributions available. The Open Source, community supported GraalVM Community Edition, the commercial, Oracle supported Oracle GraalVM which requires a license in production and the ReedHat backed Mandrel. Mandrel is advertised as "specifically to support Quarkus". The Java 21 version was not yet available on sdkman, so I used the community edition. To stick with Mandrel I will use the container build option, later more on that

# List SDKs
sdk list java
# install GraalVM
sdk install java 21-graaalce
  1. Install Quarkus

This will install the Quarkus CLI and for good measure Apache Maven. The Quarkus CLI makes a pleasant developer experience

sdk install quarkus
sdk install mvn
  1. Install Docker

You can use Docker desktop (required a license for larger organisations) or Rancher Desktop (which also handles Kubernetes), Podman Desktop, any of the alternatives or the command line. New to Docker? There's plenty of fish

Now we are good to go. Skipping the Code with Quarkus tutorial lets build a n app in java and native put it into a container


Read more

Posted by on 13 October 2023 | Comments (0) | categories: GraalVM Java Maven Quarkus WebDevelopment