wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

learning how nginx proxy works


A common approach for web applications is to serve them behind a reverse proxy. My favorite there is nginx. It has a fairly understandable configuration, supports http/2 and is fully supported by Letsencrypt's certbot.

proxy_pass can be tricky

The configuration for reverse proxying is a combination of location and a proxy_pass statement, as well as some headers. In it simplest form the URL path and the proxy_path URL are the same, so you don't need a translation between direct access (e.g. local testing) and access through nginx. A configuration could look like this:

location /someurl {
        proxy_pass http://127.0.0.1:3000/someurl;
        proxy_http_version 1.1;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
    }

The headers ensure your application knows the original request source and allow upgrade from http to websockets.

It gets interesting when location and proxy_pass don't match. Differences can range from different url path to regex and rewrites. Especially when inheriting (or googling) configurations, understanding can be a challenge.

Express to the rescue

A few lines of expressjs will echo back what ends up at the backend. Together with curl it provides you a practical learning and testing environment.

const express = require('express');
const app = express();
const port = 3010;

app.get('*', (req, res) => {
  res.setHeader('Content-Type', 'text/json');
  let result = JSON.stringify(
    {
      headers: req.headers,
      url: req.url,
      query: req.query,
      hostname: req.hostname
    },
    null,
    2
  );
  res.send(result);
  console.log(result);
});

app.listen(port, () => {
  console.log(`Echo app listening for GET on port ${port}`);
});

Enjoy, as usual YMMV!


Posted by on 09 January 2023 | Comments (0) | categories: nginx WebDevelopment

Java style guide: Functions, Streams and Optionals


Over the years I developed a certain style developing in Java (and other languages) that made my code more readable. This isn't about the formatting style guide (Just stick to Google) or making code spotless, but about style.

Objectives

Code needs to be testable, easy to adopt and understandable when someone else, like your future self, tries to read it. Starting with SOLID principles, the guidelines help to get there. They work for any style of development, while I strongly recommend looking at TDD, Dave agrees, so does Allen. In no particular order, let's dig in.


Read more

Posted by on 08 January 2023 | Comments (0) | categories: Development Java

Please wait until that HTTP service is ready


Our brave new world of containers and microservices runs on a combination of YAML and shell scripts.
Getting them to run in the desired sequence can be a challenge.

When ready isn't ready

All container environments have a "depends" clause, so the container runtime can determine the correct startup sequence for the zoo of containers comprising the application to be launched. Each container will usually signal when it is ready.

However ready can mean different things to different applications.

In the container world it should be: the service is available.
However it could be: service was successfully started, but might be busy with house keeping. In the later case the start scripts of the dependent services need to do their own waiting

curl to the rescue

Curl is the Swiss Army Knive of network tools. It probably takes a lifetime to understand all its possibilities. Make sure you have the current version, but at least 7.52.0 from 2016.

Whipping up a little shell script provides what we need:

#!/bin/bash
# Wait for a http service to be available
export TARGET=http://baseservice.local/
curl -v \
     --retry 10 \
     --retry-delay 20 \
     --retry-connrefused \
     $TARGET

if [ "$?" -ne 0 ]; then
  echo "Failed to reach $TARGET"
  exit 1
else
  echo "$TARGET has replied!"
fi

Let's break it down:

  • export TARGET defines the URL you want to reach. Works for both http and https. Potentially works for other protocols supported by curl as well, but I haven't tested that (yet). You might swap it out for $1 to capture the first parameter called
  • \ denominates line continuation. The whole command needs to be on one line which is hard to read. \ tells Linux to treat the following line as belonging to the current one. On Windows you need to replace \ with ^ and alter the syntax according to Powershell
  • curl -v Verbose output, so the logs tell the full story. When you work with privat CA or self signed certificates (Don't do self signed, use your own CA), you need to add -k or --insecure to make this work
  • --retry 10 try the connection 10 times
  • --retry-delay 20 Wait 20 sec until the next try
  • --retry-connrefused retry even on "connection refused" responses. By default curl retries only http responses not in the 2xx range. Connection refused is HTTP's way telling you: there's nothing taking your call, which curl, without --retry-connrefused takes as final, fatal error and terminates retries. When you have curl v7.71.0 (from 2020 or later), you can use --retry-all-errors instead. The documentation warns This option is the "sledgehammer" of retrying. Use it wisely
  • if [ "$?" -ne 0 ]; then Commands usually return a numeric result. Anything other than 0 indicates a failure condition of some sort. When curl has exhausted the number of retries a value greater 0 will be returned. To capture the return value we us $?
  • return 1 We indicate that our script failed. So it can be called from other scripts

We could swap out $TARGET for $1 which makes the script callable handing over the target URL e.g waitforme.sh http://someURL

Beyond containers

The script is also useful in "classic" situations, where you start a service in the background (which by definition is asynchonous). Your local environment uses anything like:

  • ./startdb &
  • systemctl start myservice
  • domino[_container] start

You can use the curl retry for any automation. Adjust the wait time and the retry time to your needs.

As usual YMMV


Posted by on 02 January 2023 | Comments (0) | categories: Container Development K8S

On Technical Interviews


When the HCL Labs is hiring engineers, we conduct technical interviews. I had and have the priviledge to interview, hire and work with incredible talented individuals in the process. Lets look behind the curtain of the process.

What we are looking for

The technical interview is conducted by the interviewie's potential future colleagues. We try to gauge skills, conduct and team fit. Fit includes: does it improve diversity. You got a tatoo, tell me how you picked the design? You like tailored suits, who's your tailor? Punk is your thing, what's your favorite band? - You get the drift.

Can the candidate explain technology to peer engineers and understands the fundamental concepts? We found a surprising number of developers who don't know how their platform actually works. Topics like "The Node event loop" or "What does Maven do".

Knowing your environment is essential. Have you kept uptodate with latest changes, even your current role holds you back on applying them.

Does the candidate know about our "Buzzwords" YAGNI, DRY, KISS, SOLID or pattern

What we ask

Questions come from all areas: general project conducts, APIs, Code (mostly JS, TS & Java), HTTP and databases. We are looking not only for "I have x years experience", but also general awareness.

Example: you might not have a chance to work with a graph database, but you know they exist, what they are supposed to solve. etc.

A little taste:

  • How much Scrum does one entertain to be agile and what's the essence of agile
  • How to interact and resolve ambiguity in tasks and asks
  • On object oriented vs functional programming styles
  • Types of NoSQL databases and their uses
  • How do the joins (left, right, inner, outer) in an RDBMS work
  • Use cases for SQL and NoSQL
  • HTTP Verbs: POST vs PUT vs PATCH
  • REST experience including a tool or another
  • HTTP return codes. Everyone knows 200, some 201 and the difference to 200, and a few 418
  • CORS and CSP
  • Callbacks vs. Promises
  • let vs var vs const
  • function vs =>
  • Loops vs streams
  • Testing: TDD, UnitTests and Integration Tests
  • SQL Injection and prevention
  • Maven & NPM
  • Git & GitHub
  • What do you use NodeJS for

Coding challenges

Coding challenges in interviews are controversial and they are usefull.
We keep ours simple and explicitely allow questions, dialogue or googling for answers, with the warning label: you need to explain your solution, no matter what.

Being a little sneaky we keep the challenge requirements fuzzy to check if the candidate will ask for clarifications.

The exercises are light and not too far from practical use:

  • talk to an API
  • refactor code
  • fix performance

Work with us

HCL in Manila is looking for engineers of all levels and career stages. Come join us, some current openings from Linkedin (links are temporary, grab them while they are hot):


Posted by on 12 December 2022 | Comments (0) | categories: Java JavaScript Software

Async testing with vert.x


Test driven development gets interesting when actions might or might not complete somewhen in the future. Point in case: HTTP requests (fetch returns a promise)

vert.x and jUnit5

I frequently write little tests, that spin up an http sever before the test, run individual tests using a WebClient. All these operations run asynchronous, returning a vert.x Future (In JavaScript Land the equivalent would be a promise)

To make this pain free, vert.x provides a full integration into jUnit 5. Using the annotation @ExtendWith(VertxExtension.class) vert.x provides two injectable parameters: Vertx and VertxTestContext. Add the simple mental rule: All async operations need to interact with the VertxTestContext.


Read more

Posted by on 03 November 2022 | Comments (0) | categories: Java vert.x

Beyond Java 8: The Cool Stuff


Java is evolving rapidly. Henri Tremblay is walking through the cool stuff. Henri is, besides other activities a contributor to EasyMock

From Java 7 (2011) to Java 19 (2022)

Unless stated otherwise Java 17 (2021) LTS will work for the examples. Henri used a trading app to walk through upgrading steps.

Impressive: just moving from Java 8 to Java 19 shows double digit performance improvements

Some of the tips:

  • use the java.time.Clock object for anything date/time, Don't use Date. Inject the clock object.
  • use Comparators
  • use Files.lines to read by line
  • use the Maven deamon to speed compilation
  • Be careful with stream().findAny() - in most cases a Map performs way better
  • Use """ text blocks (why o why did Java not pick the tripple backtick)
  • Use Files.writeString if it is a String you write
  • Use jlink and jpackage for distribution
  • Use switch expressions which returns values (not convinced about that yet)
  • Use Records for immutable data
  • Use the new instanceof syntax
  • Use the new RandomGenerator for better performance
  • Sealed types allow tighter controls

A quite enjoyable session


Posted by on 20 October 2022 | Comments (0) | categories: Java

Test-Driven Development Is a Paradox


TDD is favored by very succesfull engineers as a means of accellerating software development by preserving quality.

Objections are plenty. The talk by Burk Hufnagel tries to address those.

The TDD Paradox

TDD requires you to write more code, so you can be done sooner. It's like learning to drive a car. Your first mile takes, give or take, 40-50h to complete: Driving lessons, driving tests, get the license issued before you drive. You can walk a lot more in 50 hours, you get the drift...

Looking deeper at "development" we see, it has multiple components:

  • understanding and clarifying the business requirements (business rather loosely defined)
  • write code
  • manual testing
  • debugging

TDD helps with the first and third component. A well defined test communicates the desired outcome and serves as a living specification. Having the machine running tests reduces the need for manual testing and frees up time to work on functionality

Test early, test often

TDD proposes to write tests first, not as an afterthought. The strongest new argument Burk proposed was: nobody wants to write a test for code freshly debugged and proven working (at that time). There's no immediate value in it. A test, initially failing, written upfront is a reassurance to be on the right track when going green.

Using the "test first" approach, tests turn into a design process, increasing the confidence in not only doing things right, but also doing the right thing. In other words: Tests are Specifications

Best results are to be had when working in small increments and resist the urge to "code for things you think are coming next" instead of "(just) make the test pass".

It's not a cure-it-all

TDD might not be a good fit:

  • no clear business requirements - you can't test when you don't know what you want
  • exploratory coding - trying stuff out

A common objection is "we have an existing non-TDD code base, we can't switch to TDD". Burk suggest to use "Defect Driven Development". Write a test that test for defect free behavior and then fix the defect. Your test harness will grow while defects shrink

Test long and prosper

The usual subjects for testing in Java are JUnit and Mockito. Burk introduced another, very promising tool: Spock. Spock is written in Groovy and allows to specify a test's purpose more expressive and concise than Junit's @DisplayName property. Luckily it isn't an exclusive-or question, Spock interacts with JUnit and Mockito quite nicely.

In conclusion

Burk made a good case for TDD, backed by experience and tooling, if in doubt visit his arguments.


Posted by on 19 October 2022 | Comments (0) | categories: Java

CI/CD, Github Actions, and GraalVM Native Image


GraalVM is a promising polyglot runtime for Java, JavaScript, Python, Ruby and WASM.
It can produce native images. Unsurprisingly with great powers come ... lots of things to learn.

GraalVM Native Image Benefits

A native image has a number of benefits over an JVM based application, mainly owed to the extensive analysis done at build time.
Some of advantages mentioned:

  • Runtime speed ~ 25% above the same code running on an JVM
  • Ahead of Time compilation can lead to up to 9 times less memory consumption than running on an JVM
  • Code executes faster with a smaller memory footprint than comparable Go applications ( but still is bigger/slower than C by a whisk)
  • Warmup time got moved to build time, improving startup time

Building the image

Build has been greatly improved with tooling available:

  • a Maven plugin (also available for Gradle), eliminating the need of extensive commandline foo.
  • a GitHub action to install the GraalVM tooing in a pipeline
  • reduced memory consumption, so most apps will fit into the GitHub build container limit of 7GB memory
  • option for a non-optimized fast build, useful for testing correctness

Thoughts and conclusions

A big step in the right direction making native images more accessible. There's still room for improvements, like caching the build images and more samples. What Oracle solved nicely: If you are a GraalVM Enterprise customer, accessing GraalVM Enterprise edition is done using an access token, fully compatible to any build system. No messing around with downloads and manual installations required.


Posted by on 18 October 2022 | Comments (0) | categories: Java

Streamlining Lage-scale Java Development using Error Prone


It has been a while since I attended an in-person conference. Finally this October I managed to attend JavaOne in Las Vegas. I shall report on interesting sessions.

The first one is by Sander Mak on a topic near and dear to me: Code Quality

Another tool in the box

Error Prone, courtesy of Google, supplements tools like Sonar, Spotbugs or Checkstyle. Other than these tools it is a compiler plugin with > 500 checks.

It can be configured to fix a set of identified errors automatically in code, inline or as patch file. Sander used a few simple demos to drive home what is possible using the tooling. Besides the ckecks that are available out-of-the-box one can implement extensions. One interesting contribution comes from the popular mocking framework Mockito, extending the checks to your use of mocking.

Extending error-prone is possible using two complementary approaches:

  • writing your own extensions (extending BugChecker), dealing with Java's AST and all its power
  • using the included ReFaster templates that allow for powerfull pattern based code upgrades

Use cases

Just a few samples from the session:

  • implement consistency in coding style (not to confuse with code style)
  • reduce visible clutter like final before variables by using @var to denote mutable variables and consider immutable the default
  • ease the transition to higher JVM versions, e.g replace !Optional.isPresent() with Optional.isEmpty() (JDK11)
  • verify logging uses {} instead of %s
  • fail builds that violate rules and conventions

My impression

Error-prone seems to be an important tool in toolbox to produce a well groomed Java code base. I loved Sander's progression from a simple introduction example to the more complex use cases that help his company deliver (pun intended). Error-prone isn't for the faint of heart, but a component that belongs into any code base of size. The session inspired me to improve our Java tooling, well done!

The Steve pun, one last thing, was the icing on the cake

More content by Sander


Posted by on 18 October 2022 | Comments (0) | categories: Java

Calling a vert.x async method from a sync method


Made popular by NodeJS and ES6 asynchronous programming promises (pun intended) better throughput and resource usage by entertaining an Event Loop. In Java land vert.x implements exactly this approach and has proven its mettle, being the foundation of Quarkus

Your legacy app doesn't magically convert

When you start a new vert.x project using the App Generator, everything is asynchronous from the beginning. Snippets of synchronous code (a.k.a blocking code), you might need to maintain, can be wrapped into executeBlocking and handled in their own thread.

However when you are about to convert a synchronous application, e.g. a servlet to asynchronous and you can't finish in a sprint/session, things become interesting. The doGet method is synchronous by nature. There are a few steps that need to be accomplished:

  • Have vert.x running in its own thread. You can't start it on the main thread with its blocking operations
  • Have a method that returns a vert.x Future
  • Convert that into a Java CompletableFuture

Let's have a look at the moving parts:


Read more

Posted by on 03 August 2022 | Comments (0) | categories: Java vert.x