Current state of mind

/ Comments

Middleware

  • ExpressJS
  • Loopback (based on ExpressJS)
At one point during my 5-years of working with NodeJS, my go-to solution for REST middleware is Loopback, due to its apparent simplicity. We just had to run a command, follow along the wizard, and the model will be created for us, as well as the REST endpoints.

But being a framework, I learned that it's harder to make your project developed using Loopback behaves in a certain ways (according to our own 'opinion'). We had to dig through its layers and find a hook to tuck your code on. More often than we would run into 'clash of opionion', loopback way or highway, which is understandable.

So with those experience and insights gained, I decided to go back to the barebones, ExpressJS. I can make whatever middleware I deem right, and incorporate them to the Express framework in a very simple, straightforward way.

Actually, the point when I decided for "let's just go back with the basics", was when I wanted to incorporate Keycloak authentication. Also, when I wanted to incorporate Event-Sourcing/CQRS approach.

Event-Sourcing/CQRS

In the last couple of years I have made myself a fan or believer of ES/CQRS. Among several tenets in that approach, I like the idea of having audit-trails central to your system; not just afterthought / extension. This is drawn from my experience back then developing system (at that time was an inventory-management system), where we had hard-time understanding what had happened before (when bad thing happen). This was because of the design of the audit-trail was inadequate, and worse yet, not given its proper importance (e.g.: in a transaction, when the write to audit table fails, we didn't roll back the operation. Yes, sinful).

Analyzing what happened was one thing. After hours of (trying to) arrive at the right conclusion, we hit another challenge: how to fix the system? how to bring the data back to the correct state according to the system's world-view? We ended-up performing direct manipulations to the database. Really... cringeful. I don't want to commit the same sins again. So, I decided to adopt event-sourcing and CQRS.

I like it because it promotes:
  • Append only database operation; just append new event to the end of the trail. Many of major headaches involving database transactions are gone.
  • Cancellation _probably_ is easier. It employs "compensating-event", instead of rollback. It resembles real life more closely.
  • Adapting to new business requirement _probably_ is easier. A proper ES-framework normally support dealing with different versions of events, simply by routing events with certain attributes (e.g.: version) to associated process-manager. Combined with CQRS, we can build and rebuild as many read-models as required by business requirements. The need for complex database-migrations can be eliminated.
  • Communicating with less-technical stakeholders _probably_ is easier. From my experience they are not interested in discussions involving entity models, entity relationships, normalization; understandably so. Granted, their domain is functional specification, and our domain (as software developer) is translating that to the things mentioned previously. But there's a gap (of communication) there, and quite often that gaps leads to inadequate implementation. There's a technique called event-storming for ES-based system, which is more inclusive; it can & should involve both stake holders. Common language is spoken (in term of events and commands), which eventually will get everyone understand the domain model better.
To all the points I listed I put emphazis on _probably_. Because I haven't really used it in real project. So, I might be mistaken. Everything sounds too good maybe. Maybe it's not exactly a walk in the park as it may sound. I can understand ES/CQRS might be a little more complex to implement, for the following reasons:
  • It tends to be asynchronous. If we want to build web UI that interacts with ES system, probably you will change the way we react to request. Normally we would do: 
    1. take the request
    2. process in the request handler
    3. commit the transaction to the database
    4. query the database
    5. return response. Now, with ES/CQRS, where the web UI is only channel for posting command/event, and the logic is background processed, there is no guarantee that the read database already catches up to the latest command/event you just pushed. There are options though, pick one based on scenario / business requirement: poll, websocket, or adjust your UI-flow to account for the fact that the operation is asynchronous. 
  • There is possibility event processor drops / crashes in the middle of the way, and people are concerned if it will lose events. I learned that push is better than pull. I mean, that's what event-database is for, to deal with this situation. The event processor is supposed to pull the events from the event database (as opposed to listening to message queues for events). When it goes back up, it can pull events starting from latest milestone recorded. 
I would argue those two points are not really problem. It's not even that hard. Background processing, asynchronicity, etc..., it's not uncommon in non-trivial systems. ES/CQRS approach only makes it explicit, and forces us to embrace it from the beginning; which is a good thing.

So, adios CRUD? Not really. The read-side can stay in the CRUD style. Given it's just read, queries, I probably will go back using tools like Loopback to implement that side, and avoid writing too many lines of code.

I have investigated ES/CQRS in several different languages: especially NodeJS, Java, Scala. In Java there is a professional-looking framework called Axon, in Scala I have played a bit with Lagom (also available in Java), in NodeJS I used an open-source named node-cqrs-domain (and its sister projects). Granted the one I used to build my proof-of-concept in NodeJS doesn't look enticing (its webpage doesn't look enterprisey). There's other called Wolkenkit, that has a nice website and newbie guidances. But..., I have done code-dive in node-cqrs-domain project, and I'm convinced that one actually is production ready. At least I know it handles important aspects like message deduplication, aggregate locking, etc. I haven't looked into Wolkenkit's code (it's not 100% open-source), and their webpage does not have any mention of those things.

How does it have to do with my decision to drop Loopback and go back to ExpressJS? I just don't see the need to follow th REST-ful approach on the API for the frontend. On the write side, we only need to have one endpoint, to accept command. Just one endpoint, a POST endpoint. So, loopback with its automatic REST endpoints generation loses its selling-point. So, these days, my web-api is more like RPC, verb (command name) instead of nouns.

Keycloak

Also, in the last one year I have special interest in Identity & Access Management. With OAuth2.0 and then OpenID connect, single sign-on, user federation, account linkings, etc. They are all relevant and recurring theme in integration between systems, inside organizations among microservices, or with external organizations.

When I first learned the concepts of OAuth2.0, I implemented it following https://nordicapis.com/how-to-control-user-identity-within-microservices/ . I used combination of Kong API gateway (with its freely-available OAuth2.0 plugin) that performs request interceptions (to check for token), and provides the four OAuth2.0 flows..., and a custom-made authentication-server writtem in NodeJS that receives redirects from Kong, and generate the internal token (the token known by the final endpoint, the upstream, our API).

It worked..., the learning worth the effort, but... I realize IAM is a serious understanding, and that would be a waste of engineering time if we try to build it in-house. So I looked for some open-source solutions, and decided to use Keycloak from RedHat, which in my opinion is the most feature complete.

How does it have to do with my decision to drop Loopback and go back to ExpressJS? Keycloak provides libraries in different langiuage to be used in your API project. For NodeJS, it provide express middleware. Loopback was / is based on Express, so in theory it the library provided by Keycloak should work, right? I tried it, and gave up in the middle of the way. I didn't see any value digging that deep into the framework just to get this keycloak middleware, which works perfectly in plain ExpressJS, to work.

NextJS

In the two sections above we talked about the middle part of the solution. For the web frontend, my current choice of framework is NextJS. It's a combination of ExpressJS (or more accurately Connect-compatible), and ReactJS. Yes, it provides all the production-grade tool and flow to have your frontend code properly chunked (without battling with webpack configurations); it's already included and just works. It also provides server-side rendering and static-optimization whenever possible. All nice.

But what really pique my interest at the beginning was: the server-side render. It's like going back to old Java Servlet days. I have went before to the other extreme, by going full SPA, and got burned. Performance problems, state management problem (by keeping all state in the client, and it gets lost when user do refresh, etc). So I figured something that stays in the middle should be the answer, and NextJS it is.

It also has to do with my interest to secure the system using keycloak, and prefers a more secure "confidential" access type. Meaning, it has to be controlled on the middlware. It just fits perfectly with NextJS model, where each route / page is essentially an express route, which we can attach the keycloak middleware to.

Some links

Links to some videos I have created relevant to this entry: