Lack of Imagination

Time-Travel Debugging: Replaying Production Bugs Locally

Tue, 24 Feb 2026 00:00:00 +0000

We’ve all had that sinking feeling. There are multiple crash reports from production. We have the exact input parameters that caused the failures. We have the stack traces. Yet, when we run the code locally, it works perfectly.

We know where it broke, but we can’t see why. Was it a race condition? Did a database read return stale data that has since been overwritten? To find the cause, we have to mentally reconstruct the state of the world as it existed milliseconds before the crash. Welcome to debugging hell.

If we could simply rewind time and watch the code execute exactly as it did for those failed requests, life would be a lot easier.

In Testing Side Effects Without the Side Effects, we explored a JavaScript Effect System where business logic doesn’t execute actions directly. Instead, it returns a description of what it intends to do in the form of a simple Command object. For example:

const validatePromo = (cartContents) => {
    // Define the side effect, but don't run it yet
    const cmdValidatePromo = () => db.checkPromo(cartContents);

    // Define what happens with the result
    const next = (promo) =>
        (promo.isValid ? Success({...cartContents, promo}) : Failure('Invalid promo'));

    return Command(cmdValidatePromo, next);
};

outputs:

{
    type: 'Command',
    cmd: [Function: cmdValidatePromo], // The command waiting to be executed
    next: [Function: next]             // What to do after the command finishes
}

We often compose multiple commands in a pipeline to make the most of our Effect system:

const checkoutFlow = (cartSummary) =>
    effectPipe(
        fetchCart,
        validatePromo,
        (cartContents) => chargeCreditCard(cartSummary, cartContents)
    )(cartSummary);

Our effect pipeline handles the Success and Failure cases automatically. If a function returns Success, the subsequent function in line will be called. In the case of a Failure, the pipeline terminates.

The series of Command objects generated by the pipeline is then run by an interpreter using runEffect(checkoutFlow(cartSummary)). Because our business logic consists of pure functions that interact with the world only through data, we can record those interactions simply by adding a few hooks for services like OpenTelemetry. And if we can record them, we can replay them deterministically. Best of all, there’s no need to mock a single database or external service.

When a crash happens, we don’t just get an error message. We get a crash log containing the initial input and the execution trace complete with all outputs.

In the following simplified trace generated by our workflow, a customer uses a 100% off promo code. The business logic calculates the total as $0.00 and attempts to pass it to the payment gateway, but the payment gateway rejects the API call because the minimum charge amount is $0.50, causing a 500 Internal Server Error:

const traceLog = {
  "flowName": "checkout",
  "initialInput": {
    "userId": "some_user_id",
    "cartId": "cart_abc123",
    "promoCode": "FREE_YEAR_VIP"
  },
  "trace": [
    {
      "command": "cmdFetchCart",
      "result": {
        "cartId": "cart_abc123",
        "items": ["annual_subscription"],
        "totalAmount": "120.00"
      }
    },
    {
      "command": "cmdValidatePromo",
      "result": {
        "cartId": "cart_abc123",
        "items": ["annual_subscription"],
        "totalAmount": "120.00",
        "isValid": true,
        "discountType": "%",
        "discountValue": 100
      }
    },
    {
      "command": "cmdChargeCreditCard",
      "result": {
        "error": {
          "code": "invalid_amount",
          "message": "Amount must be non-zero."
        }
      }
    }
  ]
};

Compared to the cryptic stack traces common in imperative code, this execution trace makes the source of the error immediately obvious.

We can even go ahead and write a quick time-travel function like the one below to replay any execution trace locally, complete with built-in support for detecting time paradoxes!

function timeTravel(workflowFn, traceLog) {
    const { initialInput, trace, flowName } = traceLog;
    const format = (v) => JSON.stringify(v, null, 2);

    let currentStep = workflowFn(initialInput);
    let traceIndex = 0;
    console.log(`Replay started with initial input: ${format(initialInput)}`);
    while (true) {
        const stepName = currentStep.type === 'Command' ? currentStep.cmd.name || 'anonymous' : currentStep.type;

        if (currentStep.type === 'Success' || currentStep.type === 'Failure') {
            console.log(`Replay Finished with state: ${currentStep.type}`);
            console.log(
                currentStep.type === 'Failure'
                    ? `Error: ${format(currentStep.error)}`
                    : `Result: ${format(currentStep.value)}`
            );
            break;
        }

        if (currentStep.type === 'Command') {
            const recordedEvent = trace[traceIndex];
            if (!recordedEvent) {
                throw new Error(`Trace ended prematurely at step ${traceIndex}. Workflow expected command: ${stepName}`);
            }
            if (recordedEvent.command !== stepName) {
                throw new Error(
                    `Time paradox detected! Workflow asked for '${stepName}', but trace recorded '${recordedEvent.command}'`
                );
            }
            console.log(`Step ${++traceIndex}: ${recordedEvent.command} returned ${format(recordedEvent.result)}`);
            currentStep = currentStep.next(recordedEvent.result);
        }
    }
}

When we run timeTravel(checkoutFlow, traceLog), it will actually exercise our checkout workflow, and produce the following output. With that, we’ve successfully executed a production execution trace locally, all without touching any database or external service:

Replay started with initial input: {
  "userId": "some_user_id",
  "cartId": "cart_abc123",
  "promoCode": "FREE_YEAR_VIP"
}
Step 1: cmdFetchCart returned {
  "cartId": "cart_abc123",
  "items": ["annual_subscription"],
  "totalAmount": "120.00"
}
Step 2: cmdValidatePromo returned {
  "cartId": "cart_abc123",
  "items": ["annual_subscription"],
  "totalAmount": "120.00",
  "isValid": true,
  "discountType": "%",
  "discountValue": 100
}
Step 3: cmdChargeCreditCard returned {
  "error": {
    "code": "invalid_amount",
    "message": "Amount must be non-zero."
  }
}
Replay Finished with state: Failure
Error: {
  "code": "invalid_amount",
  "message": "Amount must be non-zero."
}

Time-travel debugging might sound like a complex feature reserved for heavy-duty enterprise tools, but it fundamentally comes down to architectural design; it takes less than 100 lines of code to implement, and that figure includes our Effect System.

Because every interaction passes through runEffect, we can easily implement a redaction layer to scrub personally identifiable information, like credit card numbers or emails, before they ever hit the trace log.

By pushing side effects to the edges and keeping our core logic pure, we gain a deterministic and secure execution trace. As a result, debugging shifts from guessing what might have happened to watching exactly what did happen, all without compromising user privacy.

GitHub Repository: pure-effect

Discussed on:

Hacker News

Testing Side Effects Without the Side Effects

Sat, 27 Dec 2025 00:00:00 +0000

In Managing Side Effects, we built a tiny JavaScript Effect System that treats side effects like database calls as data descriptions rather than executable promises.

In a typical imperative application, business logic and side effects are inextricably linked. For example, when you write await db.checkInventory(...), the runtime immediately reaches out to the database.

Our Effect System works differently. Instead of performing the action, our functions return a description of the action. When our code needs to check inventory, it doesn’t call the database; it returns a plain object instead, which will be executed later by an interpreter. For example:

function checkInventory(order) {
    // Define the side effect, but don't run it yet
    const cmdCheckInventory = () => db.checkInventory(order);
    
    // Define what happens with the result
    const next = (exists) => exists ? Success() : Failure('Out of stock');
    
    return Command(cmdCheckInventory, next);
}

outputs:

{
    type: 'Command',
    cmd: [Function: cmdCheckInventory], // The command waiting to be executed
    next: [Function: next]              // What to do after the command finishes
}

This is all well and good, but this pattern really shines when we have multiple pure functions we compose together. By using an effect pipeline to pass the output of one function to the subsequent one, we generate a syntax tree: a series of Command, Success, or Failure objects.

const processOrderFlow = (order) =>
    effectPipe(
        validateOrder,
        checkInventory,
        () => chargeCreditCard(order),
        (paymentId) => completeOrder(order, paymentId)
    )(order);

Notice that there’s no logic in the above declarative code. The logic is hidden inside the functions, where it belongs, and we can immediately see the steps required to place an order.

Look Ma, No Mocks!

In a standard imperative codebase, testing the order flow would require mocking the inventory database and sandboxing the payment gateway API to avoid real charges.

With our Effect System, processOrderFlow is a pure function. It returns a syntax tree that represents the order processing, without actually doing it. We can write a unit test that walks through the logic step by step:

const order = [{ itemId: 'sku-123', quantity: 1, amount: '99.00'}];

const step1 = processOrderFlow(order);

// Verify the intent to check inventory
assert.equal(step1.type, 'Command');
assert.equal(step1.cmd.name, 'cmdCheckInventory');

// Simulate a successful inventory check by calling next with 'true'
const step2 = step1.next(true);

// Verify the intent to charge the card
assert.equal(step2.type, 'Command');
assert.equal(step2.cmd.name, 'cmdChargeCreditCard');

// Simulate a successful payment
const step3 = step2.next('payment-id-123'); 

// Verify the intent to complete the order
assert.equal(step3.type, 'Command');
assert.equal(step3.cmd.name, 'cmdCompleteOrder');

There are three major benefits to this approach:

Speed: Tests run almost instantly since there are no network requests or database connections.
Determinism: We aren’t testing database operations, which can be tested separately by calling await runEffect(processOrderFlow(order)) against a test database. We are testing the business logic instead.
Safety: There is zero risk of accidentally charging a real credit card during a test run.

By separating the description of work from the execution of work, we’ve turned complex, async integration tests into simple, synchronous unit tests, and we’ve effectively tested side effects without invoking any.

An AI-Native Architecture

One of the arguments against functional patterns like our Effect System is the overhead of writing command objects and wrapper functions. However, I think it’s safe to say this architecture is effectively AI-native.

Because the business logic is expressed as data rather than imperative code with hidden state, LLMs can generate, refactor, and test these pipelines with great accuracy. An AI coding assistant doesn’t need to understand a complex database mocking setup to write a test; it just needs to predict the next object in the sequence. I’d argue that choosing an architecture that is easy for both humans and machines to reason about is a competitive advantage.

GitHub Repository: pure-effect

Managing Side Effects: A JavaScript Effect System in 30 Lines or Less

Mon, 24 Nov 2025 00:00:00 +0000

If you look at the source code of a typical application, you will likely find business logic tangled with database calls, HTTP requests firing off in the middle of validation rules, and try/catch blocks sprinkled here and there.

The biggest casualty of this coupling is testability. How do you test a function that calculates a user’s discount? That’s easy, right?

But how do you test a function that calculates a discount, looks up the user in a database to check their loyalty status, and then emails them a coupon? You can’t just run the function. You have to spin up a test database, or utilize a mocking library to intercept calls to db.findUser. Your unit tests stop testing your logic and start testing your mocks. Soon, you find yourself spending more time configuring the test environment than writing the code itself.

Fortunately, there is another way. It may feel strange at first, but the idea is deceptively simple:

Don’t do the work right away. Describe the work first.

Think about the difference between cooking a meal and writing a recipe.

Imperative: You execute the steps one by one: Go to the kitchen. Chop the onions. Stop if you cry. Turn on the stove.
Declarative: You write down the recipe instead: “Step 1: Chop onions. Step 2: Saute.” You can hand the recipe to someone else. You can also analyze it to see if it contains allergens (bugs) without cooking it.

By the end of this article, we will have a tiny but functional “Effect System” in JavaScript. We will stop writing functions that execute side effects and start writing functions that return descriptions of them instead.

But first, let’s look at a typical example of imperative code that executes the following user registration steps:

Validate the input.
Check the database to see if the email exists.
If it’s new, hash the password and save the user.

// ...

async function registerUser(input) {
    try {
        const { email, password } = input;
        if (!email?.includes('@')) {
            throw new Error('Invalid email format.');
        }
        if (password?.length < 8) {
            throw new Error('Password too short.');
        }

        const foundUser = await db.findUserByEmail(email);
        if (foundUser) {
            throw new Error('Email already in use.');
        }

        const userToSave = { email, password: hashPassword(password) };
        const savedUser = await db.saveUser(userToSave);

        console.log('User Created:', savedUser);
        return savedUser;
    } catch (error) {
        console.error('Registration error:', error);
        return { error: error.message };
    }
}

The code above is perfectly readable, but it has a number of hidden costs. You can’t test the logic without mocking the database calls. And if you don’t mock, you can’t call registerUser with the same input twice. If you do, it may fail the second time. Finally, logic jumps around via throw/catch, which introduces invisible control flow. Because any function could potentially throw, you can never look at a block of code and be 100% sure that the next line will actually execute.

The Effect System

Instead of doing the work right away, the Effect System returns an object describing the work, which will be evaluated later.

We start by defining three basic objects to represent the state of our program:

Success: The previous step worked, and here is the result (value).
Failure: Something went wrong (error).
Command: We store the async side effect in cmd as a function, but we don’t run it yet. If cmd returns Success, the function in next will be run afterwards.

const Success = (value) => ({ type: 'Success', value });
const Failure = (error) => ({ type: 'Failure', error });
const Command = (cmd, next) => ({ type: 'Command', cmd, next });

Next, we need a combinator function called chain that stitches these effects together. Its job is to take an existing effect and connect it to the next function (fn) in the pipeline.

const chain = (effect, fn) => {
    switch (effect.type) {
        case 'Success':
            // Pass the success value to function fn
            return fn(effect.value);
        case 'Failure':
            // Short-circuit on error
            return effect;  
        case 'Command':
            // Create a new command that chains the command's result to fn recursively
            const next = (result) => chain(effect.next(result), fn);
            return Command(effect.cmd, next);
    }
};

We will also need a simple effectPipe helper. This accepts a list of functions and runs them one by one, passing the output of the previous function into the next, but only if the previous one succeeded (thanks to chain).

const effectPipe = (...fns) => {
    return (start) => fns.reduce(chain, Success(start));
};

To wrap it up, we need one last function: the interpreter. So far, we haven’t run anything; we have just built a big object describing what we want to do. Our runEffect function is the only place where async/await lives. It loops through our commands and executes them one by one:

async function runEffect(effect) {
    while (effect.type === 'Command') {
        try {
            effect = effect.next(await effect.cmd());
        } catch (e) {
            return Failure(e);
        }
    }
    return effect;
}

And that’s it! If you don’t quite understand how these all work, it’s ok. Let’s refactor the original registerUser function using our Effect system, and see how the resulting code will look like.

From Imperative to Declarative Code

We first put the validation logic in its own function and return Success or Failure objects instead of throwing exceptions:

function validateRegistration(input) {
    const { email, password } = input;
    if (!email?.includes('@')) {
        return Failure('Invalid email format.');
    }
    if (password?.length < 8) {
        return Failure('Password must be at least 8 characters long.');
    }
    return Success(input);
}

Next, we handle the database check. We wrap the db.findUserByEmail execution in a function and pass it as the cmd. In the next argument, we define a function that will receive the foundUser (in the future) and wrap it in a Success object. Note that nothing is run here. We are simply describing the command and what will happen next once it is run.

function findUserByEmail(email) {
    const cmdFindUser = () => db.findUserByEmail(email);
    const next = (foundUser) => Success(foundUser);
    return Command(cmdFindUser, next);
}

We put the email availability check in its own function:

function ensureEmailIsAvailable(foundUser) {
    return foundUser ? Failure('Email already in use.') : Success(true);
}

Finally, the save operation. After we hash the password, just like the read operation earlier, we wrap the db.saveUser execution in a function and pass it as the cmd. In the next argument, we define a function that will receive the savedUser and wrap it in a Success object. Once again nothing is run here.

function saveUser(input) {
    const { email, password } = input;
    const userToSave = { email, password: hashPassword(password) };
    
    const cmdSaveUser = () => db.saveUser(userToSave);
    const next = (savedUser) => Success(savedUser);
    return Command(cmdSaveUser, next);
}

Now, we use effectPipe to stitch these steps together into a single “recipe”, so to speak. In the pipeline below, we use arrow functions () => ... to capture the original input variable. This ensures we can access the initial data even if the previous step returned a different result (like true).

const registerUserFlow = (input) =>
    effectPipe(
        validateRegistration,
        () => findUserByEmail(input.email), 
        ensureEmailIsAvailable,
        () => saveUser(input) 
    )(input);

Finally, we run our program:

async function registerUser(input) {
    return await runEffect(registerUserFlow(input));
}

OK, let’s slow down here a bit and reflect on what we have done so far. We haven’t just moved things around the code. We have fundamentally changed its architecture. By decoupling the description of the workflow from its execution, we gain two distinct advantages that are often difficult to achieve with standard async/await.

Testing without Mocks

In our new version, registerUserFlow is a pure function. It doesn’t touch the database; it returns a data structure representing the database interaction. This means we can test our business logic without spinning up a test database or using complex mocking libraries.

Testing a failure case is as simple as checking the Failure condition:

const input = { email: 'bad-email', password: '123' };
const result = registerUserFlow(input);
assert.deepEqual(result, Failure('Invalid email format.'));

We can even inspect the intent of the code. We can verify that the code intends to look up a user, without actually doing it.

const input = { email: 'test@test.com', password: 'password123' };
const step1 = registerUserFlow(input);
assert.equal(step1.type, 'Command');
assert.equal(step1.cmd.name, 'cmdFindUser');

const step2 = step1.next(null);
assert.equal(step2.type, 'Command');
assert.equal(step2.cmd.name, 'cmdSaveUser');

Free Logging & Profiling

Because every database call and API request must flow through the runEffect interpreter, we can handle cross-cutting concerns like logging, performance profiling, and error reporting in exactly one place.

Instead of polluting the business logic with scattered console.log statements, we can simply plug these features into the engine. We update the interpreter once, and suddenly the entire application gets smarter.

Here’s a quick example of how to log every async operation and measure how long each operation takes:

async function runEffect(effect) {
    console.log('--- Starting Pipeline ---');
    while (effect.type === 'Command') {
        const start = performance.now();
        try {
            const result = await effect.cmd();
            const duration = (performance.now() - start).toFixed(1);
            console.log(`✅ [${effect.cmd.name}] completed in ${duration} ms`);

            effect = effect.next(result);
        } catch (e) {
            console.error(`❌ [${effect.cmd.name}] failed:`, e);
            return Failure(e);
        }
    }
    console.log('--- Pipeline Finished ---');
    return effect;
}

Now, when we run our unchanged registerUser code, we get this output:

--- Starting Pipeline ---
✅ [cmdFindUser] completed in 12.2 ms
✅ [cmdSaveUser] completed in 32.5 ms
--- Pipeline Finished ---

The M Word

Those who have a background in functional programming languages like Haskell will notice that we have created not one, but two Monads in just 30 lines of code:

The Either Monad (Error Handling): By distinguishing between Success and Failure, our pipeline automatically manages the “Sad Path.”
The Free Monad (Side Effects): By representing our side effects as data objects (Command) rather than opaque functions, we built a transparent syntax tree of our program. This is what allows us to inspect, test, and interpret our code without actually running it.

Monad is a functional programming pattern, and is often treated as a scary word, associated with complex math. But at its core, a Monad is a method to put a value in a container along with some metadata to handle complexity.

The Functional Core and the Imperative Shell

What we have built here is essentially a micro-implementation of a classic architectural pattern known as Functional Core, Imperative Shell.

The goal of this architecture is to push all side effects to the boundaries of our application, leaving the center pure and predictable.

The Functional Core: In our example, registerUserFlow represents the core. It contains all our important business rules, validation logic, and decision trees. It is 100% pure, has no dependencies, and is trivial to test because it deals only with data.
The Imperative Shell: The runEffect function acts as the engine of the shell. Along with our database adapters and entry points, it handles the chaotic reality of the outside world.

By strictly separating these two worlds, we minimize the surface area for bugs. The logic that defines what the application does is safe inside the core, while the logic that defines how to execute it is isolated in the shell.

GitHub Repository: pure-effect

Little Big Line Array Build

Tue, 17 Jun 2025 00:00:00 +0000

After building quite a few subwoofers and 2-way speakers over the years, I was looking for something different, and line arrays fit the bill nicely. They use a large number of small-diameter drivers –sometimes as many as hundreds– and have some intriguing characteristics such as little loss of volume as you get further away, low distortion, and very slim enclosures.

Designing one isn’t simple, though, since drivers interfere with each other, causing anomalies in both frequency response and sound dispersion. To minimize destructive interference and use it to our advantage, it’s vital to minimize driver-to-driver spacing.

When making line array speakers, some companies use a large number of 1" tweeters coupled with several small mid-bass drivers. Unfortunately, the distance between two 1" tweeters can’t be made much smaller than 40 mm, and they don’t really go much below 2 kHz without a waveguide anyway. On the other hand, while 2" full-range drivers result in a slightly wider driver-to-driver spacing of around 60 mm, this is a small price to pay given their ability to reproduce frequencies down to 150 Hz, which is particularly useful for a home theater surround speaker use case I have in mind.

I’ve settled on using eight 2" Lavoce FSN020.71F drivers, featuring neodymium magnets and 3/4" voice coils. I could have used more, but in a typical sized room, they should be more than enough.

The drivers will be arranged in four banks, with all but the center bank’s high-frequency responses shaped by 2nd-order low-pass filters. This aims to reduce high-frequency interference and achieve very narrow vertical directivity, providing near-constant volume regardless of listening position (within reason, of course). This is especially useful if the speakers are angled towards the furthest listeners. That way, you wouldn’t be blowing the ears of people close to the speakers, something that’s only possible with line arrays. More on this later.

A line array using 2" drivers will never produce enough bass by itself, so it’s only natural to couple it with a subwoofer. And since my latest subwoofer build had generous rounded edges, I thought adding curves to the slim line array enclosure would not only make it look less like a column, but also help it match the subwoofer’s appearance better.

The cabinet walls are made from nine layers of 18 mm MDF.

Curves certainly look nice, but they aren’t easy to make. One method for creating a curved enclosure is to form it from layers cut from a sheet of wood. I’m not very fond of this technique, as a lot of material goes to waste, but in this case, the enclosure will be very slim, resulting in little waste. Overall, I’ve cut nine layers from 18 mm MDF on a CNC router. Some layers have inner lips to mount the speaker baffle and the rear panel, plus one in the middle for central support. I also applied 1/2" roundovers to the speaker cutouts.

Full-range drivers rear-mounted to the baffle.

Eight drivers mean a lot of unsightly screws, which led me to mount the drivers from the back. However, that requires either a removable rear panel or baffle (or both) to access the drivers in the future. Since space is tight, I decided to make both removable by using threaded inserts at the back of the baffle and long screws, inserted from the rear panel, to attach the front and rear together.

Baffle front view.

I painted the baffle and rear panel black and the side walls, which form the shell, white for what I think is a nice contrasting look.

Baffle and rear panel, inside view.

There are four banks of drivers, each with two 8-ohm drivers connected in series. The banks are then connected in parallel, resulting in a nominal load of 4 ohms. However, two filters are necessary to flatten the frequency response for all drivers: a low-shelf cut and a notch. These bring the nominal impedance up to 6 ohms.

From top to bottom, drivers 1 and 8, 2 and 7, 3 and 6, and finally 4 and 5 form each bank. High frequencies are increasingly shaded for each bank as they move further from the center, which helps reduce destructive interference and control vertical directivity.

There are two very short ports made from 45 mm inner diameter PVC pipe to extend the low-frequency response.

Rear of the cabinet on a scrap wood stand.

The speaker was originally designed for wall mounting, but I made a quick stand from scrap wood, incorporating the wall mount, so that it could stand on its own.

Line array speaker placed on a subwoofer.

So, how’s the sound? The speaker needs a subwoofer to fill in the missing low end, but once I’d dialed in the subwoofer integration, I was greeted with clean, smooth sound, particularly in the mid-range, and the speaker can go loud for sure. At 95 dB/1m, there was no noticeable distortion thanks to the eight drivers working in tandem. As for the controlled vertical directivity, you can tell the upper frequencies decrease in volume when your ears are not on the speaker’s center axis. The effect is subtle but quite noticeable, especially when you go increasingly off-axis.

Measured outside at ~2 meters from the ground.

It’s difficult to make quasi-anechoic measurements of the line array due to the number of drivers involved. So, the speaker was measured outside, roughly 2 meters from the ground, to minimize reflections.

On-axis frequency response measured outside at 1 meter (gated at 10 ms).

The frequency response is mostly within ±3 dB between 150 Hz and 17 kHz. The critical 1 to 5 kHz region, where the ear is most sensitive, is quite flat with only a ±1.5 dB variation. There are some response anomalies starting at 10 kHz since the 2" driver isn’t really suited to reproducing these frequencies well, and there are also some destructive effects from nearby drivers. Fortunately, there’s very little content at these frequencies.

Horizontal directivity is wide up until 5 kHz, but narrows somewhat after that.

Horizontal directivity is wide at ±60 degrees nominal up until 5 kHz, but it starts to narrow somewhat after that.

Vertical directivity is very narrow as intended and controlled well for the most part.

Vertical directivity is near ruler flat at ±15 degrees nominal down to 1 kHz and gradually gets wider below that. This extremely well-controlled and narrow directivity –if I may say so myself– allows precise steering of sound, which is especially well suited for applications such as surround sound speakers in a home theater, where some listeners are closer to a speaker than others.

The nominal impedance is 6 ohms.

The impedance and phase plot shows no major issues other than a few minor resonances. The minimum impedance stays above 5 ohms across the band. The bump centered at around 2.3 kHz is due to the notch filter that flattens the response.

Finally, a complete build video is available below:

The Innocent Loop

Wed, 30 Apr 2025 00:00:00 +0000

Loops are so common in programming that we often don’t give them a second thought, but a buggy loop has potentially infinite destructive power as it mindlessly repeats its task however many times it’s asked.

Back in the good or bad old days, depending on your perspective, the for loop with its manual index management was often the only game in town, with the occasional while loop thrown in for good measure. It was our responsibility to make sure those loops actually terminated by using explicit control conditions. These days, though, with functional programming principles in wide use, we enjoy a rich collection of functions that iterate through arrays, maps, sets, etc., making infinite loops rare.

With power comes responsibility, though. Can you spot what’s wrong with the following piece of code that filters application log entries coming from a database?

const entries = await getLogEntries(startDate, endDate, logLevel);
const results = entries.filter(entry => entry?.service === serviceName);

If you’ve guessed potential memory or performance issues, then you’re absolutely right. If getLogEntries returns millions of log entries, the application can run out of memory before we get to the loop. Even if memory isn’t exhausted, iterating through a massive array with the filter method can take a lot of time. This could block JavaScript’s own event loop and make the application unresponsive during the filtering process.

The first solution that comes to mind is to make the database do the filtering, but if we are unable to do that, we can try switching to chunked processing, which should look similar to the code below, assuming that getLogEntries always returns an array to simplify error handling:

const results = [];
const chunkSize = 10000;
let offset = 0;
while (true) {
    let entries = await getLogEntries(startDate, endDate, logLevel, offset, chunkSize);
	results.push(...entries.filter(entry => entry?.service === serviceName));
    if (entries.length < chunkSize) {
        break;
    }
    offset += entries.length;
}

Unfortunately, our code is no longer as neat as it once was, but at least it can now process large amounts of log entries without issues, or can it? If the results array becomes too large, we still risk memory exhaustion. In cases like these, it might help to impose hard limits on the number of results returned and either return an error or state that only the first X results were returned. Reasonable limits also offer some level of protection against malicious users.

Performance-wise, the above code can easily make hundreds of separate requests sequentially, which can increase the total processing time quite a bit, especially when those requests are made over a network connection.

Fortunately, JavaScript has Promise.all to execute requests in parallel, but since it has no built-in limit for the number of concurrent requests it makes, it’s all too easy to accidentally DoS your database server. Using a library like BlueBird or p-map that allows setting concurrency limits will be a lot safer. In the following example, we assume we can get the total number of log entries reasonably quickly to allow parallel fetching of chunks, with a concurrency limit of 10:

const chunkSize = 10000;
const totalEntries = await getTotalLogEntries(startDate, endDate, logLevel);
const pageCount = Math.ceil(totalEntries / chunkSize);
const allPages = Array.from({length: pageCount}, (_, i) => i);

const fetchPage = async page => {
    const entries = await getLogEntries(startDate, endDate, logLevel, page * chunkSize, chunkSize);
    return entries.filter(entry => entry?.service === serviceName);
};

const chunkedResults = await Promise.map(allPages, fetchPage, {concurrency: 10});
const results = chunkedResults.flat();

Up until now, we’ve seen how to process a large number of records without running out of memory and with reasonable performance, but there is one area where we must be extra careful: making destructive changes like database updates inside a loop, which can be difficult to revert if something goes wrong.

When there’s some business logic involved during batch database updates, it’s a good idea to have a dry run functionality not only for easier automated testing but also for manually verifying the results of an update before running it for real.

The following example demonstrates synchronizing employee data coming from an external source, such as a company’s Active Directory system, with the user data stored in an application’s database. If a user isn’t found in the database, a new record is created; if a user’s information has changed, such as the branch they belong to, their record is updated; and finally, if a user is no longer working at the company, their record is marked as disabled.

const usersToInsert = [];
const usersToUpdate = [];
const usersToDisable = [];

externalUsers.forEach(externalUser => {
	let existingUser = users.get(externalUser.id);
    if (existingUser) {
        let changes = getChanges(existingUser, externalUser);
        if (changes) {
            usersToUpdate.push({id: existingUser.id, changes});
        }
    } else {
        usersToInsert.push(externalUser);
    }
});

users.forEach(user => {
    if (!externalUsers.has(user.id) && user.status === 'active') {
        usersToDisable.push(user.id);
    }
});

By storing the necessary data for database operations in their respective arrays, we effectively separate our intent from execution. This dry run approach not only makes testing and verification easier but, perhaps even more importantly, also permits the auditing of past changes, assuming those changes have been logged.

Summary:

While modern loop constructs abstract away manual index management, performance and memory issues can still arise when processing large datasets.
Offloading work to the database is often the preferred solution. When feasible, leveraging the database’s optimization capabilities is generally more efficient.
Chunked processing is well-suited for handling large datasets. It allows for processing data in manageable portions, mitigating memory and performance issues.
Parallel execution can improve performance. However, uncontrolled parallelism can easily lead to performance problems.
Destructive operations in loops require extra caution: Separating intent from execution allows for easier testing and verification.

Little Big 18" Subwoofer Build

Fri, 28 Mar 2025 00:00:00 +0000

I’ve been building subwoofers for years now, and I’ve always used car audio drivers. However, I recently started building speakers using pro-audio drivers, specifically mid-woofers and horn loaded compression drivers. I love how dynamic and clean-sounding these drivers are, so it was only natural that I decided to give pro-audio subwoofers a try.

Unlike subwoofer drivers designed for car/home use, which typically favor deep bass output over efficiency, pro-audio subwoofer drivers excel at mid-bass punch that’s difficult for the less efficient drivers to match. Pro-audio drivers also feature strong motors that perform well in small enclosures. Their main limitation, however, is reduced output below 40 Hz. But by using multiple pro-audio drivers, it’s easy to achieve low-end performance comparable to their car/home audio counterparts, and when it comes to mid-bass punch, there’s simply no contest if both are driven with similar power.

I did some research on pro-audio subwoofer drivers with relatively low Fs (resonant frequency) and high Xmax (maximum linear cone excursion), and the Lavoce SAF184.03 stood out as a promising candidate. It features a beefy 4" voice coil and 1500W power handling. Some might scoff at the Lavoce’s 30 Hz Fs and 13 mm Xmax, but keep in mind that I plan to build multiple subwoofers in tiny sealed enclosures – at least by 18" subwoofer standards, so deep bass output likely won’t be an issue, and I’ll still be able to keep enclosure sizes in check.

The smallest enclosure that can house an 18" driver without sacrificing low-end output appears to be a 19" (~48 cm) cube, but rectangular enclosures are boring. I’ve built enough of those already. So, I made some quick sketches exploring alternative enclosure shapes. Although the octagon looked promising, the design featuring generous 4" (~10 cm) radius rounded corners was much more appealing to me, even though I knew it would be somewhat challenging to make.

Compact subwoofer cabinet ideas.

There are several methods to make wide-radius rounded corners, but kerf-bending a single sheet of wood is probably the most straightforward. You basically make deep parallel cuts in the wood in close succession, wet the cut areas, and then bend the wood around a form. I could have used MDF for everything, but 15 mm Baltic birch plywood not only keeps the enclosure weight in check but also is stronger than MDF –not to mention the wood grain looks very nice when stained a light color. The braces and baffle were made from 18 mm (~3/4") MDF.

Kerf-bent Baltic birch plywood.

I used a biscuit joiner to cut slots in the plywood. The two MDF braces are attached to the plywood using biscuits, essentially serving as the form around which the plywood was bent.

Kerf-bent Baltic birch plywood glued up.

The braces also feature circular openings to support the heavy subwoofer motor, which probably weighs close to 15 kg (~33 lbs) by itself.

The subwoofer motor brace.

The front baffle and rear panel were painted black, and the birch plywood was stained white with a two-component oil. I lined the walls with polyfill sheets.

The cabinet was stained, painted, and lined with polyfill.

I placed four pillows filled with polyfill inside to improve deep bass response. In total, I used about 1 lb of polyfill per cubic foot. The gross internal volume, before accounting for driver displacement, is approximately 3 cubic feet (~85 liters), which is about 25% smaller than the typical minimum enclosure size recommended for a standard 18" car/home audio subwoofer.

Driver and cabinet side by side.

With the addition of polyfill, the subwoofer’s resonant frequency decreased from about 61 Hz to 56 Hz. The impedance curve reveals that inductance is well-controlled, allowing this subwoofer to be easily crossed over as high as 500 Hz. The nominal impedance is 8 ohms.

The impedance curve reveals that inductance is well-controlled.

The near-field frequency response indicates a roll off that’s less steep than the expected 12 dB/octave for a sealed enclosure, thanks to the driver’s strong motor. When placed along a wall in a typical sized room, the subwoofer should deliver flat frequency response down into the mid-twenties, especially after reducing the ample mid-bass output using EQ or an AVR’s room correction capabilities.

Near-field frequency response.

Cone excursion utilization is close to optimal. With just 550W RMS of power, assuming some amplifier roll off below 10 Hz, the subwoofer reaches about 70% of its maximum linear excursion at 50 Hz, and stays below 100% at lower frequencies. Note that Lavoce specifies around 10 mm of additional excursion beyond Xmax, providing a comfortable safety margin, though with increased distortion.

Near optimal excursion utilization with just 550W RMS.

Very little amplifier power is needed above 50 Hz with a 100 Hz crossover. No wonder this subwoofer delivers such great mid-bass punch.

Very little amplifier power is needed above 50 Hz with a 100 Hz crossover.

Finally, a complete build video is available below:

Lessons from David Lynch: A Software Developer's Perspective

Sat, 15 Feb 2025 00:00:00 +0000

David Lynch passed away in January 2025, shortly after being evacuated from his Los Angeles home due to the Southern California wildfires. He’s perhaps best known for the groundbreaking TV series Twin Peaks, which inspired countless shows, including The X-Files, The Sopranos, and Lost.

Lynch was genuinely a good human being who cared deeply for his actors and crew. He discovered extraordinary talent like Naomi Watts, who had struggled to land a major role in a Hollywood movie after 10 years of auditioning. From the interviews he gave, it quickly becomes apparent that he respected people of all kinds and never put anyone down – even those who truly deserved it.

Lynch is famous for refusing to explain his movies. Although not a fan of his previous work, the great film critic Roger Ebert once wrote that Mulholland Drive remained compulsively watchable while refusing to yield to interpretation.

While Lynch offered very little in terms of what his movies meant, he was generous in sharing his views on creativity, work, and life in general. As a tribute to Lynch, I’d like to share my perspective on his life lessons from a software developer’s viewpoint.

Ideas are like fish. If you want to catch little fish, you can stay in the shallow water. But if you want to catch the big fish, you’ve got to go deeper.

We’ve all got hundreds or even thousands of ideas floating around in our brains. But the really big ones are few and far between. Once you catch a good one –because they’re so rare– write it down immediately, says Lynch. From there, ideas attract other ideas and start to grow from their initial seed state. The final job is to translate those ideas into a medium, whether it’s a film, a painting, or software.

The idea is the whole thing. If you stay true to the idea, it tells you everything you need to know, really. You just keep working to make it look like that idea looked, feel like it felt, sound like it sounded, and be the way it was.

Software development is part art, part engineering. We don’t build the same software over and over again – virtually all software is crafted by hand, sometimes with help from AI. If you ask two developers to create a non-trivial program, it’s very likely that the programs they produce will be different, even if the functionality is the same. Under the hood, the programming language, data structures, and overall architecture may be completely different. And on the surface, the user interfaces may look nothing alike.

It’s a good habit to listen to what users have to say, but they often can only describe their problems – they rarely come up with good ideas to solve them. And that’s OK. It’s our job to find the right ideas, implement them well, and solve tricky problems in a way we, and hopefully the users, will love.

My friend Bushnell Keeler, who was really responsible for me wanting to be a painter, said you need four hours of uninterrupted time to get one hour of good painting in, and that is really true.

Like other creative fields, writing code requires deep concentration. We need to hold complex structures in our minds while working through problems. Switching between coding and other tasks disrupts flow – that magical state of mind where we lose track of time and produce code effortlessly. That’s why many developers hate meetings – they are toxic to our productivity.

I believe you need technical knowledge. And also, it’s really, really great to learn by doing. So, you should make a film.

Software development is one of those rare fields where a college degree isn’t required to succeed. Yes, we should all know the basics, but in my experience, new college graduates often lack the practical knowledge to be effective developers.

The real learning happens through hands-on experience: building real projects, debugging tricky problems, collaborating with teams, and maintaining code over time. It’s crucial to never stop learning, experimenting, and iterating on our craft.

Happy accidents are real gifts, and they can open the door to a future that didn’t even exist.

Tim Berners-Lee invented the web in 1989, while working at CERN, the European Organization for Nuclear Research. Originally conceived to meet the demand for information sharing between scientists around the world, the web went mainstream within just a few years.

Linus Torvalds created Git due to a licensing dispute over BitKeeper, the original version control system used for Linux development. The need for a new tool led to Git becoming the most widely used version control system today.

I feel that a set should be like a happy family. Almost like Thanksgiving every day, happily going down the road together.

Be kind to your teammates, don’t embarrass them. They may not be perfect, but accept them for who they are. The most important trait of an effective software development team is psychological safety –that is, team members feel safe to take risks and be vulnerable in front of each other, as corroborated by Google’s research on the subject.

It’s OK to make mistakes, as long as you learn from them. Knowing that your team has your back when things go south is a wonderful feeling.

Most of Hollywood is about making money - and I love money, but I don’t make the films thinking about money.

Just like Lynch prioritizes creativity over financial gain, some of the most impactful software projects started with an open source model, and they literally changed the world, such as Linux, PostgreSQL, and Node.js, just to name a few.

What makes these projects remarkable is that they didn’t emerge from corporate boardrooms – they were built by communities of passionate developers, collaborating across the world.

Money is just a means to an end. Unfortunately, many get this confused.

David, thank you for making the world a better place!

David Lynch in 2007 via Wikimedia Commons

Discussed on:

Runtime Diagnostics: Catching Bugs as They Happen

Mon, 13 Jan 2025 00:00:00 +0000

A popular trope in science fiction movies set in space is running diagnostics whenever something goes wrong with a ship’s systems. The chief engineer might announce something like, “We’ve run a level 2 diagnostic on the thermal regulators.” This is technobabble of course, but in software development, the closest parallel to starship diagnostics we have is probably automated regression tests. Checking application logs for errors and, although less common, running integrity checks on data stores also come to mind when diagnosing tricky software bugs.

Automated test suites play a vital role in validating the correctness of code. However, they come with certain drawbacks: they exist separately from the application code they are designed to test, and they can also take a significant amount of time to execute.

Tests can easily fall out of sync with the application code if they are not maintained well. Shifting some checks from tests into the application code itself is a powerful technique that not only makes tests less brittle but also ensures runtime validation of internal application logic and data structure integrity.

Perhaps the most common method for implementing internal consistency checks is using assert statements, which throw an exception when an internal condition is violated. In my experience, assertions are most effective when applied systematically rather than in an ad hoc manner. Design by Contract, a programming approach originally introduced by Bertrand Meyer in connection with his Eiffel language offers such a system.

To illustrate the principles of Design by Contract, let’s create a simple JavaScript program that generates a data structure for building site organization charts, like the one shown below:

A sample site organization chart.

The following code is probably as bare bones as it gets, with just three methods and virtually no checks other than what is absolutely necessary. A lot of things could go wrong with this code, some of which might not be immediately obvious:

const SiteOrganizer = function () {
    const siteMap = new Map();

    this.addSite = (site) => {
        siteMap.set(site.id, { ...site, children: new Set() });
        if (site.parentId !== null) {
            siteMap.get(site.parentId).children.add(site.id);
        }
        return this;
    };

    this.removeSite = (siteId) => {
        const site = siteMap.get(siteId);
        if (site.parentId !== null) {
            siteMap.get(site.parentId).children.delete(siteId);
        }
        siteMap.delete(siteId);
        return this;
    };

    this.getSites = () => siteMap;
};

And here’s how we add the sites in our sample organization chart:

const mySites = new SiteOrganizer()
    .addSite({ id: 1, name: 'HQ', parentId: null })
    .addSite({ id: 2, name: 'Regional Office A', parentId: 1 })
    .addSite({ id: 3, name: 'Regional Office B', parentId: 1 })
    .addSite({ id: 4, name: 'Local Site A1', parentId: 2 })
    .addSite({ id: 5, name: 'Local Site A2', parentId: 2 })
    .addSite({ id: 6, name: 'Local Site B1', parentId: 3 })
    .addSite({ id: 7, name: 'Local Site B2', parentId: 3 });

Preconditions

The first thing we will add are preconditions, which consist of assertions that enforce the assumptions a particular method makes about the input it receives. These checks explicitly document the conditions that must be true for the method to function as intended, such as ensuring that a site’s parent cannot be itself.

function assert(cond, msg = '') {
    if (!cond) {
        throw new Error(msg);
    }
}

const SiteOrganizer = function () {
    const siteMap = new Map();

    this.addSite = (site) => {
        assert(!siteMap.has(site.id), `Site ${site.name} should not have been added before.`);
        assert(site.id !== site.parentId, `Site ${site.name} cannot be its own parent.`);
        assert(site.parentId === null || siteMap.has(site.parentId), `Parent site of ${site.name} must exist.`);
        // ...
    };

    this.removeSite = (siteId) => {
        assert(siteMap.has(siteId), `Site #${siteId} must exist.`);
        assert(siteMap.get(siteId).children.size === 0, `Site #${siteId} must have no children.`);
        // ...
    };

    this.getSites = () => siteMap;
};

One can also use regular if-else blocks to test the negative versions of conditions instead of assertions. However, a major advantage of assertions is that they can be easily disabled in production, especially in performance-critical parts of the code, as not all assertions will be as fast as the ones shown above.

Postconditions

Postconditions are the guarantees a method provides to its caller when all preconditions are met. In the example below, the addSite method ensures that the site has been successfully added.

const SiteOrganizer = function () {
    const siteMap = new Map();

    this.addSite = (site) => {
        // ...

        siteMap.set(site.id, { ...site, children: new Set() });
        if (site.parentId !== null) {
            siteMap.get(site.parentId).children.add(site.id);
        }

        assert(siteMap.has(site.id), `Added site ${site.name}.`);
        assert(
            site.parentId !== null ? siteMap.get(site.parentId).children.has(site.id) : true,
            `Added site ${site.name} as sub site.`
        );
        return this;
    };

    // ...
};

It’s not always practical to test all possible outcomes, this is where test suites excel. However, using postconditions whenever possible clarifies what a method’s output should be. Even better, postconditions ensure the outcome is tested in runtime.

Class Invariants

Class invariants can be thought of as guarantees about the consistency of an object’s state. They must hold true between calls to public methods. Code within functions is allowed to temporarily break invariants, as long as they are restored before the function ends. In the example below, all the sites stored in siteMap are checked to ensure they satisfy certain conditions.

const SiteOrganizer = function () {
    const siteMap = new Map();

    this.diagnostics = function {
        function checkChildren(children) {
            let status = true;
            children.forEach((siteId) => siteMap.has(siteId) || (status = false));
            return status;
        }

        siteMap.forEach((site) => {
            assert(site.id !== site.parentId, `Site ${site.name} cannot be its own parent.`);
            assert(site.parentId === null || siteMap.has(site.parentId), `Parent site of ${site.name} must exist.`);
            assert(checkChildren(site.children), `Site ${site.name} must have valid children.`);
        });
    }

    // ...
}

Since JavaScript doesn’t natively support class invariants (or object invariants in this case), you can call the diagnostics method at the end of each public method or use a library to automate invariant checks (more on this later).

Do we really need all these checks?

The short answer is, probably not–at least not in every part of the codebase. Of the three, preconditions are the most important, and most developers implement them in some form or another. Postconditions are particularly useful in mission-critical parts of the code, ensuring that methods conform to their specifications.

As for class invariants, it might seem wasteful to check an object’s internal state every time a method is called. However, these checks can save significant time when debugging tricky state-related bugs, and they can be disabled in production code.

Automating invariant checks

“If you get the data structures and their invariants right, most of the code will write itself.” – L. Peter Deutsch

There are a few JavaScript libraries that add support for automated invariant checks, such as this one, but I haven’t found a library that works without requiring code transpilation, so I created my own: Object Diagnostics.

Using JavaScript proxies, Object Diagnostics intercepts all method calls and property accesses, including modifications and deletions. It expects a public method named diagnostics to be defined on the object, which contains the invariant checks. After each intercepted operation, it automatically calls this method.

Object Diagnostics also includes an environment-aware assert function, but you can use another library for assertions as well.

import { ObjectDiagnostics, assert } from 'object-diagnostics';

const mySites = new ObjectDiagnostics().addTo(new SiteOrganizer());

Bonus: Avoiding inadvertent mutations

Careful readers may have already noticed a fundamental flaw in our example program: its internal data structure, siteMap, though technically a private property, can still be mutated. Here’s a quick example of how:

// Set the parent id of 'HQ' to a non-existent id
mySites.getSites().get(1).parentId = 100;

When we return siteMap from getSites, JavaScript doesn’t create a copy; it shares a reference instead. This makes it possible to directly modify the object. However, as soon as another operation is performed on mySites, the diagnostics method will detect the integrity issue and throw an error.

mySites.addSite({ id: 8, name: 'Local Site B3', parentId: 3 });
// Error: Parent site of HQ must exist.

Tip: Even with invariant checks in place, it’s a good idea to use copies of objects whenever possible. A utility function like Lodash’s cloneDeep can help with this.

import { cloneDeep } from 'lodash-es';

const SiteOrganizer = function () {
    const siteMap = new Map();

    this.addSite = (site) => {
        // ...
        siteMap.set(site.id, { ...cloneDeep(site), children: new Set() });
        // ...
    }
   
   // ...
   this.getSites = () => cloneDeep(siteMap);
}

Summary

Preconditions verify that inputs meet the required conditions before a method executes, while postconditions ensure that the outputs are correct after the method completes.
Class invariants guarantee the consistency of an object’s internal state across its life cycle.
Runtime diagnostics complement test suites by catching integrity issues while a program runs rather than relying solely on separate tests.

Building Valkyrie: A Compact High-Output Speaker with Smooth Directivity

Thu, 12 Dec 2024 00:00:00 +0000

Well-made hi-fi speakers sound great at low to moderate volumes, but many begin to suffer at high volume levels. Since hi-fi speakers are frequently used without a subwoofer, their designers often have to make certain trade-offs. To achieve reasonable bass, they have to use inefficient drivers that tend to distort at high volumes, assuming there is enough amplifier power before reaching that stage. Even some large floorstanding models with multiple bass drivers aren’t immune. The same applies to high-frequency drivers; many dome tweeters can’t handle high volume levels and distort noticeably.

In my opinion, music and especially home theater playback should be done at moderate to high levels to do the content justice. With subwoofers, speakers are relieved of low-bass duties and can concentrate on reproducing the rest of the frequency band. This allows the use of efficient drivers that require significantly less amplifier power to get loud. The pro-audio world has been using such drivers in their speakers for decades, and at a cinema or concert, what you are listening to is obviously not hi-fi speakers.

I previously built Valeria, a compact bookshelf speaker using pro-audio drivers, namely a 6.5" mid-bass and a 1" horn-loaded compression driver. The dynamics offered by this small speaker are impressive, if I may say so myself. It’s currently serving as the center channel in my home theater.

Lately, I’ve been thinking of building speakers with better bass response and improved horizontal and especially vertical directivity. This necessitated using an 8" mid-bass driver, a compression driver that rolls off below 1 kHz, and a horn that can control directivity better than the B&C ME10 I used in Valeria.

The mid-bass driver I selected for Valkyrie is the Lavoce WAF082.00. It has a 2" voice coil, a paper cone, and a cast-aluminum basket. The compression driver, Lavoce DF10.171K, features a 1.75" voice coil and a polyimide diaphragm. As for the horn, I decided to use the RCF HF101, a cast aluminum horn with relatively compact dimensions, which is important to minimize driver-to-driver spacing and consequently improving vertical directivity.

8" mid-bass and 1" horn-loaded compression drivers are used.

Cabinet

I designed a compact cabinet with internal bracing that couples to the drivers, making the drivers part of the brace and reinforcing the front and rear sides of the cabinet.

Valkyrie 3D model

I cut the pieces with my CNC router using 18 mm (~3/4") MDF and then added 45-degree bevels to all sides on my router table. I doubled the thickness of the front baffle to make it stronger since the mid-bass driver weighs a healthy 4 kg (~9 lb).

The pieces are simple enough that they can be cut with a circular saw or jigsaw, and the holes can be cut with a handheld router. There is no need to flush mount the horn or the mid-bass driver. The bevels are optional as well.

I glued everything together using wood glue, aligning the parts with painter’s tape. I recommend using band clamps if you have some at hand.

Gluing sides using band clamps.

The braces were made of 16 mm beech plywood. I attached the braces to the cabinet walls with 3M VHB tape instead of regular wood glue. The soft layer in between acts as constrained-layer damping to reduce resonances.

Dry-fitting braces.

I glued the port, made from a PVC pipe (68 mm inner diameter), to the back wall with hot glue. The crossover board is installed on the side wall with standoffs.

I placed M6 threaded inserts into the front baffle so that I could remove and install the drivers as many times as needed without having to worry about stripping the holes, which is quite easy with MDF.

Crossover

I decided to use a third-order low-pass filter with two laminated silicon steel core inductors (2.5 mH and 0.56 mH) in series and a 47 µF capacitor in parallel for the mid-bass driver.

For the compression driver, I used a second-order high-pass filter with an 8.2 µF capacitor in series and a 2.5 mH air core inductor in parallel.

I also added an L-pad circuit with a 10 ohm resistor in series and a 4.7 ohm resistor in parallel to reduce the level of the super efficient compression driver by 12 dB on average and flatten its response.

Finally, I used two parallel notch filters, each consisting of an air core inductor, a capacitor, and a resistor in series, to further flatten the compression driver’s response. The notch filter with the lower component values is optional if a peak at 18 KHz is acceptable since there is little content in music at that frequency and many people can’t hear it anyway.

Crossover board close-up view. The 2.5 mH air core inductor isn't pictured.

Painting & Acoustic Damping

I applied 19 mm (3/4") roundovers to all sides using my router table. Then, I took the speaker outside and painted it with several coats of MDF primer using a spray gun. Once the primer had dried, I sprayed several coats of top coat (steel blue). Since I was using water-based paints, I added some paint conditioner (pouring medium) to help reduce orange peel.

Cabinet Painted

It’s critical to add absorption material to manage the reflections and standing waves that form inside the cabinet. I lined the walls with felt and placed extra-thick 60 mm pyramid foam on top while leaving enough space for the drivers.

Absorption materials placed in cabinet.

Finally, Valkyrie is complete, standing next to Valeria. Valkyrie’s dimensions are 24 x 42 x 21.5 cm (WxHxD), while Valeria measures 19 x 33 x 22.5 cm (WxHxD).

Valeria next to Valkyrie.

Measurements

So, how is the sound? By itself, the bass is lacking, as expected. However, as soon as you add a subwoofer to the mix, the missing low-end is filled in, and what you get is effortless dynamics throughout the entire frequency range assuming the subwoofer can keep up with Valkyrie at higher volumes.

In listening tests, it’s immediately apparent that, compared to Valeria, the bass is considerably louder and deeper below 200 Hz. Otherwise, they’re well matched.

The impedance plot looks clean, with the exception of a resonance at around 300 Hz. This appears to be inherent to the mid-bass driver’s construction, as the free-air impedance measurements of the driver show it as well, albeit not as pronounced. The nominal impedance is 6 ohms dipping to 4.4 ohms at 210 Hz.

Impedance and phase plot.

The frequency response is within ±3 dB from 100 to 20,000 Hz (quasi-anechoic measurement). The mid-bass to compression driver crossover frequency is quite low at 1 kHz to help make vertical directivity well behaved. Please note that it’s not advisable to use such a low crossover frequency for this particular compression driver in a commercial setting where speakers are routinely run at their limits, but in a residential setting, it should be fine.

As for the subwoofer crossover frequency, 100 Hz looks like a good starting point. You can probably use a slightly lower crossover frequency if you place the speaker close to walls, as this will result in some bass boost from nearby boundaries. It’s also a good idea to run the subwoofer(s) a few dB hot for optimal integration.

On-axis quasi-anechoic frequency response.

The horizontal directivity is ±45 degrees nominal, smooth down to 800 Hz. Below that frequency, the directivity quickly becomes omnidirectional. Thanks to the uniform directivity, the frequency response can be equalized without any issues.

Smooth horizontal directivity (normalized).

The vertical directivity is ±35 degrees nominal, and it’s smooth down to a respectable 1200 Hz. As long as your ears aren’t below around 15 degrees of the horn’s center axis, the timbre of the speaker shouldn’t change noticeably regardless of where you sit or stand.

Smooth vertical directivity down to 1200 Hz (normalized).

Finally, a complete build video is available below:

Writing Composable SQL using Knex and Pipelines

Mon, 25 Nov 2024 00:00:00 +0000

SQL is great! It can handle complex queries involving multiple tables, aggregations, subqueries, and joins. It can perform CRUD (Create, Read, Update, Delete) operations. It enforces data integrity through constraints like primary and foreign keys. It supports transactions, allowing multiple operations to be executed as a single unit, which can be rolled back if one of the operations fails.

Despite all its strengths, which are too many to list here, SQL can be awkward to integrate with host languages such as JavaScript and Python. There’s often an impedance mismatch between SQL’s declarative nature and the host language’s object-oriented or functional paradigms since SQL queries are typically written as strings within the host language, making it difficult to apply composable programming techniques.

Several solutions have been developed over the years to address the impedance mismatch, with ORM (Object-Relational Mapping) libraries being one of them. ORMs such as Java’s Hibernate are widely used to model a relational database using objects, but there are various difficulties in making them work well.

Fortunately, there exists a category of libraries called query builders like Knex for Node.js. Query builders allow the construction of queries programmatically without resorting to string concatenation and, unlike ORMs, they don’t suffer from the object-relational impedance mismatch.

Knex

To demonstrate how a query builder works, let’s model a simple discussion forum. To keep things simple, we will use just three tables: post, topic and user.

The database tables for a simple discussion forum.

To retrieve all the posts in a topic along with information about the posters, we can use the following piece of code:

const query = require('knex')({
    client: 'pg',
    connection: 'postgresql://localhost/my_forum',
});

const getPostsInTopic = (topicId) =>
    query
        .from('post')
        .select('post.*', 'user.username', 'user.country')
        .join('user', 'user.id', 'post.user_id')
        .where('post.topic_id', topicId)
        .orderBy('post.id');

And the result should look something like this:

[
  {
    id: 1210,
    user_id: 327,
    topic_id: 42,
    content: 'Hi there!',
    created_at: '2024-11-22T14:34:26.454Z',
    updated_at: '2024-11-22T14:34:26.454Z',
    username: 'Asuka',
    country: 'JP'
  },
  // ...
]

Knex uses a fluent interface to build queries, making it possible to modify them by simply chaining additional methods. In the example below, we’ve made retrieving details about posters optional:

const getPostsInTopic = (topicId, withPosterDetails = false) => {
    let myQuery = query
    	.from('post')
    	.select('post.*')
    	.where('post.topic_id', topicId)
    	.orderBy('post.id');

    if (withPosterDetails) {
        myQuery = myQuery
        	.select('user.username', 'user.country')
        	.join('user', 'user.id', 'post.user_id');
    }

    return myQuery;
};

Pipelines

Knex’s built-in object-oriented fluent interface works well, but for those who prefer a functional-style approach and the advantages that come with it, it’s possible to use an alternative interface based on pipelines.

“It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.” – Alan Perlis

It’s surprisingly simple to implement a generic pipeline function that accepts multiple functions as input, passes an initial value to the first function, and executes the functions sequentially, passing the output of each function to the next:

const pipe = (...functions) => (initialValue) =>
	functions.reduce((accumulator, fn) => fn(accumulator), initialValue);

We will also need an identity function, which simply returns its input, acting similarly to a no-operation (no-op) function.

const identity = (x) => x;

We can now utilize the pipe function to generate our SQL query using only function composition. The query modifier functions below return functions that accept the query object as their input:

const postTable = query.from('post');

const postsInTopic = (topicId) => (query) => query
	.select('post.*')
	.where('post.topic_id', topicId);

const includePosterDetails = () => (query) => query
	.select('user.username', 'user.country')
	.join('user', 'user.id', 'post.user_id');

const orderBy = (...fields) => (query) => query.orderBy(...fields);

const getPostsInTopic = (topicId, withPosterDetails = false) =>
    pipe(
        postsInTopic(topicId),
        withPosterDetails ? includePosterDetails() : identity,
        orderBy('post.id')
    )(postTable);

Summary

We have used a query builder to programmatically build SQL queries, and by using higher-order functions (functions that take other functions as arguments) and pipelines, we have:

Eliminated the need to pass the query object explicitly.
Made it easier to add, remove, or rearrange query modifiers without disrupting the overall structure.
Improved maintainability and simplified testing since each query modifier function is a self-contained unit that can be reused across various queries and scenarios.

Discussed on:

Hacker News

Self-documenting Code

Sun, 20 Oct 2024 00:00:00 +0000

Think back to the last time you looked at an unfamiliar block of code. Did you immediately understand what it was doing? If not, you’re not alone – many software developers, including myself, find it challenging to grasp unfamiliar code quickly.

Let’s take a look at a simple JavaScript function that creates a user account:

async function createUser(user) {
    if (!validateUserInput(user)) {
        throw new Error('u105');
    }

    const rules = [/[a-z]{1,}/, /[A-Z]{1,}/, /[0-9]{1,}/, /\W{1,}/];
    if (user.password.length >= 8 && rules.every((rule) => rule.test(user.password))) {
        if (await userService.getUserByEmail(user.email)) {
            throw new Error('u212');
        }
    } else {
        throw new Error('u201');
    }

    user.password = await hashPassword(user.password);
    return userService.create(user);
}

At first glance, this function doesn’t look too bad apart from its use of cryptic error message codes. The argument user is apparently an object that contains information about the user to be created. There are a few lines of code that checks if the password conforms to the password policy, using regular expressions. Then, there is a check to see if the user account already exists. Finally, if all the checks pass, the user’s password is hashed, and a function to create the new user is called; it probably returns something on success.

One can certainly do a lot worse than this example, but there is also quite a bit of room for improvement.

“There are only two hard things in Computer Science: cache invalidation and naming things.” – Phil Karlton

Software developers frequently deal with abstract ideas and complex systems. Translating these abstractions into concrete, meaningful names that accurately reflect their behavior isn’t always straightforward. However, that isn’t really an excuse when we’re dealing with well-understood processes like user account creation, as in our example.

Named Constants and Doing One Thing Only

The first change I would make is to use named constants instead of cryptic error codes. Also, putting complex logic, such as the password check, into its own function makes the code easier to read. After all, a function should ideally do one thing and one thing only. Implementing the password check separately allows it to be called from other functions as well.

const err = {
    userValidationFailed: 'u105',
    userExists: 'u212',
    invalidPassword: 'u201',
};

function isPasswordValid(password) {
    const rules = [/[a-z]{1,}/, /[A-Z]{1,}/, /[0-9]{1,}/, /\W{1,}/];
    return password.length >= 8 && rules.every((rule) => rule.test(password));
}

After these changes, createUser should look something like this. We can now immediately tell what would happen if a check fails.

async function createUser(user) {
    if (!validateUserInput(user)) {
        throw new Error(err.userValidationFailed);
    }

    if (isPasswordValid(user.password)) {
        if (await userService.getUserByEmail(user.email)) {
            throw new Error(err.userExists);
        }
    } else {
        throw new Error(err.invalidPassword);
    }

    user.password = await hashPassword(user.password);
    return userService.create(user);
}

Short-circuit Evaluation

The revised code above is now easily readable, and we can leave it as is. However, in this case there’s an opportunity to make the code flow more linear by using short-circuit evaluation.

Short-circuit evaluation allows us to simplify conditional statements by using logical operators. In this case, the || operator checks the condition on the left, and if it’s false, it executes the function on the right.

We have also flattened the password validation and existing user checks. The resulting code is shorter and has no nested logic.

function throwError(error) {
    throw new Error(error);
}

async function createUser(user) {
    validateUserInput(user) || throwError(err.userValidationFailed);
    isPasswordValid(user.password) || throwError(err.invalidPassword);
    !(await userService.getUserByEmail(user.email)) || throwError(err.userExists);

    user.password = await hashPassword(user.password);
    return userService.create(user);
}

Type Annotations

Self-documenting code involves writing code with minimal comments, but there is a category of comments that is useful not only for developers but also for compilers and IDEs, and that is annotations about the types of variables and arguments used.

I’m not a fan of TypeScript, but I appreciate its ability to perform static type checks. Fortunately, there’s a way to add static type checking to JavaScript using only JSDoc comments. Alex Harri did an excellent job explaining how you can do that in this article if you’re interested.

The following JSDoc comments make it clear what type of argument createUser accepts and what it returns. The type definitions can be automatically picked up by the TypeScript compiler or an IDE like VS Code, giving you real-time feedback when you pass a value with the wrong type.

/** @typedef {{ id?: number, birthDate: Date, email: string, password: string }} User */

/**
 * Creates a user and returns the newly created user's id on success
 * @param {User} user
 * @returns <Promise{any}>
 */
async function createUser(user) {
    validateUserInput(user) || throwError(err.userValidationFailed);
    isPasswordValid(user.password) || throwError(err.invalidPassword);
    !(await userService.getUserByEmail(user.email)) || throwError(err.userExists);

    user.password = await hashPassword(user.password);
    return userService.create(user);
}

Summary

We have turned a seemingly simple but somewhat hard-to-follow function into a self-documenting function by:

Using named constants instead of cryptic error codes,
Extracting complex logic and putting it in its own function,
Using short-circuit evaluation to make the code flow linear,
Introducing type annotations to help with static type checking and real-time coding feedback.

Discussed on:

Avoiding if-else Hell: The Functional Style

Mon, 23 Sep 2024 00:00:00 +0000

Many years ago, I took part in the development of a taxi-hailing mobile app that is still widely used today. I don’t know what kind of code they’re running now, but in those early days, the driver assignment code –if I remember it correctly– was similar in spirit to the grossly simplified example shown below:

async function assignDriver(rider, availableDrivers) {
    const driverDistances = await calculateDistances(rider.location, availableDrivers);
    let assignedDriver = null;

    for (let driver of availableDrivers) {
        if (driverDistances[driver.id] <= 5) {
            if (!rider.preferredVehicle || rider.preferredVehicle === driver.vehicle) {
                if (driver.rating >= 4.5) {
                    if (rider.preferences.includes('Premium Driver')) {
                        if (driver.isPremiumDriver) {
                            assignedDriver = driver;
                            break;
                        } else {
                            continue;
                        }
                    } else {
                        assignedDriver = driver;
                        break;
                    }
                } else if (driver.rating >= 4.0) {
                    assignedDriver = driver;
                    break;
                }
            }
        }
    }

    return assignedDriver;
}

There are five levels of nested if statements in less than 30 lines of code. It doesn’t look so bad, some might say, but it’s not difficult to imagine how complicated this code can become with just a few more checks, such as surge pricing and loyalty programs, among other things.

Fortunately, there are ways to flatten code, and you might be surprised by the end result as there will be no if statements left when we are finished refactoring it.

Early Returns with Guard Clauses

Let’s start with the easy pickings. The very first if statement, which checks the maximum allowed distance between a driver and the rider, can be converted into a guard clause to remove one level of nesting since it applies to all cases. We can similarly add one more guard clause for the preferred vehicle check to remove an additional level of nesting. The resulting code is shown below:

async function assignDriver(rider, availableDrivers) {
    const driverDistances = await calculateDistances(rider.location, availableDrivers);
    let assignedDriver = null;

    for (let driver of availableDrivers) {
		if (driverDistances[driver.id] > 5) {
			continue;
		}
		if (rider.preferredVehicle && rider.preferredVehicle !== driver.vehicle) {
		    continue;
		}
		if (driver.rating >= 4.5) {
            if (rider.preferences.includes('Premium Driver')) {
                if (driver.isPremiumDriver) {
		            assignedDriver = driver;
		            break;
		        } else {
		            continue;
		        }
		    } else {
		        assignedDriver = driver;
		        break;
		    }
		} else if (driver.rating >= 4.0) {
		    assignedDriver = driver;
		    break;
		}
    }

    return assignedDriver;
}

Decision Tables

Instead of hardcoding the logic inside our function, we can place each if-else block in its own function and put those functions in an array, forming a decision table. Then, we run each function in the decision table until we get a positive response. Obviously, the entries in the table must be sorted from most specific to least specific for it to function properly in our case.

const conditions = [
    (rider, driver) => driver.rating >= 4.5 && rider.preferences.includes('Premium Driver') && driver.isPremiumDriver,
    (rider, driver) => driver.rating >= 4.5 && !rider.preferences.includes('Premium Driver'),
    (rider, driver) => driver.rating >= 4.0 && driver.rating < 4.5,
];

async function assignDriver(rider, availableDrivers, conditions) {
    const driverDistances = await calculateDistances(rider.location, availableDrivers);
    let assignedDriver = null;

    for (let driver of availableDrivers) {
        if (driverDistances[driver.id] > 5) {
            continue;
        }
        if (rider.preferredVehicle && rider.preferredVehicle !== driver.vehicle) {
            continue;
        }
        if (conditions.find((condition) => condition(rider, driver))) {
            assignedDriver = driver;
            break;
        }
    }

    return assignedDriver;
}

With the addition of a decision table, we have completely eliminated nested ifs, and as an added bonus, the driver assignment logic can now be changed by simply editing the conditions array.

Function Composition

Let’s get rid of the remaining if statements by converting the for loop to Array.find() and creating individual functions for each if statement inside the loop:

const conditions = [
    (rider, driver) => driver.rating >= 4.5 && rider.preferences.includes('Premium Driver') && driver.isPremiumDriver,
    (rider, driver) => driver.rating >= 4.5 && !rider.preferences.includes('Premium Driver'),
    (rider, driver) => driver.rating >= 4.0 && driver.rating < 4.5,
];

async function assignDriver(rider, availableDrivers, conditions) {
    const driverDistances = await calculateDistances(rider.location, availableDrivers);

    const isDriverClose = (driver) => driverDistances[driver.id] <= 5;
    const isVehicleOK = (rider, driver) => !rider.preferredVehicle || rider.preferredVehicle === driver.vehicle;
    const isAConditionSatisfied = (rider, driver) => conditions.find((condition) => condition(rider, driver));

    const assignedDriver = availableDrivers.find(
        (driver) => isDriverClose(driver) && isVehicleOK(rider, driver) && isAConditionSatisfied(rider, driver)
    );

    return assignedDriver || null;
}

Summary

By utilizing basic functional programming principles, we have:

Removed all nested if statements,
Decoupled most of the logic and made it easily modifiable,
Made the code easier to follow by placing individual if-else blocks in their own functions,
Finally, as a nice side effect, cut the program size by one-third.

Discussed on:

Firewalling Your Code

Tue, 20 Aug 2024 00:00:00 +0000

When writing code, you can call any function as long as it’s public, and similarly, you can access any object’s public properties or methods. Usually, access to code is all or none – a piece of code can be either public or private. Access to private code is typically controlled by the upper layer that encapsulates it. Although some languages support semi-private (also known as protected) object members, that’s a merely variation on the public/private theme.

Lately, I’ve been thinking about ways to implement more fine-grained access controls and have looked to the networking world for inspiration. In computer networks, firewalls are used to restrict access to network resources. For example, you can set firewall rules to allow only the application and analytics servers to access the database server – or more specifically, the port on which the database service is running. There’s no need for encapsulation, inheritance or interfaces to implement; you simply define access rules.

Unlike network firewalls, when accessing a public property or method in a program, we don’t normally check whether the caller has the right to do so. Since the callee is public, any part of the program should be able to call it without any restrictions. However, I believe there are cases when more fine-grained controls could be useful.

In the commonly used multi-tier architecture, code flows in one direction only – from the upper-most layer to the layers below. Modules can call the other modules in the same layer. They can also call modules in the layer directly below, but never above.

How can you ensure that the modules follow these rules? Unless you’re conducting in-depth code reviews, it isn’t too difficult to end up with haphazard code paths that might have maintenance and security implications.

As a proof of concept, I’ve created a Node.js library called firewall-js using JavaScript proxies. It relies on the filesystem structure of a code base to limit access to certain parts.

Take a simple backend application with three layers: routes > controllers > services. Each layer has its own directory, and each file in a directory houses a module. The directory listing should look something like this:

> controllers
> routes
v services
   auth.js
   log.js
   user.js

Now, if you want all the controller modules (upper layer) and all other service modules (same layer) to have access to a particular service module, we can do it with a single line:

// services/user.js
// ...
const firewall = require('firewall-js');

const userService = {
    hashPassword: function (password) {
        return bcrypt.hash(password, 8);
    },

    getUserByEmail: function (email) {
        return db('user').where('email', email).then(_.head);
    },

    // ...
};

module.exports = firewall.allow(['controllers', 'services'], userService);

If you attempt to call, for example, userService.hashPassword() from a file in any other directory, an exception will be thrown:

Error: Access denied for hashPassword from /Users/me/my-app/routes/main.js:51:19

You can go one step further, and allow access not just from directories, but from files too. In the following example, only the userProfile controller can access userService, and no one else:

module.exports = firewall.allow(['controllers/userProfile.js'], userService);

I’ve deliberately made the filesystem structure the basis of the access control system. This, I believe, has two benefits:

A clear-cut organization of code with directories acting as layers and files as modules within those layers.
Permissions that are easy to understand, since most everyone is familiar with how a filesystem works.

An access control system like the one implemented here can help prevent maintenance issues, such as dependencies or coupling between unrelated parts of the application, and enhance security by limiting the potential for unauthorized actions. Additionally, it doesn’t require any drastic changes to the existing code base, as long as a sensible directory hierarchy is in place. Firewalling a module can then be as easy as adding a single line of code.

GitHub Repository: firewall-js

Discussed on:

Hacker News

Building Valeria: A Compact Horn-Loaded Speaker

Sat, 27 Jul 2024 00:00:00 +0000

I’ve been using a pair of KEF LS50 Metas as my front speakers and my DIY KEF LS50 clone as the center channel speaker in my home theater for a while now. I’ve been generally happy with the smooth sound they produce, but there’s one thing lacking: dynamics.

Granted, these are small bookshelf speakers, but they receive assistance from my DIY subwoofers, so they don’t have to reproduce the most difficult frequencies of all: sub-bass. Unfortunately, at my listening distance of approximately 3.5 meters (~11.5 feet), they start to distort noticeably at higher volumes, which reduces the impact of certain movies, especially ones with epic soundtracks.

Lately, I’ve been thinking about building high-sensitivity speakers using pro-audio drivers. Horn-loaded compression drivers, which are used to reproduce high frequencies in the pro-audio world, can get really loud with little distortion. A well-designed horn can also achieve constant directivity – the horizontal and vertical dispersion of high frequencies are consistent and limited to specific angles (e.g., nominally ±45 degrees horizontal and ±30 degrees vertical).

The horizontal directivity plot of a constant directivity horn as measured by mtg-designs.

Some hi-fi speakers that use waveguides can also control high-frequency directivity similarly to horns, but the waveguides they employ are usually not deep enough to boost the efficiency of the dome tweeters as regular horns do.

Coming from coaxial speakers, where the tweeter is located at the center of the mid-bass driver like the KEFs, my main goal, apart from improved dynamics, is to achieve decent vertical directivity.

With non-coaxial speakers, you need to position the mid-bass driver as close as possible to the tweeter to minimize driver center-to-center spacing. However, this is often not sufficient because you also need to use a lower crossover frequency than usual. Unfortunately, many tweeters will self-destruct if the crossover frequency is too low and the volume is too high.

With pro-audio compression drivers mounted on a horn, you can easily get them to play anywhere from half to a full octave below the lower limit of a typical dome tweeter (e.g., 1.6 kHz instead of 2.2 kHz). This helps with vertical directivity as wavelengths get longer the lower you go in frequency, resulting in less destructive interference between the drivers.

A typical pro-audio speaker, often seen at events like weddings and live music venues, usually features a large mid-bass driver (12" and 15" are common) and a large horn. However, as you might guess, this results in high driver center-to-center spacing. Fortunately, you don’t need concert-level output in a typical home environment, especially when one is sitting within a few meters of the speakers. You want it loud, but not deafeningly so, if you value your hearing.

After doing some research, I came across small horns like the 13 cm (~5") B&C ME10 and a relatively high output 6.5" mid-bass driver: Lavoce WSF061.52. By using physically small components, a driver center-to-center spacing of 16 cm (~6.3") that corresponds to about 3/4 wavelength distance at the crossover frequency turned out to be possible. You can reduce the distance even more if you’re willing to mount the mid-bass driver from the inside, but 16 cm is good enough for me.

The only problem left is that the mid-bass driver’s frequency response takes a nosedive starting around 120 Hz, even after adding a port. No problem, though – I have multiple subwoofers with dual 12" drivers for low-bass duty. Let’s get to work!

I designed a compact cabinet with internal bracing that touches the drivers, making the drivers part of the brace and reinforcing the front and rear sides of the cabinet.

Cabinet drawing with dimensions.

I cut the pieces on my CNC router using 18 mm (~3/4") MDF and then added 45-degree bevels to all sides on my router table. I also added a 6 mm MDF backing to the front baffle to make it stronger.

The pieces are simple enough that they can be cut with a circular saw or jigsaw, and the holes can be cut with a handheld router. The bevels are optional – one can also use regular butt joints after resizing the pieces, of course. I glued the pieces together using wood glue, aligning them with painter’s tape. Band clamps really came in handy for gluing.

Parts ready for dry fit

Borrowing an idea from KEF, just like in my KEF LS50 clone build, I attached the braces to the walls with 3M VHB tape instead of regular wood glue. The soft layer in between acts as constrained-layer damping to reduce resonances.

I glued the port, made out of a plastic water pipe (5.7 cm inner diameter), to the back wall with hot glue. The port is quite short (only 3 cm, not counting the outside MDF flare). The crossover board is installed on the side wall with standoffs. I placed M4 threaded inserts into the front baffle so that I could remove and install the drivers as many times as needed without having to worry about stripping the holes, which is all too easy with MDF.

Braces installed, drivers and crossover board mounted.

I was originally going to use an active crossover, but after doing a few simulations in VituixCAD, I noticed that a simple passive crossover with only five parts would work nicely. With active crossovers, you can get the frequency response ruler-flat, but you also need two channels of amplification and DSP –one for the mid-bass and one for the compression driver– whereas a passive crossover can make do with only one amplifier channel and no DSP. Additionally, active crossovers can have more background noise than passive crossovers because the amplifiers are connected directly to the drivers.

I used a second order low-pass (a 2.5 mH laminated silicon steel core inductor in series and a 15 µF capacitor in parallel) for the mid-bass driver. For the compression driver, I used a first order high-pass (a 3.3 µF capacitor in series) and an L-pad circuit (two 4.7 ohm resistors in series/parallel) to reduce its level and flatten the response. You wouldn’t normally use a first order high-pass with compression drivers, but once again we are going to use it in a home environment where it will unlikely to be stressed, and besides, the electrical roll-off starts quite early. Finally, the compression driver is connected in reverse polarity to avoid a dip in the frequency response at the crossover frequency.

Crossover board close-up view.

I patched the rough spots on the cabinet with wood filler, sanded it smooth, and applied 19 mm (3/4") roundovers to all sides on my router table. Then I took the speaker outside and painted it with several coats of MDF primer using a spray gun. Once the primer had dried, I sprayed several coats of top coat (steel blue). Since I was using water-based paints, I added some paint conditioner (pouring medium) to help reduce orange peel.

Cabinet Painted

Once the paint had dried, it was time to add absorption material to deal with the standing waves that would form inside the cabinet. I lined the walls with felt (5 to 10 mm thick) and placed extra-thick 60 mm pyramid foam on top, leaving enough space for the drivers, of course.

Absorption materials placed in cabinet

Here’s the side view of the cabinet with drivers waiting to be mounted. The mid-bass driver, Lavoce WSF061.52, has a 1.5" voice coil and a paper cone, while the compression driver, Lavoce DF10.142LM, has a 1.4" voice coil and a polyester (likely Mylar) diaphragm.

Cabinet side-view with drivers to be installed.

Finally, Valeria is complete standing next to one of my KEF LS50 Metas. They’re roughly similar in size. Valeria’s dimensions are 19 x 33 x 22.5 cm (WxHxD), while the KEF LS50 Meta is 20 x 30 x 28 cm (WxHxD).

Valeria next to a KEF LS50 Meta

So, how does it sound? By itself, the bass is anemic, as expected. However, as soon as you add a subwoofer to the mix, this little speaker transforms into something special. Female vocals are particularly lifelike, thanks to the effortless dynamics and silky smooth sound of the compression driver. With tracks like “Can’t Catch Me Now” by Olivia Rodrigo, I found myself turning up the volume higher and higher, yet the sound remained clean throughout.

Valeria really shines at high volumes. In fact, on some bass-heavy tracks, I discovered bass resonances in my room that I hadn’t noticed before. With the LS50 Metas, I couldn’t turn the volume up that high without seriously distorting the sound.

Now, some measurements… The average sensitivity (300 Hz-3,000 Hz) is approximately 92 dB/2.83V/1m, measured with a UMIK-1 microphone. This means that only 1 to 2 Watts of amplifier power is needed to achieve very loud volumes at 1 meter from the speaker.

The impedance plot looks relatively clean, with the exception of a resonance at around 950 Hz. How audible this is, it’s hard to say. The nominal impedance is 8 ohms, and the worst-case EPDR (Equivalent Peak Dissipation Resistance) is around 3 ohms at 200 Hz, making Valeria a relatively easy load for the partnering amplifier, as John Atkinson would say.

Impedance and phase plot.

The frequency response is mostly within ±3 dB from 120 to 20,000 Hz (quasi-anechoic measurement). The mid-bass to compression driver crossover frequency is 1.6 kHz. As for the subwoofer crossover frequency, 120 Hz looks like a good starting point. You can probably go a bit lower if you place the speaker close to walls, as this will result in some bass boost from nearby boundaries. It’s also a good idea to run the subwoofer(s) a few dB hot for optimal integration.

On-axis quasi-anechoic frequency response.

The horizontal directivity is ±45 degrees nominal, but starts to narrow gradually above 3.5 kHz. Thanks to the smooth directivity, the frequency response can be equalized without any ill effects.

Smooth horizontal directivity (normalized).

It’s surprising how a small speaker like this one can get so loud yet manage to stay so clean. The trade-off, of course, is little to no low-bass output by itself, but that’s precisely what subwoofers are for.

Update (August 3, 2024):

DZCreeper over on Reddit suggested adding a foam plug to the throat of the horn and pointed me to an expired patent by Earl Geddes, who’s well known for his extensive research on waveguides.

The idea is simple: Placing some open-cell foam inside the horn will absorb internal reflections, leading to cleaner sound. The downside is the loss of 2 to 3 dB of high-frequency output, but since the crossover is already running the compression driver a few dB hot relative to the mid-bass driver, this shouldn’t pose a problem.

I got some 30 pores-per-inch (ppi) open-cell foam from a pet store, typically used for things like aquarium filters. I then cut the foam with scissors, and placed it loosely inside the horn throat. It’s important not to pack it tightly, otherwise you will lose too much output above 10 kHz.

Horn phase plug made of open-cell foam.

With the foam plug in place, I was expecting consistent attenuation in the frequency range that the compression driver is producing, but I was pleasantly surprised to see the response immediately above the crossover frequency become smoother, not just lower in volume. It appears that the foam plug actually helped with the cancellation caused by the phase mismatch between the drivers over an octave above the crossover frequency.

As for any improvement in sound quality beyond the frequency response change, I can’t really tell. Geddes has written on several online forums that the effect would be subtle and not easy to measure, especially in well designed horns.

Frequency response after foam plug inserted.

With the foam plug, another pleasant surprise is the improved horizontal directivity, which has become more balanced.

Improved horizontal directivity with foam plug.

Alternative Crossover:
Finally, DZCreeper has graciously designed an alternative third order crossover for Valeria. It has a few more parts, but offers a relatively flatter response.

Update (Dec 2, 2024):

The frequency response graphs and the sensitivity figure have been updated to account for the speaker baffle effect in near-field measurements.

Teaching Programming with BASIC

Sun, 14 Jul 2024 00:00:00 +0000

I started programming in GW-BASIC on an IBM PC clone running MS-DOS. Back then, many so-called home and business computers came bundled with a BASIC interpreter, mostly made by or licensed from Microsoft.

And they all looked similar. You were greeted by a screen with a READY or OK prompt and a blinking cursor waiting for your input. The “screen editor” and interpreter were all in one in the true sense of the word – they weren’t bolted together like the separate text editors and interpreters/compilers we use these days. You wrote your programs inside the BASIC interpreter, which also doubled as a line-oriented text editor.

To change a line, you first listed the approximate range of lines you were interested in with the LIST command. For example, LIST 100-200 would list all the lines starting with 100 and ending with 200, assuming they fit on the typical 25-line screens of the day. There was no scrolling, so if the line you wanted to change wasn’t visible, you had to specify a smaller range. Then, you would go to the line with the cursor keys, make your changes, and, this is very important, press the Enter key to save the new line. After that, you would go to an empty line and enter RUN to run your program.

Here’s a simple BASIC program, which prints out the squares of integers from 1 to N as specified by the user:

10 INPUT "How many numbers to square? Enter 0 to exit. ", N
20 IF N = 0 THEN END
30 FOR I = 1 TO N
40   PRINT I ^ 2
50 NEXT I
60 GOTO 10

The line numbers are typically incremented by 10 to allow for the addition of more lines in between at a later time. Thankfully, GW-BASIC had a useful RENUM command that would automatically renumber some or all of the lines in your program and update any referenced lines. You can think of it as an early example of automatic refactoring.

Enough with the peculiarities of GW-BASIC, though. Let’s go back to our BASIC program. I’ve written code in quite a few languages over some three decades, and I believe the BASIC program above is probably as short yet thorough as you can get in demonstrating how computers actually work to absolute beginners.

In just 6 lines, you get user input, store it in a variable, compare the variable with a value, run an operation inside a loop, and finally jump back to the beginning. There are no functions, no object oriented programming, no libraries, no frameworks – just a few simple instructions and jumps that can be directly mapped to machine code.

Once the beginner grasps the building blocks of programming –variables, conditionals, loops, and branching– you can start showing them how to reuse code. BASIC doesn’t have local variables or real functions, but it does have subroutines, which can be utilized with the GOSUB statement.

The following example program calls the text mode window drawing subroutine starting at line 1000 with 7 variables set: X, Y, W (Width), H (Height), FG (Foreground color), BG (Background color), TITLE$ (Title text of the window). The subroutine draws windows by using the ASCII codes of the line drawing characters.

5 COLOR 7,0: CLS
10 X=10: Y=5: W=30: H=8: FG=15: BG=1: TITLE$="Info": GOSUB 1000
20 X=20: Y=8: W=40: H=10: FG=15: BG=4: TITLE$="Warning": GOSUB 1000
30 COLOR 7,0: END
1000 'Draws a window with title bar
1005 COLOR FG,BG
1010 LOCATE Y,X: PRINT CHR$(213) 'Top left
1020 FOR I=1 TO W-2: LOCATE Y,X+I: PRINT CHR$(205): NEXT I 'Top
1030 LOCATE Y,X+W-1: PRINT CHR$(184) 'Top right
1040 FOR I=1 TO H-2: LOCATE Y+I,X: PRINT CHR$(179): NEXT I 'Left
1050 FOR I=1 TO H-2: LOCATE Y+I,X+W-1: PRINT CHR$(179): NEXT I 'Right
1060 LOCATE Y+H-1,X: PRINT CHR$(192) 'Bottom Left
1070 FOR I=1 TO W-2: LOCATE Y+H-1,X+I: PRINT CHR$(196): NEXT I 'Bottom
1080 LOCATE Y+H-1,X+W-1: PRINT CHR$(217) 'Bottom right
1085 FOR I=1 TO H-2: FOR J=1 TO W-2: LOCATE Y+I,X+J: PRINT " ": NEXT J,I 'Inside
1090 LOCATE Y,X+(W-LEN(TITLE$))/2-1: PRINT " "TITLE$" " 'Title bar
1100 RETURN

We just drew our own windows from top to bottom, complete with borders, using only 17 lines of code. Once again, we didn’t rely on any libraries, and just used the built-in BASIC statements. Here is the output of the program:

Two text-mode windows drawn on screen by a GW-BASIC subroutine.

“It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration.” – Edsger Dijkstra

Now, obviously I don’t agree with Dijkstra here. Yes, BASIC lacks some universally agreed-upon programming concepts, and it does encourage the use of GOTO, which can easily lead to spaghetti code. However, at the lowest level, computer machine code doesn’t employ any of those concepts either. It’s just a bunch of CPU instructions and lots of jumping around.

If one wants to understand how computers work, I think it’s easier to start closer to the level they operate, and introduce useful abstractions like functional and object oriented programming as time goes on. Machine code and Assembly are too low level for beginners and even for veteran programmers. Popular programming languages like JavaScript and Python, on the other hand, are too high level – they’re far removed from the underlying hardware.

For me, BASIC feels about right as the first programming language. However, as soon as you start to feel its limitations, you should graduate to a modern language. I’m sure you will appreciate modern languages more once you’ve spent some time in BASIC, but you might also find that some modern languages and frameworks are more complicated than necessary.

Worth a look:

PC-BASIC, a cross-platform interpreter for GW-BASIC
Writing a Tetris clone in GW-BASIC demonstrates advanced GW-BASIC features

Discussed on:

I Don't Trust My Own Code

Mon, 27 May 2024 00:00:00 +0000

Recently, I had a conversation with a junior developer on my team. Let’s call him Alan. We were talking about a new notification feature that was going to be used to send reminder e-mails to potentially thousands of people if they had forgotten to enter certain data in the last month or so.

Alan was confident that the code he had written was correct. “I’ve tested it well,” he said. I had heard some variation of this statement from so many developers over the years that I sighed: “I’ve been writing code in a professional capacity for over two decades now, and yet even I don’t trust the code I’ve written.”

You can probably guess what happened to Alan’s “well-tested” code. In testing, a number of bugs were discovered, many of which were caused by incorrect assumptions on his part.

Now, I’m not picking on Alan. In fact, I hired him and the rest of the team for that matter. He may be short on experience and a little overconfident, but generally writes decent code, and constantly tries to improve himself, a trait that is more important than experience in my book.

As for me, I’m certainly not one of the best software developers in the world. I may not be a great one either. I’ve seen quite a few developers who have written qualitatively worse code than I have, and relatively few who have written better code. So, statistically speaking, I consider myself above average. How much above average, it’s hard to say, but it doesn’t really matter as long as one is aware of their limitations.

“The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.” – Edsger Dijkstra

Oh yes, clever tricks. It’s tempting to use them from time to time, but are you going to remember how you solved a problem when you revisit your code six months later? If you need to go through your code line by line just to understand what it’s doing, another developer reading it is unlikely to fare better.

It’s often said that writing code is easier than reading it, but that doesn’t have to be the case. When writing long sections of code, I find it helpful to stop every once in a while and review the code as if I were someone seeing it for the first time. I ask myself a few basic questions: Are my functions short? Do they have descriptive names? Is complex logic encapsulated within their own functions? Am I passing too many arguments? And so on.

Assuming that one writes reasonably easy-to-understand code, a major source of errors typically stems from the assumptions made by developers. Reality is messy. There are almost always edge cases that will bite you in the back.

Numerous “falsehoods programmers believe about X” pages on the web list commonly held but incorrect beliefs. For brevity, I will only include some I had the displeasure to deal with:

Falsehoods About Time:

A timestamp represents the time that an event actually occurred.
The time zone in which a program has to run will never change.
Daylight saving time happens at the same time every year.
Weeks start on Monday.
Time always moves forwards.
My software stack will handle it without me needing to do anything special.
[More]

Falsehoods About Text:

ASCII data represents text.
Interpreting common ASCII data such as tabs, carriage returns, and line-feeds is defined.
A Unicode file that only contains ASCII letters, encoded in UTF-8, is a valid ASCII file.
Latin-1 can be mapped directly into Unicode.
[More]

Falsehoods About Prices:

Prices can’t have more precision than the smallest sub-unit of the currency.
You can always use a dot (or a comma, etc.) as a decimal separator.
Given two currencies, there is only one exchange rate between them at any given point in time.
[More]

Finally, for a curated list of a great many falsehoods, try this.

“Only the paranoid survive.” – Andy Grove

A piece of advice, if I may: Assume nothing. Don’t trust anything including your own code. List your assumptions. Double, triple-check every single one of them. If some of your assumptions are wrong, it doesn’t matter if your code is correct – it will inevitably fail. Challenge your assumptions, test them, and adjust as necessary. Trust me, you will sleep better at night.

Easy Application Deployments with Linux

Sun, 19 May 2024 00:00:00 +0000

Linux comes with all the basic tools necessary to deploy an application to development and production environments, and to roll back to any past version if something goes wrong. It takes just a few commands to set everything up.

Creating Application User

It’s a good idea to create a user with limited privileges to deploy and run your application. Typically, your main user account has root-level access via sudo, which poses a security risk if your application is compromised.

The following command will create a user named ‘app’ with no password, requiring an SSH key pair to log in for increased security:

me@server:~$ sudo adduser --disabled-password --gecos '' app
Adding user `app' ...
Adding new group `app' (1001) ...
Adding new user `app' (1001) with group `app' ...
Creating home directory `/home/app' ...
Copying files from `/etc/skel' ...

We also need to ensure that the services started by ‘app’ continue to run after logout:

me@server:~$ sudo loginctl enable-linger app

We can now switch to our newly created account and generate an SSH key pair. It’s fine to press Enter to accept the default settings. We will use this key pair exclusively to pull code from our application’s Git repository:

me@server:~$ sudo su - app
app@server:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/app/.ssh/id_rsa): 
Created directory '/home/app/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/app/.ssh/id_rsa
Your public key has been saved in /home/app/.ssh/id_rsa.pub
...

To view the contents of the public key file, run the following command and copy its output to the online Git repository of your choice, such as GitHub or GitLab, as a deploy key (normally found under Settings):

app@server:~$ cat ~/.ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAA... app@server

Once the deploy key is set up, we can clone our application’s repository with:

app@server:~$ git clone git@github.com:me/my-app.git

Before we continue, it’s important to add the SSH public key used for accessing our regular account to the ‘app’ account’s authorized_keys file, enabling direct SSH access to ‘app’.

app@server:~$ nano ~/.ssh/authorized_keys

From this point on, we won’t be needing our regular user account anymore. The rest of the commands must be run after SSH’ing to ‘app’.

Deploy Script

Since we have our application code in place, it’s time to create a deploy script. Here’s one for a typical Node.js application:

app@server:~$ nano deploy
#!/bin/bash

APP_REPO='/home/app/my-app'
APP_DEPLOYMENTS='/home/app/deployments'

# Exit immediately if a command fails
set -e

# Pull changes
cd $APP_REPO
git reset --hard
git pull

# Install dependencies and run tests
npm install
npm run test

# Create new deployment directory and copy the code over
NOW=$(date --iso-8601=minutes | sed 's/:/-/g' | cut -c 1-16)
mkdir -p $APP_DEPLOYMENTS
cd $APP_DEPLOYMENTS
cp -a $APP_REPO $NOW

# Point to new deployment using a symbolic link
ln -vfns $NOW current

# Restart app
systemctl --user restart my-app

We make the deploy script executable with:

app@server:~$ chmod +x deploy

In the deploy script, we pull changes, install dependencies, run tests, and create a new deployment directory named with the current date and time, such as 2024-05-19T16-42. Finally, we update the symbolic link named current to point to the new deployment directory and then restart the application.

After a few executions of our deploy script, the deployments directory should look like the following:

app@server:~$ ~/deploy
...
app@server:~$ ls -l ~/deployments/
drwxrwxr-x 4 app app 4096 May 18 09:53 2024-05-18T09-53
drwxrwxr-x 4 app app 4096 May 19 13:25 2024-05-19T13-25
drwxrwxr-x 4 app app 4096 May 19 16:42 2024-05-19T16-42
lrwxrwxrwx 1 app app   16 May 19 16:42 current -> 2024-05-19T16-42

If anything goes wrong with the deploy script, such as failing tests, no harm will be done because the script exits upon the first error encountered.

Rolling Back Deployments

To roll back to a previous deployment, just two commands are needed:

Update the symbolic link to point to a different deployment directory.
Restart the application.

app@server:~$ ln -vfns ~/deployments/2024-05-19T13-25 ~/deployments/current
app@server:~$ systemctl --user restart my-app

Service File

In the deploy script and the rollback example above, we restart our application by using systemctl, a widely used service manager in Linux. However, we haven’t created a service file for our application yet, so let’s do that:

app@server:~$ mkdir -p  ~/.config/systemd/user/
app@server:~$ nano ~/.config/systemd/user/my-app.service
[Unit]
Description=My App

[Service]
WorkingDirectory=/home/app/deployments/current
Environment=DB_HOST='localhost'
Environment=DB_PORT=5432
Environment=DB_USERNAME='my_app'
Environment=DB_PASSWORD='...'
ExecStart=/home/app/.nvm/versions/node/v20.13.1/bin/node server.js
RestartSec=30
Restart=always

[Install]
WantedBy=multi-user.target

The service file is mostly self explanatory. We define the working directory of our application, any environment variables the application needs, the command to start the application, and the application restart behavior in case of a crash.

The WantedBy=multi-user.target line ensures that our service is launched only after the server can connect to the network in the event of a server restart.

For a complete list of options, you might want to take a look at Systemd service unit configuration.

Finally, we can enable and start our service with the following commands:

app@server:~$ systemctl --user enable my-app
app@server:~$ systemctl --user start my-app

The status of the service can be checked with:

app@server:~$ systemctl --user status my-app
● my-app.service - My App
     Loaded: loaded (/home/app/.config/systemd/user/my-app.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-05-19 09:27:08 UTC; 4s ago
   Main PID: 1582880 (node)
     CGroup: /user.slice/user-1001.slice/user@1001.service/my-app.service
             └─1582880 /home/app/.nvm/versions/node/v20.13.1/bin/node server.js

May 19 09:27:08 server systemd[1563720]: Started My App.
May 19 09:27:08 server node[1582880]: Server is running on http://localhost:8000

We can review the logs our application generates with journalctl. For example, the following command shows the last 100 entries:

app@server:~$ journalctl --user -n100 -u my-app

Bonus: Putting our Application Behind a Web Server

Our application is now up and running on http://localhost:8000, but it lacks HTTPS support and features such as HTTP access logs. We can set up a web server as a reverse proxy to redirect requests to internal ports.

Let’s switch to our main user account, install the nginx web server, and create a configuration for our application:

me@server:~$ sudo apt install nginx
me@server:~$ sudo nano /etc/nginx/sites-available/default

The following configuration redirects all HTTPS requests to our application. HTTP requests are automatically redirected to HTTPS as well. If you are using a DNS provider like CloudFlare, you can use the SSL certificate files they provide, or you can use a free SSL certificate service like Let’s Encrypt’s Certbot.

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name example.com;
    ssl_certificate     /etc/ssl/certs/example.com.crt;
    ssl_certificate_key /etc/ssl/certs/example.com.key;

    location / {
        proxy_pass          http://localhost:8000/;
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
    }
}

server {
    listen 80;
    server_name example.com;
    return 301 https://$host$request_uri;
}

After we’ve saved the configuration, it’s a good idea to check its integrity with nginx configtest. If all is well, we can then tell nginx to reload its configuration:

me@server:~$ sudo service nginx configtest 
 * Testing nginx configuration                                                     [ OK ] 
me@server:~$ sudo service nginx reload

Web server access and error logs can be found in /var/log/nginx.

Wrap up

We’ve made application deployments and rollbacks possible with nothing more than the built-in Linux tools. One can easily create multiple environments for development, staging, and production. All it takes is a deploy script, a service file, and a few commands run over SSH.

Back to Basics in Web Apps

Wed, 24 Apr 2024 00:00:00 +0000

In the beginning, there was only HTML. The first official HTML specification focused on semantic markup. There were minimal styling tags and attributes. It was up to the web browser how to render the markup in an HTML document.

The whole specification was refreshingly simple. You could easily read it in one sitting. Over three decades later, we are still using the same HTML tags and then some.

In Principles of Design, Tim Berners-Lee, explains why he came up with HTML as the language to define web pages:

Computer Science in the 1960s to 80s spent a lot of effort making languages that were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The reason for this is that the less powerful the language, the more you can do with the data stored in that language. If you write it in a simple declarative form, anyone can write a program to analyze it in many ways. (…) I chose HTML not to be a programming language because I wanted different programs to do different things with it: present it differently, extract tables of contents, index it, and so on.

CSS

There were attempts to expand the styling capabilities of HTML with tags like <font>, but a better solution arrived a few years later in 1996: CSS.

CSS enabled the separation of content and presentation by using style sheets. CSS relied on rules (e.g. h1, h2, h3 { font-family: sans-serif }) to define styles for HTML markup. It was generally easy to understand, except for the concept of “specificity”, which caused countless headaches for generations of web developers. As a result, a number of workarounds such as BEM and Tailwind emerged over the years.

JavaScript

As the final cornerstone technology of the web, the confusingly named JavaScript actually arrived shortly before CSS, but became a standard under the name ECMAScript later in 1997.

Despite the name, JavaScript has virtually nothing to do with Java except for a somewhat similar syntax. Unlike Java, JavaScript has dynamic typing, prototype-based object-orientation, and first-class functions (Java added support for anonymous functions much later).

JavaScript was designed to add dynamic behavior to otherwise static web pages, but despite its many strengths, it was arguably the world’s most misunderstood language for quite some time.

Ajax

JavaScript received a massive boost of popularity from Ajax, a browser technology that allows a web page to exchange data with servers in the background. Before Ajax, many user actions like submitting a form required a complete page refresh. Ajax enabled the separation of data exchange and presentation, and dynamic web apps as we know them today were born!

The first popular web apps started to appear in the mid-2000s, with Google’s Gmail and Maps as prime examples.

jQuery

As web apps started to appear, the tooling for web apps was seriously lacking. Enter jQuery, a JavaScript library which allowed developers to use CSS selectors to access any DOM element. jQuery supports a large number of chainable methods for DOM manipulation, event handling, and Ajax.

Here’s a typical piece of jQuery code. It’s probably not hard to guess what it does:

$('ul.tablist li').addClass('tab').on('click', selectTab);

Thanks to its simplicity and power, it’s no wonder jQuery is still widely used nearly two decades after its initial release. However, a major issue with jQuery is the close coupling of presentation (DOM elements) and JavaScript logic, which can make maintenance and scalability hard.

AngularJS

Starting with the 2010s, frameworks specifically designed to create Single Page Applications started to appear with the arrival of AngularJS.

AngularJS’s main selling point was two-way data binding. AngularJS keeps the data model and the corresponding HTML (view) in sync automatically. In other words, whenever one changes, the other changes too. This reduces the need for direct DOM manipulation via libraries like jQuery at the expense of additional complexity and potential performance issues introduced by the framework.

Here’s how a simple counter can be implemented in AngularJS:

<div ng-app="myApp" ng-controller="counterCtrl as counter">
    <button ng-click="counter.increment()">Increment</button>
    <div>Count: {{ counter.count }}</div>
</div>

<script>
    angular.module('myApp', []).controller('counterCtrl', function () {
        this.count = 0;
        this.increment = function () {
            this.count++;
        };
    });
</script>

React

React, which emerged a few years after AngularJS, is now the most popular web frontend library.

Unlike AngularJS, which uses the existing HTML as a template, React components dynamically generate HTML using JavaScript. JSX, a JavaScript syntax extension, is typically used for HTML generation but requires a transpiling step before use.

The creators of React argued that “instead of artificially separating technologies by putting markup and logic in separate files, React separates concerns with loosely coupled units called ‘components’ that contain both.”

Under the hood, React utilizes a virtual DOM to optimize updates, altering only the parts of the browser DOM that have changed, thus enhancing performance. There’s no direct support for two-way data-binding, but it’s possible to achieve a similar result by other means.

The following example is a counter implemented in React with hooks, a later addition to the library:

import React, { useState } from 'react';

const App = () => {
    const [counter, setCounter] = useState(0);
    const increment = () => {
        setCounter(counter + 1);
    };
    return (
        <div>
            <button onClick={increment}>Increment</button>
            <div>Count: {counter}</div>
        </div>
    );
};

export default App;

Vue.js

Shortly after React came out, Evan You created Vue.js with the goal of creating a lightweight version of AngularJS. It’s currently the second most-popular web frontend framework, following React. Just like React, Vue internally uses a virtual DOM, but it also supports two-way binding.

Vue supports single-file components (SFCs), which encapsulate HTML templates, JavaScript, and CSS in one file, keeping them clearly separated. The following example is in SFC format, using Vue’s original Options API. Vue also supports a new API called “Composition” that works similar to React hooks.

<template>
    <button @click="increment">Increment</button>
    <div>Count: {{ count }}</div>
</template>

<script>
    export default {
        data() {
            return {
                count: 0,
            };
        },
        methods: {
            increment() {
                this.count++;
            },
        },
    };
</script>

<style scoped>
    button {
        color: blue;
    }
</style>

Other popular web frontend frameworks include Angular, which is quite different than the original AngularJS, and Swelte.

Bundlers

Web bundlers such as esbuild and webpack are often used to manage the varying levels of support for JavaScript and CSS features across web browsers. These tools transpile modern JavaScript and CSS code to formats old web browsers can understand.

Bundlers also decrease the size of JavaScript and CSS code by performing tasks like tree shaking for unused code elimination, removing whitespace, and shortening variable names.

Bundlers typically consolidate hundreds or even thousands of JavaScript and CSS files from a project into a few optimized files for easier distribution.

Until recently, using web bundlers as code transpiling tools was mostly a necessity, but with the retirement of Internet Explorer 11, modern evergreen browsers (Chrome, Edge, Firefox, and Safari) no longer require transpilation.

Progressive Enhancement

The idea of progressive enhancement is about prioritizing HTML content over everything else. Progressive enhancement has been around for quite some time, but gained renewed attention with the proliferation of frontend frameworks.

Not all web apps are single-page applications, and not all single-page applications are complex enough to warrant a full-blown frontend framework.

Starting with HTML and CSS, and enhancing them with lightweight JavaScript frameworks like Alpine.js and htmx offers numerous advantages, including faster initial load times, better accessibility, easier indexing of page content by search engines, and reduced or no need for bundlers.

Here’s our counter implemented in Alpine.js:

<div x-data="{ count: 0 }">
    <button @click="count++">Increment</button>
    <div>Count: <span x-text="count"></span></div>
</div>

Although Alpine encourages “sprinkling” JavaScript into HTML as a form of progressive enhancement, the above example can be expanded into a reusable component, as shown below:

<div x-data="counter">
    <button @click="increment">Increment</button>
    <div>Count: <span x-text="count"></span></div>
</div>

<script>
	const app = () => {
	    Alpine.data('counter', () => ({
	        count: 0,
	        increment() {
	            this.count++;
	        },
	    }));
	};

	document.addEventListener('alpine:init', app);
</script>

Too Much of a Good Thing

We’ve come a long way since the early days of the web, yet we are still using the same three core technologies: HTML, CSS, and JavaScript, all of which have vastly improved over the past three decades. To give a few examples, HTML now natively supports web components, CSS layouts have been enhanced by flexbox and grid, and JavaScript proxies have simplified the implementation of two-way binding.

Despite this progress, it seems we rely too much on JavaScript, the most powerful of the three core web technologies. Had Tim Berners-Lee chosen a more powerful language as the foundation of the web, instead of HTML, would the web have gained as much traction? How would search engines even work? They couldn’t merely parse web pages; they would need to execute code and evaluate the output somehow. Does anyone even remember websites built entirely with Flash, let alone Java applets?

Coming Full Circle

As much as I like JavaScript, I think it’s time to return to the basics. I find it refreshing to write, not generate, HTML and CSS, and then sprinkle some JavaScript for interactivity. I’m sure many web developers would feel the same. Let’s give it a try.

Beyond Foreign Keys

Fri, 05 Apr 2024 00:00:00 +0000

In a relational database, foreign keys are normally used to associate records stored in different tables. Let’s say we want to add a forum to our website where people can discuss matters related to our product. To keep things simple, we will model our forum with just three tables: user, post and topic. The relationships between the tables can be modeled by asking a few questions:

Who wrote a given post? post.user_id references user.id.
Which posts are under a given topic? post.topic_id references topic.id.
Who started a given topic? topic.user_id references user.id.

Using foreign keys to associate tables for a simple forum.

This is all well and good, but each relationship requires adding a new column to a table, and we don’t even have many-to-many relationships in this example, which will necessitate creating additional tables.

Wouldn’t it be nice to define relationships dynamically without having to add extra columns or tables? And while we’re at it, how about having sparse relationships by associating a record directly with any other record like “post X was last edited by user #123” or “post X was flagged for review by user #456” (who happens to be a moderator)?

With that idea in mind, I’ve created Tie-in, a relational data component library for Node.js. Tie-in lets you store and query records that can be related to any other record.

Let’s create a user, a post, and a topic by using data components (more on how to define them later) in the relational database of our choice (in this case PostgreSQL):

const dbConfig = {
    client: 'pg',
    connection: 'postgresql://localhost/me'
};

// Load Tie-in and data component definitions
const tie = require('tie-in')(dbConfig);
const user = require('./components/user')(tie);
const post = require('./components/post')(tie);
const topic = require('./components/topic')(tie);

async function firstSteps() {
    // Register the components we will use
    await tie.register([user, post, topic]);

    // Create a user named Asuka
    const userId = await tie.create(user({ username: 'Asuka', email: 'asuka@localhost', country: 'JP' }));

    // Create a post and make its author Asuka
    const postId = await tie.create(post({ content: 'Hi!' }), {
        upstream: [user({ id: userId, relType: 'author' })],
    });

    // Create a topic and make the topic starter Asuka, also make the post a child of this topic
    const topicId = await tie.create(topic({ title: 'First Topic' }), {
        upstream: [user({ id: userId, relType: 'starter' })],
        downstream: [post({ id: postId, relType: 'child' })],
    });

    // Retrieve topic and related records
    const topicRecs = await tie.get(topic({ id: topicId }));
    console.log(JSON.stringify(topicRecs, null, 2));
}

firstSteps();

Once the individual records are in place, we retrieve the newly created topic with tie.get, which recursively retrieves all related records and groups them together. In Tie-in, related records can be upstream (referencing a record) and/or downstream (referenced from another record). Related records can optionally have types specified by relType.

{
  "topic": [
    {
      "self": {
        "id": 1,
        "title": "First Topic",
        "createdAt": "2024-03-28T12:27:51.542Z",
        "updatedAt": "2024-03-28T12:27:51.542Z"
      },
      "user": [
        {
          "self": {
            "relType": "starter",
            "id": 1,
            "username": "Asuka",
            "email": "asuka@localhost",
            "country": "JP",
            "createdAt": "2024-03-28T12:27:51.531Z",
            "updatedAt": "2024-03-28T12:27:51.531Z"
          }
        }
      ],
      "post": [
        {
          "self": {
            "relType": "child",
            "id": 1,
            "content": "Hi!",
            "createdAt": "2024-03-28T12:27:51.538Z",
            "updatedAt": "2024-03-28T12:27:51.538Z"
          }
        }
      ]
    }
  ]
}

Defining Components

To define a component, you call tie.define with the following arguments:

name: Name of the component
schema: A function that defines the database table schema. Tie-in uses knex under the hood. Table field names must be in snake_case for maximum compatibility across different database systems. Tie-in does the snake_case to camelCase conversions and vice versa automatically.
data: A function that accepts an object with field names in camelCase, maps those fields to the database table fields created with schema, and returns the resulting object.

In its simplest form, a component definition should look like the example below. The only requirement is that there must be a field named id that uniquely identifies each record.

module.exports = (tie) => {
    const name = 'post';

    async function schema(knex, tablePrefix) {
        const tableName = tablePrefix + name;
        if (!(await knex.schema.hasTable(tableName))) {
            return knex.schema.createTable(tableName, function (table) {
                table.increments('id').primary();
                table.text('content').notNullable();
                table.timestamps(false, true);
            });
        }
    }

    function data(input) {
        return {
            id: input?.id,
            content: input?.content,
            created_at: input?.createdAt,
            updated_at: input?.updatedAt,
        };
    }

    return tie.define(name, schema, data);
};

You might probably want to validate the input passed to the data function. Tie-in uses fn-arg-validator for internal data validation, and exposes it through tie.is, but you can of course use any other library you would like.

tie.get

Syntax: tie.get(comp, filters = {})

When you pass a component instance (comp) to tie.get, it uses the component instance’s data for search. Here are some examples:

await tie.get(user( {country: 'JP'} )) returns the users from Japan.
await tie.get(user( {country: 'JP', username: 'Asuka'} )) returns the users from Japan having the username ‘Asuka’.
await tie.get(user()) returns all users.

tie.get supports a wide range of options to filter the results returned. For example:

async function getPostCountsGroupedByUser(topicId) {
    const filters = {
        aggregate: [{ fn: 'count', args: '*' }],
        group: { by: user(), columns: ['id', 'username'] },
        filterUpstreamBy: [topic({ id: topicId })],
        orderBy: [{ column: 'username', order: 'asc' }],
        limit: 20,
    };
    return tie.get(post(), filters);
}

For a complete list of the available functions and options, you can visit Tie-in’s GitHub page.

How is this Different than an ORM?

Tie-in isn’t an ORM (Object-Relational Mapper). In fact, Tie-in’s data component instances are immutable – once set, you can’t change the values a component instance holds.

Tie-in components have embedded schema definitions. In a sense, they are self-contained.

Tie-in doesn’t suffer from the worst performance issues that plague various ORMs such as the infamous N+1 query problem. Tie-in also tries its best to reduce the number of queries it generates by merging related queries together.

Can I still use Foreign Keys?

Absolutely. There’s nothing preventing you from using them. Foreign keys are great! Tie-in doesn’t replace foreign keys, but supplements them.

Fun with Relationships

In my opinion, Tie-in makes it fun to write database-level code by removing the need to define relationships for anything and everything. You can easily have an ad-hoc relationship between two records and nothing else.

Tie-in’s recursive record retrieval functionality makes it almost too easy to get the data you need without having to worry about all the joins, and dare I say, it’s a little like magic.

GitHub Repository: Tie-in

Software Architectures in a Nutshell

Tue, 26 Mar 2024 00:00:00 +0000

All problems in computer science can be solved by another level of indirection." – Butler Lampson

The great majority of the software architectures currently in use are variations of the layered architecture, and what really sets them apart is the implementation details. Some might find this statement controversial, but in my experience, most software applications rely on code organized in layers to manage complexity. Some of the layers may utilize message queues or microservices, but that doesn’t necessarily make the architecture event or microservices based.

In-n-Out

All non-trivial computer programs, regardless of their complexity, accept some input, do something with it, and produce an output.

Modules

Most programs have some internal structure in the form of modules. Modules can be implemented by using functions, objects, or a mixture of both. Ideally, each module should be responsible for a well defined task and nothing else. This principle is called Separation of Concerns.

Layers

Once you reach a certain number of modules, it often makes sense to introduce a hierarchy, and put related modules in separate layers. This is known as Multi-tier Architecture. Notice the direction of arrows. Communication is normally unidirectional, and usually obeys the following rules:

Modules can call the other modules in the same layer.
They can also call the modules in the layer directly below.
The modules in lower layers cannot call the modules in upper layers at all. In fact, they don’t even know if an upper layer actually exists unless they receive a call from it.

Since the modules in a layer cannot make any assumptions about the structure of upper layers, it’s possible to make changes to the individual layers without affecting others. You can even completely replace a layer with something else. For example, the calls from the upper layer can come from a web page or a mobile app, and the lower layer answering those calls doesn’t care where they come from as long as they are in a format it can understand.

If you don’t follow the general rules above, layers will eventually intertwine, and as a result, your code will likely turn into spaghetti or a big ball of mud. Yes, these are technical terms :)

Events

At Faradai, we collect millions of sensor readings from thousands of IoT devices every day, but we don’t process them the moment they arrive at our data collection layer. We store the readings in a message queue first, and process them at a rate we know our servers can comfortably handle. Otherwise, unexpected spikes in traffic could place a heavy burden on the servers.

You can think of message queues as an example of event-based architecture in which the so-called producers (in our case, IoT devices) store events (sensor readings) that are then consumed by the respective modules in our data processing layer.

We could have sent the sensor readings directly from our data collection layer to the data processing layer, and the early versions of our platform had actually worked that way, but the amount of data we processed increased significantly over time, so we put the message queue as an intermediate layer between the two.

Microservices

When you “carve out” some of your modules or even layers and turn them into independent programs that you can call from your main application or any other application for that matter, you get yourself microservices.

Unlike modules that rely on the environment provided by your application, microservices are standalone entities. The usual way to access them is over the network, so it’s easier to place them in separate servers. They can be developed separately by different teams in different programming languages or databases if necessary.

There’s a price for the flexibility offered by microservices though. Instead of a single application to worry about, you now have multiple dependencies that may go down or be slow to respond, which can negatively affect the responsiveness of your main application.

There are tools to “orchestrate” multiple microservices such as Kubernetes, but it’s a good idea to do a cost-benefit analysis before committing to microservices as the added complexity might not justify the potential benefits.

Plug-ins

Plug-ins offer a way to dynamically change the behavior of an application through components that act on the data provided by the application (aka host). For example, at Faradai, we generate alarms upon detecting abnormal energy consumption patterns based on a complex set of rules unique to each client. Since every alarm scenario can be unique, we provide a plug-in architecture called RulesEngine. Each alarm scenario can be thought of as a plug-in that is coded in an embedded scripting language, and can retrieve and process data from the host.

Notice the bidirectional nature of the arrows between the application and plug-ins. Plug-ins have access to various application modules, and supplement the capabilities of the application.

Other Architectures

There are software architectures other than the ones we’ve gone over so far, but as we approach software architecture from a code organization perspective, these are the major ones. Other architectures such as pipes and REST focus more on data flow and access respectively rather than code organization.

Layers, Events, Microservices, Plug-ins all Combined

There is a time and place for all architectures, and they can usually live together as a hybrid architecture. It’s probably not a good idea to exclusively commit to a single architecture as each architecture has its strengths and weaknesses, and we can get the most out of them by using each one in places where they excel.

Making Technical Decisions

Tue, 05 Mar 2024 00:00:00 +0000

Over the past two decades, I’ve worked as a software developer, tech lead, and CTO at various startups, including my own. I’ve accumulated a number of principles I use to make major technical decisions. Most of the principles outlined here were learned the hard way. You might not fully agree with all of the principles, but I hope at least some of them could help those who struggle with tough technical decisions.

Focus on your core business

Before implementing a new functionality, the very first question you should ask is whether that functionality will be a part of your company’s core business. If the answer is no, you should look for a third-party library or tool, preferably open-source. If the available options on the market don’t quite solve your needs, then you might consider implementing the functionality yourself, but only after doing a careful cost-benefit analysis. Whenever possible, avoid writing new code – the best code is no code.

When I first joined Faradai, the company had the first version of its energy management software platform up and running, successfully serving customers. The platform generally worked fine, but AML, its custom web frontend framework, was hard to use and extend.

Since no other company used AML, there weren’t any outside developers exposed to it, so we had to train new developers before they became productive in it. Worse, some of the developers complained that their newly learned AML skills wouldn’t be useful elsewhere, and you know what, they are right!

In a startup, time to market is critical, and training new developers slowed us down. Had we used a popular frontend framework, we could have simply hired React or Vue developers and had them write code from day one.

Finally, when our AML framework couldn’t do something we needed, we had to stop working on the core application code and work on AML instead. As a software company operating in the carbon accounting and energy management domains, maintaining a custom frontend framework was clearly not our core business.

No rewrites unless absolutely necessary

One of the first major technical decisions I made at Faradai was to switch from AML, our custom web frontend framework, to Vue. However, there was a lot of code written in AML, so the decision to rewrite the frontend in Vue wasn’t made lightly as explained above. Since the rewrite would take some time, we did the bare-minimum to maintain AML until we reached feature parity in the Vue version, which also added features like dashboard widgets that could be dragged and dropped anywhere on the screen, something AML lacked.

Although we had to rewrite the frontend code, the backend stayed mostly as it was before because the code was solid and needed just a bit of refactoring and performance tweaks.

Rewriting large sections of code is highly risky. Not only is it easy to break existing functionality, but it wastes precious time that could have been used to add new features users need.

Have a simple tech stack

We selected Vue over React because, for us, Vue was simpler to learn and use. There were other, even simpler web frontend frameworks, but they didn’t quite meet our needs. As Einstein had once said, “Everything should be made as simple as possible, but no simpler,” it’s best to avoid introducing unnecessary complexity since it might come back to bite you in the future in the form of high maintenance and financial costs.

When in doubt, choose a monolith over a bunch of microservices, a load balanced application server over a Kubernetes cluster, PostgreSQL over the database flavor of the day, and so on. If you ever start to outgrow your current tech stack, you can always add more complexity later, but you cannot always simplify an overly complex tech stack.

Minimize your dependencies

In today’s highly connected world, software applications don’t exist in a vacuum. There can be a vast number of dependencies as far as the eye can see. Just looking at the dependency tree of a typical Node.js application can reveal tens of primary dependencies, with each dependency having its own dependencies and those dependencies having their own, ad infinitum. If you count them all, you can easily end up with hundreds or even thousands of dependencies in total, and this is just the code side. The dependency trees on the server side can be even deeper if you go all the way down to kernel level.

With so many dependencies, it’s a miracle that anything works. As software developers, we really are standing on the shoulders of millions of software developers.

Since it’s next to impossible to have control over your entire dependency tree, the only thing that you can realistically control is your primary dependencies, and it’s a good idea to keep them as few as possible. Statistically speaking, the more dependencies you have, the greater the chances of something going wrong.

Avoid scope creep

Software is infinitely malleable, and there’s always the temptation of adding one more feature. Don’t, under any circumstances, accept change requests if the previously agreed-upon timeline for development would stay the same.

Many software developers are eternal optimists, a good trait that’s necessary to deal with a seemingly unending string of technical challenges, but it can also be our undoing. Software development is a complex process, and there are oftentimes previously unrealized technical issues that need to be addressed before implementing a feature. There’s a good chance that a seemingly minor change may require more time than originally thought.

Finally, don’t fall into the trap of non-technical people trying to convince you with shaky arguments like “How hard could it be? App X already has that feature.” The truth is they don’t have the technical background to speculate about the difficulty of technical problems. However, you still need to explain the technical aspects to them in plain language. A curt “No, that’s not possible.” won’t suffice.

Prioritize by taking business realities into account

There are many methods to choose for prioritizing the items in your roadmap. It’s not critically important which one you use, but you should pick one and stick with it. I personally like Cost of Delay because it focuses on the financial impact of feature requests – a seemingly important feature may easily turn out to be not so important for your company from a financial perspective.

Jim Rohn said, “Time is our most valuable asset, yet we tend to waste it, kill it, and spend it rather than invest it.” Companies need a steady cash flow to stay healthy, and implementing software features that don’t bring in revenue is not only a waste of time but also incurs a significant opportunity cost.

Build prototypes before making risky decisions

Some of the decisions you make may involve a lot of unknowns that need to be explored before making a final decision. In cases like that, it pays to do some exploratory work before fully committing to a plan of action.

For example, digitizing energy bills is fraught with risks. It’s all too easy to get the numbers wrong. We weren’t comfortable with the potentially high error rate. So, we explored various technologies. OCR was the obvious choice, but it performed poorly in our experiments.

Switching to large language models improved reliability significantly, but if we hadn’t tried exploring different technologies and built prototypes, we might have made the error of committing to using OCR, costing us a great deal of time and money.

Work in cross-functional teams

A successful software application isn’t just written – it’s the joint effort of a cross-functional team of product managers, UX designers, software developers, and QA engineers.

If a UX designer creates a screen design that would take twice as long to implement as a more traditional design, the software developers can offer some revisions for a faster turnaround as long as the original design isn’t compromised much.

Similarly, if the scope of a feature as outlined by the product manager is too complex to implement in a reasonable amount of time, the scope can be streamlined according to the feedback of the rest of the team.

Finally, if the software developers say they need to pay off some of the accumulated technical depth by doing some refactoring work, the development schedule can be revised to allow time for the refactoring.

Trust your team

This is the most important principle by far. If you can’t trust your team to make the necessary day-to-day technical decisions, then you will be quickly overwhelmed and your team will likely be demoralized as well. As the Russian proverb says, “Trust, but verify”.

Economics of Bugs

Tue, 20 Feb 2024 00:00:00 +0000

Software development would be a lot easier without so many bugs. In a typical software application, there’s a never-ending stream of bugs as far as the eye can see. Some of the bugs make users’ lives miserable, some of them are annoying but can be dealt with workarounds, and the rest are mostly minor inconveniences. If that sounded depressing, I’m sorry, that wasn’t my intention, I’m only trying to survey the current software quality landscape not only as a software developer but a user.

When writing code, unless there’s the possibility of putting the lives of people in danger or losing a lot of money, software developers are often incentivized to produce code quickly since time to market is almost always deemed to be critical by the powers that be. So, developers write code that works anywhere between 0.1% of the time (works on my machine) to 99.9% of the time (mostly bug-free). Given that the actual application code software developers write is only a fraction of the total code that is running on a computer (there are many dependencies such as the operating system, databases, third-party libraries, web services, and so on), it’s next to impossible to expect 100% bug-free applications. But, we often fall far short of the hypothetical 100%.

To me, nothing is more frustrating than a critical bug in a feature you rely on, which had been working all this time, but the developers managed to break it in the latest update. How can they mess it up and release something so obviously broken? Aren’t the developers using their own software? The answer is… no, not really, at least not the way you are using it. They’re not the pilots flying the plane; they’re the engineers building it. In fact, as a user, you probably know more about the intricacies of your favorite feature than its developers do.

Since bugs are a part of life, you might be prepared to forgive a critical bug if they fix the bug quickly, but what if they don’t? A bug might only affect a small subset of users, so the developers might have a hard time reproducing it. Still, isn’t the developer’s job to anticipate all the things that could go wrong? The answer is, once again, not really. In an ideal world, where money is no object and you can spend as much time as you need to write high quality code, it may be possible to produce almost perfect software. But, we’re not living in an ideal world. If there aren’t many users who are willing to pay a premium for highly reliable software, then the financial incentive isn’t obviously there.

Can we still write mostly bug-free software even when the immediate financial concerns aren’t in our favor? After all, producing high quality software could, in the long run, turn out to be a significant competitive advantage since developers will be able to spend more time in improving their software than fixing bugs, which brings us to the use of the scientific method in software development, or more often than not, the lack of it.

The scientific method is about forming a falsifiable hypothesis and trying to refute it by finding at least one conflicting condition. In software development, however, a more optimistic route is usually taken. Yes, the code is tested to see if it works in certain conditions, but those conditions usually entail the happy path. Testing edge cases, the source of most bugs, is often a lower priority.

So, are we doing it wrong? Shouldn’t we pay more attention to finding and testing edge cases? The answer again lies in economics. The proper handling of a large number of edge cases means doing a lot of work that isn’t always appreciated by users who need a feature yesterday. So, what do software developers under time pressure do? They enclose the happy path code in a try-catch block, and return an “Unknown error” message if something goes wrong. You now know where those cryptic error messages come from.

It’s easy to criticize software that is full of bugs, but it might not make much financial sense to produce highly reliable software with only a few features if most users in your market prefer software that, despite having more bugs, offers twice as many.

If I’ve come across as someone defending mediocrity in software development just because the financial incentives aren’t there, that’s not what I meant. No software developer who takes pride in their work, by definition, wants to produce less than great software, but good engineering is a balancing game. You are usually given a time budget, and are expected to do your best within that budget. Good engineers know how to negotiate rather than doing blindly what they’ve been told. They might offer cutting or simplifying features in order not to spread too thin, and good management listens to what they have to say.

Works On My Machine

Wed, 07 Feb 2024 00:00:00 +0000

But it works on my machine!

When you work with software developers, especially those with less experience, it’s virtually guaranteed to hear one of them utter a version of the above expression when someone presents them with a situation they think is impossible (or more likely, improbable) to happen in the program they’ve written. Now, the great thing about software is that it’s mostly deterministic – if something strange happens, it’s very likely caused by a bug in the code rather than some unexplained cosmic coincidence. I’ve written about debugging in general before, so I’d like to focus on a set of bugs specifically caused by subsystems.

Hardware/OS/Compiler
Programs don’t run in isolation. They depend on many subsystems to function. Obviously, you need a computer and a network connection. Your program also needs to be converted into a form the hardware/operating system combo can run, through the use of a compiler or interpreter.

Libraries
Next come the libraries. Nowadays, it’s rare for anyone to write code entirely from scratch, thanks to the vast array of freely available libraries that cover tasks from the simplest to the most complex. Just as your program has a version, the libraries it depends on also have their own versions. Fortunately, there are useful programs called package managers that automatically download and install the correct library versions your program needs.

Filesystems/Databases
If people use your program to store data, you will need to keep it somewhere. Popular choices are filesystems and databases that can be located on the same machine as your program or on another machine accessible over the network. In the case of relational databases, schemas are used to define the structure of the data stored.

Remote Services
In the interconnected world we live in, your program may need to interface with remote services. These services can either be internal or third-party. Remote service availability and the expected behavior of each service are important considerations since you may not have direct control over them.

Web/Mobile/Desktop
Most users access programs via web browsers or mobile devices. Yes, there are still quite a few widely used desktop (or should we call them laptop?) applications, but they are in the minority. Unlike desktop applications, web applications have less control over the environments they run in – different web browsers or even different versions of a particular browser may behave differently. As for mobile devices, they have widely varying capabilities that can be difficult to manage. This is especially true in the relatively fragmented Android world.

Program Configuration
Finally, many programs have configuration options that can be used to change their behavior, sometimes quite markedly, and these options may even interact with each other in unexpected ways.

Aside from the subsystems mentioned so far, there are other subsystems specific to certain kinds of programs such as games and embedded applications, but I think it’s safe to say that we’ve covered the most common ones. Statistically speaking, the more the subsystems your program relies on, the greater the chances of something going wrong.

Why does a feature only work on a developer’s machine?
Unless the developer’s machine has supernatural capabilities, the reason is most likely because one or more subsystems on their machine behave differently than the equivalent subsystems on other machines.

Automated Deploys
First, let’s get the obvious out of the way. Do you have an automated deploy method? Do you use a deploy script, Docker image, or some other automated mechanism to deploy software, which has the added benefit of easily rolling back to a previous deployment when something goes wrong? If not, you will probably waste a lot of time chasing bugs that could easily be prevented by automated deploys in the first place.

Package Managers
Using a package manager is essential. Otherwise, it’s too easy to lose track of which library versions your program is compatible with. When you use the wrong version of a library, your program may not fail right away; it may even appear to be working fine, but things can start to go wrong in subtle ways, which makes it hard to track the root cause of the bugs you will inevitably encounter.

Database Snapshots
As the primary data storage system used in many applications, the database probably holds the distinction of causing the most “works on my machine” bugs. Data stored in databases are living things. They change over time, sometimes a lot, but many developers tend to use outdated or unrealistic datasets during development and testing.

It greatly helps developers track down hard-to-reproduce bugs if you provide them with recent database backups from the production environment, after anonymizing and encrypting sensitive data, of course. If backups are too large, creating a remote database environment that developers can connect to is a good idea. Using a filesystem like ZFS makes it incredibly easy to spin up a database from a past snapshot. The database is ready in seconds, no matter how large it is.

Message Queues
When remote services work as expected, it’s great; when they don’t, all sorts of strange things can happen. It’s tempting to assume that remote services will respond back quickly, but in the production environment, they can get overloaded and may respond late or not at all. Also, you wouldn’t really want to accidentally DoS yourself by repeatedly making requests to an already slow service, making things even slower. Sensible timeouts and exponential backoff are often used to relieve the load on remote services if they fail to respond in a timely manner. Finally, whenever possible, it’s recommended not to query remote services directly, and use a message queue as a buffer instead.

Testing with Multiple Browsers/Devices
It’s not always possible to make a web app work exactly the same across all major browsers, but the differences should be small enough not to warrant attention. Whenever you develop a new feature, it pays to test with all major browsers including mobile browsers if your application is responsive. As for mobile apps, screen sizes are more or less standard these days, but it’s still easy to fall in the trap of assuming everyone has big screens.

Sensible Defaults
Not everyone can keep in mind what each configuration option does especially when you have hundreds of them. So, it’s important to have sensible defaults, or really strange things can happen like the inability to send e-mail farther than 500 miles, as one university system administrator found out.

Decoding Complex-Sounding Programming Terms with Examples

Wed, 24 Jan 2024 00:00:00 +0000

Richard Feynman, the Nobel Prize-winning physicist, developed an interest in biology while he was a graduate student at Princeton. He started attending a course in cell physiology, which required him to report on papers assigned to him. One such paper was about measuring the voltages generated by the nerves in cats:

I began to read the paper. It kept talking about extensors and flexors, the gastrocnemius muscle, and so on. This and that muscle were named, but I hadn’t the foggiest idea of where they were located in relation to the nerves or to the cat. So I went to the librarian in the biology section and asked her if she could find me a map of the cat.

“A map of the cat, sir?” she asked, horrified. “You mean a zoological chart!” From then on there were rumors about some dumb biology graduate student who was looking for a “map of the cat.”

In the software world, we don’t have as many complicated terms as there are in biology, but we aren’t completely immune to giving complex-sounding names to what should be simple concepts – some of us even enjoy getting pedantic about them. Here are some of my favorites, with examples in JavaScript:

Dependency Injection

When you pass objects as arguments to an object, it’s called dependency injection. Yes, that’s all there is to it.

The following constructor function can be used to generate UserService objects with a logger dependency. The dependency can be injected from the outside, which allows the use of different logging libraries.

var UserService = function (logger) {
    this.create = (firstName, lastName) => {
        let getFullName = () => firstName + ' ' + lastName;
        logger.info('Created User:', getFullName());
        return { firstName, lastName, getFullName };
    };
};

var myUserService = new UserService(console);
var newUser = myUserService.create('Katniss', 'Everdeen');
// [INFO] Created User: Katniss Everdeen

Factory Method

Despite the lofty name, a factory method just returns an object.

var userService = {
    create: (firstName, lastName) => {
        let getFullName = () => firstName + ' ' + lastName;
        return { firstName, lastName, getFullName };
    },
};

var newUser = userService.create('Asuka', 'Langley');
console.log(newUser.getFullName());
// Prints Asuka Langley

Higher-Order Function

A function that returns a function or takes other functions as arguments is called a higher-order function.

Most popular programming languages natively support higher-order functions, or added support for them.

In the code below, _.pickBy passes each property in the collection to the provided functions (_.isNumber or _.isString), and if those functions return true, it “picks” the corresponding properties from the collection.

const _ = require('lodash');
const collection = {'a': 1, 'b': '2', 'c': 3};

console.log(_.pickBy(collection, _.isNumber));
// { 'a': 1, 'c': 3 }

console.log(_.pickBy(collection, _.isString));
// { b: '2' }

Idempotence

A function is called “idempotent” if multiple calls to the function have the same effect on the program state as a single call. In other words, no matter how many times you call the function with the same arguments, the program state will not change after the first call.

The following function updates the status of a message with the given id in the database. The program state will not change after the first call. So, this function is idempotent.

// (...)
async function updateMessageStatus(id, status) {
    return db('message').update({ status }).where({ id });
}

await updateMessageStatus(1, 'read');
await updateMessageStatus(1, 'read');
await updateMessageStatus(1, 'read');
// The message #1's status will always be 'read'.

Immutability

All variable assignments are final. Once you assign a value to a variable, you aren’t allowed to change it in any way.

In the example below, JavaScript’s native array sort method sorts an array in place, replacing the values of the original array. Lodash’s sortBy method, on the other hand, doesn’t mutate the original array, and returns a new array instead. It’s good programming practice to avoid changing variable values whenever you can even if the programming language you use only has partial immutability support like JavaScript.

const _ = require('lodash');

const mutableElements = ['earth', 'water', 'air', 'fire'];
mutableElements.sort();
console.log(mutableElements);
// [ 'air', 'earth', 'fire', 'water' ]

const immutableElements = ['earth', 'water', 'air', 'fire'];
sortedImmutableElements = _.sortBy(immutableElements);
console.log(immutableElements, sortedImmutableElements);
// [ 'earth', 'water', 'air', 'fire' ] [ 'air', 'earth', 'fire', 'water' ]

Memoization

As a form of caching, a memoized function “remembers” the result it has previously returned for a given set of inputs. Subsequent calls with the same inputs return the remembered result rather than recalculating it.

Lodash has a convenient memoize method that can be used to turn any function into a memoized function:

const _ = require('lodash');
// (...)
const memoizedSlowFunction = _.memoize(slowFunction, (...args) => JSON.stringify(args));

That’s all for now. Let me know what your favorite complex-sounding yet simple programming terms are.

On Unix Philosophy

Sun, 14 Jan 2024 00:00:00 +0000

I will never forget the time I discovered how web pages were made. The year was 1995. I was using Windows 3.1 like most everyone, and it didn’t even support the TCP/IP network stack out of the box. I had to install a program called Trumpet Winsock and a hot new web browser called Netscape to connect to the Internet.

Netscape Navigator 1.2 on Windows 3.1 - Source: Wikipedia

In Netscape, there was an option called “Source…” under the View menu, and when I clicked it, I was half expecting to see some sort of cryptic binary file format that would require special software to generate. What I got instead was the actual text of the web page I was viewing, with a bunch of HTML tags thrown in for styling. That was a revelation – with a simple text editor, anyone could create a web page!

I didn’t know at the time, but it was part of the Unix philosophy to use text streams to exchange data between programs. Had Microsoft or some other company managed to invent the web instead of Tim Berners-Lee, I bet they would have opted for a proprietary binary format for web page delivery, as it was customary in the non-Unix world.

In those early days, surfing the web was an amazing experience for me. I had, for the first time in my life, instant access to people anywhere in the world. There was little content back then of course, and connection speeds were very slow. “Image-heavy” web pages could easily take several minutes to download on my 0.014 megabit landline phone/Internet connection.

These days, we have incomparably fast Internet speeds – multi-megabyte web pages download in seconds, and a lot of people watch TV online thanks to the wide availability of on-demand TV series and movies in high resolution. To top it off, all these feats are possible with no wires in sight. How far we’ve come, and yet one thing hasn’t changed – web pages are still delivered in plain text.

Text is the universal interface all humans can understand. You can use literally thousands of different programs to view and process text files. Binary files, on the other hand, require specialized programs that greatly reduce their utility. Granted, not everything can be a text file. Images and sound recordings are better stored in binary, but there’s usually no good reason to use proprietary binary file formats when a simple text file is enough, especially when large text files can easily be compressed on the fly.

Many software developers fall in the trap of writing a program for every problem they encounter, even when there are better alternatives to achieve the same outcome. Modern operating systems based on Unix such as Linux and MacOS come with a large number of built-in Command-Line-Interface (CLI) programs that can be chained together to solve seemingly complex tasks with ease, often with nothing more than one-liners. But first, let’s hear what Master Foo has to say about the subject :)

Master Foo and the Ten Thousand Lines

Master Foo once said to a visiting programmer: “There is more Unix-nature in one line of shell script than there is in ten thousand lines of C”.

The programmer, who was very proud of his mastery of C, said: “How can this be? C is the language in which the very kernel of Unix is implemented!”

Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C”.

The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!”

Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C”.

The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”

The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so.

“And how many hours would you require to implement and debug that C program?” asked Nubi.

“Many”, admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him”.

“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”

Upon hearing this, the programmer was enlightened.

I don’t know which script Nubi wrote on the whiteboard, but the following one-liner, when run in Linux or MacOS returns the 5 most frequently used words in a file (in this case LICENSE), and prints out a sorted list of those words along with their frequencies:

cat LICENSE | egrep -o "\w+" | awk '{print tolower($0)}' | sort | uniq -c | sort -rn | head -n5
  14 the
   9 software
   9 or
   8 to
   8 of

Some of you might recognize the script I wrote above as the somewhat modernized version of Doug McIlroy’s version of Donald Knuth’s 10+ page Pascal program. I’m certainly no Donald Knuth, but I think it’s correct to say that even the most brilliant minds may sometimes make things more complicated than they should be.

The power of shell scripts is a direct manifestation of the Unix Philosophy, as succinctly summarized by Doug McIlroy, the inventor of Unix pipes:

Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.

As software developers, what can we learn from the Unix philosophy? Isaac Z. Schumpeter, the creator of npm, suggests the following principles in the context of Node.js modules, but I think they are general enough to cover software modules written in other programming languages as well:

Write modules that do one thing well. Write a new module rather than complicate an old one.
Write modules that encourage composition rather than extension.
Write modules that handle data streams, because that is the universal interface.
Write modules that are agnostic about the source of their input or the destination of their output.
Write modules that solve a problem you know, so you can learn about the ones you don’t.
Write modules that are small. Iterate quickly. Refactor ruthlessly. Rewrite bravely.
Write modules quickly, to meet your needs, with just a few tests for compliance. Avoid extensive specifications. Add a test for each bug you fix.
Write modules for publication, even if you only use them privately. You will appreciate documentation in the future.

Unix has been around in one form or another for over half a century now, and the philosophy behind it remains as relevant today as when it first emerged, not just for operating systems but for software development in general. While combining different modules in software development isn’t quite as straightforward as snapping Lego bricks together, the Unix philosophy is guiding us towards this goal, one brick at a time.

Worth a Read:

DIY Dual Column Speaker Stands

Sun, 07 Jan 2024 00:00:00 +0000

I was looking to buy speaker stands for my KEF LS50 Metas and the DIY KEF LS50 Clone I use as my center channel speaker. KEF’s matching extruded aluminum S2 stands were a natural choice. Unfortunately, they were quite expensive, and I’m not really a fan of single column stands anyway, so I decided to build a few dual column stands myself.

I picked 8 mm MDF for the sides of the columns, 12 mm MDF for the top plate, and 18 mm MDF for the bottom plate. The columns will be hollow, so they can be filled with sand to make the stands acoustically more inert. Finally, the top and bottom sides of the columns will be made of 15 mm scrap birch plywood I had on hand.

Pieces cut for one speaker stand.

The stands will be approximately 65 cm (25.6") tall, including the spikes. Each column will be 5 cm wide and 15 cm deep. The distance between the columns will be 2 cm. After cutting the pieces, I dry-fitted the columns to make sure the pieces fit each other, then glued the pieces together with wood glue, and held everything in place using clamps.

Column dry fit.

After the glue dried, I used my router table to flush trim all edges and applied a 3/8" (~9.5 mm) roundover to the columns for a softer look. Next, I drilled holes into the top and bottom sides of the columns and put in M6 threaded inserts. This will allow me to secure the top and bottom plates in place using M6 countersunk socket head screws.

I then performed a test assembly to ensure that the holes lined up. Notice the different top plate sizes in the picture below. The stand on the left uses a smaller top plate for my KEF LS50 Metas and the one on the right is for my bigger KEF LS50 clone.

Test assembly of my DIY speaker stands with different top plates.

After sanding, the stands were sprayed with several coats of black paint.

Stands painted black.

After the paint dried and cured, I assembled the stands and screwed brass spikes into the M8 threaded inserts I had placed in the corners of the bottom plate. Rubber feet could be used in place of spikes on hardwood/concrete floors.

Brass spikes.

Each stand, fully assembled, weighs roughly 4.5 KG (~10 pounds), incidentally about the same weight as empty KEF S2 stands. I haven’t filled them with sand yet, but they should weigh about twice as much when fully filled.

Stands fully assembled.

I think the stands turned out well. There’s a little too much orange peel in the paint for my taste, and there are a few minor blemishes that I might touch up in the future, but you need to get really close to see them, which is next to impossible in my in home theater with all the dark surfaces around.

One Function Per Line

Wed, 20 Dec 2023 00:00:00 +0000

It’s often said that reading code is harder than writing it, but is it really? If that were the case for books, everyone would be writing their own rather than reading the books others wrote. But, a page of code isn’t equivalent to a page of text. Code, by nature, is denser. To effectively explain what a page of code does may easily take several pages, and in some cases a lot more.

Is it possible to make code easier to read than to write? After all, code is read more often than it is written when multiple developers work on the same codebase. If developers spend less time to understand how the existing code works, they can start writing code faster.

In his book Code Complete, Steve McConnell describes a process called pseudocode programming that starts with writing pseudocode where each line, written in English, describes a specific operation in relatively high level. You then turn the pseudocode into comments, and finally fill in the code below each comment. Here is an example in JavaScript:

async function createUser(email, password, firstName, lastName) {
	// Validate input fields, and return error if validation fails
	...
	// Check if the password conforms to the password policy, if not return error
	...
	// Check the user table to see if email is available, if not return error
	...
	// Hash user password by using a strong hashing algorithm
	...
	// Create a new record in the user table, return user's id on success or return error
	...
}

Though it lacks a few details, I don’t think any developer would have a problem understanding the above pseudocode. Comments also come handy in AI powered code generation tools to auto generate some of the code.

Can we go one step further, and actually write code that looks like pseudocode? By using a variation of the Chain of Responsibility pattern, it’s possible to write code that appears remarkably similar to pseudocode:

const chain = require('fn-one');

// (...)

async function createUser(email, password, firstName, lastName) {
	return await chain(
		() => validateInput(email, password, firstName, lastName),
		() => isPasswordValid(password),
		() => userService.isEmailAvailable(email),
		() => userService.hashPassword(password),
		(hashedPassword) => userService.create(email, hashedPassword, firstName, lastName)
	);
}

For chaining functions, I used fn-one, a tiny library I wrote, but there are quite a few function chaining libraries out there. It doesn’t matter which one you use, what matters is, whenever possible, to call one function per line. This style of coding may require you to write a lot of small functions, but the code can become significantly easier to follow as a result.

When chaining functions, each function’s output is passed to the next function in line. If a function returns false or throws an exception, the chain breaks. There are no explicit control structures such as if-else statements and loops that clutter the code. They are hidden inside lower-level functions where they belong.

Writing Robust Code via Idempotence

Thu, 14 Dec 2023 00:00:00 +0000

What happens when a function call fails? Can you retry without causing any unintended side effects? If the answer is no, then your function is probably not robust enough.

Here is a scenario that is very likely happening to someone somewhere right at this moment. Let’s call this person Midori. Midori has been eyeing a pair of headphones, fancy noise cancelling ones, no less, and as soon as there’s a sale on them, she adds the phones to her shopping cart, and with a touch of excitement, clicks the Order button. A few seconds pass, but nothing happens, and then she waits some more, but still nothing. Has the order gone through? Does she need to order again?

There can be different reasons as to why the order page might get stuck – Midori’s Internet connection might have dropped right before or after she clicked the Order button, or maybe her connection was OK, but the server was busy.

Midori goes back to her cart and notices the headphones are still there. Can she safely assume that the order hasn’t gone through? Maybe the cart hasn’t been updated to reflect the order, and is showing outdated information.

A function is called “idempotent” if multiple calls to the function have the same effect on the program state as a single call. In other words, no matter how many times you call the function with the same arguments, the program state will not change after the first call.

In Midori’s case, if the server has received the order, but has no way to respond due to the dropped connection, a second order won’t be executed if the function responsible for the ordering process was designed to be idempotent. For example, if the shopping web site sends a unique “idempotency” key to the server for each order, the server will realize that it has already executed the order, and won’t do anything.

Idempotency is often discussed in the context of HTTP methods. According to the HTTP 1.1 specification, in addition to the normally idempotent GET method, the methods PUT (update), and DELETE should be implemented in an idempotent manner, but this is only possible if unique identifiers are used. For example, the function handling the following request

PUT /messages/id/123
{"status": "read"}

can be made idempotent because only the message with the id 123 will be updated. However, if your API supports updating messages using non-unique identifiers, such as a date range, it may not be truly idempotent because the same date range can update different messages in the future.

Resuming Interrupted Operations

Idempotent functions aren’t just about limiting state changes – they may also allow resuming interrupted operations. For example, if you are synchronizing files across computers, and the process is interrupted, a second call can start where the previous call left off. Long-running batch SQL jobs that process millions of records are also prime candidates for idempotency. If something goes wrong, they won’t have to start from the beginning.

Pure Functions

So far, we’ve talked only about functions that change the program state, but idempotency is also a natural property of “pure” functions. Pure functions always return the same results for a given set of arguments. They don’t modify the passed arguments or depend on external components like non-local variables, databases, or network connections that could change outside the function’s control.

function getFullName(firstName, lastName) {
	return firstName + ' ' + lastName;
}

is a pure function, but the following isn’t because it refers to non-local variables that can be changed by other functions at any time, and yet it’s still idempotent because it doesn’t change the program state at all.

var firstName, lastName;

function getFullName() {
	return firstName + ' ' + lastName;
}

Utilizing idempotent functions, both pure and impure, not only makes your code more robust by minimizing unintended side effects but also makes it easier to reason about your code.

Common SQL Mistakes Developers Make

Tue, 05 Dec 2023 00:00:00 +0000

Over the years I’ve seen all sorts of SQL mistakes made by developers on my teams, and I’ve made quite a few of them myself. I’d like to share what I’ve learned with you, starting with security first.

Security

Writing Raw SQL queries

Even if everyone on your team is careful, sooner or later, someone will inadvertently pass unsanitized user input to a query, inviting various SQL injection attacks. Use an SQL query builder like Knex instead. SQL query builders allow you to programmatically create queries and automatically sanitize all inputs using prepared statements, but they do still allow running raw SQL queries for those rare occasions that you need them.

Not Using Prepared Statements

If you really have to run raw SQL queries, make sure that you use prepared statements. Yes, it’s more work, but that’s one of the reasons why we are using SQL query builders in the first place.

Data Integrity

Missing Primary Keys

Primary keys serve as unique identifiers for each row in a database table. For example, each customer must have a unique, usually numeric, id. Other fields uniquely identifying a customer such as e-mail addresses and phone numbers might change, but a customer’s id never changes. If a table doesn’t explicitly rely on the primary key for relationships such as many-to-many intermediary tables, you may omit them, but those cases are few and far between. When in doubt, use primary keys.

Missing Foreign Keys

Foreign keys link data in one table to the data in another table. Their primary purpose is to enforce relationships between tables. For example, a foreign key in an “Order” table may reference the primary key in a “Customer” table. It’s not possible to delete a customer leaving their orders orphaned when such a foreign key is in place. The opposite is true as well. It’s not possible to add an order from a non-existent customer.

When you have a good set of foreign keys in your database, you know that your database has your back when your application attempts to do something it shouldn’t.

Not Using Transactions

Transactions allow you to group multiple operations into a single logical unit of work. They are essential in dealing with complex operations that involve multiple steps. Either all steps in the transaction are completed successfully, or none of them are. For example, when placing an order, you may add an entry to the “Order” table and update the stock values of the products purchased in the “Product” table. Without transactions, if an error occurs after the order is created in the database, you end up with incorrect stock numbers.

No ORDER BY

This one is relatively innocuous because it doesn’t directly affect data integrity, but queries without ORDER BY might return results in unpredictable ways. Developers often make the assumption that query results are always returned in the same order, but this isn’t guaranteed unless you explicitly use ORDER BY.

Performance

Inadequate Indexes

Having just a primary key in a table isn’t enough. As soon as you start to use columns other than the primary key in WHERE conditions, the database won’t be able to use indexes, and performance might suffer greatly as a result. Queries that take milliseconds to complete might take seconds or even minutes. If you are having performance issues, the first place to look at is indexes. Adding the right index can significantly improve an application’s performance, often by thousands of times.

It’s tempting to add indexes to every single column in a table, but there are drawbacks to that approach. Databases can’t always use multiple indexes at the same time. Also, they need to update every single index after every write operation (INSERT, UPDATE, DELETE). Finally, your data storage requirements can easily double or more, so it pays to be selective when adding indexes to a table.

Overusing SELECT *

Retrieving all columns using “SELECT *” may be convenient, but it can negatively impact performance. The database needs to retrieve and transfer entire rows, and your application needs to convert all incoming data to an internal format. These operations consume more CPU time, memory, and network bandwidth when there are many unnecessary columns included in the result set.

Not Using LIMIT Clauses

When you need only a small subset of a result, using LIMIT may greatly improve performance rather than retrieving the whole result and pruning the unnecessary parts in code.

Running SQL Queries Inside a Loop

It may not always be obvious which functions access the database to return a result. Database operations are naturally slow. Accidentally calling SQL queries inside a loop may cause your application to come to a crawl, whereas converting the function to return multiple results, either using WHERE IN or some other method, will likely work much faster.

Using WHERE IN with a Large List of Values

While there are good use cases for WHERE IN, it’s a good idea to keep its limits in mind. When there are a large list of values, some databases might not use indexes properly, and some also impose a limit on the number of values that can be used.

Running SQL Queries One by One

SQL queries are slow. Whenever you can, you should run SQL queries in parallel. For example, in Node.js you can use Promise.all to run multiple SQL queries at the same time.

Too many UPDATEs

Many relational databases don’t normally update rows in place – they create new rows instead and mark the old ones “dead”. Dead rows are usually reclaimed by the database after some time. If you need to make a great number of updates for things like increasing product view counters, database performance might start to suffer. In cases like this, it’s a good idea to augment your relational database with a data store like Redis that can handle frequent updates.

Testing with Small Data Sets

A query may run just fine when there are 1K rows in a table, but what happens when there are 100K, 1M, or 10M rows? Critical parts of the code must be tested with realistic data sets.

Using ORMs

Unlike SQL query builders, using Object-Relational Mapping (ORM) frameworks like Hibernate in Java that automatically generate SQL queries for you to populate objects might add an unacceptable overhead for some use cases. A seemingly simple query in an ORM can easily trigger tens of side queries. It’s also difficult to optimize or customize the generated queries to achieve better performance.

Change Management

Not Using Migrations

Changes are made to the database schema over the lifetime of an application. All schema changes must be stored in immutable migration files so that database schema changes can be made and tracked automatically. Ad-hoc changes without explicit change management might lead to a lot of headaches during deployment.

That’s a wrap. If there are other common SQL mistakes I’ve missed, please let me know, and I will update the article.

Practical Defensive Programming

Sun, 26 Nov 2023 00:00:00 +0000

“Only the paranoid survive.” – Andy Grove

No software developer in their right mind would trust user inputs in an application. All user inputs need to be validated and sanitized, otherwise nasty security vulnerabilities like SQL injection and Cross-Site Scripting (XSS) attacks might happen. Once user inputs pass validation and start to go through internal layers of code, many developers assume that there isn’t much need to validate inputs anymore, but validation isn’t just about security, it’s also about correctness.

Many bugs are caused by passing the wrong type of data to a function such as passing a string when an integer is expected or passing an array with fewer fields than necessary. Language features like static type checking is good at detecting many type errors during compilation, but can’t catch everything in runtime like Java’s infamous NullPointerException.

As for TypeScript, which adds type annotations to JavaScript, there are no runtime checks at all after the code is transpiled to JavaScript. Even if you have type annotations for your entire code base, any inputs coming from a database or an external API can still be of a wrong type.

One of the most effective methods for detecting invalid types is to validate function arguments in runtime. Validating every single argument can get tedious fast though, with line after line of if statements. Fortunately, there are some good data validation libraries out there like Joi and Zod to make data validation easier, but it can still get long-winded at times, and you wouldn’t really want to give developers any excuse to skip data validation :)

Instead of using an all-purpose validation library designed for validating a wide range of data types, it might make more sense to use a library tailor-made to validate function arguments. With that idea in mind, I’ve created a lightweight JavaScript function argument validation library called fn-arg-validator. Here’s a quick example of how it works:

const is = require('fn-arg-validator');

function createUser(firstName, lastName, birthDate) {
	is.valid(is.string, is.string, is.date, arguments);
	// ...
}

createUser('Thomas', 'Anderson', '1971-09-13');
// [WARN] 1971-09-13 failed date check

I’ve decided to use a functional style interface where you pass type check functions (built-in or your own) for each function argument, and as the final argument, you simply pass the arguments object to avoid typing the function parameters again.

When validating arrow function arguments, the arguments object isn’t available, so you need to provide the parameter names in an array like [firstName, lastName, birthDate].

The example code above will normally log a warning and continue to run, but you may choose to check the return value of is.valid to stop the execution of the rest of the code. The decision depends on how you want to handle type errors. Sometimes, a warning is enough, and sometimes you need to be strict and throw an error.

Boundary Checks and Maybe Types

In addition to strict type checks, it’s possible to do things like string length checks, and use maybe types that also accept undefined/null values:

function createUser(firstName, lastName, birthDate) {
	is.valid(is.stringBetween(1, 20), is.stringLTE(20), is.maybeDate, arguments);
	// ...
}

Object Property and Type Checks

is.objectWithProps can be used to check if an object has the specified properties and those properties have the correct types.

const userObjectProps = { id: is.number, firstName: is.string, lastName: is.string, birthDate: is.date };

function updateUser(user) {
    if (!is.valid(is.objectWithProps(userObjectProps), arguments)) {
        throw new Error('Invalid user object');
    }
    // ...
}

updateUser({ id: 1, firstName: 'Thomas', lastName: 'Anderson', birthDate: '1971-09-13' });
/*
[WARN] {"id":1,"firstName":"Thomas","lastName":"Anderson","birthDate":"1971-09-13"} failed objectWithProps check

Error: Invalid user object
...
*/

You can have one-level of nested object property checks as shown below:

is.valid(
    is.objectWithProps({
        a: is.number,
        b: is.objectWithProps({ c: is.string, d: is.number }),
    }),
    arguments
);

Passing your Own Type Check Functions

The beauty of a functional style interface is that you aren’t limited to the built-in validation functions, you can simply pass your own. The only requirement is to give your functions names since is.valid uses function names for logging purposes.

Log Configuration

By default, fn-arg-validator uses the console object for logging. However, this can be configured by assigning a different logger to is.config.log.

The log level can be set by changing the value of is.config.logLevel. The default log level is ‘WARN’, which only logs failed checks. If you would like to see successful validations, you need to set the log level to ‘DEBUG’ or higher. To disable all logging, set the log level to ‘OFF’.

Finally, to see the complete list of built-in functions, you can take a look at fn-arg-validator’s GitHub page.

Making a KEF LS50 Clone

Fri, 17 Nov 2023 00:00:00 +0000

KEF makes some of the best coaxial speakers in the market, and they use the same driver platform in all their speakers ranging from the entry-level Q series to all the way up to Blade. The drivers in their higher end speakers usually have stronger motors and better crossovers, but the higher prices don’t always justify the level of performance improvements.

KEF Q150 and LS50 Meta side by side.

KEF’s acclaimed LS50 (and later LS50 Meta) speakers use a single coaxial driver which looks identical to the one used in the entry level Q150 from the outside, but the speakers sound quite different. If you equalize both speakers to have about the same frequency response to level the playing field, the LS50 will still have a better defined, less diffuse sound.

Inside KEF LS50 Meta's cabinet. Image © KEF

The LS50 has a superior cabinet that features thicker walls, substantial cross braces, a flexible port, and finally KEF’s version of Constrained Layer Damping (CLD) to minimize internal resonances. Since you can buy two to three pairs of the Q150 for the price of one pair of LS50 Meta depending on whether there’s a sale or not, I decided to build an LS50 clone using a Q150 driver to see if they would sound similar after equalization. The clone wouldn’t have the beautiful curved front baffle and better crossover of LS50 Meta, but it would otherwise be very similar in construction.

The KEF Q150 is a narrow speaker – only 18 cm (7") wide while the LS50 is slightly wider at 20 cm. My version would be even wider at 22 cm as I decided to reuse the original port, which had such a wide flare that it wouldn’t fit if the cabinet was narrower.

The KEF Q150’s front baffle is quite thick at 36 mm (1.4"), but the remaining walls are rather thin at 12.7 mm (1/2"). There’s only one small horizontal brace and very little stuffing inside, so the cabinet is naturally prone to resonances.

KEF Q150 has a bare-bones cabinet with one small horizontal brace.

I decided to use 18 mm thick MDF just like the LS50 for all cabinet walls except for the front baffle which would stay the same at 36 mm. The cross braces would also be made out of 18 mm MDF. KEF uses something similar to 3M VHB tape to have a flexible connection between the braces and the cabinet walls as a form of CLD, so I decided to give it a try.

I started with cutting the interlocking braces and attached them to the surrounding walls by using double sided 3M VHB tape instead of wood glue. The VHB tape is supposed to act as a CLD layer to loosely couple the braces to the cabinet walls.

Braces support all six walls and the driver.

The outside walls are glued to each other with wood glue.

The cabinet is coming together.

The front baffle was made out of multiple layers of MDF. I opted to use M4 threaded inserts and screws in place of the original’s wood screws, as all it takes a few driver test fits to strip wood screw holes. I also rounded over all cabinet edges except the rear side with a 12.7 mm (1/2") radius router bit, and sanded everything smooth. The cabinet weighs around 6 kg (~13 pounds).

Front baffle was finished and cabinet edges were rounded over.

The cabinet was painted white with a spray gun with the paint I already had at hand, but I may use a different color scheme in the future.

The cabinet painted white.

The KEF Q150 uses a simple first-order crossover board to divide the audio signal between the mid-bass driver and tweeter. Although the crossover board had clearly been designed to be mounted on the back side of the terminal cups, KEF chose not to do that for some reason. They mounted the board on the bottom side of the KEF Q150 cabinet instead. Since the original terminal cups didn’t have the necessary standoffs to mount the crossover board, I purchased another set with standoffs, and used M3 nylon screws to mount the board after tapping the standoffs of course.

The KEF Q150 crossover is mounted to the terminal cups in my version.

Here is the finished cabinet with the driver and trim ring mounted. I reused the rubber grommets from the original to mount the trim ring. I was excited to test the speaker out, so there is no clear coat to protect the paint on the speaker yet.

In my version, the crossover board is mounted on the backside of the terminal cups.

The center of the port is positioned one-fourth of the way from the right and the top to cancel some of the internal standing waves. The terminal cups are also mounted vertically. I think they look better that way.

Finished cabinet - rear.

So, how does it sound? Quite good actually. The original had a great sound stage thanks to the well designed coaxial driver, but was undermined by the cabinet. With the new cabinet, I’m getting better bass definition, and the upper notes have better clarity.

After listening to the speaker, I made impedance measurements of the original KEF Q150 and my LS50 clone. My measurement setup isn’t super accurate, but the wiggles that indicate the resonances in the original (orange) are mostly gone in my LS50 clone (green). The difference is especially visible at around 700 Hz.

Impedance measurement (Orange: KEF Q150, Green: LS50 Clone)

Update (August 2, 2025):

Plans for the horizontal and vertical braces are available below (units are in mm):

Plan for horizontal brace.

Plan for vertical brace.

Worth a read:

KEF LS50 Meta / Wireless II White Paper

Design Patterns vs. First Class Functions

Wed, 15 Nov 2023 00:00:00 +0000

Design Patterns: Elements of Reusable Object-Oriented Software is an influential book that describes around two dozen software programming patterns. Over the years much has been written about those patterns to make the confusing parts easier for generations of developers – additional books got published and endless discussions on things like the difference between “Abstract Factory” and “Factory Method” took place in various online forums. But really, they shouldn’t have been this complicated – the purpose of the most design patterns can be summarized in just 3 words:

“Hide implementation details.”

Or more specifically in 7 words:

“Hide implementation details behind a common interface.”

Granted, the authors of the Design Patterns book famously stated: “Program to an interface, not an implementation.”, which means about the same thing, but then they went on to explain in great length how to achieve that goal within the bounds of traditional object oriented languages employing classes, and confused a lot of people along the way.

When Design Patterns first got published, the year was 1994, a rather primitive time when functional programming languages that support first class functions were rarely used outside academia. These days, though, we have mainstream languages such as JavaScript and Python that can be used to trivially implement many of the design patterns described in the book with only a few lines of code.

Let’s go over five popular design patterns and see how easy it is to implement them in JavaScript, the lingua franca of the web:

Factory

Despite the lofty name, a factory function just returns an object. We do that all the time in JavaScript without giving it a second thought.

var userService = {
	create: function (firstName, lastName) {
		function getFullName() {
			return firstName + ' ' + lastName;
		}
		return {firstName, lastName, getFullName};
	}
}

var newUser = userService.create('Asuka', 'Langley');
console.log(newUser.getFullName());
// Prints Asuka Langley

Decorator

Another pattern with a fancy name, the decorator pattern is quite simple: modify an object while keeping its original interface the same. In JavaScript, we modify objects all the time, so I suppose we are all decorators.

// The user service object has been "decorated" with a shiny name property
userService.name = 'userService';

// You can also create a new "decorated" object (use _.cloneDeep for nested objects)
var anotherUserService = {...userService, name: 'anotherUserService'};

Command

The Command pattern can actually be quite useful when there is a need to transparently change the behavior of certain functions. Instead of calling a function directly, you call a command function, which in turn calls the original function for you. When there is an intermediate layer between the caller and callee, there are a myriad of possibilities – you can

queue function calls instead of calling them right away,
log function calls complete with arguments passed,
cache function results,
route calls to other functions, great for creating test doubles among other things.

Here is a generic command runner object that logs function calls before calling the actual functions. Just make sure to add a name property to the objects you pass to the runner, if it cannot automatically determine the objects’ names.

var fn = {
    log: [],
    run: function () {
	    // To preserve 'this', we need the parent object
	    var parentObj = arguments[0];
	    // If the second argument is empty, the parent object is called as is
	    var func = arguments[1] ? parentObj[arguments[1]] : parentObj;
	    // Get arguments to pass to the function
	    var argList = [].slice.call(arguments, 2);
	     // Log function calls
	    fn.log.push([parentObj.name, func.name, ...argList]);
	    // Call function
	    return func.apply(parentObj, argList);
	}
};

var user01 = fn.run(userService, 'create', 'Shinji', 'Ikari');
var user02 = fn.run(userService, 'create', 'Asuka', 'Langley');
console.log(fn.log);
/* Prints:
[
    [ 'userService', 'create', 'Shinji', 'Ikari' ],
    [ 'userService', 'create', 'Asuka', 'Langley' ]
]
*/

Strategy

In JavaScript, it’s common to pass a function as an argument to customize the behavior of another function. Here’s an example from Lodash, a popular utility library:

var _ = require('lodash');
var collection = {'a': 1, 'b': '2', 'c': 3};

console.log(_.pickBy(collection, _.isNumber));
// Prints { 'a': 1, 'c': 3 }

console.log(_.pickBy(collection, _.isString));
// Prints { b: '2' }

Chain of Responsibility

If you’ve used a popular web service framework like Express or Restify, you’ve used chain of responsibility. Each web service request needs to go through several handler functions (aka middleware) to handle things like logging, rate limiting, authentication, and authorization before processing the actual request. Any handler can break the chain and throw an error. The implementation is left as an exercise to the reader, but the process is quite simple:

Register handler functions.
Call them in order.
Check each handler function’s return value or the exception it has thrown to decide whether to call the next handler or exit.

Here is an example of the pattern in action from Restify:

// ...

var server = restify.createServer();

// Register global handler functions that are called for all routes
server.use(log.webRequest);
server.use(restify.plugins.throttle({burst: 100, rate: 50, ip: true}));
server.use(authentication.verify);

// Add custom handlers for each route below that are run after the global handlers

// Run an authorization handler and if all is well, return user profile data
server.get('/profile/:userId',
	authorization.canViewProfile,
    userController.getProfile
);

// Only admins can update admin settings
server.put('/admin/settings',
    authorization.isAdmin,
    adminController.updateSettings
);

// ...

David Wheeler famously said: “We can solve any problem by introducing an extra level of indirection.” Many a classic design pattern is nothing but an extra layer of abstraction. Classes aren’t usually the best way to implement those abstractions, but first class functions that can be passed around as arguments are.

Debugging is Programming

Tue, 07 Nov 2023 00:00:00 +0000

First things first…

How to write production quality code in 12 steps:

Understand the problem.
Divide the problem into smaller chunks if possible.
Choose the data structures and algorithms to use for the solution.
Write the code.
Run the code. Be surprised if it runs on the first try. Otherwise, fix any errors the compiler has thrown at you.
Test the program behavior. If it functions correctly, go to step 9.
Guess which part of your code is responsible for the incorrect behavior. If necessary, add breakpoints or print statements to your code to pinpoint the bug.
Fix the bug. Go to step 5.
You’re almost there. Commit your code and open a merge request for code review.
Assuming there’s a Continuous Integration (CI) process in place, see if some of the automated tests failed. If so, go to step 7.
Your code has been deployed to the test environment. The product manager and the QA team test your code. If they discover any bugs you have missed, go to step 7.
Your code is merged into the main branch and deployed to production. A user finds a bug that you, the product manager, and the QA team have missed. Go to step 7.

See any patterns above?

No?

Hint: There are quite a few references to step 7.

You’ve probably seen programmers calmly writing code at 200 keystrokes per minute to solve hideously complex problems under strict deadlines, and the programs they wrote run perfectly on the first try, often only a few seconds before something blows up real good. Wait, what? I’m talking about Hollywood movies of course in which programmers can routinely achieve amazing feats while in the real world, we mere mortals have to do something called debugging, and do lots of it.

Debugging is often described as something separate from writing code as I did in the 12-step coding process above, but that’s not quite true. Unless you’re writing very simple programs, writing code involves a lot of trial and error, and it’s not uncommon to write, run, and revise a program hundreds of times before submitting it for review. In other words, debugging is done from day one.

Some software developers like to debug code, some don’t. As for me, I’m firmly in the like camp. I think of debugging as doing detective work, and unlike detectives who can successfully solve only some of their cases, our success rate is much higher, approaching close to 100% because we have one magic trick up our sleeves – the ability to reproduce bugs. If you can consistently reproduce the conditions that trigger a bug, it’s inevitable to find the reason because code is deterministic. It will behave the same every time. Granted, it may not be exactly deterministic when multiple branches of code are run in parallel, but there are ways to write well behaved parallel code.

Some software developers tend to fault compilers, databases, and other external components for errors in their programs, but almost all the time there is a fault in the programs they’ve written. If you have the steps to reproduce a bug, you’re half way finding what causes a bug. Just running the relevant sections of your code step by step and seeing how it behaves should be enough in most cases. Remember, determinism is your friend.

What causes bugs?

From most common to less common:

Syntax errors: They are the most obvious because your code won’t run at all, and there is a compiler error message telling why. In dynamic languages like JavaScript, your code may still run at first, but it will fail with a compiler error message when the code block containing the syntax error is executed. Today’s Integrated Development Environments (IDEs) can usually show syntax errors as you are typing code, so they’re easy to spot.
Type errors: Passing a variable with a different type than expected such as passing a string value to a variable expecting an integer may cause issues depending on the programming language used and whether the receiving code can handle different types. Passing undefined values is also a problem that may cause errors such as the infamous NullPointerException error in Java, and the “Cannot read property ‘x’ of undefined” errors in JavaScript.
Logic errors: In most programs, the most important part is the “business logic” as implemented in code. Either the logic is implemented incorrectly or it is implemented correctly, but some edge cases are overlooked. Extensive unit and integration testing is the key to identify logic errors.
State errors: When a program starts to behave erratically, if restarting it solves the problem, the cause is likely to be the corrupt internal state of the program. Over time, the internal shared state may change unexpectedly when different parts of the program updates it. Using immutable data structures whenever it’s practical helps keep internal state consistent.
Concurrency errors: One common error is race conditions – function A may depend on the result returned by function B running in parallel, and if the function B runs late function A fails. Developers who are coming from traditional programming languages may experience quite a surprise when they write their first programs in JavaScript/Node.js, which doesn’t normally wait for asynchronous statements to finish and go on executing the next line. As for programming languages that rely on threads for concurrency, writing thread-safe code is critical.
Environment errors: These errors are caused by the environment a program is running in such as out of memory/disk space errors and network time outs. Some of these errors can be solved by simply increasing server resources.

Debugging Methods

There are many ways to debug a program, but the following are the most common:

Print statements: Temporarily sprinkling a few print statements here and there to display variable values and track control flow is one of the oldest methods of debugging, but it’s still effective especially when debugging concurrent code.
Breakpoints: Pausing program execution when it reaches a certain point, and then inspecting variables as you step through the code line by line can be quite useful to understand code behavior. For debugging concurrent code though, use print statements since you can’t really step through concurrent code.
Logging: When you make your print statements more descriptive, categorize them by levels of severity such as INFO and ERR, and store them permanently in an external log file or database, what you get is logging – an indispensable tool to trace code execution flow. Logging comes handy for other purposes such as user analytics, but that’s outside the scope of this article.
Version Control: With version control systems like Git, you can easily revert to a past version of your code if one of the recent changes you had made introduced a bug.
Profiling: If you are having performance problems, you can use profiling tools to discover in which functions your program consumes CPU time the most.

It’s hard to get good at programming without getting good at debugging since debugging is such an essential part of the process. Debugging the code others have written is also a great way to learn how an unfamiliar code base works. Granted, debugging isn’t always pleasant, but the better you get at debugging, the faster you can write code – maybe not as fast as our Hollywood programmers, but we can only hope!

Writing Testable Code: A Simpler Way

Mon, 30 Oct 2023 00:00:00 +0000

There are many ways to write automated tests, and you may have heard about test doubles as a way to make testing easier. Some parts of our code can be hard to test, so we replace those parts with fake test doubles. The main reason for this is that external components, such as databases and web services, cannot easily be restored to a consistent state for each test run.

For example, if you’re testing whether a new user account is created successfully, you’ll need to provide a randomly generated email address or phone number during each test. Otherwise, the test would fail. There are workarounds, of course –you could delete the user account after the test– but that adds an extra step to an already slow process. This brings me to the second reason why test doubles are preferred: external components can be much slower than a test double simply returning a success value.

“Now wait,” you might ask, “aren’t we supposed to test whether a new user has actually been created in the database? Isn’t that the whole point?” Well, not really. The actual account creation process is probably the least interesting part to test because the database will almost always do its job and create the corresponding user record. What we really want to test is under what conditions the account creation would fail – for example, when a duplicate email address is used or when some fields are missing or contain invalid data. These control conditions, which fall under “business logic,” are typically the most important to test.

It appears that there are about half a dozen types of test doubles with interesting names like fakes, spies, and mocks. There are also lengthy articles explaining the differences between them. Apparently, there’s a need for all those variations, but for me a test double is a test double. In an expressive language like JavaScript, with its support for first-class functions and closures, you can likely go a long way by simply tracking which functions are called with which arguments to determine whether your code dealing with external components is working as it should.

So, I came up with a tiny JavaScript utility called fn-tester to record function calls and define test doubles. I deliberately kept it simple – the first version is only 14 lines long (except for tests of course), and I hope it stays that way.

fn-tester has only three options and one method. It requires running functions indirectly, using a variation of the Command design pattern. This approach is necessary to intercept all function calls and preserve the value of “this”.

const fn = require('fn-tester');

// Instead of parentObj.methodName(argument1, argument2, ...), use:
fn.run(parentObj, 'methodName', argument1, argument2, ...);

// For simple functions, use:
fn.run(myFunction, null, argument1, argument2, ...);

Now, you might argue that changing all function calls like that isn’t simple at all. However, there’s no need to modify every single function call. In a typical Model-View-Controller (MVC) application, it’s usually enough to focus on controller-level functions that manage interactions between multiple services. Services that handle databases and remote APIs will need test doubles in any case.

fn-tester can be configured by simply changing the test property, and the defaults are:

fn.test = { enabled: false, calls: [], doubles: [] };

When enabled is true, fn-tester starts to record function calls in calls. It also runs the test double functions stored in doubles (if any) instead of the originals.

Here is how to define test doubles for a simple user account creation controller:

// (...)

const userService = {
    hashPassword: function (password) {
        return bcrypt.hash(password, 8);
    },

    getUserByEmail: function (email) {
        return db('user').where('email', email).then(_.head);
    },

    insertUser: function (email, name, password) {
        return db('user').insert({ email, name, password }).returning('id');
    },
};

function createUser(email, name, password) {
    return fn.run(userService, 'getUserByEmail', email).then((existingUser) => {
        if (existingUser) {
            return Promise.reject('error.duplicateEmail');
        }
        return fn
            .run(userService, 'hashPassword', password)
            .then((passwordHash) => fn.run(userService, 'insertUser', email, name, passwordHash));
    });
}

fn.test = {
    enabled: true,
    calls: [],
    doubles: [
        function getUserByEmail() {
            return Promise.resolve();
        },
        function hashPassword() {
            return Promise.resolve('hash');
        },
        function insertUser() {
            return Promise.resolve(1);
        },
    ],
};

fn.run(createUser, null, 'some@email', 'name', 'pass').then(() => console.log(fn.test.calls));

When fn-tester finds a function in test.doubles with the same name as the function called, it runs the test double instead of the original. test.calls contains the function calls complete with the arguments passed.

[
    ['createUser', 'some@email', 'name', 'pass'],
    ['getUserByEmail', 'some@email'],
    ['hashPassword', 'pass'],
    ['insertUser', 'some@email', 'name', 'hash'],
];

test.calls is useful to determine if a function is called and has received the correct arguments.

Finally, here is an example of how fn-tester can be used with Chai and Mocha, a popular assertion library and testing framework respectively:

const fn = require('fn-tester');
const chai = require('chai');
const chaiAsPromised = require('chai-as-promised');
const should = chai.should();
chai.use(chaiAsPromised).should();

// (...)

describe('createUser', function () {
    it('should go through with account creation', async function () {
        fn.test = {
            enabled: true,
            calls: [],
            doubles: [
                function getUserByEmail() {
                    return Promise.resolve();
                },
                function hashPassword() {
                    return Promise.resolve('hash');
                },
                function insertUser() {
                    return Promise.resolve(1);
                },
            ],
        };

        const result = await fn.run(createUser, null, 'some@email', 'name', 'pass');
        fn.test.calls.should.be
            .an('array')
            .that.deep.includes(['getUserByEmail', 'some@email'])
            .and.that.deep.includes(['hashPassword', 'pass'])
            .and.that.deep.includes(['insertUser', 'some@email', 'name', 'hash']);
    });
});

Three Aspects of B2B Software Development

Mon, 23 Oct 2023 00:00:00 +0000

B2B (Business-to-Business) software is different from B2C (Business-to-Consumer) software for several reasons, but the most important one is this: In B2B, each customer isn’t an individual but a company with its own unique needs and workflows. If your software isn’t compatible with the way a company operates and there are no convenient workarounds, you risk losing their business, which brings me to the first important aspect of B2B software…

Customizations

In the B2B software world, your software may solve every problem a company has, but it may still end up coming short because it doesn’t solve those problems just the way they need. As a result, both prospective and current customers often request changes to the software. The most common requests we encounter as a maker of carbon accounting and energy management platforms include:

Custom dashboards
Custom reports
Custom fields or options when entering data
Custom notifications
Custom logos and wording

Having a built-in ability to create modular, preferably drag and drop dashboards is an important asset in meeting custom dashboard requests. Custom reports are equally important, and it’s often possible to arrive at a common reporting format, which may benefit more than one customer.

If you haven’t designed your B2B software with customization in mind, you will likely have a lot of problems down the line with your software looking like the car designed by Homer Simpson with seemingly unrelated features sticking out like a sore thumb.

Custom car design by none other than Homer Simpson. Image by Electronic Arts

In order to be a product company, not a consultancy, you should avoid one-off custom features as much as possible and focus on features that can be customized instead. It’s a tight rope to walk on – features that support customization introduce complexity, and that may not only make your software harder to use for the end users, but also make it harder to maintain for you. One-off features, on the other hand, can easily turn out to be a maintenance nightmare because they are specific to one customer only, are not usually tested as well as the core software features, and are easier to break after software upgrades.

Integrations

After customizations, integrating with company systems comes up as a common customer requirement, and the most requested integration is probably SSO (Single Sign-On). If a large number of employees are going to use your software, Active Directory integration (usually via LDAP) is the most popular SSO request. It’s a relatively easy integration to do, isn’t unique to a single customer, and saves you from dealing with support requests about user account issues like forgotten passwords.

There are basically two types of integrations from a technical point of view – integrations using your software’s API (Application Programming Interface) and integrations using everything else. The former is the ideal one because you have the most control and usually the least amount of additional work to do other than documenting your API well, but it requires the company to have their own software developers or contractors to do the integration.

Other integrations may require connecting to company systems by using APIs (SOAP used to be common, but REST APIs using JSON is getting popular for data exchange) and reading and parsing files (usually CSV) on an FTP site, which in this day and age is still popular. You may also push data to other software products such as internal company dashboards. Integrations that depend on internal company systems tend to be more fragile due to the increased number of dependencies and may occasionally break for reasons as simple as the company IT department’s decision to change the address of an internal API service without telling you.

These days B2B cloud software is quite common, but some customers may still request on-premise installations of your software due to regulations or their 3rd party software policies. While on-premise installations are not really integrations per se, they still require you to adhere to the internal software deployment and testing procedures of a company, and can turn out to be gargantuan tasks in some cases, so beware.

Backward Compatibility

Unlike many B2C software where “fickle” users can switch from one software to another on a whim, switching from B2B software can be quite challenging especially when there are significant switching and training costs are involved. When your software is customized to the needs of a company, has several integrations with that company’s proprietary systems, and after who knows how many meetings that spanned months or even years to arrive at this successful outcome, there has to be a really good reason for a company’s decision makers to switch to another software provider and go through the whole process all over again.

With B2C software you are allowed to move fast and break things especially in the beginning. In B2B software, however, backward compatibility is important. Your software doesn’t sit in isolation – it’s connected to other systems, and a lot of people depend on your software to do their jobs. Just like it’s hard for a company to switch to another provider, it’s also hard to make major changes to existing B2B software with a large user base. And there lies the dilemma – how to change or improve functionality without making potentially breaking changes and requiring extensive training. At the end of the day, if some things need to change in your software, it all depends on whether the new features/improvements are worth the trouble for the customer. At large companies it’s not uncommon to come across software with user interfaces that look like they were designed 20 years ago, and you can probably guess why.

In my opinion, over time the relationship of B2B software providers with their customers starts to resemble a marriage for lack of a better word, especially the software in question is vital to the customers’ operations. A well designed B2B software attuned to each customer’s specific needs may not win any user interface design awards, but it gets the job done, and that’s what really counts in the end.

Small Software Development Team Structure

Sun, 15 Oct 2023 00:00:00 +0000

I’ve worked at software development startups including my own throughout my career. I’ve worked as a software developer, tech lead, and CTO. I’ve written code, managed servers, created user interfaces, designed web sites, prepared documentation, and even done some marketing. I’ve hired and assisted in hiring dozens of people: both developers and non-developers. But, first things first, if you are a cash-strapped startup, you can usually get away with a team of just one or two developers and hope for the best. There’s absolutely nothing wrong with that. But if you have the funds to add new members to your team, don’t just go with more developers. Developers aren’t generally very good at designing products people want to use – they are good at implementing well-specced features. That leads to the first non-developer position I recommend filling in:

Product Manager

A good product manager is worth their weight in gold (OK maybe not that much given the price of gold these days, but you get the idea). The product manager’s job involves:

Knowing the product inside out. She should be the person everyone goes to if they have a product question.
Providing input on which features to develop first. This requires a thorough understanding of company objectives and user expectations.
Dividing new feature developments into manageable chunks with the help of the software tech lead. Great product managers can also make suggestions about which parts of the features should not be developed right away and can be safely deferred.
Writing detailed specifications for each new feature to develop.
Manually testing the newly developed features to see if they conform to the specification.
Writing documentation about features.
Helping with release planning.

UX Designer

Good product managers can double as UX designers to a certain extent especially in the early days of a startup, but as soon as you get the funds to hire a proper UX designer, don’t think – hire one. A good UX designer can design user interfaces that look nice and a joy to use. From a user’s perspective your product is simply what’s visible to her. A user can’t be expected to know how things work in the background. If she sees a button on the screen and presses it, she expects it to behave as she thinks it should. A good UX designer can make your product behave as the user expects it to, and that leads to happy users and fewer support requests.

A UX designer’s job involves:

Designing screens and deciding on how the user navigates from one screen to another.
Doing usability tests preferably on prototypes in software like Figma before developers implement a feature.
Iteratively improving existing designs by observing actual user interactions with the product.
Working closely with the software developers and modifying the designs if they are technically difficult to implement. Not all designs are practical to implement, and changes may be necessary to reduce development time as long as they don’t negatively affect user experience.

QA Engineer

Once you have your product manager and UX designer in place, it’s time to hire your first QA engineer. Sure, software developers can write their own automated tests, but their main job is to implement features. A dedicated QA engineer can be much more productive than developers in writing tests, especially the all important integration tests that touch multiple features. A QA engineer has a birds-eye view of your code base while developers often focus on individual parts of the code.

A QA Engineer’s job involves:

Having a good understanding of how each feature under test is supposed to work.
Writing maintainable test cases that make sure that your code base functions correctly. Using a Behavior Driven Development (BDD) tool like Cucumber helps design test flows that can be modified easily.
Making small modifications to your code base such as adding a class attribute to an HTML tag so that a feature is easier to test. When bigger modifications are necessary, asking software developers to write code more suited to testing.
Keeping test suites up to date. It’s normal for some tests to fail for reasons other than bugs, but those tests should be fixed as soon as possible, and the number of failing tests should be kept to a minimum.

Software Tech Lead

This role is usually given to the most experienced software developer on the team, but experience alone isn’t sufficient. The tech lead must have some “soft” skills in order to effectively prioritize and distribute work on her team. If your tech lead doesn’t have enough hard or soft skills, she can receive mentoring from a more experienced software developer if you have one like the CTO or a tech lead on another team.

A tech lead’s job involves:

Deciding on the architecture and technologies to use for the features to develop. A good tech lead is practical – she chooses simple architectures over complex ones and prefers battle-tested technologies over the latest buzzword ones.
Prioritizing work in coordination with the product manager.
Making quick estimates on how much she thinks each feature would take to complete taking into account the inevitable bugs that would need to be fixed.
Writing code that deals with core functionality.
Coming up with suggestions to improve existing code and processes.
Helping team members with their technical and personal issues.

Software Developer

In my experience, for small teams, having an experienced software tech lead supported by several junior software developers tends to work well. Junior developers often need assistance from the tech lead or from each other, so I have them work in groups of two or three, and make it clear that they should help each other as much as possible. The technical hurdles to cross can be overwhelming for junior developers especially when they are fresh out of school. Working in groups make them feel that they aren’t alone and it’s OK to seek help when they’re stuck.

For software developers, it isn’t important what they know as long as they have the basics covered – what’s important is that they are highly motivated to learn. It also helps if they’re quick learners.

DevOps Engineer

Strictly speaking, this position isn’t necessary if your product has few users, doesn’t require large servers to function, and your users are fine with the occasional downtime if it’s not too long. If you have a software developer on your team who can willingly devote some of her time to managing your servers then you can probably get away without hiring a DevOps engineer until you really need one.

A DevOps Engineer’s job involves:

Having separate test and production servers for your product.
Having “one-click” software deploy and rollback processes.
Backing up all critical data such as databases and git repos regularly, and testing those backups actually work.
Monitoring servers for abnormal conditions and taking corrective actions.

As a final note, for a startup it’s especially important to hire people who can take the initiative and work under minimal supervision. People who are accustomed to large corporate environments where it’s common to have a career ladder and strict processes to follow may not do well at a startup.

Three Power Amp Fan Mods

Fri, 29 Sep 2023 00:00:00 +0000

I use pro audio power amps in my home theater. They provide tons of power for little money. There is one major downside though. Most models feature noisy fans, and for a good reason – pro audio power amps are often run at their limits, and many manufacturers prefer to err on the side of caution. For home use though, you can usually get away with replacing the built-in fans with quieter ones as long as the amp isn’t housed inside closed spaces with poor or no ventilation.

Quiet fans move less air, but movies rarely demand the level of cooling required in playing very loud, very compressed music for hours at a time, which is exactly the thing that taxes power amps the most.

I have done three power amp fan mods so far, and each one was different. Let’s start with the easiest one first:

The Behringer iNuke NU6000DSP, a class D amp with built-in DSP. It can do about 2 x 2,000W into 4 ohm short-term, an inexpensive amp well suited for subwoofer duty.

iNuke nu6000DSP, a class D amp with built-in DSP. It can do about 2 x 2,000W short term into 4 ohm.

The NU6000DSP uses dual 80 mm 12V fans. For this one, I used USB powered 80 mm fans and mounted the fans from the outside. Internal mounting meant drilling holes in the case, which I preferred not to. The USB connector is plugged into the AVR, and the fans start spinning when the AVR is turned on.

iNuke nu6000DSP USB fan replacement.

Difficulty: Easy

Up next, the Behringer iNuke NU3000DSP, smaller brother of iNuke NU6000DSP, is again a class D amp with built-in DSP. It can do about 2 x 500W into 4 ohm. When bridged, it can produce as much as 2,000W short-term into 4 ohm.

The NU3000DSP looks exactly like the NU6000DSP, but is shorter in depth. It uses an 80 mm 12V fan covered by a black cardboard shroud, which is lightly glued to the fan. The shroud appears to be designed to concentrate air flow to four transistors with small yellow heat-sinks. Once the shroud is removed, replacing the existing fan with a quieter version is easy. The fan connector needs to be fixed in place with a dab of hot glue though.

iNuke NU3000DSP's fan covered by a black cardboard shroud.

iNuke NU3000DSP quiet fan replacement.

Difficulty: Easy

The fan mod that required the most effort is a Behringer KM750, a traditional class AB amp. I use this one for my center channel speaker. It can do about 2 x 180W RMS into 4 ohm.

Behringer KM750, a traditional class AB power amp that can do 2 x 180W RMS into 4 ohm.

The KM750 has a 90 mm 24V fan mounted on an internal heat sink. As the heat sink is wide enough, I decided to use a 120 mm fan, which can spin slower (and quieter), but still move quite a bit of air. So, how do you mount a 120 mm fan to a heat sink with holes drilled for a 90 mm fan?

Behringer KM750 uses a 90 mm 24V fan with a 2-pin connector.

I had four options:

Mount the fan by using just one hole and hope the fan wouldn’t move over time.
Drill and tap new holes in the heat sink.
Buy an adapter.
3D print an adapter.

I happen to have a 3D printer, and decided to print a 3D model I found online. I had to print it slightly thinner though. Otherwise it wouldn’t fit under the top cover of the amp.

3D printed 120 mm to 90 mm fan adapter.

Since most quiet fans are 12V, an inexpensive buck converter is necessary to lower the 24V provided by the amp to 12V or less. In the end, the 120 mm fan wasn’t quiet enough at 12V, so I lowered the voltage to less than 10V until the fan noise was acceptable. Even in cases when you don’t need a buck converter, they can come handy to lower fan noise to exactly the level you want.

120 mm fan installed. A buck converter lowers the voltage to less than 12V.

In the picture above, the buck converter can be seen on the bottom-right corner mounted on a suitable spot. I didn’t feel like drilling holes in the bottom of the case. There was already a screw hole for one of the feet, so I used that screw, a 3D-printed piece to mount the buck converter and some hot glue to hold everything in place.

Difficulty: Medium

…and that’s a wrap. Three different amps, three different ways to do fan mods.

Disclaimer: Working on electronics connected to AC power can kill you. If you don’t know what you are doing, please don’t.

CNC Router Control Box Wiring

Sun, 24 Sep 2023 00:00:00 +0000

A few months ago, I bought an entry-level CNC router from a local supplier, knowing full well that I would probably need to modify it extensively. The price of the machine was right, and in case I decide to build a better one on my own, this machine would be a good starting point to learn how all the parts are connected together.

The machine performed OK as long as I didn’t push it hard. I added drag chains to the Y-axis to prevent the step motor cables from moving all over the place while the machine was running. I also added a dust boot which was not only essential for dust extraction, but also freed me from standing right next to the machine holding a vacuum hose while it was running.

After these modifications, connecting a simple height probe that would be used to automatically calculate the distance of the router bit to the waste board seemed easy enough – just two wires connected to the control board located inside the CNC router control box. But, try as I might, I couldn’t get my hands to the red control board even after removing some of the cabling. It was very cramped inside. I also needed to reduce the current to the Y axis motors (they got too hot), but I couldn’t access the jumpers on the step motor drivers either. There were other things that I wasn’t happy with the control box – there were no jacks outside the box to connect cables. All cables went straight inside through bushings.

Inside the original cramped CNC router control box.

So, I thought it wouldn’t be too hard to buy a bigger box and move everything from the old box to the new one, making a few improvements along the way. A CNC router control box is relatively simple to make. It consists of a power supply, step motor drivers and a control board. I replaced the generic brand 12V power supply with a Mean Well 24V power supply I had lying around. The extra voltage should help the motors generate more torque at higher speeds.

I changed the component layout so that the control board’s USB input could be accessed directly from the outside. I also placed the step motor drivers between the power supply and control board to separate signal wires coming from the control board and power wires coming from the power supply as much as possible. Finally, I added cable ducts with slits to keep the wiring neat. Can’t recommend these ducts enough – they make wire management very easy.

The pictures below show the testing of the components outside the box, and a close-up view of the step motor driver wiring.

The new electronics layout was tested outside the box.

Signal wires come from one side and power wires come from the other side through wire ducts.

For outside connections, I used 4-pole Speakon connectors for step motor cables, and 3-pole XLR connectors for things like height probes and limit switches. These connectors are normally used in professional audio, but I really like the easy locking mechanism of these types of audio connectors.

In the picture below, only the AC power socket and the Speakon jacks are installed. With the “eyes and mouth”, the box kind of looks cute though, don’t you think? :)

Outside view of the new CNC router control box with Speakon jacks installed.

The picture below shows the components installed in the box. There’s plenty of space now for future expansion. It’s probably a good idea to install a fan to keep the components cool especially if the machine is expected to run long hours.

Inside the new CNC router control box. No bushings! :)

Tools of the trade: Wire stripper, crimpers, and a cordless drill with the necessary bits. I especially like the look of the big step bit installed in the cordless drill. Looks like a ray gun :) You can buy the cheapest step drill bit you can find. They are perfectly adequate for drilling plastic, which is all I needed here.

Wire crimpers.

Step drill bit is great for circular cutouts on plastic.

Finally, the finished CNC router control box with four step motor connectors and one height probe connected.

The new CNC router control box standing under the CNC router table.

Disclaimer: Working on electronics connected to AC power can kill you. If you don’t know what you are doing, please don’t. Also, keep in mind that I’m a hobbyist, but I did consult professionals including an electrical engineer friend for my CNC control box work.

Three Stages of Software Development

Wed, 20 Sep 2023 00:00:00 +0000

There are a lot of things to consider before starting to write code: Which software libraries to choose? Use SQL or NoSQL? What would the database schema look like? Would we need a caching system to handle heavy traffic? How should the code be structured? And so on…

These questions are certainly important, but we live in an era of fast-changing and often ill defined user requirements. The conventional software development model of plan > code > test can more often than not fail to produce software that meets user requirements in a reasonable amount of time. One alternative is a three-stage process that allows us to iteratively learn what the users really want, and if all goes well develop better software faster.

Stage 1: Make it Work

This is similar to Stephen King’s approach to writing the first draft of a new book. King tries to finish the first draft as quickly as possible, with little concern for sentence structure or vocabulary. The important thing is to create a draft that, though rough around the edges, can be refined and polished later. Similarly, in software development the aim is to produce a prototype that can be shared with users without worrying about code quality or technical architecture very much. It’s acceptable for some parts to be flawed, and performance may not be optimal at this stage – these concerns can be addressed later.

There’s a saying that the best code is no code. Depending on user feedback, we may find that we don’t need to work on features we initially thought would be challenging, or we may uncover new areas to work on that we haven’t anticipated earlier.

Stage 2: Make it Right

After the first stage is complete, it’s time to fix all the important bugs and address any shortcomings of the product. During this stage, we should make the user interface easier to use, ensure that there are automated tests to check all features work as expected, display proper error messages (no more “unknown errors” please), and address all security concerns. This is usually the stage where the majority of development time is spent. Good luck!

Stage 3: Make it Fast

This is the stage I personally find the most fun. At this stage, we should have a software product that has been shaped by user feedback and is functioning correctly. However, we may start encountering or have already encountered performance issues. So, it’s finally time to optimize.

Many software developers have a tendency to do performance optimizations early in development, but there’s a good reason to delay optimizations until the final stage: predicting what would be slow within a complex software system before it reaches a certain level of maturity and is tested by actual users is hard even for experienced software developers. Also, a relatively mature software product is essential for establishing a baseline for performance before making any changes to the code.

In my experience, it’s more efficient to achieve perfection step by step, rather than trying to write beautiful and performant code from day one. So, please try to ignore the perfectionist voice in the back of your head, at least early on.

Subwoofers for Home Use: An In-depth Guide

Sun, 12 Feb 2023 00:00:00 +0000

Subwoofers are probably the most misunderstood and difficult-to-integrate components of a music or home theater system. However, when integrated properly, they have the potential to elevate your speakers to a whole new level. Over the years, I’ve bought and built quite a few subwoofers, and I’d like to share what I’ve learned about them with you.

The first few sections of this guide cover the basics, such as how to choose the right subwoofer(s) and how to make them sound good (they aren’t exactly plug-and-play). Later sections discuss building your own subwoofers for a fraction of the cost of commercial units. Finally, for endgame deep bass reproduction, you might want to read the section on the double bass array (DBA).

First things first, what’s a subwoofer?

The image of a large black box housing a big speaker driver that pumps out deep bass is probably the first thing that comes to mind. A subwoofer’s purpose is to complement the main speakers, many of which struggle to reproduce the lower octaves of music and sound effects at satisfactory levels.

Types of subwoofers

For home use, the most popular types are sealed and ported (also known as vented). Sealed subwoofers have only the driver that produces bass while ported subs have one or more vents usually in the shape of a circle or rectangle that produce bass in addition to the driver.

Sealed subwoofers are compact. Although large sealed subwoofers exist, a sealed subwoofer can be made only slightly larger than the driver dimensions (imagine a 13-inch cube housing a 12-inch driver).

Ported subwoofers are generally quite a bit larger, but they can produce significantly more output –up to 10 dB– within a certain frequency range compared to sealed subwoofers. However, some may suffer from port noise (also known as chuffing) when pushed to their limits.

A rather nice looking 15" slot ported subwoofer. Not all ported subwoofers look this good. In fact, featureless black boxes are the norm. Image © Seaton Sound, Inc.

What to look for in a subwoofer?

Box dimensions: It depends on where your priorities lie. If compact size is important to you, then large ported subwoofers are probably out of the question. That doesn’t mean that all ported subwoofers are large – there are relatively compact ported subwoofers, but they have to sacrifice some deep bass output in order to fit in a compact cabinet.

Driver diameter: All other things being equal, as the driver diameter gets larger, the bass output produced by the driver increases. Subwoofer drivers typically come in nominal diameters of 8", 10", 12", 15" and 18" (20, 25, 30, 38, 46 cm respectively). The most popular diameters for home use are 10" and 12". A subwoofer can have more than one driver, but most have one.

15", 12" and 10" drivers from a car subwoofer driver line. Some home subwoofer manufacturers use car drivers, and they are also a popular choice for DIY subwoofer builders. More on that later.

A good rule of thumb is that as you increase the size of the driver, you get about twice as much bass output. For example, a 12" driver has roughly double the output of a 10" driver, assuming both drivers have similar technical parameters other than surface area, and the same amount of amplifier power is used. However, there are exceptions. Some high-performance 10" drivers can outperform an average 12" driver, but these can be expensive and often require substantial amplifier power to reach their full potential.

Amplifier power: After driver size, the next important factor to consider is the available amplifier power. Most commercial home subwoofers come with built-in amplifiers. The amplifier needs to provide enough power for the driver to achieve satisfactory output levels. It’s wise to disregard any peak power ratings, as they are often exaggerated and difficult to compare across different brands. What matters is RMS power (also known as continuous power), preferably provided with a distortion rating (like 1% THD) to allow for accurate comparisons.

A home subwoofer usually has a built-in amplifier to supply power to the driver.

There’s no need to sweat over small differences in amplifier power. For example a 25% difference may look big on paper when comparing a 400W RMS amplifier to a 500W RMS amplifier, but it only amounts to 1 db more output all other things being equal, which is so small that it’s hard to tell the difference in practice. For a meaningful output difference, an amplifier should be able to provide twice the power which is equivalent to 3 db of additional output.

Frequency response: Finally, take a look at the +/- 3db frequency response specification of the subwoofer. A subwoofer that can authoritatively reach down to 25 Hz or lower is ideal. However, frequency response alone isn’t very telling. Nowadays, many sealed subwoofers utilize digital signal processing (DSP) or other means to enhance lower frequency output at lower volumes. If the driver lacks sufficient surface area and the amplifier power is limited, a subwoofer’s ability to reach low frequencies doesn’t matter much, as these frequencies won’t be loud enough to hear or feel at higher volumes.

How many subwoofers?

How about zero?
It might seem odd to suggest going without a subwoofer in a guide about home subwoofers, but if you have terrible room acoustics that are difficult to manage, it might actually be a good idea to go without one. Contrary to common belief, boomy bass isn’t always caused by cheap subwoofers. More often than not, the main culprit is poor room acoustics – walls made of brick or concrete and minimal furnishings can create severe resonance issues.

If you clearly hear an echo when you clap your hands inside a room, you will likely experience boomy bass with a subwoofer in that space regardless of the subwoofer’s quality. While there are ways to address some acoustical issues, not all solutions are practical. However, don’t get discouraged. Just be aware that in some cases, it might be better to have no subwoofer at all than to have one.

One subwoofer placed right
If you are going to have only one subwoofer, like most people, achieving the most even response from your subwoofer is all about placement. There are really only two optimal locations for a single subwoofer to accomplish this (more on this later).

Going dual: Buy two get two free
If you have the funds and space for two subwoofers, having dual subwoofers offers several advantages over a single subwoofer, assuming their placement in the room is correct:

Quadruple the output over a single subwoofer if they are placed relatively close to each other. You literally get two additional subwoofers worth of output for free, thanks to a physical phenomenon known as mutual coupling of sound waves.
Consistent bass output across a horizontal plane. In other words, no matter where you sit on the sofa, the bass will sound about the same.
Instead of one large subwoofer that is difficult to place or hide, two smaller subwoofers can be used.
Two subwoofers can function optimally in many more locations.

More than two subwoofers?
While there is no upper limit to the number of subwoofers you can use, for most people, having two subwoofers hits the sweet spot. If you simply need more bass output, you can colocate subwoofers (a fancy term for placing multiple subwoofers in close proximity), and they will act as one large subwoofer.

You can also have more than two subwoofers distributed across the room, but they require careful measurements to function properly. If you are curious about non-colocated subwoofer setups with more than two subwoofers, head over to the double bass array section.

How to setup subwoofer(s)

It’s extremely unlikely that randomly placing a subwoofer will result in good sound. Subwoofers require some effort to sound their best.

Where to place your subwoofer(s)

One subwoofer
Try to avoid corner placement as it will produce uneven bass across the room. If possible, place your subwoofer along the front wall, preferably right in the center. If the center of the front wall isn’t available, you can try placing your subwoofer along the center of the rear wall.

Center front or rear wall placement is ideal for a single subwoofer.

Two subwoofers
Having two subwoofers offers a lot of flexibility with placement. They can be placed in each corner of your front wall. You can also move them closer toward the center of the wall as long as each subwoofer is roughly an equal distance from the side walls. Placing the subwoofers at locations of 1/4 the width of the front wall is recommended to achieve the smoothest response. For example, if your front wall is 4 meters wide, try placing your subwoofers about 1 meter away from the side walls.

Dual subwoofers can be placed symmetrically along the front wall for a smooth bass response.

Where to sit

If possible, avoid sitting close to walls (especially the rear wall), where bass can become overwhelmingly intense in a negative way. Sitting in the middle of the room isn’t a good idea either, as this is where bass energy is typically at its weakest.

The Crossover Frequency

The crossover frequency determines at which point your main speakers start to hand off bass reproduction to your subwoofer(s). An 80 Hz crossover frequency is usually recommended, but if your speakers are particularly lacking in bass, you might want to increase this to 100 or 120 Hz for a smoother transition. Virtually all AV receivers (AVRs) have built-in crossover capabilities. If you are using an AVR to set the crossover, it’s best to disable the crossover setting on your subwoofer(s) or set it to the maximum setting possible.

Room Correction

Once the subwoofer placement is optimal and you are ideally not sitting too close to the rear wall, there is one more step left to enjoy smooth bass reproduction in your room: applying room correction. This is necessary to address room resonances that can’t be solely fixed by subwoofer placement.

If you own a modern AV receiver, it’s a good idea to try out your AVR’s automatic room correction capabilities. Many AVRs produced in the last decade support some form of automatic room correction, and they generally work in a similar way, although the results may vary depending on the quality of the room correction algorithm used (some are very basic) and the severity of the acoustic issues.

You should place the calibration microphone that comes with the AVR in one or more listening positions, preferably attached to a tripod, and allow the AVR to take measurements by playing test tones for each speaker, including your subwoofer. After the measurements are completed, corrections are applied at certain frequencies to reduce the audibility of resonances in your listening position. Although it’s called room correction, the effect of the corrections is often limited to the small listening area where the calibration microphone is placed.

For advanced users who have a good measurement microphone (you can probably get away with the AVR’s microphone although some caveats apply), there is a powerful free software application called REW (Room EQ Wizard) that can be used to measure your subwoofer’s response. Then, you can apply corrections using the parametric EQ capabilities available on your computer or an external device called a DSP. More on that in the DIY section.

DIY Subwoofers

Granted, DIY (Do-It-Yourself) isn’t for everyone, but for those interested in building their own subwoofers and making their own acoustic measurements, the potential rewards are significant. You can achieve high-quality bass reproduction for a fraction of the cost of an equivalent commercial subwoofer setup and professional calibration. Not to mention the personal satisfaction of building something yourself that will be put to good use for years to come.

Building a subwoofer isn’t really hard. It can be done with little to no woodworking skills and minimal tools. Having had no prior experience, I built my first subwoofer using nothing more than a power drill, a jigsaw, and a few wood clamps. Since then, I’ve built many subwoofers over the years, and although the later ones look better, they’ve all functioned well.

Building a subwoofer can be done with little to no woodworking skills.

A subwoofer consists of three components: one or more subwoofer drivers, an amplifier, and a box. Sourcing the driver(s) and amplifier(s) is relatively easy. The only remaining task is to build a box from MDF (Medium Density Fiberboard) or high-quality plywood. If you don’t have the tools to make the cuts, your local home improvement store will likely do the job for you for a small fee. You may also purchase ready-made subwoofer flat-packs online if they are available in your region. Once you have the pieces cut, they can simply be glued together with wood glue and held with a few clamps until the glue dries. More on box construction later.

How to choose a driver

Among DIY subwoofer builders, the driver of choice is often a car subwoofer driver for a number of reasons:

They are widely available and many are reasonably priced.
They have the necessary moderate to high excursion capabilities to reproduce special sound effects like explosions at high volumes, a desired feature for home theaters.
Many are designed to work in relatively compact enclosures, which is always a plus.
Finally, many of them are tolerant of abuse since car subwoofer manufacturers know it all too well that car subwoofers are pushed beyond their limits quite often.

10" and 8" car subwoofer drivers with substantial motor structures.

Thiele/Small parameters
Most manufacturers provide the Thiele/Small (T/S) parameters for their subwoofers. These parameters can be used to objectively compare drivers and are also essential for determining the appropriate box size. For home theater use, the most important parameters are:

Maximum linear excursion (Xmax), the maximum distance a subwoofer cone can travel in one way without gross distortion. Xmax is important because as you go down one octave such as from 40 hz to 20 hz, the subwoofer cone needs to travel 4 times the distance to have the same level of output.
Usable surface area of the driver (Sd), which is the area of the driver excluding parts of the surround and outer frame. The greater the surface area, the easier it is to produce bass.
Resonant frequency (Fs), which is the natural resonant frequency of the cone. This is the frequency where the driver is most efficient (outside of a box).

You will want Xmax to be as high as possible and Fs to be as low as possible. Sd does not change significantly across drivers of the same diameter, but it is very important because multiplying Xmax by Sd gives you Vd (volume of displacement).

You’ve probably heard the expression “there’s no replacement for displacement” used for car engines. Similarly for subwoofers, the larger the Vd of a driver, the greater the deep bass output will be, all other factors being equal. For example, if the Sd of two drivers are similar (they both have the same diameter), the one with twice the Xmax will produce twice the deep bass output, again assuming all other factors are equal. A large diameter driver with little Xmax and a small diameter driver with plenty of Xmax can produce about the same amount of deep bass, although you will likely need to provide the smaller driver with considerably more power to achieve this.

Other T/S parameters such as QTS and Vas are also important in determining the optimum box size for a subwoofer. If you require a very compact box size without sacrificing too much deep bass output, make sure these two parameters are as low as possible.

Box volume

The easiest subwoofer box to build is a sealed box with no ports. They can be made fairly compact and are tolerant of mistakes. There are free software applications such as WinISD that can help you simulate the behavior of subwoofer enclosures after you enter your driver’s T/S parameters. They also let you see whether you exceed the excursion limits of a driver when you feed too much power to it, but for most drivers, unless the manufacturer recommends otherwise, you can simply use the following approximate sealed box volumes:

10" driver: 14 liters (0.5 cubic foot)
12" driver: 28 liters (1 cubic foot)
15" driver: 56 liters (2 cubic foot)

There are software applications that can help you simulate subwoofer behavior.

If you are going to put two drivers in the same box (such as in a dual opposed subwoofer build), you need to double the required volume. Please keep in mind that when calculating the volume of a box, you should use its internal width, height, and depth dimensions.

Amplifier selection

While car subwoofer drivers are popular among DIY subwoofer builders, when it comes to amplifiers, inexpensive multi-kilowatt professional audio (pro audio) amplifiers are clearly the preferred choice. Pro audio amplifiers may not have the best signal-to-noise specifications, but they excel at providing substantial power, which is essential for power-hungry subwoofers. Some of them even have built-in DSP capabilities, which simplify setting crossovers and correcting response anomalies caused by room acoustics.

There is one major downside to using pro audio amplifiers – virtually all of them have built-in fans, and some of these fans can get quite noisy. It’s challenging to keep an amplifier that provides kilowatts of power cool without a fan. Fortunately, there are models with relatively quiet fans that only start spinning when the amplifier is driven hard. For those amplifiers with loud fans, putting them in another room or replacing the existing fans with quieter ones are options to consider, but the latter would likely void your warranty.

Another potential downside is that unlike commercial subwoofers with built-in plate amplifiers, pro audio amplifiers are often stand-alone units. While plate amplifiers do exist for DIY subwoofers, they aren’t very cost-effective. On the upside, stand-alone amplifiers are potentially more reliable because they offer better cooling.

Building a box

Building a basic sealed subwoofer box isn’t difficult. It requires six sides cut from 18 mm (~3/4") MDF or high-quality plywood (preferably made from void-free hardwoods such as Baltic birch), a cutout for the driver, one or two braces to strengthen the box, some wood glue, and a few clamps to hold everything together until the glue dries. Using silicone caulk to seal all internal joints will ensure the box is airtight.

Some wood glue and a few clamps are enough to make a box.

While it’s not strictly necessary, adding some polyester fiber stuffing inside the box can make the internal volume appear slightly larger to the driver(s), allowing them to play a bit deeper. For large boxes, stuffing also helps eliminate most resonances caused by sound waves bouncing around inside the box.

Drivers can be mounted using regular wood screws (pre-drilling pilot holes is highly recommended to avoid splitting the wood). Heavy-duty drivers weighing over 10 kg (~22 pounds) may require thicker baffles (mounting surfaces) and thicker screws.

Marking the cutout for a speaker terminal cup.

Dual opposed subwoofers:
Subwoofer boxes housing a single driver tend to “walk around” when pushed hard. To fully eliminate this behavior, you can use two drivers placed on opposite sides of the box. Any movement made by one driver will be immediately canceled out by the opposing driver. You can even balance a coin on top, no matter how much the drivers are moving! Dual opposed subwoofers also have the added benefit of providing more even bass output across the room, regardless of where the drivers are facing.

Sealed enclosure for dual opposed subwoofers with a center brace.

When using two drivers, you can connect them in series (which doubles the resistance) or in parallel (which halves the resistance). For example, two 4 ohm drivers connected in series would present an 8 ohm load to the amplifier. When connected in parallel, they would present a 2 ohm load. Subwoofer drivers often come with instructions on how to connect multiple drivers together. Having a multimeter is helpful to verify the final resistance. Just keep in mind that some amplifiers may not support low resistances, especially when they are set to bridge mode.

Making it look good:
Building a box isn’t difficult, but making it presentable may take some effort. Of course, there’s nothing stopping you from leaving the box as is, which won’t affect the performance one bit.

If you have a router, you can apply a roundover or chamfer to the edges for a nicer look. A router also comes in handy when making perfect circular cuts for the drivers. If you plan to flush mount the drivers, a router may even become a necessity.

There are basically two options for finishing: you can paint the box or apply a thin veneer made of vinyl or wood. Whichever path you choose, the surface needs to be sanded smooth first. This can be done by hand, but if you plan to build more than one box, you might want to invest in or borrow a random orbital sander, which can significantly speed up the process.

A dual opposed subwoofer box painted white with a paint gun.

If the box is made of MDF and you’ve decided to paint it, the edges must be sealed first – a mixture of half water and half wood glue can be used for this purpose. Otherwise, the edges will absorb the paint, resulting in an uneven surface finish. You can use a foam brush to apply the paint if you don’t mind some brush marks, but spraying is clearly superior. If you don’t have a paint gun, spray cans can also be used effectively. After the paint dries, it’s a good idea to apply several coats of clear coat to protect the paint, which also add a nice satin or gloss sheen.

Measuring and correcting output

The in-room frequency response of subwoofers has nothing to do with their natural outdoor response. Resonances (also called standing waves) occur inside the room, and the resonant frequencies are determined by the room dimensions. The smooth open-air response of a subwoofer is replaced by peaks and valleys (also called nulls). Peaks cause boomy bass, while nulls result in weak bass. To make matters worse, the magnitude of these peaks and nulls varies from one location to another within the room – any corrections made using the automatic room correction features of an AV receiver or through DIY parametric EQ will only be applicable to the small area where the measurements are taken.

Simulated frequency response of two subwoofers in-room (solid line) vs open-air (dashed line) in listening position. The colored vertical lines correspond to this particular room's resonant frequencies.

A powerful free software application called REW (Room EQ Wizard) is a popular choice to make measurements and to automatically calculate the corrections necessary to reduce the magnitude of peaks. Nulls are better left alone because attempting to boost them can easily overload your subwoofer’s amplifier. REW also features a quite accurate Room Sim tool, which can simulate the frequency response of your subwoofers in different parts of your room without the need to physically move the subwoofers or make hundreds of measurements.

End Game: Double Bass Array

If you have a rectangular room with all parallel surfaces (no sloped ceilings) and access to at least two subwoofers, there is a special configuration known as a double bass array (DBA) that can virtually eliminate all room resonances up to a certain frequency.

For two subwoofers:

Center a subwoofer (halfway from the side wall and floor) against the front wall and another subwoofer against the rear wall.

For four subwoofers

Raise the front two subwoofers halfway from the floor, and position them at locations 1/4 of the front wall width from each side. Repeat this setup for the rear wall.

Elevation view of double bass array subwoofer locations for four subwoofers (two against front wall, two against rear wall). For a 4 meter (~13 feet) wide and 2.4 meter (~8 feet) high room.

Now, you need to delay the signal going to the rear subwoofers by a certain amount, which can be calculated by simply dividing the length of the room by the speed of sound (about 0.343 meters per millisecond). For example, in a 5-meter-long room, the delay would be 5 / 0.343 = about 14.5 milliseconds. You will also need to reverse the polarity of the rear subwoofers, which can be achieved by toggling the polarity switch on powered subwoofers or by switching the positive and negative wires for passive subwoofers powered by an external amplifier.

Here’s what happens: The sound wave generated by the front subwoofers is canceled out by the rear subwoofers at the exact moment it reaches the rear wall, due to the delayed rear signal and reversed polarity. It’s as if there is no rear wall, thus eliminating resonances between the front and rear walls. The resonances between the remaining walls aren’t excited either, thanks to the strategic placement of the subwoofers.

The great thing about a DBA is that the bass output across the room will mostly be free of resonances, independent of where you sit. However, you should still avoid sitting too close to the walls.

Finally, keep in mind that delaying the signal to the rear subwoofers may require the use of an external DSP device or an amplifier with built-in signal delay features. It also helps to have the capability to make measurements to fine-tune the setup. For more information, you can refer to the DBA Wikipedia page.

All Software have Bugs

Thu, 13 Oct 2022 00:00:00 +0000

All Software have Bugs

It’s inevitable.
You fix some,
New ones appear.
You fix some more,
Old ones come back from the dead
Unless your tests catch them first.
Do you even have time for tests
When the feature backlog is a mile long,
And time to market is everything.
There is this satisfying feeling though
When you have the scalpel in your hand,
Removing features left and right,
Cutting corners no one would notice,
Or so you hope.
But it must be done,
You can always fix things later.
This is software after all –
Infinitely malleable, infinitely soft.

Favorite Exercise

Thu, 23 Jun 2022 00:00:00 +0000

I don’t like to exercise. I mostly think of it as a distraction, but what bothers me the most is the amount of time spent for exercise could have been dedicated to other, you know, more fun activities.

We all need to exercise regularly to stay in shape and be healthy. I don’t think anyone would disagree with that, but if exercise takes too much time, I know I wouldn’t do it. So, I needed a compromise. I needed to find an exercise that would take minimal time, and yet still be as effective as others that would take a long time to complete. Here is my list of criteria for an ideal exercise:

Should take minimal time.
Should work as many muscles as possible at the same time.
Can be done anywhere with minimal or no equipment (I’m certainly not going to a gym and spend even more time on the way to and from there. I also find it strange to pay for doing physical exercise. Not long ago, if you wanted to exercise, you just chopped wood for your fireplace or worked at a construction site and got paid for it, but I digress.

These criteria can be fortunately met with just one exercise, and that’s pull-ups. Yes, pull-ups. I know they are hard. Many people can’t even do one pull-up. I couldn’t do one when I first started either. But, as soon as you can do one pull-up (or its slightly easier variation: chin-up), the rest should become easier.

What is great about pull-ups is that since they are so hard to do (think of pulling 100% of your body weight), it often takes less than a minute to complete one set, and they work your arms and almost your entire upper body too. Doing pull-ups is probably the easiest way to achieve the coveted V body shape.

Now, I must say that pull-ups are significantly harder for women than men. Women tend to have less muscle mass on their upper bodies. But, they can still do pull-ups. My wife, after seeing me doing pull-ups, tried them herself. She can barely do one pull-up, and yet after about a month of casual daily pull-up sessions, her posture is noticeably more erect now. That’s because not only do pull-ups build muscle, but they are great for stretching too. You literally stretch your entire body while doing pull-ups. In fact, Arnold Schwarzenegger aka the Terminator used to do pull-ups for warming up before doing his regular routine of body building. We are lucky that we could just stop after doing pull-ups and call it a day while he had to go through hours and hours of exercise every day. Well, he clearly enjoyed doing body building unlike many of us. Each to his own, as they say.

To improve quickly in pull ups, there is a trick, and it’s a simple trick too: don’t push yourself to failure. That is, don’t just follow the common advice of trying to do as many repetitions as you can. If you do that, you can brag to your friends that you can do, say, 10 pull-ups, but you can do that many only once or maybe twice a day, and you will always feel exhausted after each session. In other words, it will start to feel like a chore, rather than something you look forward to doing.

I read about that trick in one of Pavel Tsatsouline’s books. He is a well known fitness instructor who also happens to write fun to read books about strength training. What he recommends for any kind of exercise, not just pull-ups, is basically this: Do as many repetitions as you can, but stop a few repetitions short of failure. That’s all there is to it.

I stop doing pull-ups at about 3/4 of the maximum repetitions I can do, and I try to do as many sets as possible spread throughout the day. The pull-up bar is attached to the door frame of my wood workshop in the basement, and the room next to it has my home theater. So, whenever I go down there I try to do one set, and then do some woodworking, watch a movie, or just go back upstairs to do something else, and I would still feel fresh. That way, it doesn’t feel like I’m doing work, I want to do more of it, and my body (or should I say brain) isn’t conditioned against its limits prematurely. When my usual 3/4 max repetitions start to feel too easy, I just add another repetition.

Lately, I decided to work on my abdominal muscles that the all-mighty pull-ups could sadly not reach. The obvious exercise choice would be sit-ups, but they don’t quite meet my 3-item criteria, and a back condition I have makes doing them somewhat uncomfortable. So, I decided to give push-ups a try. Not only do they work the abdominal muscles, but also work a few of the muscles on the upper body pull-ups don’t quite work. Another reason why I chose push-ups is that combined with pull-ups, they are the staple exercises of the militaries around the world. So, all those drill sergeants certainly know what they are doing. Right?

Anyway, after getting my upper body in shape, I thought doing push-ups would be a piece of cake, but it didn’t quite work that way, at least initially. After my first try, during which I apparently pushed myself too hard, I was sore for two days. So, after that I followed my own advice of stopping well short of the maximum repetitions I could do, and things started to improve from there.

With push-ups, you lift about 70% of your body weight, and that seemingly small 30% difference allows your body to do about three to four times as many repetitions as compared to pull-ups. In other words, if you can do 5 pull-ups, you should be able to do 15 to 20 push-ups, but let’s forget those numbers. I think what really matters isn’t the number of repetitions one could do (after all we are talking about doing exercise casually), but keeping the motivation to exercise high. With only a few minutes total to spend on exercise each day, I can certainly manage to keep my motivation high. If you keep doing something every day, even if you are really bad at it at first, it’s inevitable that you will improve over time.

Finally, a word of caution: I’m not an expert at strength training, and I’m just sharing my informed opinions with you. Although it’s unlikely to seriously injure yourself by doing pull-ups or push-ups, it isn’t impossible either. Please use some common sense while you exercise, listen to your body, and if some part of it hurts badly, it’s probably a good idea to rest a day or two (or more) while your body heals. If you have some health condition, it’s wise to talk to your doctor first before following the advice you read on some random web site.

Lodash, a Middle Ground between JavaScript and TypeScript

Sun, 19 Jun 2022 00:00:00 +0000

I still remember a quote from a computer magazine I read when I was 12. In an article comparing the programming languages of the time, a C developer described the Pascal programming language as “having a mother hen hovering over you, watching your every move, making sure you don’t screw up.” I didn’t fully understand it at the time, but he was most likely referring to the strong type checking in Pascal, which could lead to compiler and runtime errors. If you mixed and matched types in a C program, the program would probably continue to run and fail only when something went terribly wrong.

I think our C developer was mostly right in retrospect. Generally speaking, there isn’t an overwhelming need for strict type checking. Coding is hard as it is. Developers spend a significant amount of time fighting with compilers just to get their code run. Dynamically typed programming languages like JavaScript, PHP, and Python are popular for a good reason. It’s a lot easier to get your code running in them. Now, please don’t get me wrong – I think strong type checking is important in avoiding quite a few bugs before they occur, but is the price we pay worth the benefits?

I think there is a happy middle ground between plain JavaScript and its strongly typed variant TypeScript, and that is Lodash, the popular Swiss Army knife JavaScript library. At Faradai we make heavy use of Lodash. The number one reason why we use it is because Lodash eliminates a whole range of type checks. If you pass an undefined variable to a Lodash function, the function will return gracefully, and the execution will continue from the next line without throwing any errors.

Lodash has a large number of handy functions, but probably the most important one is the simplest: _.get(). It allows you to access an arbitrary property of an object no matter how deep it is like so: _.get(table, 'data.0.childNodes.3') and if the property doesn’t exist, no harm is done. This is especially useful in frontend code – displaying a table cell empty isn’t usually the end of the world when the alternative is throwing the infamous ‘undefined’ is not an object error, breaking the whole table and maybe some other things too.

There are times, of course, especially when you are dealing with mission-critical code, you need to make sure that the data you are processing is in the right format. However, in such cases, you need to have a good automated test suite anyway. All the type checking in the world can’t save poorly tested code. It’s the age old validity vs. reliability dilemma. Your code may be 100% valid, but it can still be incorrect.

Another reason we prefer to use Lodash is that Lodash/fp, in my opinion, is the most convenient way to do functional programming in JavaScript. Yes, the documentation for Lodash/fp is terrible, but all you really need to know are:

The first argument to a Lodash function is now passed from “outside” like _.get('property')(myObject) instead of _.get(myObject, 'property').
Use _.flow() liberally to pass one function’s output to the next. Lodash handles the necessary “auto-currying” for you in the background.

It is possible to do a great deal of functional programming in a mixed-paradigm language like JavaScript without knowing a thing about function composition, currying, or point-free style. I think there are quite a few developers out there who would appreciate the elegance of the following code:

var _ = require('lodash/fp');

var users = [
	{firstName: 'Katniss', lastName: 'Everdeen', status: 'active'},
	{firstName: 'Peeta',   lastName: 'Mellark',  status: 'active'},
	{firstName: 'Seneca',  lastName: 'Crane',    status: 'disabled'}
];

function getFullNamesOfActiveUsers(users) {
	return _.flow(
		_.filter(user => _.get('status')(user) === 'active'),
		_.map(user => _.get('firstName')(user) + ' ' + _.get('lastName')(user))
	)(users);
}

console.log(getFullNamesOfActiveUsers(users));

The code is read naturally from left to right. The lodash functions simply transform the passed data without changing it, and finally and perhaps most importantly the code is easy to follow.

I think Lodash/fp has the potential to transform JavaScript (and Node.js) into a dynamically typed functional programming language with the least amount of effort. If you wish to add some strong type checking on top, that’s fine too.

Worth a Read:

Dipping a toe into functional JS with lodash/fp

Writing from a Software Developer's Perspective

Sat, 23 Apr 2022 00:00:00 +0000

I write because I want to share what I know with others. Yet the software developer in me makes writing hard for me. See, in coding the internal structure needs to be consistent. The code we write is like a house of cards. If we make even one tiny mistake, the whole thing may come falling down. So, when I write, I try to be consistent, which is not a bad thing in itself of course, but the desire for consistency makes writing a lot more difficult for me than it should be.

Haruki Murakami once said you should not be too intelligent to write fiction. Well, I only write non-fiction, but strictly speaking, nothing we write is entirely non-fiction. Even in scientific writing –especially the popular science kind– there can be quite a lot that is assumed to be true. The software developer in me gets pedantic at times, and tries really hard to ground every aspect of an article in reality. Most of the time, this attitude means I have to put a lot of extra work into making sure everything is accurate and makes sense, and yet the end result may turn out to be quite boring.

I’ve always wanted to be able to write the way I speak. Apparently there are authors who can write like that, seemingly never lifting a finger from the keyboard so to speak, but there are those like Stephen King who can’t. Stephen King was brave enough to share excerpts from some of his early drafts, and they are pretty bad. But, what makes the end result infinitely better is the intense polishing phase. When I try to write and polish at the same time, which I often fall into the habit of doing, I rarely get anywhere.

The best way to write, at least for me, is to write as quickly as possible as Stephen King suggested, and polish later. Polishing doesn’t just mean editing the text though. It involves adding new viewpoints or removing passages that slows down pacing – especially the pedantic parts I’m so prone to include.

Our Experience with PostgreSQL on ZFS

Sat, 16 Apr 2022 00:00:00 +0000

First a little context: At Faradai we had been using PostgreSQL on the ext4 file system for a number of years until it recently became evident that the constantly growing database size (over 5 TB at the time) would start to cause issues in general performance, cloud costs, and the ability to have reliable backups. We reviewed a number of alternative file systems, and chose ZFS mainly for its following three features:

Instantenous Filesystem Snapshots
This makes creating incremental database backups a piece of cake, something we didn’t have before. It never fails to amaze me how fast and easy it is to revert to a snapshot. This is doubly important if one is managing a write-heavy database that changes every second. We use the snapshot feature to transfer hourly snapshots to a backup server which also doubles as our development database server. More on that later.

Built-in Compression
Most of the data we collect are time-series sensor measurements, and they compress really well. In fact, the compression ratio is over 4.5x in our case! That naturally decreases our disk storage needs by a similar ratio. Another side benefit is that the disk throughput would be higher because data takes less space on disk and would require fewer I/O operations and less bandwidth. CPU usage is increased slightly when compression is enabled, which is fine with us.

Built-in Support for Secondary Caching on SSD Disks
Block storage on the cloud is great until it isn’t. Yes, we are getting virtually infinite storage that can be dynamically allocated and resized, and it tends to be more reliable than individual physical disks, but the network interface to storage is significantly slower. That is the price we pay for flexibility. So, we needed an easy way to utilize the fast physical disks attached to VMs for caching. We don’t care if they fail. We want to use them as a secondary cache, and ZFS makes that really easy.

ZFS has other neat features such as the ability to detect bit rot and its own implementation of software RAID, which aren’t really critical for us, but they are still very nice to have.

How we Do Backups on ZFS

There are a number of open-source tools that are used to automate ZFS backups, but our needs aren’t complex, and we prefer our critical tooling like backup scripts as simple as possible to avoid unpleasant surprises when things go south. So, we went ahead and wrote a short script and added 2 cron entries. Less than 20 lines in total including comments. It can’t probably get simpler than that thanks to ZFS’ amazing incremental snapshot feature that finishes its job in seconds. We take 23 hourly and 14 daily snapshots on the production server, and those snapshots are transferred to the backup server. Our backup server initiates and controls the entire process, and it is essentially a perfect replica of our production server, but is of course up to an hour behind.

We Live with our Backups

Test your backups! How many times have we heard that? Too many to count, but it’s absolutely true. Probably one of the most famous examples of the untested backups went bad is Toy Story 2. The entire movie production could have been lost due to corrupt backups and the infamous ‘rm -rf’ command in the wrong directory. Fortunately, they discovered a few weeks old copy of the film assets on the computer of a pregnant employee working from home, and saved the production. It’s not hard to imagine that others in a similar situation might not be so lucky.

Probably one of the most effective ways to test the backups is to actually use their contents, and use them every day. Here is how we do it:

Every few weeks, we pick a recent daily snaphot on the backup server, and create a ZFS clone from the snapshot to make it writable.
We then make the PostgreSQL instance running on the backup server use the clone’s mount point as its data directory.
Finally, we restart PostgreSQL to have it recognize the new data directory.

The backup server is now doubling as our development database server, and the process of taking hourly snapshots from the production server continues to run in the background.

Technical Bits

Installation

It takes only a few commands to install ZFS and PostgreSQL on a Linux machine (Ubuntu in our case).

sudo apt install zfsutils-linux                                          # Install zfs filesystem packages
sudo fdisk -l                                                            # Take note of disk names for the next step
sudo zpool create -o autoexpand=on db /dev/nvme1n1 /dev/nvme2n1          # Create a striped RAID0 pool from 2 block storage volumes
sudo zpool add db cache /dev/nvme3n1                                     # Add local SSD drive as cache
sudo zfs create db/data -o mountpoint=/var/lib/postgresql                # Create dataset and mount it to use for PostgreSQL
sudo zfs set compression=lz4 db                                          # Enable zfs compression
sudo zfs set atime=off db                                                # Disable atime for improved performance
sudo zfs set recordsize=32k db                                           # Set zfs record size to 32 KB (default: 128 KB)
sudo reboot                                                              # Reboot
sudo zpool import db                                                     # Import zfs pool; has to be done only once after first reboot
wget --quiet -O - \
https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -  # Add PostgreSQL repository key
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt \
$(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'       # Add PostgreSQL repository
sudo apt update                                                          # Update apt packages
sudo apt install postgresql-14 postgresql-client-14                      # Install PostgreSQL 14
sudo sysctl -w vm.swappiness=1                                           # Tell Linux not to use swap unless absolutely necessary
echo 'vm.swappiness=1' | sudo tee -a /etc/sysctl.conf                    # Append 'vm.swappiness=1' to systctl.conf for permanent effect

Careful readers will notice that we have created a striped RAID0 pool out of two disks on a production server, which should be verboten, and I would normally agree with you if these were actual physical disks, but they aren’t. The disks are commissioned from network attached storage, and they are managed by a tier-1 cloud provider. We think they know what they are doing, and they have the necessary redundancy in place. In any case, we have a backup/development server that is ready to switch to production in a moment’s notice and a separate set of backups stored on object storage.

After the installation, the settings in postgresql.conf needs to be optimized to make better use of the number of CPUs and amount of RAM on the machine. PGTune’s recommendations is generally a good starting point.

Finally, we disable full page writes in PostgreSQL, and restart the service to eliminate unnecessary disk writes. By design it is impossible to write partial pages on ZFS.

ALTER SYSTEM set full_page_writes=off;
CHECKPOINT;

Backup Script

Here is our backup script: “remote_db_snapshot” running on our backup server. The script accesses the production server (remote) over SSH, switches PostgreSQL to online backup mode, takes a snapshot, exits backup mode, prunes old snapshots, and finally copies the new snapshot over.

#!/bin/bash

HOST="some-user@prod-db"
NOW=$(date --iso-8601=minutes | sed 's/:/-/g' | cut -c 1-16)
DELETE_OLDEST=$(date -d "$1 ago" --iso-8601=minutes | sed 's/:/-/g' | cut -c 1-16)
LAST_SNAPSHOT=$(sudo zfs list -t snapshot | tail -n1 | cut -d " " -f 1)

# Switch remote PostgreSQL to backup mode, take a ZFS snapshot, and exit from PostgreSQL backup mode
ssh $HOST "sudo -u postgres psql -c \"SELECT pg_start_backup('$NOW', true);\""
ssh $HOST "sudo zfs snapshot db/data@$NOW"
ssh $HOST "sudo -u postgres psql -c 'SELECT pg_stop_backup();'"

# Destroy oldest snapshot on local server, and if successful delete on remote
sudo zfs destroy -v db/data@$DELETE_OLDEST && ssh $HOST "sudo zfs destroy -v db/data@$DELETE_OLDEST"

# Copy newly taken snapshot from remote server
ssh $HOST "sudo zfs send -cRi $LAST_SNAPSHOT db/data@$NOW" | sudo zfs receive -vF db/data

The cron entries:

0 1-23 * * *	. $HOME/.profile; ~/scripts/remote_db_snapshot '1 day'   2>&1 | logger -t remote_db_snapshot_hourly
0    0 * * *	. $HOME/.profile; ~/scripts/remote_db_snapshot '2 weeks' 2>&1 | logger -t remote_db_snapshot_daily

To make it easy to keep track, our snapshots are named with the time they are taken:

Daily snapshots

...
db/data@2022-03-30T00-00
db/data@2022-03-31T00-00
db/data@2022-04-01T00-00
db/data@2022-04-02T00-00
db/data@2022-04-03T00-00
db/data@2022-04-04T00-00
db/data@2022-04-05T00-00
db/data@2022-04-06T00-00

24-hour rolling window snapshots:

db/data@2022-04-06T18-00
db/data@2022-04-06T19-00
db/data@2022-04-06T20-00
db/data@2022-04-06T21-00
db/data@2022-04-06T22-00
db/data@2022-04-06T23-00
db/data@2022-04-07T00-00
...
db/data@2022-04-07T11-00
db/data@2022-04-07T12-00
db/data@2022-04-07T13-00
db/data@2022-04-07T14-00
db/data@2022-04-07T15-00
db/data@2022-04-07T16-00
db/data@2022-04-07T17-00

Conclusion and Disclaimer

Our experience with PostgreSQL on ZFS has been absolutely positive so far, but we certainly don’t know everything there is to know about the duo. We are confident that we have a handle on the important bits though. So, please do take our experience with a grain of salt, and if you think we are doing something terribly wrong or have any recommendations, please let me know, and I will update this article.

Worth a Read:

Shared by:

Choose Your Cloud Dependencies Wisely

Fri, 15 Apr 2022 00:00:00 +0000

Vendor lock-in: An unfortunate condition that forces a person or company to continue using a disliked product or service just because they invested in it so much. Vendor lock-in used to affect mostly large companies using mission critical legacy applications, but as more and more companies move to the cloud, cloud providers started to rediscover the joys of vendor lock-in by offering “value-added” services that simplify the lives of software developers who have better things in life than managing servers and databases.

I think most anyone would agree that virtual machines (VMs) that one could launch and destroy in seconds, and easily resizable block storage attached to such machines are generally good things. We don’t have to buy or rent servers which have more capacity than we need. We also don’t do really low-level machine maintenance tasks like replacing failed hard drives anymore. These days we are not really managing servers; they are already managed for us, but the thing is, given so much competition, there isn’t probably much profit left to cloud providers for offering plain-vanilla VMs. Where is the value-add in that, they would ask.

Put yourself in the shoes of a cloud provider eager to increase their stock price, and start pondering questions like “what are the things that are very hard to change in software development once you are committed to them?” Easy, the programming language itself, certain libraries such as ORMs, …and the databases used. Ah, the epiphany!

Cloud providers can’t really create their own languages or libraries, but they can all offer their own versions of popular databases in managed form. Managing and scaling databases is scary, they say. “You don’t have to worry about all that anymore. Just use us, and focus on your code.” Sounds tempting, doesn’t it? In reality, a managed database offering that is hard to migrate from probably isn’t such a good idea. When in doubt, using a PostgreSQL database on a simple VM can get you a long way, and doesn’t really require much maintenance, if at all. Sure, you need to monitor the resource usage of the VM from time to time, but all cloud providers offer built-in monitoring services, and it’s impossible to get a surprise bill for a simple VM as opposed to managed database servers that can auto scale. The worst you could do with a VM is to overload it. If that happens, it’s probably because your code is inefficient or you forgot to add an index to a column, or maybe everything is fine, but your app is getting popular. If you really are getting popular, that’s a good problem to have: upgrade the VM with more cores and memory, and go back to coding or whatever you were doing.

Come to think of it, although I said cloud providers couldn’t really create their own languages, they really came close by offering “serverless” code running environments. Again, it could be even harder to migrate from those offerings than migrating from managed databases, and surprise bills, once again, start to become a very real possibility. Again, when in doubt, running your code on a simple VM deployed by “git pull” can get you a long, long way.

About Lack of Imagination

Thu, 14 Apr 2022 00:00:00 +0000

“Imagination is more important than knowledge. For knowledge is limited to all we now know and understand, while imagination embraces the entire world, and all there ever will be to know and understand.” - Albert Einstein

This is Aycan Gulez, your host (LinkedIn | GitHub | YouTube | Mastodon). I’m the Izmir Lab Director at Efsora, a software and AI development company that partners with innovative enterprises and scaleups to deliver large, cutting-edge R&D projects. Previously, I served as CTO at Faradai, leading the development of Faradai’s carbon accounting and energy management software platforms, which are utilized by a number of mid- to large-sized companies to monitor and reduce their energy consumption.

Why the name “Lack of Imagination” you might ask. I’ve noticed that the more I think I know about a subject, the more confident I get, and the less likely I become to entertain alternative views. This, in my opinion, leads to a dangerous state of mind: a lack of imagination. So, I chose to name this site “Lack of Imagination” to remember that the world is big and there are so many things I don’t know, no matter how much I think I know.

I’m writing this blog to share my views about subjects on which I consider myself at least somewhat qualified to comment. Hopefully, some of the entries on this site will attract the attention of people who can contribute their own viewpoints.

I would like to thank Alper Gayretoglu, Bilal Ceylan, Edward Hubbard, Emre Yildiz, and Tugce Aker for their feedback and also the people behind the following tools and services powering this web site:


Static site generator:	Hugo with Tranquilpeak theme
Comments:	Coral
Mailing list:	listmonk
Web server:	nginx
Web stats:	GoAccess
Containers:	Docker
Operating system:	Ubuntu Linux
DDOS protection, CDN, SSL:	CloudFlare
Hosting:	DigitalOcean