Jeff Riggle

How deep does the callback go?

Tue, 09 Jun 2026 19:50:47 GMT

Building a scheduler Despite being single-threaded, JavaScript has concurrency as a...

Understanding Event Loops Through File I/O

Fri, 29 May 2026 00:00:00 GMT

In the last blog series, I referenced a presentation by Ryan Dahl on NodeJS and wanted to explore the concepts in that talk more. It can be easy to dismiss JavaScript as a low-performance language based on the observation that it is a single-threaded environment. However, this talk directly opposes that mindset. Instead, it suggests that the real problem is removing I/O from the hot path. Data locality is often the place where performance tradeoffs live. Generally speaking, even the slowest arithmetic operations, like division, cosine, and sine, are significantly faster than reading from RAM. Reading from disk or the network is orders of magnitude slower than reading from RAM. The key argument is that if you keep a single loop completely saturated while pushing blocking IO out of the hot path, you can get better performance at a reduced memory cost. This is because threading comes at a performance cost. For starters, each thread comes with its own memory. At a minimum, every thread has its own stack, which easily consumes 1 megabyte or more of memory. Also, once you have shared memory between threads, you have mutexes that can create bigger problems than single-threaded code.

Thinking about this a bit more, I decided it might be fun to try building out a simple event loop on a contained problem to better understand this space.

Doing some research

Before I got too into this, I wanted to get a better understanding of what was under the hood of NodeJS. Going into this, I knew that NodeJS was largely a C++ project with V8 bindings. However, I was a bit less familiar with the machinery it used to handle its non-blocking IO. After some research, I found that modern NodeJS uses libuv to handle this. This library handles everything from file access and child processes to sockets and DNS resolution. This struck me as a little bit odd. In the presentation I referenced, Ryan mentioned using libev to handle eventing. In an attempt to figure out what happened, I started digging. Looked into that library I found it was related to yet another eventing library, libevent. Each library added its own unique features to improve performance in event loops. As it turns out, Ryan Dahl even wrote about all of these libraries and why they decided to create libuv.

In many of these libraries, the key difference revolves around the OS support they provide and the APIs they build on. Many point to the difference between select, poll, and epoll. Each gives different performance benefits along the way. Since many of these optimizations operate at the boundary between userland and the kernel, it can be quite difficult to create a multi-platform, performant, and coherent API.

Setting up an experiment

While most of the literature on these eventing libraries had focused on sockets, I was a bit more preoccupied with file access. I was still fixated on the original code that brought this all up, forcing a synchronous call for file access in NodeJS. Unfortunately, this was the one case where not all kernels provided good non-blocking IO APIs. Because of this, utilizing things like epoll didn't fit naturally into this problem. However, there was still a way to move forward with building an event loop and taking blocking IO off of the hot path.

Building up the scenario

Before I could do anything, I had to set something up to test file access. I decided it would be easy to test reading JavaScript files and doing some very basic parsing. Each file would have a collection of imports, a handful of constants, and some functions. The processing code would count the total constants and functions found across all files and queue up processing the imports.

Doing some basic research, I found that a reasonable range of files for a project would be around 10 files for a smaller project and closer to 1000 for a larger project. Since I didn't want to manually create that many files, I decided the best way to bootstrap this experiment was to write a NodeJS program to generate these projects. I wanted imports, but I also didn't want any circular references. I assumed circular references could lead to complications during processing. I pulled out Tarjan's algorithm to make sure no circular references got generated. Also, to make it easier to parse, I parsed the graph at the very end and I referenced all roots in an index.js.

Also, to entertain myself, I created a list of predefined words to combine for the naming of files and variables. This led to funny file names like help-overwhelming-random.js, malware-darwin.js, and sqs-agentic-wheel-tokenizer.js.

Figuring out what to test

Armed with the ability to create JavaScript projects, I now needed to decide what to test. Part of my interest was comparing the performance of different cases across NodeJS and C. This resulted in the following applications being created.

A NodeJS application that blocks on IO by using the sync APIs.
A NodeJS application that used the async libraries correctly.
A C application that blocked on IO.
A C application that used an event loop.

Also, since projects could be of different sizes, I decided each program would be tested against a handful of different projects.

A tiny project with 10 files.
A small project with 50 files.
A medium project with 100 files.
A large project with 500 files.
An extra-large project with 1500 files.

With all of these, I would gather timings and compare how they did.

Benchmarking file access is tricky

Before I go into the specifics of each example, I want to preface with a little benchmarking tidbit. While I was conducting this experiment, I realized that I was getting some crazy outliers in my initial testing. Those who are closer to the OS and file-based performance will know where I am going with this. OS's know that file access is slow and attempt to compensate for this. In some cases, quite aggressively. In my case, I would see a massive boost on the second run of a program. For these tests, I was running on a MacBook using an M3 chip. The best way I could find to get the lowest variance in these tests was to sudo purge between each run.

I was a bit concerned about how specialized the M series architecture was, so I conducted another round of tests on a Linux-based machine. Linux also had similar caching that needed to be cleared between each run.

I suggest taking the exact numbers with a grain of salt, and instead focus more on the trend than the result.

Running the test

Each of these programs had required a bit of different effort. Admittedly, I hadn't written much C in the past 15 years, so it took me a while to get back into the swing of things.

Blocking NodeJS example

This was the easiest one to write. I didn't want to do a full JavaScript parse for any implementation. The problem set was very well contained, so I just needed to write something that worked. I abused that to its fullest. So instead of pulling out a proper parser, building an AST, and traversing the tree, I did a simple line-by-line check. The original form of this used a regular expression to get the import file path, but I ended up scrapping it. I didn't like the Regex differences between JavaScript and C, and didn't want this to be a comparison of Regex processing engines.

This program can be found here. The most important part is that I used a while loop to do the processing.

while (filesToProcess.length > 0) {
  let file = filesToProcess.pop();
  // Do processing
}

This took surprisingly little time to write, and even at the largest scale, it was a fairly quick program. The slowest execution time for 1500 files was ~273ms on Mac and ~363 on Linux.

Async Node example

Moving on, I implemented the callback-compatible version of the same program to see the difference with just using the event loop properly. The biggest change I had to make was around that while loop. Now, instead of blocking each file, I would block in batches as I found them.

async function main() {
  while (filesToProcess.length > 0) {
    let fileRequests = [];
    for (let file of filesToProcess) {
      fileRequests.push(processFile(file));
    }

    await Promise.all(fileRequests);
  }
}

This program can be found here.

The resulting difference in performance was notable. In all but the tiny dataset, the program ran nearly twice as fast. The slowest observed time was ~124ms on Mac and ~278ms on Linux.

Basic C example

This is where implementation speed started to slow down a bit. Since it had been so long since I had written C, it took me way too long to get back into it. Eventually, I produced a working program. Again, I used a similar loop as seen before.

  while (FILES_TO_PROCESS[FILE_TO_PROCESS_INDEX] != NULL)
  {
    processFile(FILES_TO_PROCESS[FILE_TO_PROCESS_INDEX]);
    FILE_TO_PROCESS_INDEX++;
  }

The program can be found here.

Unfortunately, due to inefficient array scanning used in this program, execution slows down at larger scales. This had performance as bad as ~350ms on Mac and ~363ms on Linux.

Eventing C example

This one required a bit of work. Originally, I just tried putting some threads on the problem without building a legible abstraction. The main execution was done on a single thread, but the way file memory was managed created terrible performance due to the inefficiencies of the threading and mutexes required. In the end, I scrapped that implemenation. Eventually, I got to a much more stable version version that had a dynamically sized thread pool and a proper event loop. Now, as I mentioned before, the file APIs provided are a bit awkward, so it's not a perfect event loop. In this case, the implementation looks more like a managed thread pool for reading files and an event loop that dispatches work to the pool. All of the processing work still resides directly in a single thread in a loop.

The main event loop ended up being similar to this.

void run(EventLoop* loop)
{
  while (loop->completed == 0)
  {
    // grow pool to maximum if we are not at the maximum
    if (threadPool->size < MAX_POOL_SIZE)
    {
        // Calculate idle time.

        if (idleTime > GROW_THRESHOLD)
        {
            GrowPool(threadPool, 1);
        }
    }
    
    // dispatch events
    if (loop->lastDispatchedEvent < loop->nextEvent)
    {
        // Find and dispatch pending read file events to idle threads.
    }

    for (int i = 0; i < threadPool->size; i++)
    {
        // Determine if the thread has a valid result
        if (isValid)
        {
            // Invoke file handler on a single main thread
            loop->handler(loop, threadPool->readers[i]->buffer, threadPool->readers[i]->fileSize);
            
            // Reset thread state
        }
    }
  }
}

In the end, this version produced the fastest overall results, with the slowest result being ~113ms on Mac and ~195ms on Linux. Similar to the other C example, since I didn't use an efficient array search algorithm, this could likely be faster.

Considering the results

With all the runs completed, it was time to look at the results and see how things turned out. First, the original hypothesis of NodeJS tells us that using async APIs instead of sync APIs should produce a meaningful difference.

In this chart, we can see the difference between the sync node implementation (red) and the async node implementation (blue). At the smallest scale, the difference is negligible, but at larger scales, the difference is notable. The real benefit of the async pattern is distributing the wait time for file IO. This means the performance benefit scales with the number of files to be read, and we can see that clearly in this example.

Now, if we compare the same in the C implementations, we see a similar pattern. Even though we are using a single thread for all the processing, we are still getting the same performance benefits we see in Node.

Conventional wisdom states that low-level languages are faster, so let's take a look at how the C implementation fairs against the Node implementation.

This shows that the C implementation is consistently faster. However, these differences are not nearly as significant as one might expect. If we consider this workload more deeply, a majority of the required time complexity comes from reading from the file system. This problem is not directly owned by either language. The time to load a file is largely dictated by the kernel and underlying hardware. Now, where things can get interesting and where some of the benefits lie is that C gives you direct control over memory. In the C implementation, I was able to get just a bit more performance by being clever with my memory allocations.

Lastly, let's consider the cost of getting things wrong. Remember how I mentioned the original C abstraction I created wasn't working well? If we compare this to the Node example, we can see that a poor implementation in C is quite detrimental.

In this case, the results clearly show that the misuse of allocations and creation of additional thread contention on shared data will result in performance problems. Just because you have access to low-level primitives doesn't mean you are faster by default. If you are not careful, you have more ways to slow yourself down.

If you would like to look at the data and draw your own conclusions, you can find the raw data here.

What I learned from this experiment

In this project, I got to have a lot of fun learning about the lower level of event loops. I also got to see firsthand how building something in an event loop correctly can produce much better results. While writing C again was a bit of a learning curve, I was reminded just how much fun it is for me to think about low-level details like memory allocation and pointers. However, it was also a reminder that just because something is written in a low-level language doesn't mean it will be fast. It is all too easy to write a low-level program that runs way slower than a high-level language because you didn't think about the low-level primitives correctly. In the next blog, we will see that in action all over again.

Farm Scene

Thu, 07 May 2026 00:00:00 GMT

View animation here

More details

Description Basic study of motion and procedural generation in a farm setting

Medium: 2D Canvas

Topics Researched

Basic motion (acceleration, deceleration)
Basic predictive collision detection
Procedural generation
Time cycles based on radial movement

First excursion into Electron

Sat, 18 Apr 2026 00:00:00 GMT

Why use Electron anyway?

After a bit of a meandering journey through a silly little Discord bot, we now find ourselves evaluating the Electron application. My first experience with Electron was using the Atom editor. At the time, Atom was my daily driver. It was faster than Visual Studio, more feature-rich than Notepad++, and it was free, unlike Sublime Text. I was intrigued by the idea of combining Node and Chrome to develop desktop applications. This sounded like a much better experience than I had previously using Swing, JavaFX, WinForms, WPF, or Qt.

What made this a uniquely good opportunity to use Electron was that I wanted to manage another application running on my local machine. Since Node has access to the system and can spawn child processes, this was my chance to finally give the tool a go.

Diving back in

In my first attempt to review the code, I found myself fixated on issues that amounted to preferences about code style. Upon reflection, I realized that many of these changes would amount to nearly identical bytecode. For example, many of them were new features, such as optional optional chaining or destructuring.

As I moved past the superficial critiques, I noticed a few things that seemed non-standard. The first was that I had no unit tests in any of the code in my Electron application or rendered application code. The second was that I was using plain JavaScript, no TypeScript anywhere to be seen. At first, I saw this as an issue, but upon reflection, I think this was wise.

Unit tests are wonderful when you are building a production product. However, when you are just tinkering and learning, it only gets in your way. Making assertions on fundamentally flawed prototypes is a perfect waste of time.

When the domain is not well understood, as is common in learning and prototyping, a type system provides little benefit at a real cost. In these cases, types haven’t had time to solidify, and the usage of any, unknown, and Function are likely to come up. It’s hard to defend using Typescript in these cases. TypeScript is not a first-class citizen. If you give raw TypeScript to the browser or Node, it will fail. Leveraging TypeScript means adding a build step to compile your TypeScript. This adds complexity and additional time, both in compile time and setup.

Getting started with Electron

Now that I have had a good rant about the general ecosystem, let’s focus more on this Electron application. The first choice I had to make for the electron application was how to render the UI. Most examples will have you open a new BrowserWindow to show content. Then you will display your application using standard web technologies. Generally, the hardest choice is managing these static assets. Generally, I have seen two patterns. The first involves building your static assets and running the browser to load those local files. The other approach involves using a web server to host the static assets. Then the browser window can leverage those via a URL.

In this case, I opted for the web server hosting the static assets. This allowed me to create two different experiences. The first experience was in-browser. If you viewed the website, you would be able to create a configuration file and download it. The other experience was loading the website from the Electron application. This experience would layer on the option to start and manage the Robit server on your local machine.

Enter IPC

To support these kinds of features, you need to have an interprocess communication channel between the browser and the Node process. At the time, the only way to do this was by using ipcMain. This gave you an event emitter to handle messages between the browser and the Node process.

This worked well, but it didn’t fit my mental model. I wanted to be able to express two modes of communication. A client should be able to send a request and await its response. It should also be able to subscribe to some event and get notifications over time. Since I couldn’t find an easy way to do that with the standard feature set, I built ipc-bridge to support my use cases. This abstraction allowed me to use my mental model across the IPC provided by Electron.

let {client} = require('@jeffriggle/ipc-bridge-client');

// Sending single request to Node runtime.
client.sendMessage('doSomething', 'someData').then(() => {
    // Logic for when the request passes
}).catch((err) => {
    // Logic for when the request failed
});

// Getting incremental updates from Node runtime
client.subscribeEvent('customizablemessage', () => {
    // Logic for when a message comes from the main process
});

Adding in the dual experience

As I mentioned before, I wanted to create multiple experiences on the website using the same static assets. This was something I had built into the ipc-bridge. This library exposes a property that lets the frontend know if the Electron application was hooked up.

const handleElectronState = (available) => console.log('Is electron available? ', available ? 'Yes' : 'No');
handleElectronState(client.available);
client.on(client.availableChanged, handleElectronState);

By having the available state and a handler, I was able to dynamically load additional routes into my application.

Building the front-end

Since Electron uses Chromium as its rendering layer, you have all the same choices you would have for building any other website. The first is choosing whether to use a framework and which framework you would want to use. In this case, I decided to use React. Before this project, I had rarely used React. I was still skeptical of the memory required for the virtual DOM. However, what I did find compelling was the mental model of a single render function. To make this as easy as possible, I used the react-start-app that existed at the time. This opted into a default test framework, build process, and even a service worker.

Class versus function

When I started this project, class-based React components were the only option. As a result, I constantly ran into problems with callbacks. I kept running into the common problem of forgetting to bind callback functions. The way I dealt with this problem was problematic from a memory use perspective.

class MyComponent {
    constructor(props) {
        super(props);

        this.state = {
            stateString: 'sup'
        };
    }

    doSomething() {
        console.log(this.state.stateString);
    }

    render() {
        return <button onClick={this.doSomething.bind(this)}></button>
    }
}

This would create a new bound instance of doSomething on every render call. Some alternatives I have seen cache the bind or utilize arrow functions.

class MyComponent {
    constructor(props) {
        super(props);

        this.state = {
            stateString: 'sup'
        };

        this.doSomethingBound = this.doSomething.bind(this);
    }

    doSomething() {
        console.log(this.state.stateString);
    }

    render() {
        return <button onClick={this.doSomethingBound}></button>
    }
}

class MyComponent {
    constructor(props) {
        super(props);

        this.state = {
            stateString: 'sup'
        };
    }

    doSomething = () => {
        console.log(this.state.stateString);
    }

    render() {
        return <button onClick={this.doSomething}></button>
    }
}

Functional components handle this problem much better by completely avoiding this class of problem, as there is no this to contend with.

const MyComponent = () => {
    const [stateString, _] = useState('sup');

    const doSomething = useCallback(() => {
        console.log(stateString);
    }, []);

    return <button onClick={doSomething}></button>
}

Every time I come back to this project, I have to fight the urge to update all components to be functional components. While it would feel better, it is not strictly required.

A missed opportunity

Most of the frontend code in this project was building a dynamic configuration file using components. The way I approached this problem was interesting in hindsight. I created a configManager . This was not a class but a closure that housed a global configuration object. As you added actions, you would mutate the global actions array. A similar thing would happen for updating an action. Every time you created an action component, it would pull the related action metadata from the global state and mutate it in place.

// relevant bits of configManager.js
let config = {
    greeting: "Robit online. Boop Boop",
    audioSources: [],
    access: {
        default: [],
        users: [],
        deniedMessage: "You do not have the rights to perform this action"
    },
    deferredactions: [],
    actions: []
};

export const addAction = (action) => {
    config.actions.push(action);
}

export const getActionById = (id) => {
    let retVal;

    config.actions.forEach(v => {
        if (!retVal && v.id === id) {
            retVal = v;
        }
    })

    return retVal;
}

// relevant bits of Action component.
class Action extends Component {
  constructor(props) {
    super(props);

    this.action = getActionById(params.id);
  }

  // event handler to mutate state
  typeChanged(event) {
      let newType = event.target.value;

      this.action.type = newType;
      this.setState({
          selectedType: newType
      });
  }
}

There are a couple of issues I have with this. I didn’t store the action in the state. Instead, it was a member of the class. However, the more egregious offender was the implicit changes. This built up its configuration by modifying the same underlying action object as was in the global. A much more modern and acceptable solution to this same problem would have been to use Redux. By using Redux, I would have the same global state I wanted, but with a couple of benefits. The first would be that I wouldn’t have implicit changes. The Redux pattern results in a global immutable state being changed over time by deterministic actions. The second benefit would be state consistency in the component. In this example, I am managing the state of the action outside the props or state. Going with a Redux pattern, this would not have been the case.

Finally, a better way

As I was reviewing the code, I happened upon this pattern I am all too familiar with seeing.

const save = (data, type, fileName) => {
    let el = document.createElement('a');

    el.setAttribute('href', `data:${type};charset=utf-8,${encodeURIComponent(data)}`);
    el.setAttribute('download', fileName);

    document.body.appendChild(el);
    el.click();
    document.body.removeChild(el);
}

Convinced that there had to be a better way, I set out to survey the landscape. Much to my excitement, a year ago, that better way is finally here.

Getting to the Node runtime

Up until now, almost all of the logic I covered has been for the client assets. These would run fine with or without the desktop part. However, I think it is worthwhile to evaluate some of the choices in the Node runtime as well. There is far less of this code, so there is a bit less to uncover here.

Detecting environment

Debugging the application was something I needed early on. I wanted the full application running on my local machine. I also wanted access to the devtools in the Chromium view. I was able to control this behavior by setting a NODE_ENV variable. When the variable was production, I would be able to do both of these behaviors.

const isDev = process.env.NODE_ENV !== 'production';
const appUrl = isDev ? 'http://localhost:3000/robit' : 'http://ilusr.com/robit';

// opening the browser window
if (isDev) {
    window.webContents.openDevTools();
}

window.loadURL(appUrl);

An excessive dependency

A key flow involved starting the web server locally. To do this, I would download the compiled Robit “application”, more like a JavaScript file, from GitHub. That’s right, I checked in the compiled version as well as the source, blasphemy. To download this single asset, I added octokit as a dependency. However, I could have made an HTTP GET and prevented including the extra dependency. The effort to use the HTTP library would have been low. To avoid this low effort, I took on the risk of including a large dependency and its current 222 dependencies. That wildly expanded the amount of code I had consumed. This, in turn, expanded the surface area for security issues.

An overly aggressive timeout

Coming back to this, the Docker startup logic caught my attention. If the application determined you had Docker, it would allow you to start Robit as a Docker container. To detect Docker, a child process would be spawned.

const dockerEnabled = () => {
    return new Promise((resolve, reject) => {
        exec('docker --version', (err, stdout, stderr) => {
            resolve(!err && !stderr);
        });
    });
}

Once eligibility was determined, you could choose to start this application in Docker. This was done by pulling the Docker container, starting it, and waiting for a health check to succeed. The problem was that this was a very aggressive health check. If you couldn’t start the application in 5 seconds, it would abandon the attempt.

function waitUp(timeout) {
    const startTime = Date.now();
    return new Promise((resolve, reject) => {
        pollingInterval = setInterval(() => {
            request.get('http://localhost:8080/robit/state', (err, response, body) => {
                let json = JSON.parse(body);
                
                if (json.state === 'Stopped') {
                    clearInterval(pollingInterval);
                    resolve();
                } else if (timeout < Date.now() - startTime) {
                    clearInterval(pollingInterval);
                    reject('timed out waiting for docker server');
                }
            })
        }, 500);
    });
}

function startRunningDockerServer(config) {
    return new Promise((resolve, reject) => {
        exec('docker pull jeffriggle/robit && docker run -p 8080:8080 -d jeffriggle/robit', (err, stdout, stderr) => {
            if (err) {
                reject(err)
                return;
            }
    
            waitUp(5000).then(() => {
                request.post('http://localhost:8080/robit/start', {
                    json: config
                }, (err, response , body) => {
                    if (err) {
                        console.log(`Failed to set robit config. error ${err}, response: ${JSON.stringify(response)}. body: ${JSON.stringify(body)}`);
                        reject(err);
                        return;
                    }

                    resolve();
                });
            }).catch(err => {
                reject(err);
            });
        });
    });
}

I cannot think of many Docker containers I have pulled from the internet, started, and interacted with in under 5 seconds.

Composition of services

This electron application was a multi-process architecture that required consideration at multiple levels. This had a web server hosting a frontend that interfaced with a desktop backend, which in turn interacted with a web server. This was a neat little problem all rolled up into one silly little application. What was especially interesting in this project was how many of the processes had been built to work in isolation. Despite it being the core focus area, you could run this entire application without ever using the electron part. While this sounds like it would have added a lot of extra scope, it was just a few lines of JavaScript to hide some visual elements controlled by a single event from the ipc-bridge abstraction I made. In this case, a small decision on how to load static assets made a world of difference in the final feature set. In retrospect, this was one of the more impactful decisions in the project, not because of its complexity, but because of how it shaped the way the system could be composed.

Building a Discord Bot

Thu, 09 Apr 2026 00:00:00 GMT

With the high-level overview of Robit out of the way, it’s time to cover how the server was built. As I mentioned in the last blog, the main goal of this project was to learn Electron, so working on the server was not the primary focus. What I wanted the bot to do was handle messages from Discord channels and perform a set of actions based on the content. To create the path of least resistance, the server was written in Node.js. By this point in my career, I was completely preoccupied with JavaScript. Setting up a proper C# or Java server simply required more effort than using Node.

Choosing a package

To interface with Discord, I did a quick search and found discord.js. As is typical, I spent the bare minimum time to find the popular package. I did not consider any alternatives and learned the bare minimum about the package. The result: I built a system that ignored most of what the package I was pulling in provided.

The way I used the package involved subscribing to message events on all channels for the server in question. Then Robit would parse each message. If the message started with !robit the server would proceed to process the message content. Much of what I did was supported by default in Discord.js with commands. A little more time spent researching, and I could have narrowed the surface area of my backend even more.

In retrospect, this was a failure on my part to understand what I had chosen to interface with.

Running the server

Once again, the runtime of this project changed over time. The entire configuration of this application was determined by a single JSON file. This configuration file had all the permissions, actions, etc defined inside of it. In the beginning, this would only run on a local machine. It assumed the program would be run using Node with the configuration file as an argument.

> node main.js myconfig.json

In retrospect, the argument processing is a bit humorous to me.

let configFile;

process.argv.forEach((val, i, arr) => {
    if (val.endsWith('json')) {
        configFile = val;
    }
});

As you could see, this will enumerate all arguments and then take the last JSON file. If you provided multiple JSON files, it would just read the last one. This may or may not be the desired case. Now there are many ways I could have handled this better. I could have made this an option instead of a positional argument. For this, I could have gone with an arguments parser like Yargs. However, positional arguments are not the worst thing ever. What bothers me most now is just how much I felt compelled to use forEach. There are many other ways to get the same effect with more precision and efficiency.

// So you just have to use functional programming.
const configFile = process.argv.findLast(val => val.endsWith('json'))

// The forgotten for loop.
let configFile;

for (let i = process.argv.length - 1; i > 0; i--) {
    const val = process.argv[i];
    if (val.endsWith('json')) {
        configFile = val;
        break;
    }
}

// Why not just apply a convention?
// Maybe a forced file name
const configFile = './robit-config.json';

// Possibly a proper positional variable
const configFile = process.argv[process.argv.length - 1];

Eventually, I decided to expand the scope of this. Docker is cool, so everything, even toy projects, should run in Docker, right? Instead, this added deployment complexity without having a distribution problem to justify it.

Containerization

With containers comes an interesting problem. If I build an artifact and give it to you, I have created an issue. How can I get your specific config into my built container? A naive solution would be to provide the Dockerfile. Then you could define your configuration, provide it to the image, and build your own Docker container. However, if you are going to do that, why use Docker at all?

There are many ways I could have solved this problem: use bind mounts, pass the configuration JSON as an insanely long argument, require the configuration to be hosted so CURL could be used to get the file, the list goes on. Instead, my application was extended to have an HTTP API using Express. This API provided a way to initialize and update the configuration.

Building the API

The primary goal of the API was to allow a user to create a configuration file and send it to a running Robit server to update the configuration. This also added the ability to start and stop the server. However, this also added another seemingly weird API, log collection.

While not the topic of this particular blog post, the reason for this was rooted in the Electron application. The Electron application would interface with this API to manage the server. This included a view of the log messages from the running process. I could have piped the stdout from the container back to the electron application using the child process. What I did instead was pull the logs via an HTTP request. The result was the loss of access to the console API for my logging. There are plenty of libraries that I could have used, like Winston, but instead, I just built my own thing.

let watching = false;
let batchedLogs = [];

const setWatching = (watch) => {
    watching = watch;
}

const log = (message) => {
    if (watching) {
        batchedLogs.push(message);
    }

    console.log(message);
}

const flush = () => {
    let retVal = batchedLogs;
    batchedLogs = [];

    return retVal;
}

module.exports = {
    setWatching,
    log,
    flush
}

This solution is missing all sorts of features. Differing log levels and log formatting are just not part of the deal here. There are many options I could have evaluated to make my life easier, including using a third party or properly leveraging stdout. This was another case of over-engineering instead of using the ecosystem I had already bought into.

Security strikes again

Once again, I see that security was not my primary consideration. This surfaced in two different ways. The first and more unfortunate one pertained to command permissions. With these commands, I built a simple permission system. The idea was that there might be some commands that you didn't want everyone on the server to be able to invoke. The solution I created for permissions was a key-value pair. The key was the user's name, and the value was an array of all the commands the user had access to.

{
    "permissions": {
        "steve": ["help", "currentSong"],
        "sally": ["help", "coolMessage"],
        "admin": ["*"]
    }
}

If you haven't figured it out yet, the big problem is that the solution keyed off the display name. The funny thing about display names in Discord is that you can change them. So, if, for instance, Sally changed their Discord name to Admin, they would get permission for all actions.

The other area where security issues surfaced was on the web server. The interaction with the server when using Docker was via an HTTP API. This API had no permissioning. That choice "might" be considered safe if you are running on your local machine without the port exposed to the outside world. However, it is certainly poor form and not secure. To make matters worse, regardless of whether you ran in Docker, the HTTP server was always used. This was despite the fact that it wasn't needed. This reminds me of a Zoom security issue in which a local web server was the culprit. If you don't need the web server, maybe you shouldn't run it.

Get that bundler out of here

Another regret could be summed up with webpack.config.js. I don't have a great memory of this decision. Either I thought it would be efficient to bundle my Node application, or it was bundled to make it easier to download from GitHub. Whatever the reason, I ended up using webpack to bundle my Node.js application. As a result, strange tools like Babel ended up being a part of my system.

Every time I upgrade a project, the biggest pain point is around the build tooling. Sadly, in this case, I inflicted the pain on myself when it was not required. This introduced build complexity without a clear need for doing so.

The features I forgot

Looking through the code, I was reminded of some interesting features I had not mentioned before.

Harassment as a feature

I am not proud of this, and I don't think I realized the issue I was creating. However, I created what I can only describe as the perfect harassment feature. You could configure an action to repeat any message you wanted to another channel. This could allow a user to send a message directly to Robit, and it would happily repeat that message without a record of the originating user. I even created the action on my local server for a while and called it "whisperbroadcast". In practice, the following message, "!robit whisperbroadcast Lenny's mother was a hamster and his father smelt of elderberries," sent directly to the bot, would have broadcast to a larger channel to harass poor Lenny.

Why shouldn't Discord be a music player?

As it turns out, I spend an extraordinary amount of time turning Discord into a music player. I created a set of actions that would allow Robit to read the file system, gather all the files, and then provide commands for you to play those in a voice channel. If you configured that correctly in the JSON, you could do something like "!robit playMusic" which would start a voice channel to play music in that channel. You could then control the music with commands like "next", "shuffle", and "stop".

Looking at this code, I realized I committed the cardinal sin of Node.js programming.

function getMusicFilesFromDirectory(dir) {
    let files = fs.readdirSync(dir);

    files.forEach(file => {
        let path = `${dir}/${file}`;
        if (fs.statSync(path).isDirectory()) {
            walkDir(path);
            return;
        }

        if (!validAudioFile.test(file)) {
            return;
        }

        addMusicFile(file, path);
    });
};

That's right, I used a "sync" API from the fs module. The initial thesis of Node.js was that even though JavaScript is single-threaded, it is not the thread that makes things faster. Instead, proper handling of IO is the primary concern. This talk does a far better job explaining this than I could ever do. Since this happened only once, it is fine, but a solution more in the spirit of Node would have been the following.

async function getMusicFilesFromDirectory(dir) {
    let files = await fs.readdir(dir);

    for (let file of files) {
        const path = `${dir}/${file}`;
        const fStat = await fs.stat(path);

        if (fStat.isDirectory()) {
            await getMusicFilesFromDirectory(path);
            continue;
        }

        if (validAudioFile.test(file)) {
            addMusicFile(file, path);
        }
    }
}

JavaScript is a weird language

This project reminds me that JavaScript is a melting pot. In this project alone, I see an interesting mix of functional and object-oriented programming paradigms. The HTTP handling code and Discord message processing code are written in a functional way. However, the composition of actions, their metadata, and associated actions was done in an object-oriented manner. These actions even had a base class that handled the sending of messages back to channels. For these actions, I could just as easily have used a series of functions without a class, but instead I decided to mix paradigms. I assume this difference is because libraries like Express adopt a functional paradigm, but I was thinking in an object-oriented fashion. The parts closer to the library followed a more functional paradigm, but I was more comfortable with an object-oriented paradigm. As a result, core business logic would use a more object-oriented style.

Rounding it out

I consistently introduced tools and abstractions before I had a concrete need for them. Most of the complexity in this system came from decisions like that, not from the problem itself. These problems appear to be all too common in these projects I have taken on. Also, chasing fads caused me issues later on. Having to support the choice of yarn and Webpack created continued friction over time. As we close out this chapter of Robit's development, we prepare for a dive into the client that powered this server. This will focus on learnings specific to Electron and React.

What is a robit?

Sun, 29 Mar 2026 00:00:00 GMT

Building for someone else

Continuing the thread of past projects in creation order, we arrive at robit. Robit was never anything I took too seriously, and yet somehow it ended up being another multi-repo application that was barely used. This started as some projects do: someone, in passing, was unhappy with Discord's feature set. The problem they had was that they wanted to set up scheduled daily messages. I did no research at the time to know if that feature is part of the application. However, given the size of the community, I would be surprised if this wasn't a standard feature by now.

So then why build it?

There was nothing to gain from this work. I was not and still am not a daily Discord user. While I provided steps on how to run this application to the person who started it, I am unaware if it was ever used. I certainly didn't ask for any payment for it.

So what did I stand to gain from this? I was interested in learning how to build with Electron, and this felt like a perfect excuse. Since the core focus was on learning, I decided to take the path of least resistance on everything else.

Setting up the project

By this point, I was quite proficient in JavaScript, so I decided to write all the code in JavaScript. Creating the server in Java or C# would have been possible, but I found that it was way faster to bootstrap and prototype with JavaScript. Given those reasons, I decided to use a Node.js server.

Below is the architecture I created for this project.

On the server machine, which in this case was a random computer I owned, there was the Electron application. That communicated with my personal domain to access the static assets. Now, depending on whether you viewed the web page in the browser or on the Electron application, you would get a different experience.

If you viewed the site in a browser, you would see the UI, which would build a config JSON file that you could provide to the server via the command line.

However, if you used the Electron application, you would get that experience plus the ability to start the Robit server directly, either by spawning a node process or by starting a Docker container.

The typical scope expansion

Once again, the scope expanded far beyond what I needed. The original goal was just to learn Electron and build a Discord bot that could send scheduled messages. By the time I was done, I had gone overboard on features again. In this case, I had built a bot that supported three types of communication, eight different configurable actions, and an access control system to top it off.

Of course, knowing me, the scope creep was not just in features. This project resurfaced the problem of having too many run options. While it was more constrained than in the past, having the option to run in a local node process or in Docker was more than was required.

Getting back into this project

Before getting back into this project, I wasn't sure how I would feel about it. As I have mentioned before, I like to run these applications again before writing about them. There are a couple of things that struck me with this one in particular. First, I am not a Discord user, so trying to remember how the application worked was a bit of a challenge. This featured issues like: "What email did I register my account under again?" "How do I create a private server again?", and who can forget "How has the bot permission model changed over the last 6 years?"

Once all those questions had been answered, I had to get the application working again. All repositories in question hadn't been updated since Node 10 was LTS. Six years is also an eternity in JavaScript. The tooling landscape had, of course, changed. This began the drudgery of upgrading every Node version to be compatible with LTS, fixing all the project dependencies, and dropping yarn.

Not everything needs to be maintained

As I reflect on this project as a whole, it's clear I learned a fair amount about Electron and Discord. However, upon returning to this project, what was once interesting had become a chore. Because I am not a frequent Discord user, it is hard for me to form a strong attachment to this project. Having projects to learn a new tool or framework is incredibly valuable, but maintaining those projects after they have provided the desired learning is a recipe for discontentment. Sometimes, keeping a project as a static asset locked in time is best. Without this blog series, I would agree that treating Robit more like an ancient artifact to be observed, not touched, would have been for the best.

Now that I have endured some pain and dissatisfaction, the real fun can begin. What I have been enjoying with these blogs is seeing what I can learn from my past projects. Now that I have done the drudgery, I can get on to the fun. In the next blog, I will start exploring the build-out of the Node.js server.

JavaScript’s Promise Rejected Its Namesake

Tue, 17 Mar 2026 11:30:00 GMT

JavaScript promises aren’t actually the promises they were named after. The concept of a promise...

Read on dev.to

A Promise diverged

Thu, 12 Mar 2026 00:00:00 GMT

An unexpected journey

As I was finishing my last blog, I felt compelled to understand the relationship between promises and async/await. What I expected was a quick review of JavaScript history. Instead, I found something stranger.

The concept of a "promise" in JavaScript traces back to a 1988 paper by Barbara Liskov. However, what JavaScript eventually implemented diverges in several important ways from the original concept. What began as a tool for structured distributed computation slowly evolved into a callback orchestration library.

The following is a strange walk through history.

Setting a starting point

Early on, I realized if I kept pulling this thread, I would probably land at the dawn of computation. I decided to draw the line at the first mention of the word promise. This puts our starting point at a paper published by Barbara Liskov in 1988. I had to stop myself from following MultiLisp and its concept of futures. If you would like to follow this and risk going mad digging through history, this would be your starting point.

Coining Promises

The foundational paper that started it all is "Promises: linguistic support for efficient asynchronous procedure calls in distributed systems". In this paper, Barbara discusses the difficulty of managing message exchange between systems. The message passing that is focused on is RPC and call streams. Barbara argues that we need a linguistic abstraction to assign to these concepts, making them easier to work with. Then the “promise” was born. These promises are not quite like the promises that you may be used to if you come from JavaScript. In these promises, you create a promise, and then you claim that promise. There is no then, catch, fulfill, or reject.

The following are the benefits of Promises.

Strong types: the result and failure modes of a promise were strongly typed.
Multiple Claims: the result of a promise could be claimed any number of times.
Blocking: When a promise is claimed, the process blocks until the value is available.

The abstraction for promises also needed to work with RPC and call streams.

When a promise is claimed, it blocks the current executing process until the claim is ready. Reading a claim that is already ready would not result in blocking. Instead, it would immediately return the known value. The following is an example directly from the paper written in the Argus language.

sinfo record [ stu string, grade: grade ]
info array [sinfo]

pt = promise returns (real)
averages = array [pt]

grades: info := % some pre-recorded info not of interest

begin
	a: averages := averages$create(info$low(grades)) % create empty array with lower bound
	
	% record grades
	for s: sinfo in info$elements(grades) do
		averages$addh(a, stream record_grade (s.stu, s.grade))
		end
	flush record_grade

	%print
	for i: int in averages$indexes(a) do
		stream print(make_string(grades[i].stu, pt$claim(a[i])))
		end
	synch print
	end expect .. end

In this example, recording a grade updates an average and returns a promise for when the grade is recorded. The flush record_grade waits for the call-stream to be cleared so that future claim operations do not have to block. If you come from JavaScript, you could loosely think of this as waiting for all outstanding asynchronous work to finish, similar to Promise.all. Finally, it claims all the promises and writes the averages.

Advent of a new language

Now we are going to jump forward to 1997. At this time, Mark S. Miller was working on the programming language E. In this programming language, promises are a language feature. This language aimed to address the problems of distributed computing better than other languages of the time. A full paper surrounding this language is here. Unlike Argus, E was not trying to hide asynchrony. Instead, it embraced it as a core part of the programming model. As a result, promises shifted to represent modern-day promises in JavaScript.

def asyncAnd(answers) {
    var countDown := answers.size()
    if (countDown == 0) { return true }
    def [result, resolver] := Ref.promise()
    for answer in answers {
        when (answer) -> {
            if (answer) {
                countDown -= 1
                if (countDown == 0) {
                    resolver.resolve(true)
                }
            } else {
                resolver.resolve(false)
            }
        } catch exception {
        	resolver.smash(exception)
    	}
	}
	return result
}

We can see the callbacks start to emerge a bit. You when a promise and you catch its error. Also note the emergence of the resolve concept.

Getting twisted

The work Mark did inspired our next character. In 2001, Glyph Lefkowitz was working on the Twisted library in Python. In this library, the concept of deferred was added. This was heavily inspired by Mark's work, and at PyCon 2003, he wrote a paper on this topic that cited Mark’s paper on the E language. In this initial implementation, you would create a deferred and register callbacks as well as error callbacks.

from twisted.internet import reactor, defer

def getDummyData(inputData):
    print('getDummyData called')
    deferred = defer.Deferred()
    reactor.callLater(2, deferred.callback, inputData * 3)
    return deferred

def cbPrintData(result):
    print('Result received: {}'.format(result))

deferred = getDummyData(3)
deferred.addCallback(cbPrintData)

reactor.callLater(4, reactor.stop)
print('Starting the reactor')
reactor.run()

This condensed example from the twisted documentation site shows the new pattern. Now you can chain multiple callbacks on a deferred. These callbacks take the form of a success or error callback. Notice that this was called a deferred, not a promise. This seems like a deliberate choice. Notice how this isn't quite the same as a promise but is heavily inspired by them.

Dojo did it first

In the early 2000's, Alex Russell was busy working on Dojo. At the time, the only way to handle asynchronous workloads was with callbacks in JavaScript. The dojo team looked to Twisted for their solution in the deferred feature. This was originally created sometime around 2005.

var deferred = new dojo.Deferred();
setTimeout(function(){ deferred.resolve({success: true}); }, 1000);
deferred.then(function(value){
    // Do something on success.
  },
  function(error){
    // Do something on failure.
});

In this, we can see even more patterns emerge. Now, addCallback has been replaced with then, and this then contains both the success and error callbacks.

Promises/A

Starting in 2009, Kris Zyp was busy trying to bring promises to CommonJS. What started off as a spirited discussion resulted in a Promises/A document. If you had been paying close attention, you would have noticed that Mark S. Miller corresponded on that discussion. Also of note, Kris and Russel had some interaction in the past, per this blog post. Kris went on to create promise-io and node-promise. In promises-io, the original approach was to continue using the term deferred, but that appears to have been superceeded by node-promise, which uses a very familiar syntax.

promise.then(function(result){
   ... when the action is complete this is executed ...
},
function(error){
    ... executed when the promise fails
});

From this work, many other libraries arose, such as Q with its wizard creator Kris Kowal, Bluebird with its creator Petka Antonov, and When with its creator Brian Cavalier.

Standardization

Due to the popularity of these languages, ES2015 added support for promises. The API was defined by the libraries from the CommonJS community. This was shortly followed by Node.js v4, which added Promises directly into the Node.js ecosystem. This marks the entry point for most modern web developers.

This particular snip from the Promises documentation seems to indicate a history of tension in the term "promise."

Note: Several other languages have mechanisms for lazy evaluation and deferring a computation, which they also call "promises", e.g., Scheme. Promises in JavaScript represent processes that are already happening, which can be chained with callback functions. If you are looking to lazily evaluate an expression, consider using a function with no arguments e.g., f = () => expression to create the lazily-evaluated expression, and f() to evaluate the expression immediately.

The gap

What was once a distributed computation solution snowballed into a library in JavaScript, hated for the callback hell it created. However, in its roots, there was no callback hell. This callback hell was a result of building a library when a native language feature was needed. Async is closer in spirit to the original promise concept. However, I would argue that all implementations in the JavaScript ecosystem have failed to meet the original requirement of error handling. If we think back, JavaScript Promises lost some incredibly valuable capabilities.

The strong typing on failures was lost.
The semantics of a claim no longer exist
The visibility of a promise state is now gone.

To me, the most unfortunate among these is the loss of failure types. Even TypeScript wasn't able to resurrect this, as the types for promise only include a result type and do not have an error type.

SONA frontend and changing times

Thu, 05 Mar 2026 00:00:00 GMT

The face of SONA

To round out the SONA blog series, I will be focusing on the front-end. Looking back on this project, I’m reminded how quickly the front-end ecosystem evolves. A while back, I wrote a short piece on Aurelia. This piece lamented the state of Aurelia. In this blog, I will instead focus on general issues with my front-end. Much of what I noticed during my post-mortem review was patterns that have fallen out of favor and mistakes that exist regardless of framework choice.

Event passing

In modern frameworks, the way state is passed between components is largely decided for us. This passing of state is so well abstracted in the framework that you might not realize there is a way to pass state without a framework. For example, in Angular, you can use either an event emitter or an output signal to pass state between components.

@Component({
  selector: 'component-a',
  template: `<component-b (action)="handle($event)" />`
})
export class ComponentA {
  handle(evt) {
    console.log('Got event ', evt);
  }
}

@Component({
  selector: 'component-b',
  template: `<button (click)="actioned()">Click Me</button>`
})
export class ComponentB {
  action = output<string>();
  actioned() {
    action.emit('Yo!');
  }
}

Even in React, this is a straightforward pattern, assuming you don't want to use a Redux-style pattern.

function ComponentB({ action }) {
    const act = useCallback(() => {
        action('Yo!');
    }, [action]);
    return <button onClick={act}>Click me</button>
}

function ComponentA() {
    return <ComponentB action={evt => console.log('Got event ', evt)} />;
}

However, there is another way to pass state without these frameworks. You can emit a CustomEvent from DOM elements directly. This is the pattern used in Aurelia. This pattern looks like the following.

<!--component-b template-->
<button click.delegate="actioned">Click me</button>

@customElement('component-b')
@inject(Element)
export class ComponentB {
    actioned() {
        let evt = new window.CustomEvent('action', { detail: 'Yo!' });
        this.element.dispatchEvent(evt);
    }
}

<!--Component A template-->
<component-b actioned.trigger="handle($event)">

@customElement('component-a')
@inject(Element)
export class ComponentA {
    handle(evt) {
        console.log('Got event ', evt.detail)
    }
}

Style preprocessors

In this project, I used Less. From what I can tell, Less has lost a lot of popularity. Angular has embraced a similar framework, Sass. What React is doing seems to be constantly shifting. The last I knew, JSS-style frameworks like emotion were the popular choice. I wouldn't be surprised to find another community that doesn't use a preprocessor but instead uses TailwindCSS.

With how this project used Less, I could have dropped the preprocessor completely. Everything I needed is now handled with the introduction of CSS custom properties. Increasingly, I find it hard to find a compelling case for a CSS preprocessor. The base CSS feature set has gotten better over the years. Many of these abstractions should be phased out by native browser functionality, similar to what happened with jQuery and the querySelector API.

Accessibility issues

One major issue I see in this project revolves around accessibility. I found countless accessibility violations, but decided to cover two.

Button contents affect screen readers

A major violation I see repeated in this project is the continued use of x as the text for a button. While this works for many users, the screen reader experience is terrible. Here is one such example.

<button class="attachment-remove" click.delegate="remove()">x</button>

Using the not-so-handy rule guide, we can see this should have been.

<button class="attachment-remove" click.delegate="remove()" aria-label="remove">x</button>

Applying this treatment would cause the screen reader to read something closer to "button remove" instead of "button x".

Live text

The notification system is another place where I had issues. In this application, if a request failed, the user was informed by a failure notification. This notification was a toast that would appear on the screen after an error occurred. Then it would slowly fade out. The problem with this is that it didn't use a live region. Because of this, it is unlikely that a screen reader would have alerted the user to the failure.

Here is the example from the notification area component.

<!-- Notification Area -->
<template>
    <require from="./notification-item"></require>
    <div id="notification-area" class="${notifications.length > 0 ? 'show-notification' : 'hide-notification'}">
        <notification-item repeat.for="notification of notifications" notification.bind="notification"></notification-item>
    </div>
</template>

A version of this that would have been compliant would be the following.

<!-- Notification Area -->
<template>
    <require from="./notification-item"></require>
    <div id="notification-area" class="${notifications.length > 0 ? 'show-notification' : 'hide-notification'}" role="alert" aria-atomic="true">
        <notification-item repeat.for="notification of notifications" notification.bind="notification"></notification-item>
    </div>
</template>

Promises vs async

This project was written during an interesting period in JavaScript history. Promises had recently been introduced, but async/await had not landed yet. As a result, all asynchronous code in the project used the Promises API. This resulted in callback chaining that was very hard to follow.

httpManager.get(`/sona/v1/incidents/${value.id}`).then((incident) => {
     // Process incident

     httpManager.get(`/sona/v1/incidents/${incident.id}/attachments`).then(data => {
         // process attachments
     }).catch(err => {
         // handle attachment failure
     });
 }).catch(err => {
     // handle incident failure
 });

The async version is a bit easier to follow, but it would be more legible if the error handling in both cases were the same.

try {
    const incident = await httpManager.get(`/sona/v1/incidents/${value.id}`);
    // process incident
    try {
        const data = await httpManager.get(`/sona/v1/incidents/${incident.id}/attachments`);
    }
    catch {
        // handle attachment failure
    }
}
catch {
    // handle incident failure
}

Tools I don't miss

I also used Webpack and gulp. Webpack, in particular, is powerful, but it creates something that is hard to maintain. Effective use of Webpack requires many plugins. For example, if you need to ship in multiple browsers, you will need the Babel plugin. The same holds true for CSS preprocessors like Less. It also applies to the minification of JavaScript. Installing all these plugins to build a web app creates a problem. Over time, plugins stop being maintained or fall out of favor. This is the inevitable nightmare I face when returning to a side project years later. Hopefully, up-and-coming build tools like Vite will be better, but plugins are still involved.

Getting more specific on SONA

For the most part, I have been rambling on about how the JavaScript ecosystem has changed. However, it would be beneficial to elaborate on some specific learnings from this project.

My design isn't getting better

I can see I was exploring with this UI, but when I use this project, it looks like a watered-down version of JIRA. The design flair I added was some gradients and a few animations. This project continues to showcase that I could use improvement in my design skills.

Auth pain

As I mentioned previously, the authorization pattern involved the following. First, you need to log in. Then you will get a "token". Finally, you need to send that token in an X-Sona-Token header. This, however, created a problem. To keep the token between page refreshes, it had to be stored somewhere. What I ended up doing was managing the token manually in JavaScript by putting the token in localStorage. This would have been much simpler if the token had been set in a cookie instead. The client-side JavaScript wouldn't have had to interact with local storage, nor would it have needed custom logic in its fetch calls to make sure the header was set.

A lack of routes

This is SPA, but it barely uses routes. There were three different routes: login, incidents list, and incident. Some routes that could have been helpful would have been: a 404 page, a 401 page, and a user profile page. Not having the 404 and 401 pages seems strange to me anymore.

What I learned from SONA

I have other critiques of my code, like specific CSS choices and some other miscellaneous issues. However, I didn't want to spend a lot of time talking about these very specific opinions.

What has stuck with me when considering Aurelia again was why I was drawn to it in the first place. It seemed reasonable that some subset of what Aurelia would become standardized in the browsers. This creates an interesting observation for me. I also enjoyed using HTMX. I know it also has similar aspirations. While I hope they get traction, this is a cautionary tale that HTMX may never find its way into the standards.

Over the last decade, the front-end ecosystem has undergone massive change. If you extend that period by another decade, the changes are even more shocking. Layers of abstraction are added, then removed. Configuration vs convention is a constantly evolving battle. In the meantime, the browser primitives have been improving. A fixation on frameworks can hide this fact. On your next project, try JavaScript without a framework; you might be surprised at what you can do.

Lessons from the SONA Backend

Tue, 03 Mar 2026 00:00:00 GMT

The engine behind SONA

Building on the last blog, I want to focus on the backend decisions behind the SONA project.. Looking back on this project, I see growth in some areas, but I also see new areas of weakness emerge. Now there are enough weaknesses in this that I cannot cover them all. Since I approached this like a code review, things like commenting style made it to my notes. To avoid making this a painful, drawn-out code review, I will focus on the most egregious issues.

Starting off on a good note

While I didn't realize it at the time, I showed some maturity in my process. Because my goal went beyond simply learning or a sense of misplaced pride in building everything myself, I used third-party libraries. Had I followed the path of my last project, I might have been tempted to build my own abstraction to create a RESTful API; instead, I used gorilla/mux. This saved me hours, if not days, of dealing with the HTTP library directly. It also allowed me to focus on the core problem. In addition to this library, I used many others, like database drivers and AWS clients.

Another very positive aspect was avoiding the creation of a premature abstraction in an attempt to solve all problems. In the past, I might have been tempted to create a webhook library that aimed to solve everyone's problems with webhooks. This would have inevitably failed. The result would have been an abstraction that failed to meet even my own needs.

Database hell

In this project, I noticed a pattern. There was a deliberate effort to allow the end user to pick their desired technology. Now I see this as a failed attempt to remove vendor lock-in.

I supported three major databases: Datastore, MySQL, and DynamoDB. This created a terrible straddle between relational and non-relational databases. If the goal was to avoid vendor lock-in, the usual culprit is the cloud vendor, not the database vendor. Making the system work well on a single database, such as MySQL, Postgres, or MongoDB, would have accomplished the same goal without the consequences.

The pitfall of abstraction

There is no shortage of documentation and blogs talking about the differences between relational and non-relational databases. However, my experience in this project taught me something a bit different. While you can make either model work behind a single interface, one or both implementations will be inefficient. I don't think I respected that enough going into this. Abstraction is a powerful tool, but it can also make you blind to inefficiencies inherent in it. These may be acceptable, but there is no such thing as a free abstraction. In my abstraction, I wanted to support any type of query on any column. The net result was implementation misses in both implementations.

Failures of Non-Relational Representation

To limit scope, I will focus only on the failings of my DynamoDB implementation. If you know anything about DynamoDB, you will know that you can either query or scan the database. Queries are the optimal route for retrieving data from the database because they use the database’s partitions effectively. However, you can do a suboptimal scan to perform other queries you may need. Notice the key focus is on performance. To use DynamoDB effectively, you should query, not scan. However, to query, you need to filter on an index, and indexes have to be predefined. This means a well-designed DynamoDB deployment requires planned query patterns. This abstraction allowed the creation of dynamic queries on any attribute that might not have an index. The net result was having to scan for all my incident search requests. At scale, this would have shown as a performance problem.

But you did better on MySQL, right?

The relational implementation fared better, but the abstraction still leaked in some interesting ways. Additionally, I do see some implementation misses. To support dynamic queries, I created the following data structure.

type Filter struct {
	Property       string `json:"property"`
	ComparisonType string `json:"comparison"`
	Value          string `json:"value"`
}

type ComplexFilter struct {
	Children []*ComplexFilter `json:"children"`
	Filter   []Filter         `json:"filters"`
	Junction string           `json:"junction"`
}

type FilterRequest struct {
	Filters  []ComplexFilter `json:"complexfilters"`
	Junction string          `json:"union"`
}

As you can see, this should allow you to chain together filter conditions with a junction (AND/OR). This has two issues. The first is similar to the NoSQL problem. While relational databases can do these queries, they tend to do better when you generate indexes. In this case, there is no good index to create. The secondary issue is somewhat painful for me to look at. It appears that the implementation missed a requirement. Do you see the issue with this code?

for i, filter := range filter.Filters {
		if i != 0 {
			buffer.WriteString("AND ")
		}
		for iter, complexFilter := range filter.Filter {
			if iter != 0 {
				buffer.WriteString("AND ")
			}
			buffer.WriteString(complexFilter.Property + convertToSQLComparisonType(complexFilter))
			args = append(args, complexFilter.Value)
		}
	}

One last commentary on this solution. As defined in the prior blog post, any incident can have any number of attributes. Here is how I built the data model.

CREATE TABLE Incidents (
    Id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    Type VARCHAR(255),
	Description VARCHAR(1048),
	Reporter VARCHAR(255),
	State VARCHAR(255)
)

CREATE TABLE IncidentAttributes (
    IncidentId INT UNSIGNED NOT NULL,
	AttributeName VARCHAR(255),
	AttributeValue VARCHAR(255),
	PRIMARY KEY(IncidentId, AttributeName),
	FOREIGN KEY (IncidentId) REFERENCES Incidents(Id)
)

Conventional wisdom says this is the correct solution, and it likely was. However, it may have been interesting to benchmark the difference between this model and a more non-standard model.

CREATE TABLE Incidents (
    Id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    Type VARCHAR(255),
	Description VARCHAR(1048),
	Reporter VARCHAR(255),
	State VARCHAR(255),
    Attributes JSON
)

Because every attribute query required joining these tables, the flexibility came at a cost. A JSON column may have simplified reads at the expense of more complicated inserts and updates.

Identity as an afterthought

A feature I bolted on later was authentication and authorization. This project highlights the dangers of rolling your own authentication and authorization. Now, before I completely rip apart the solution, I would like to point out one thing I handled passably. I did not store user passwords as plaintext in the database. Instead, I peppered the password with the user’s email and ran that through a SHA256 hash before storing it in the database, which is better than nothing.

Tokens matter.

The way I permissioned the application was the following. Once a user completed a login, they would be handed a token. No, not a sensible JWT. This was a special token of my own creation. This token contained some key data that was base64 encoded. The encoded value was then sent back to the client. The client would then have to provide this token in subsequent API requests via an X-Sona-Token header.

At this point, you might start asking some valid questions, including "Why was this base64 encoded?" This appears to be a classic case of security through obscurity. What I effectively wanted was to prevent the user from knowing what was in the token. However, a halfway decent security researcher would immediately realize this and attempt to abuse the fact. The token contains the following attributes: the user's ID, the expiration time for the token, and the permissions associated with the token. At this point in my read-through of the code, I was quite concerned. One could imagine the following cases that could cause unexpected token outcomes: extending the expiration date, masquerading as another user by changing the user ID, or even privilege escalation by changing the permissions in the token.

Oddly enough, these cases didn't end up applying as I feared they might. I did something a bit clever, not to be confused with good, that reduced the surface area of the attack. When a user logs in, a token will be generated. This token would then actually be stored in the database in its initially created state. When a user used the X-Sona-Token, it would check to see if that specific token string was issued in the past. If it weren't, it would reject the request. It would also prune the requests over time. So, while it is not good in practice, what this would mean is that to escalate your privileges, you would have to use a formerly issued, valid, and non-expired token in your request. I think this code best summarizes this almost seemingly accidental handling of these escalation concerns.

func (manager MySQLUserManager) ValidateUser(token string) bool {
	userId := GetTokenUser(token)
	tokens, _ := manager.getTokens(userId)
	found := -1

	for i, v := range tokens {
		if v == token {
			found = i
		}
	}

	if found == -1 {
		logManager.LogPrintf("Token not found for user %v", userId)
		return false
	}

	expired := TokenExpired(token)
	logManager.LogPrintf("Token expired %v", expired)
	go manager.pruneTokens(userId, tokens)

	return !expired
}

Some things that would have made this better are the following. The first group would be focused on reducing the risk of token interception. Storing the token in an HttpOnly cookie would have prevented JavaScript from reading it, reducing the risk of token theft via XSS. There would still be a risk of man-in-the-middle attacks, but since the server deployment used an SSL certificate, even that would be limited. Another thing that would have helped would be encrypting or signing the token with a server-side key. This would have prevented users from inspecting or modifying the token's contents, including user IDs and permissions.

Looking forward

In many ways, I saw improvements from my last project, but in other ways, I saw a failure to properly build a web application. I had some obvious failings in deployment strategy, database management, and authorization. I found that many of these helped form my opinions on later projects. The risk was isolated to a project that was never commercialized, talk about a win. As this evaluation of the backend comes to a close, we look to the future with an analysis of the front-end in the next blog.

Creating an issue tracker

Fri, 27 Feb 2026 00:00:00 GMT

Who needs an issue tracker anyway?

In the last series of blogs, I wrote about the first public project I worked on. It only seems natural that my next long-winded series would be on the second application to join my graveyard. So begins the tale of the Support Orient Notification Application, or SONA for short.

What the heck was I making?

While working on the text adventure creator application, I had the thought, "What if someone actually used this?" Certainly, they should be able to flag issues with it. Then the even more absurd thought occurred to me, "What if that user doesn't even know what GitHub is?" Clearly, if the user were unaware of GitHub, they wouldn't be able to create a GitHub issue. Those two thoughts alone caused me to create my own issue tracker.

What's in a name?

It was time to level up. "issuetracker" is about as good a name as "textadventurecreator". Coming up with cool names for projects is not a skill I possess. I am just not cut out to create the next BRB-DLL. Despite my best efforts, I landed on IONS, the Incident Oriented Notification System. That was until a quick search revealed several existing companies with the same idea. Instead, I landed on SONA, which, as it turns out, was already globally indexed by a video game instead.

Getting the requirements sorted

What I wanted was a hybrid between JIRA and GitHub issues. If you had been paying attention, there was a constant emphasis on "notification system". The plan was that this application would interact with whatever application you needed to by using webhooks. Created an issue, cool, there is a webhook for that. Aside from that, these were the features I wanted

Unauthenticted creation of issues. I didn't want users to have to create an account to file an issue.
Dynamic attribution. Each ticket should be allowed to have any attributes you want.
File upload. The original use case involved uploading a log file.
Notes per ticket. Each ticket should have a collection of notes to track the ticket status. Just plain text would do. No need for fancy HTML or markdown.

Choosing the right tech

Having a working service was a nice-to-have. The primary goal for me in this project was to learn some new skills. At the time, I was itching to use Go, so I decided the back-end for this application would be Go.

The front-end was a bit more complicated. By this time, the industry was starting to fall out of love with good ol' fashioned thick clients. For those who are unaware, these applications used a TCP socket to send data to a desktop client. These applications were usually written in C++, C#, or Java. My day job was primarily Angular.js with obscure pockets of Dojo and Ember. This was also a turbulent time in the industry. React was rising, the death of Angular.js was on the horizon, and the uncertain future of Angular 2 was present. Why would anyone want to add a compiler when JavaScript ran in the browser without it? One framework that appealed to me at the time was Aurelia. As this was a project that only impacted me, I decided to use Aurelia.

Over time, this morphed to include NGINX and Docker. While the front-end was drowning in options, the back-end was dumping virtual machines to chase after Docker. At least that was what I was experiencing in my bubble.

Some lessons take a while to sink in

In this project, I was starting to learn from my past mistakes. This time, I was building software to solve my problems. Even better, the scope was limited to my desires and requirements. However, there was one little thing I couldn't decide on: how should I run this application?

I fell into the trap of making the backend work for all sorts of technologies. I had created a JSON config file that would be loaded at application startup. This would control a wide variety of options. For the file storage, you could use the file system or an S3 bucket. Where things got more complicated was on the persistent storage side. By the end of this project, you could store issues in RAM (ephemeral runtime space), MySQL, DynamoDB, or DataStore. If you hadn't noticed, this was a mix of relational and non-relational databases. This forced me to torture these databases to support a very dynamic filter system.

Running the system

For a while, I had an instance of this running. I even used Swagger to generate a Java client for my text adventure creator to use. This was running in AWS on a single t1.small instance for a time. However, after a year of no tickets, I decided it was time to save the small amount of money the system was costing me.

Coming back years later

To prepare for writing these blogs, I needed to remember how it worked. I find the best way to do this is to run the application. Normally, coming back to projects 6+ years later can be exceptionally painful. This time it wasn't too bad. I only ended up having to solve two problems, and it only took me an hour or so. The first issue was that Go 1.8 and Go 1.26 are very different. As it so happens, you cannot run an application without a go.mod file anymore. Getting this working was as simple as running two commands.

go mod init <repo>
go mod tidy

I cannot overstate my joy in how simple this was compared to what I am used to for upgrades. I am looking at you, Jigsaw. The second issue was around my Node version on the front-end. The last time I ran Aurelia, I was using Node 10. As it turns out, Apple silicon and Node 10 don't play nicely together. After a bit of digging, I updated to Node 16, and I was up and running.

The feature I didn't remember

I began to interrogate the source code for a postmortem code review. Then I noticed something strange in my nginx config.

location /repos {
    proxy_pass https://api.github.com/;
}

What was I doing here? I searched the front-end and back-end documentation I created to no avail. Finally, I found a component in the front-end, and it started to click. I had a GitIssueViewer component, and I remembered. The original pain point was, "What if a user couldn't file a GitHub issue?" I had created a hook in my use of the application to create GitHub issues when SONA created a ticket. Then I would link them back to the SONA issue via a gitissue attribute. If a ticket had that attribute present, the front-end would call GitHub's public API to load the issue and its comments, which would be displayed in the application.

This might seem obscure, and it was. However, there was a justification for my madness. I wanted to display this in my application without a link. Originally, I wanted to iframe the issue, but if you know anything about security, you know where this is going. Clickjacking is a problem, and GitHub does protect against it. As a result, I rebuilt a minimal set of GitHub issues in my application just for this flow.

Looking to the future

Much like my other posts, I wanted to establish the context before revisiting the code with a more critical lens. That’s where the next blog begins.

Best animation you have seen or created?

Tue, 24 Feb 2026 22:48:47 GMT

It could be the way I interact with the internet, but most sites I use today follow a pretty...

Read on dev.to

I Assumed Java Streams Had Minimal Overhead. They Didn’t

Wed, 11 Feb 2026 02:57:38 GMT

Many of us have heard the same mantra: idiomatic code comes at little to no cost. The benefit to...

Read on dev.to

Descending the Java Stack

Tue, 03 Feb 2026 00:00:00 GMT

Some ideas won't leave you alone

As I was publishing the last series of blogs, I had a constant nagging feeling that I left the biggest pain point underexplored. In the blog, I often lament the decision I made to support multiple attributes with dynamic values. At some point, I even suggested an approach to make the code more maintainable. While I enjoy the developer ergonomics of the solution, the thing I couldn't let go of was, "What is the cost of this abstraction?"

In general, I find this to be an undervalued consideration that can lead to designs that have questionable performance characteristics. The common argument is that the developer velocity makes up for it. In many cases, that is a reasonable tradeoff. However, doing a postmortem on a project that is unused gives me the flexibility to be as unreasonable as I want and stretch the limits.

Setting up the experiment

In this experiment, I wanted to test the cases discussed in the blog.

Original implementation (Base Case)

While this is not the exact same implementation of the original code the project was using, this is very close to what it would have been doing. The important thing to note is that the value is just stored as an Object. There is no help for the consumer to know what the underlying value is.

class Attribute {
    private String name;
    private Object value;

    public Attribute(String name, Object value)
    {
        this.name = name;
        this.value = value;
    }

    public Object getValue() {
        return this.value;
    }
}

Implemented solution (Generics Case)

As mentioned in the blog, the solution that is currently in the project is based on generics. This case is intended to mimic the implemented case, but with a data type field. This allows for slightly better handling on the consumer side, but still requires the primitive to be stored as an Object.

class GenericCaseAttribute {
    private String name;
    private Object value;
    private DataType dataType;

    public GenericCaseAttribute(String name, Object value, DataType dataType)
    {
        this.name = name;
        this.value = value;
        this.dataType = dataType;
    }

    public <T> T getValue() {
        return (T) this.value;
    }

    public DataType getDataType() {
        return dataType;
    }
}

Optimal Solution (Best case)

In this case, I just did the solution I assumed would be the fastest. This breaks the requirements in an attempt to speed up the algorithm. This does not allow for multiple types and forces every value to be a double.

class BestCaseAttribute {
    private String name;
    private double value;

    public BestCaseAttribute(String name, double value)
    {
        this.name = name;
        this.value = value;
    }

    public double getValue() {
        return this.value;
    }
}

The testing methodology

Now that I knew the implementations I wanted to test, I needed to set up a way to test the implementations. I settled on testing three different paths through the implementations. The first would be a full test that would build a list of 1,000,000 entities in the related data structure. I called this the population test. The second test would start with the population, but then iterate over all the values. However, only the iteration was measured. That test was referred to as the iteration test. Lastly, a full test would be done. This combined the population and iteration cases in what I call the full test.

Finding the benchmark

Originally, I thought I could do a basic stopwatch test. In this case, I would grab the time, run the test, and then compute the delta to get the time the implementation took. Simple enough, right?

public static void main(String args[]) {
    long startTime = System.nanoTime();
    // Test here
    long endTime = System.nanoTime();
    System.out.println("Test took " + (endTime - startTime).toString());
}

Upon some reflection, I knew this was insufficient. The problem with timing Java is that it is a JIT-compiled language and as such it requires multiple passes over code before it’s optimized. If I really wanted a good representation of the real impact, I needed to do something better. As it turns out, people have spent a lot more time thinking about this than I have. There is an entire tool JMH built for doing this type of test. Armed with a new tool, I decided to go with a pretty barebones setup in which I would test operations per second. As I mentioned, many benchmarks had to be created.

Full run benchmark

@Benchmark
public void bestCase() {
    BestCase c = new BestCase();
    c.populate(1000000, 4);
    c.aggregateTotal();
}

Population benchmark

@Benchmark
public void bestCase() {
    BestCase c = new BestCase();
    c.populate(1000000, 4);
}

Iteration benchmark

@State(Scope.Benchmark)
private static BestCase c = new BestCase();

@Setup
public void setup() {
    c.populate(1_000_000, 4);
}

public class MyBenchmark {
    @Benchmark
    public void bestCase() {
        c.aggregateTotal();
    }
}

Looking at the results

After spending more time than I care to admit running all of these tests, I ended up with a dataset I could play around with. If you filter out the V2 data points, you will notice something shocking. The best case doesn't just beat the other cases; it completely demolishes them. See, there is one thing I didn't mention. In the best case, I actually use a plain Attribute[] data type instead of an ArrayList<Attribute>, which has a huge implication. I will come back to this, but I made a minor change and generated some V2 benchmarks that lined the values up better.

Impact of different approaches

Considering all of the originally considered approaches, we can see that the difference between them is more or less within the margin of error.

Best case consideration

Much like I assumed, the best case outperformed the other cases, and it did so by a pretty noticeable amount.

So wait, what was the big difference between V1 and V2?

I am so glad you asked. In the first version, all implementations except the base case used an unsized ArrayList. Now, some of you may know where this is going. It turns out that if you don't pre-size your ArrayList and grow it on insert, you pay a real performance cost. In this case, I insert 1,000,000 values sequentially, growing the ArrayList in the most inefficient way. As it turns out, the base size of an ArrayList in Java is 10 entries, and when you exceed the boundary, it grows by 1.4x. This means in the end my arrays were resized nearly 30 times and produced an excess of 250k entries. This adds up to an insane amount of time, as the resizing event requires array copying and is a costly operation.

But why stop there?

Now I could stop there, but where is the fun in that? At this point, I knew approximately the time it required, but I noticed that I was getting variability in my test runs with JMH. While it does provide direction, it is not the end source of truth. The question I had was, “How can you get closer to the source of truth?“ I decided the best way to find out what is going on is to see the actual disassembly generated by the JVM and interpret that.

Getting the tools

This required a fair amount of setup. First, to reason about the assembly generated, I set up IntelliJ IDEA with Jitwatch4j. Since the benchmarking was run as a jar, I had to ensure the run generated a HotSpot log file. These log files contain all of the disassembly. To do this, I ran some crazy command like this.

$ java \
  -XX:+UnlockDiagnosticVMOptions \
  -XX:+PrintAssembly \
  -XX:+LogCompilation \
  -jar target/benchmarks.jar

Once I had the log files, I could then load them into Jitwatch4j and look at the disassembly. However, I still wasn't finding everything I wanted to see. As it turns out, some instructions are predefined in the JVM. The result is that the disassembly might just jump to some instruction you cannot see. After some searching, I found hsdis which gave me everything I was looking for.

Looking at the disassembly

Now that I had the disassembly, it was time to find anything interesting in there. The first challenge I had to overcome was learning the ARM architecture. In the past, the only assembly I had done was based on the x86 architecture, and ARM was a learning curve for me. After spending a while figuring out how to read the manual and using this cheat sheet, I was able to piece together some basics.

I had understood that there was a cost in some languages when storing as an object and casting it back to a primitive. What I found interesting in the disassembly is that it does not appear to be the case. Instead, we just find the underlying data, put it in a register, and add it.

Let's consider this Java source.

public double aggregateTotal() {
        double retVal = 0;

        for (GenericCaseAttribute attr : this.data) {
            double value = 0;
            switch (attr.getDataType()) {
                case Double:
                    value = (double)attr.getValue();
                    break;
                case Integer:
                    value = (double)(int)attr.getValue();
            }
            retVal += value;
        }

        return retVal;
    }

When the double branch is hit, we do a straight-forward addition.

0x0000000115a623e4: ldr d16, [x0,  #16] ; Get field containing double value and store in d16
0x0000000115a623e8: b 0x0000000115a62400
; intermediate steps truncated
0x0000000115a62400: add w17, w10  #0x1 ; Increment ArrayList iterator
0x0000000115a62404: fadd d0, d0, d16  ; Add double attribute value d16 to retVal d0 and store in d0

However, converting an integer double is not free and requires an extra operation.

0x0000000115a62470: ldr w10, [x0,  #12] ; Get field containing int value and store in w10
0x0000000115a62474: scvtf d16, w10  ; convert int value in w10 to double value in d16
0x0000000115a62478: add w10, w17  #0x1 ; Increment ArrayList iterator
0x0000000115a6247c: fadd d0, d0, d16  ; Add double value in d16 to retVal d0 and store in d0

What I noticed was that in both cases, there hadn't been special instructions to go from Object to double or Object to int.

Finding metrics to pull from the disassembly

After spending time inspecting the disassembly, I realized it was hard to find significant findings. Looking at 800+ lines of assembly and comparing them to another 800+ lines where there is an illegible diff just wasn't cutting it. At that point, I remembered something: the slowest operations on the CPU should be memory access, especially. These are even more pronounced if memory access misses the L1 cache. In the disassembly of this program, most operations outside of memory access should have fairly low time complexity. All operations had been very simple arithmetic. Things like sin, cos, or even division didn't show up.

With this new insight, I decided to count all of the memory access operations for each case and plot them in this dataset.

Correlation between data sets

I realize that just the number of memory access operations or the number of lines of assembly code is a flawed metric for determining performance. This is because you don't actually know how the branches will be used at runtime. However, I found it to be shockingly correlated to the timing dataset from JMH. Basically, the more memory access operations you have defined in the routine, the fewer operations you can do per second.

Correlation between memory access and operations in iteration runs

Correlation between memory access and operations in population runs

Lessons learned

Depending on where you are willing to set your limit, Java performance evaluations can go much deeper than many people may realize. From intuition to slow path to a JIT-compiled path, there are many opportunities for performance to be difficult to measure. This whole test didn't even consider the differences between HotSpot optimizations for different CPU architectures, which could have been explored.

However, if the numbers tell you one thing, it is to pay close attention to your collections and how you initialize them. Failure to do so will result in a serious decrease in performance.

In this case, it turns out that the costs I cared about for legibility weren’t the ones the machine cared about.

Execution Is Easy Now. That’s the Problem

Sun, 18 Jan 2026 20:52:31 GMT

At the end of last year, I had come to a stopping point on one of my many long-running side projects....

Read on dev.to

IDE for Text Adenture Games?

Sat, 17 Jan 2026 00:00:00 GMT

Building an IDE for text adventure games

By this point, I have covered most of the core libraries used; now it is time to get into the core game building. The end product of my work was an IDE for making text adventure games. This IDE was intended to feel somewhat familiar to developers used to using an IDE.

Building a UI Framework

The last core library I haven't covered to this point was IROShell. Looking back, I don't have many grievances about this software. However, I will say its shortcoming came from overengineering. I had built a Java UI framework on top of JavaFX that I intended to use going forward. The problem was that, after this project, all of the work I did was in the web ecosystem. So, I didn't get to use this again.

This framework required a mindset where users had complete control over the layout of multiple tabs. The result was layers of abstraction between you and the end layout. You would define views, menus, and toolbars. Then the framework would put them in their correct place and allow the user to move them further.

This framework also included features for different layout strategies, namely a multiple-document interface and a single-document interface. It also had the ability to define custom pre-load screens, such as a splash screen or a login screen.

Looking back, some of the biggest issues with this had been building on Java-core and Logrunner. While it was very helpful for me to generate logs, having your framework generate logs for debugging your application shell on an end user's machine was a bad idea in hindsight.

Also, while the feature was incredibly helpful, the execution of the dynamic styling feature was problematic. I had built a feature that allowed for dynamically changing the application's styles just by changing a CSS file. This would watch the file for changes and host an internal Java protocol for reapplying the styles. The problem was that this required a new protocol to be defined, and the protocol I chose was called internal://. In hindsight, that was a terrible and careless naming decision. It also required constantly creating new URLs, which likely led to memory leaks.

Issues in the Core

The textadventurelib was the core of the game engine. This is where all the real game logic occurred, which just so happened to be turning an XML representation of some game state into a functioning game.

In pursuit of clean code

To create the cleanest code I could think of, I ended up creating multiple cases of indirection that were hard to follow but also quite common in front-end development. One specific case that comes to mind is game state processing.

To set the stage, the general operation model broke down into the following

A game has game states
A game state has options
Options have triggers and actions.

In many cases, you would want to send a message from an action to the game state. However, since the game state was the holder of the action, you wouldn't want that action to interact with the game state; that isn't clean, right? How I addressed this created a different problem. The actions would call back on an interface that the game state had. The problem was to wire all of this up; the game state had to understand all possible actions to hook up any action that might call it back later. This created a problem where the game state had to infer the type of the action at startup time.

Startup
load game state --> iterate all options --> find actions --> if action of type x add listener y

Send event
action triggered --> find all listeners --> call specific interface callback

To make matters worse, the design of these actions was overkill. These actions allowed multiple callback handlers, but in practice, there would only ever be one.

This created ugly code like the following.

for (IOption option : options) {
	if (option.action() instanceof CompletionAction) {
		CompletionAction act = (CompletionAction)option.action();
	    act.addListener(this);
	}
	if (option.action() instanceof AppendTextAction) {
		AppendTextAction action = (AppendTextAction)option.action();
	    action.addListener(this);
	}
	if (option.action() instanceof FinishAction) {
		FinishAction fAction = (FinishAction)option.action();
	    fAction.addFinishListener(this);
    }
}

Instead of creating this mess, I could have used a message broker pattern or provided all actions access to the game state.

Just let people figure out images for themselves

This one is quick and minor, but early on, I decided it was very important that images be stretched to fit the image area, aspect ratio be dammed. In hindsight, this was clearly a silly choice.

Maybe the entity model wasn't that good

So much of this code is a complicated mess. This mess was most noticeable when you needed to act on the data in an action or trigger. This complication originated from building mental models of real-world concepts in code. What I mean is that if I had simplified the player down to: a player is an entity that can have many properties, I could have saved a lot of complexity. The important distinction between that and what I build is the dimensionality. In this alternative, I would have had a flat list of properties. Mapping classes to the real-world abstraction created friction in actions, triggers, and macros. I had to build overly complex, loosely typed models just to allow you to modify or read the specific data you needed to act on. This resulted in more cases of Object instead of type, as well as some unnecessary reflection.

Macro's needlessly complicated

Another feature that was important to me was macro substitution. If you have attribution on a player, you should show it in the game at some point. For this, I created a complex substitution pattern that was somewhere between a substitution engine and a DSL. Let's consider a player with the attribute age. To get the value, you would have to create this macro {[player(PlayerName)@attribute(age)@value]}. Since that wasn't complex enough, I decided to abstract out the separators so that it could be written in other ways. At execution time, this translated to dynamically building regular expressions and using reflection to find the properties on objects that you wanted to display.

Looking back, I think there are a couple of ways I could have done this differently. First, there is the recurring observation that the entity model was too complex. However, if I couldn't have done that, there are other options, such as creating a DSL or even maintaining a simple map of a unique identifier to a substitution value.

Manual persistence code?

An overwhelming amount of code and classes were duplicated entities that had been defined for the sake of persisting with persistlib. A noticeable amount of the code written would simply not exist if metaprogramming were used to persist classes. There are ways to do this in Java, but even if I wanted to accomplish this with my own system, making persistlib metadata-driven using Java annotations would have saved a great deal of effort.

Bringing it all together in an IDE

All of this work had built up to the text adventure creator. This was an IDE for generating text adventure games. It built on all of the prior libraries, plus some JavaScript ports I didn't mention. The mistakes of the existing libraries would resonate here, but there had been a few new ones.s

Code generation is brittle

Originally, the end result asset was going to be either an XML file that could be provided to another application or a compiled jar. In the end, I instead supported all of these

An XML-based file that could be provided to another application that did not exist.
A generated jar.
A generated Maven project for a developer to later generate a jar.
A generated game in an HTML file format.
A generated game as an electron app.

The choice to support all of these with no users was painful. To make matters worse, the HTML asset generation used a Webpack build system and React. When you are not actively working on a web project every day, you will find yourself in a legacy state that is hard to get out of. In modern web development libraries, build systems, and Node.js versions change at a nauseating pace. A failure to update every 6 months means you will feel incredibly behind if you wait two or more years to update something.

Another thing that made this quite painful was that I used very large format strings for all of this. Using a framework like Handlebars or having a template file with clear substitutions would have made this much easier when making changes to the generated assets. Here is one example of the crazy code I ended up writing instead of creating a sane template.

public final static String GAMESTATEINITIALIZER = "package org.%s.%s;\r\n\r\n"
			+ "import ilusr.gamestatemanager.GameState;\r\n"
			+ "import ilusr.gamestatemanager.GameStateManager;\r\n"
			+ "import javafx.stage.Stage;\r\n"
			+ "import textadventurelib.persistence.TextAdventurePersistenceManager;\r\n\r\n"
			+ "public class GameStateInitializer {\r\n\r\n"
			+ "\tprivate Stage mainStage;\r\n"
			+ "\tprivate GameStateManager manager;\r\n\r\n"
			+ "\tpublic GameStateInitializer(Stage mainStage) {\r\n"
		    + "\t\tthis.mainStage = mainStage;\r\n\r\n"
		    + "\t\tinitialize();\r\n"
			+ "\t}\r\n\r\n"
		    + "\tprivate void initialize() {\r\n"
			+ "\t\tmanager = new <String>GameStateManager(\"NA\", new GameState(null));\r\n"
		    + "\t\tMainMenuGameState mainGameState = new MainMenuGameState(manager, mainStage);\r\n"
			+ "\t\tmanager.addGameState(GameStates.MAIN, mainGameState);\r\n"
		    + "\t\tmanager.addGameState(GameStates.TRANSITION, new TransitionGameState(mainStage, manager, buildPersistenceManager()));\r\n"
		    + "\t\tmanager.addGameState(GameStates.LOAD, new LoadGameState(manager, mainStage));\r\n"
		    + "\t\tmanager.currentGameState(mainGameState);\r\n"
	        + "\t}\r\n\r\n"
		    + "\tprivate TextAdventurePersistenceManager buildPersistenceManager() {\r\n"
	        + "\t\tTextAdventurePersistenceManager retVal = null;\r\n\r\n"
		    + "\t\ttry {\r\n"
			+ "\t\t\tretVal = new TextAdventurePersistenceManager(getClass().getResource(\"%s.xml\").toExternalForm());\r\n"
		    + "\t\t} catch (Exception e) {\r\n"
			+ "\t\t\te.printStackTrace();\r\n"
		    + "\t\t}\r\n\r\n"		
		    + "\t\treturn retVal;\r\n"
	        + "\t}\r\n\r\n"
	        + "\tpublic void run() {\r\n"
		    + "\t\tmanager.start();\r\n"
	        + "\t}\r\n"
            + "}";

Good luck trying to line up those %s substitutions!

Player model strikes again

As we now know, a complex player model made player-specific triggers, actions, and macros hard. As it turns out, this resurfaces when you are building a UI for those actions, triggers, and macros. For a decent user experience, you don't want to keep a user guessing. For example, if you want to create a trigger that fires every time a player's health is below 10, you would want the UI to help you. The player model knows what attributes the player has, their types, and even their default value. To accomplish this, I built a complex set of dynamic combo boxes, or selects if you prefer that terminology. These would be dynamically added or removed based on the combo box prior, and their values would be tied to the entity model.

Despite getting this to work functionally, the UX was still clunky.

I am not a designer

As evidenced by the beautiful artwork below, I am not a designer and probably will never be great at it.

As a result, my style choices came off looking crude and childish. Since it was not my core focus, I didn't spend much time refining it. This often appeared as unstyled UI, but in other cases, it was just styled poorly. A prime example of this is the debugging experience I built into the IDE. While I found it useful to a degree, the debugger looked terrible. It also lacked the functionality you might expect from a debugger. The result was a poorly styled entity state tracker. Every time an action was triggered that changed a player's state, you would see removed entities in red, updated entities in blue, and added entities in green. The problem was that these had been styled so big and bulky that it was hard to read the change.

Example of this visualization

While this primative debugger was helpful for me, doubling down on making this better instead of focusing on other features would have been to my benefit.

Feature bloat gets in the way of deep work

Constantly going wide on this means having to support questionable value work without improving the core experience. This often leads to making UI changes just to support a new feature that provides little value. One example where this surfaced in this project was in the initial game creation wizard. Over time, something simple grew into a three-page wizard that was constantly changing. To make matters worse, most of the options provided little value. Every time I wanted to run another test, it added unnecessary time to just create a test game.

Had I focused on the core problem, I could have spent more time making the experience better and wasted less time on UIs that reflected the ever-changing options.

Dynamic styling was very helpful

The investment in being able to dynamically style IROShell came in very handy. While developers who have only worked on Web 2.0 likely won't understand, having to wait for an entire compile of a Java application just to test one CSS change was miserable. Investing the time into that saved me a ton of recompile time.

High-level lessons learned

Prototypes are valuable! Believe it or not, you are allowed to write crappy or unclean code to learn what does and doesn't work in a system. Spending too much time making something clean can distract from understanding the core problem.

Iteration time is everything! I think this is part of why the web is doing so well. The time it takes to iterate on HTML, JS, and CSS assets is faster than that of a typical compiled application. If you disagree with this, focus on your toolchain. You don't actually need a build system for the web, and your iteration time is just a save and refresh away.

If you are hand-writing persistence code for each entity, you are probably doing something wrong!

Each feature comes at a maintenance cost! Without careful selection of features, you will end up with so much bloat that you cannot move forward. In all likelihood, most of those features don't matter anyway. Treat your features like cattle, not pets!

Failure to read

Sat, 10 Jan 2026 00:00:00 GMT

In my last blog entry, I detailed how I came to build a text adventure creator application. Much of that focused on high-level learning. These learnings were more focused on project structure and reception of the work. However, to do a retrospective, I wanted to focus more on the specific code choices to see what I think of them years later.

With the complicated web of dependencies created, breaking this analysis across core libraries and the application seemed appropriate. In this case, core libraries are anything other than iroshell, textadventurelib, and textadventurecreator.

Who needs telemetry anyway?

Logrunner is very small and serves a simple purpose: write log entries to a series of TSV files as well as stdout.

Now, you might think TSV is a funny format, and I will give you credit for that. However, at the time, I was using a log analysis tool that enabled querying TSV files in a SQL-like syntax, and I found that very beneficial.

In most cases, others would reach for log4j, but I didn't need those features for my purposes.

Looking back, I don't have many complaints about this library. A more advanced collection of data is required for some projects. In my case, I didn't need this kind of telemetry. If someone wanted help, they could just send me their log files.

Bad to the core

When I look back on java-core I see some obvious mistakes. If I were to start this project over, this would be a very different library. Since I made plenty of mistakes here, I will choose them carefully.

Failed wrapper abstractions

A common pattern I noticed, especially around the XML logic, is a failure to create a meaningful abstraction. In many cases, I took an existing standard library and wrapped it in a method that did the same thing. This can be a powerful abstraction if you want to switch out libraries, but in my case, I got this completely wrong. Firstly, one stated goal of this project was to use only the standard library. Wrapping the standard library with a limited feature added no value, as I was never going to use a different third party in the future. Secondly, I would wrap the standard library but return the standard library classes to the consumer. This created a leaky abstraction and did not effectively abstract out the standard library.

Failure to learn the ecosystem

If you are going to work in an ecosystem, you should use the tools from that ecosystem in the intended way. Instead, I decided to create my own pattern. Reevaluating at this library, I can elaborate on two examples where a little more time spent researching would have created less friction.

Swing is not WPF

None of this code is public, but at one point, I had a Swing-based application in the mix. This application loaded the created games and ran them. This was completely written in Swing and was created before I found JavaFX. I was influenced by WPF at the time. I was all in on MVVM (model-view-viewmodel), MVP (model-view-presenter), pick your acronym poison of choice. Of course, Swing didn't easily fit into that pattern. I could have just stuck with a single view class, as was common with Swing. However, I found it more fitting to create a monster.

If you don't know this pattern, this analysis is most likely foreign to you. However, if you want to know or want a refresher, this is a way to separate your view and interaction/data model logic. The often stated goal is that it is easier to have "just UI" and "just behavior" changes. The flow between components is relevant, so here is a visual aid.

To create a similar effect, I decided on using: a model with no view logic, a view that was the Swing control, and a presenter. The implementation was an absolute mess. I ended up having all views and models extend from some base classes. These base classes would then allow you to raise a change event. This event would then be picked up by the presenter, and the presenter would use reflection to invoke the appropriate method on the linked view or model. This produced completely absurd code and failed to meet the desired abstraction.

Not only did this build MVP incorrectly, but it forced you to pay the reflection tax. If I had scaled up this pattern, the application would have run much slower, as none of the UI interactions would have been able to be effectively JITted.

Localization is solved in the standard library

Another mistake I made in the implementation was creating my own language file format for localization. If I had done a little research, I would have found that Java has a robust solution for localization. Instead, I created an abstraction that was very similar to standard properties files but used the delimiter ;=; instead of =.

Pulling in Spring

I have no issue with Spring, but one of my stated goals was to only use the standard library. Instead, I pulled in one of the biggest common libraries for a very small purpose. I wanted to be able to do IoC, and Spring had it. In doing so, I saved myself a headache by not trying to implement IoC myself, but I missed out on some of the learning I wanted to achieve.

Data formats are tricky

Much of what I built takes in an abstract state, serializes it to XML, then reads it back into classes. When I created this abstraction, I assumed I would eventually use other formats. The problem is, I made a very classic mistake. I built an abstraction before I had multiple use cases. This ended in a poor abstraction that was the Java XML library with a few opinions. At one point, I had considered adding JSON support, but the abstraction I had built didn't make sense for any other data format. What the heck does it even mean to add an attribute to a JSON property?

Building an entity model

The core of playerlib is the ability to create an entity that describes how I thought a player should work. This entity had many opinions about players. Players have attributes, characteristics, inventory, and so on. This even built in some features, such as listening for changes to the player entity.

I didn't have too much of a problem with this, but then I remembered something. There was one thing that this abstraction did, which caused me much grief. The value type was not known at compile time. Let's take a simple example: a player can have attributes. In these attributes, there can be different value types. For example, you might have an attribute age that is an int and another attribute surname that is a string. The original implementation stored the value as an Object. The caller of the code had to figure out how to unwrap the value. Later on, I remembered that Java had generics; certainly, those would fix my problem! Of course, that wasn't quite right in my implementation. What I effectively did was take a method like this

Object getValue();

and turn it into this mess.

<T> T getValue();

Now, in hindsight, there are probably a couple of ways I could have made this easier to reason about. One option would have been to create a pseudo-type union.

enum ValueType {
    String,
    Integer,
    Double,
    Boolean
}

abstract class BaseValue {
    private ValueType valueType;

    public BaseValue(ValueType valueType) {
        this.valueType = valueType;
    }
}

class StringValue extends BaseValue {
    private String value;

    public StringValue(String value) {
        super(ValueType.String);
        this.value = value;
    }

    public String value() {
        return this.value;
    }
}

class IntValue extends BaseValue {
    private int value;

    public StringValue(int value) {
        super(ValueType.Integer);
        this.value = value;
    }

    public int value() {
        return this.value;
    }
}

// etc

This would have removed many type checks that had not been applied consistently. In addition, it would have allowed for unboxing primitive value types.

Building a state machine

In gsmlib, I assumed that I could represent any possible game with the abstraction created. This mental model being all games are a series of game states, and there is data that moves between states. Also, all games would have a finished state. This state would tell you the game is over. In addition, there was a concept of game state buffering. For buffering, you would be able to load a couple of game states into memory. Then, as the game states proceed, the current game state would be evaluated to load the next game states needed.

I don't have much to complain about with this abstraction. However, to be fair in this analysis, I must admit that this had the same shortcoming as playerlib. The game state data that was passed around from game state to game state was Object and did cause some issues. This also made an overgeneralization; this clearly would not apply to all possible games. In essence, I just created another state machine.

So what did I learn?

If I had to distill this long analysis down into key lessons learned, this would be the TL;DR.

Take some time to read about the ecosystem. Reinventing patterns in an ecosystem is rarely to your benefit. This is doubly true if you are a junior. You can learn a lot doing this, but more often than not, you are just creating a mess.

Have more than one implementation before creating an abstraction. Failure to do otherwise will create a premature abstraction that is almost certainly insufficient.

If your abstraction is clever, it's probably wrong. Just because you can do something and get it working doesn't mean it was a good idea. Consider your trade-offs and understand the cost you are paying.

Strong types are preferred. If you are just passing Object around or accepting a generic whose type hint cannot be statically inferred, you are going to end up paying for it in the future. The runtime cost to find the underlying type is not free, and the developer time wasted writing and maintaining that code will be staggering.

Side Project Beginnings

Sat, 03 Jan 2026 00:00:00 GMT

Public side projects, a beginning

As I mentioned in my last blog entry, I will be covering my past projects and evaluating them. The only logical step is to start from the beginning, or at least close to it. This brings me back to when I was leaving college and starting my first job as a software engineer. At this time, it became clear to me that college had provided me with core foundational knowledge in computer science, but I was lacking practical application skills. To better understand the field, I decided it was time to start building some applications instead of focusing completely on algorithms and theory.

Since I had the most cliché introduction to computer science, I decided something with games would be a great start. Given this, I decided the easiest place to start would be something with text adventure games. Since I tend to lack creativity and storyline creation, I set out to create a text adventure framework so that others could create games on my system. So began one of the largest projects I have done outside of work. This was also for an audience of zero people. As far as I know, no one has ever attempted to use it.

Hello World

Energized with a fun idea, I set out to build this framework. I had decided to go in with a couple of constraints.

Built on Java

At the time, I was mostly proficient with third-generation programming languages, and the majority of code I had written was in Java and C#. At the time, I was working heavily in C#. I had justified this choice in a few different ways. First, I wanted it to be cross-platform, and I knew that Java was built to be cross-platform. This focus on cross-platform was one of the major reasons I didn't do the project in C#. Also, I knew that the all too popular up-and-coming game Minecraft was written in Java. I figured if a 3D-rendered game could run in Java, so could a text adventure engine.

Only standard Library

I had decided that if I wanted to learn something deeply, I should only use the standard library and import nothing else. In the end, I fell short of this one. While I intend to cover this more later, the notable exceptions are Java Spring, JavaFx, and IzPack.

Evolution of the project

Before we consider the end state, it seems fitting to discuss how this project evolved over time.

Early days

One of the first decisions I made was to use Maven for this project. In the earliest versions of this, I had a single project that I was working on in Eclipse and building with Maven. This started as something that I kept on my local computer and backed up on an external hard drive and flash drive for a few years. Over time, the project grew, and before I knew it, a monorepo with around nine different libraries was created.

Growing up

At some point, I had lost my flash drive. This made me realize that I could lose the project completely. After this little scare, I decided it was time to start backing up my project online. For some reason, I was worried about sharing my code publicly. So, I decided to create a Bitbucket account to back the project up in a private repository.

During this time, the industry had decided it was no longer cool to store all code in a single repository. Instead, a ton of small repositories should be managed and versioned independently. As a good engineer, obviously, I had to follow the rest of the industry. I spent a notable amount of time creating separate projects for each of my libraries. This involved breaking the library out of the monorepo, creating a new Bitbucket repository, publishing an artifact to an S3 bucket on my AWS account, just so I could download it again via Maven for the projects that depended on that code.

Getting ready for "Release"

At some point, I had decided enough is enough! “I am going to put this code on GitHub as a public repository”, and that will be my released version. I was worried that once it was out there, people would see what I was working on, so I decided to work feverishly on the project to make it "release-ready". What I defined as release-ready was the following.

First, the project needed very high test coverage. If there was a class it needed to be unit tested. It did not matter how small or pointless. To put this into context, even the about dialog in the main application had unit tests.

The next major requirement I had decided on being a "blocker" was that every public library needed to have every single method documented with Javadoc. and that all projects needed to have generated documents. This began with spending more time than I care to admit writing questionable Javadoc for absolutely everything. Then I had to fight with Maven to make sure the docs would actually generate correctly.

Lastly, if I were going to send this out into the world, every respectable program that runs on a user's machine needs an installer, right? This began a long journey of creating an install workflow. Which basically just dropped a batch script into Windows to start the game builder. In this effort, I found IzPack and spent a long time learning how to create an install flow.

Post Release

Soon after moving everything to GitHub and making everything public, I realized that no one was watching. If you want people to actually see what you are building, you need to be able to market your work, and that was not something I was good at or wanted to get good at. From that point on, I felt free to just work on the things that I had found interesting or wanted to learn.

One of the first major changes happened when I was building my personal website. I decided I wanted that website to scrape all of my GitHub content and have pages for everything I put in the docs folder of any of my projects, along with any associated Java documentation. During that time, I moved a lot of documentation. Mostly, this involved converting HTML documents that detailed how to use the application to markdown files on GitHub.

Not long after I created my website, I decided to create GitHub workflows for all my projects. During that time, all of these repositories got their own set of actions to build and test the repository on every PR. In doing this, I realized that how I was integrating with Maven and S3 was not going to work in GitHub actions. Due to this, I switched over to having my projects published to GitHub's Java repository.

At some point along the way, Java updated to Jigsaw, and I decided it would be nice to make some of my "internals" actually internal, so I updated to Java 9. I wrote about this in the following blog.

While I never really completed it, the last thing I decided to focus on was creating end-to-end tests for the main application. For this, I was playing around with test-fx. Ultimately, this never got completed because I was having a hard time getting the tests to pass on the GitHub runners. The tests had worked just fine on my local, but, for some reason I never really cared to debug, I couldn't get them to pass in the runner.

Current state of the project

Once the dust had settled, I had been working on and off on the project for close to a decade. To my knowledge, it remains used and unheard of.

Project setup

The final makeup of the project was the following repositories. While I do not think lines of code is a great metric, I do think in a pre-AI world, it highlights the amount of time spent on the project more so than anything.

Name	Language	Purpose	Lines of code
java-core	Java	Library containing anything I deemed a core utility	3562
logrunner	Java	Abstraction to log output to csv or tsv file	293
java-persistencelib	Java	Library to save xml files to disk	1586
persist-lib	Javascript	Javascript reimplementation of java-persistencelib	270
playerlib	Java	Library for entity models and abstractions	4100
player-lib	Javascript	Javascript reimplementation of playerlib	956
playerlib-example	Java	Example application using playerlib	1459
gsmlib	Java	Library for game state models and abstractions	935
gamestate-manager	Javascript	javascript reimplemenation of gsmlib	306
iroshell	Java	Multipurpose JavaFx application shell abstraction	10940
iroshell-examples	Java	Example applications build on iroshell	1444
textadventurelib	Java	Core library required to play the text adventure games	25045
text-adventure-lib	Javascript	Javascript reimplemenation of textadventurelib	7437
textadventurecreator	Java	The main application that created text adventure games	55066

In total, that is close to 115k lines of code split between Java and JavaScript.

This breaks out into the following dependency graph.

Features

In the spirit of a fair analysis, I find it is useful to document all of the features wrapped up in this project. Much of this is documented on my personal site. As such, I will avoid repeating all of it. Instead, I will highlight some core features. If you are interested, the main documentation can be found here.

Feature	Category	high level details
Export as App	Export	The ability to export the game as a jar
Export as Web	Export	The ability to export the game as an html document
Export as Electron	Export	The ability to export as an electron app
Entity creation	Core	The ability to define players and NPC's/enemies
Macros	Core	The ability to replace text with some player details
Game State creation	Core	The ability to define all actions and triggers in a game state
Text Trigger	Core-Triggers	The ability to trigger some action on user text
Timed Trigger	Core-Triggers	The ability to trigger some action on a timer
Player Triger	Core-Triggers	The ability to trigger some action based on player state
Scripted Trigger	Core-Triggers	The ability to trigger based on some executed javascript
Multi-part trigger	Core-Triggers	The ability to union multiple triggers together
Append Action	Core-Actions	Ability to add text to the game output
Complete Action	Core-Actions	The ability to complete a game state
Execute Action	Core-Actions	The ability to trigger a process on the users machine
Modify Player Action	Core-Actions	The ability to change player state
Save action	Core-Actions	The ability to save the game
Script action	Core-Actions	Any action that could be expressed in javascript
Finish action	Core-Actions	The action that completed the game
Text and Input View	Layout	The base look and feel
Text and Button View	Layout	A look and feel that replaces a text input with buttons
Content Only	Layout	A display only view for things like transition images or videos
Custom layout	Layout	A customizable grid layout of whatever controls you want for the game state, including custom styling
Libraries	Libraries	User defined exportable collections of players/actions/triggers or layouts
Debugger	DevTools	A crude debugger to see how playing the game changes the state in the IDE
Language Packs	Localization	The ability for users to define their own localization to be used in the IDE
Mods	DevTools	The ability for a developer to add a runtime loaded mod into the IDE

Lessons learned at the project level

A warning to the reader. Many of these lessons are based on the constraints and end state of this project. If this project had been worked on by a team of engineers, my perspectives would have been different. Remember that the end state of this project is that a solo developer built something that was unused but is now in the public domain.

Just because it's on GitHub doesn't mean anyone will notice

I think this is one lesson that I wish I had learned early on. There was a fair amount of time I spent scrambling to get the code "ready for the public to view". I was concerned that once you put something on GitHub, everyone would see it. In reality, I think a handful of people have glanced at the landing page, then saw a sparse readme and moved on. The largest reader of my code may be some AI model trained on all public GitHub data.

The most important lesson for me around this is that you can waste an immense amount of time expanding code coverage and documentation on a project. If you are not working on a team and do not plan to market your work, the returns for high test coverage and documentation are questionable. The few exceptions to this would be critical or regression-prone code paths. Now, in modern times, this may be less of an investment because you could offload most of this work to an AI agent or otherwise. However, if you are going to do that, you should really scrutinize the output because it might not be documenting things correctly or testing things effectively. It is unclear to me if it is even worth the tokens or time to review to autogenerate these sorts of things for solo side projects. What is clear to me is that incorrect documentation or tests are worse than having no documentation or tests.

As far as visibility goes, what I have come to assume is that if people are going to look at your project, they will have to be led to it via some sort of marketing on social media. If you want to capture your audience, you will want a strong README with flashy images to draw their attention. I have never taken the effort to market my work, so these assumptions may be incorrect.

Industry standards are highly contextual

In this case, I made a classic junior to mid-level engineering mistake. That is adopting the mindset that the industry standard is the best practice for all use cases. Since then, I have learned that industry standards are guidelines contextualized for engineering with many active engineers working on the same project, or for domain-specific reasons.

Take, for instance, this project. I intentionally broke apart a monorepo into a series of small, targeted repositories without considering the tradeoff. In this case, I lost valuable time breaking apart a monorepo and incurred the cost of managing dependencies without getting observable benefits. Here are some cases that could have made the investment valuable, and why it remains a sunken cost.

Community involvement

I could have built a community around something like iroshell. This would have the benefit of many engineers helping me with my framework. This did not happen because I did not take an active effort to build a community. In a single instance, a developer reached out with an interest in a single part of the framework, and I chose not to engage with that engineer. If I had been truly interested in a community, I would have missed a real opportunity there.

Code Reuse

Another possible benefit I could have gotten was reuse. However, this reuse benefit would have depended on doing more Java projects that are built on iroshell, java-core, or logrunner. Instead, I have not started another Java project since this project's completion. This means I missed out on all of the hard work I put into making iroshell reusable for other possible applications.

Observing the problem space has its benefits

At no point during this work did I ever stop to look around at what other people had been doing. Had I done so, I would have found that there was already a pretty big community around tools very similar to the one I created. For instance, Twine is a rather powerful tool that can do something very similar to what I created. I do not know that this would have deterred me from taking on this project. However, taking the time to read about the project could have given me a chance to see what features others like and some of the pain points. If I wanted to create a product that got traction, that was a missed opportunity to build for the community.

Not all features are worth the investment

There are a handful of features that were either premature or questionable. In particular, the “execute action” in the game engine itself is a feature that didn’t need to be made. I am still not sure what I thought the benefit would be. As it stands, its biggest purpose is a security risk. On the premature side, features like modding and libraries had been way too early. Without a community using the tool, there was no reason to create community tools like modding.

The highlights

So far, I have been very critical of all the work I have done. However, I would be remiss if I didn't point out the benefits. On the off chance a junior to mid-level developer finds the blog and reads it, I wouldn't want to give the impression that this was a completely wasted effort. While I may have some regrets about the features I chose to work on or the way I assumed the project would be received, it was a very valuable learning lesson, even from a techinical level.

Learning JavaFX

At the start of this project, my experience with Java UIs was small college projects using Swing. At work, I was learning about C# and WPF, but those lessons had not been fully transferable. In this case, I learned how to build a framework, a UI, and auto-generated games all on top of JavaFX. I really enjoyed learning about JavaFX's use of CSS and using tools like ScenicView. While I have yet to use this knowledge in a professional setting, it was great fun to learn.

Installers

While I do not think I nailed the implementation, I did get to learn the basics of installers and the pain points of creating one. Up until this point, the software I wrote either had a team working on an installer or was just a jar/exe that someone else had to figure out how to run for themselves. In the modern era, with web-first and electron apps, this may be an irrelevant skill, but it helped remind me that different users have different machines, which may or may not have some of your prerequisites for your software to run.

It still works

With fairly minimal effort, I was able to come back to this years later and get the thing running again. The biggest pain points I had were the kinds that I expected. Mostly setting up my machine to build Java again by installing OpenJDK and Maven, as well as configuring Maven to install from GitHub.

Project completion

This was the first project I really got to play every possible role. I had the idea, oversaw all the technical decisions, implemented the entire project, designed the UX, and even did some terrible icon artwork to top it all off. I got to see this thing from a silly drawing in a notebook to a completed project that can still be used today.

In this, I learned how to stick with something even when it gets hard or boring. There were days when I would get stuck on a technical issue that I couldn't find any great documentation online. On other days, I mind-numbingly drudged through just one more Javadoc comment.

I cannot overstate how valuable that was to me. I find that when working at a company with a clear goal in mind, there are trade-offs that must be made. As an engineer, those are not always to the benefit of your technical growth. You may want to spend time optimizing a routine or playing around with a new technology/framework/language. In many cases, doing this on the job is borderline negligent and not in the interest of the company you are working for. However, when you are working on a project that is yours without direct financial incentives, you are free to make whatever choices you want. The cost incurred, in money or in time, is on you, and you do not have to consider the cost to some company.

In the modern era of AI, I am not sure if you could recreate this experience. For example, a new developer who needed an icon in a specific way might not try to draw it themselves. Instead, they might just ask an AI to generate it for them. AI can provide some excellent rubber ducking, which was just not possible when I attempted this project.

Digging deeper

By now, I have covered an exhaustive history of this application, with topics across the board. Now I want to dive deeper into the technical design choices. I will have some follow-up blogs about specific libraries and some of the technical design choices I made. This will focus on how I see those decisions now.

Evaluating past projects

Fri, 02 Jan 2026 00:00:00 GMT

Growth through reflection

Lately, I have been busy with my day job, and the work I am doing on the side is not something I am ready to share publicly. This has left me wanting to share something, but unsure of what to share. After some consideration, I found that there is something I can write about without requiring the focus of a full public project.

Spending some time looking at the work I have done in my spare time, I realized I am missing a key part of learning. In much of my work, I focus on generating a product or prototype to learn something new. Throughout this process, I have learned a fair amount through repeated efforts and failures along the way. However, I rarely go back and look at what I have done years later to reflect on what I did right or wrong. As of writing, I now have 51 source repositories on GitHub, and I think it’s about time to start reflecting.

Upcoming content

I intend to create a collection of blog posts about my past public projects on GitHub. In these blog posts, I intend to cover the following topics.

Loosely, why I started the project.
The constraints I had on the project.
An overview of using the project in its current state.
A dive into how the project exists now. This would include topics such as code structure, architectural patterns, and other considerations.
A reflection on what I like and dislike about the state of the project.

Some of these projects are big and will need to be split across several posts. I also do not intend to post regularly about this, and expect that there may be some blog posts focused on what I am learning with the other project I am not currently sharing publicly.

Web visualization

Sat, 27 Sep 2025 00:00:00 GMT

Visualizing the web

So now we come to the end of our planned journey. If you have been following along so far, here is a quick recap. What started off as a plan to work on a small project in Neovim ended with me doing all sorts of extra work. I have covered how I built a web crawler, an administrative application, and a PageRank implementation. With this done, we can explore the final visualization work I did.

For this final stage, I created a standard search-based UI and another that was more interactive with visualizations.

Basic search

As a test, I pulled back out Rust and HTMX to build out a basic search page. This is a very simplified version of what you might have seen from Google, Yahoo!, or Bing. However, I decided to do one thing a bit differently. In this search page, I added some extra details.

With an inspection twist

While working with a crawler, I found that how pages connect to other pages and assets became an interest of mine. Due to this, I added a feature to see all links in a sidebar for a given page.

Another feature that I decided to create was an inspection of the page. I was already downloading all the HTML, JavaScript, and CSS, so I figured it would be neat to see what I could get from those assets.

For CSS, I got all the links I could find in the documents. Then I decided to list all of the classes, ids, and selectors used. I also decided to find all CSS functions that had been used.

In the case of JavaScript, I decided to look for all properties and functions that had been used on the window and document. I also decided to gather every string that was defined because those strings can contain URLs.

For HTML files, I started by grabbing all links. Then I pulled in all node types, attributes, ids, and classes used. Finally, I processed all inline JavaScript and CSS using the rules above.

This work did require a bit of additional parsing in Rust. My prior experience with ASTs and visitor patterns made this relatively easy to pick up on.

Interactive search

For the interactive search feature, I decided I would try something rather different. Originally, I wanted to create a large graph of all of the pages and their connections. However, I also really wanted to show how PageRank impacts the importance of pages. I didn't really think that would highlight the importance as much as I wanted. After some consideration, I landed on a visualization that was a collection of stars. Each star was the result of a search, and each star would have a size relative to its page rank.

Soon after I got that working, I decided to do a couple of screen transitions. When you clicked on a star, you could explore the star. Exploring the star would switch to a different view. This exploration would show the search result as the center of a solar system. This solar system would contain all of the pages it linked to. On that page, you could either follow a planet and see its links, or you could explore the planet. If you explored the planet, you would see all of the assets the page used and the relative footprint of the asset on the page. For example, if a page had 10 images, each 10KB in size, and one JavaScript file 1MB in size, you would see 11 elements with the JavaScript one being larger than the others.

Tooling for visualization

While I kept the Rust backend, I completely dropped HTMX from the equation. Instead, I did all of this visualization by drawing on a 2d canvas. Now, when I say I drew to a 2d canvas, I mean I did this only using the native browser API with no library on top of it. I decided to forego the use of any build step for this page. Instead, I just used the ESM import style of the document. I found it very nice to have the benefit of multiple JavaScript files and ESM import styles without needing a build step or any JavaScript libraries.

Building on canvas is different

When you start drawing directly on a canvas and building interactive features manually after doing typical web development, you start to see things other developers already know. I am sure most developers working on games, browsers, or operating systems already know everything I found in far better than I do.

Elements and Controls are an abstraction

You don't need HTML elements or common controls for interactivity. While you get benefits like accessibility for using the DOM correctly, you don't actually need a button element to make a button. Instead, you could draw a rectangle and put some text in the center. Then you can watch the mouse events and determine if the mouse is hovering over an element or pressing an element. In these cases, you can just draw a different state in the button.

Behaviors require consideration

I alluded to this before, but handling a hover or a click requires careful consideration of the current mouse position and state. The more complex the shape gets, the harder it is to determine if the mouse is within the element. Concepts that seem simple, like word wrapping or truncation, require careful measuring of the text and the size of its container. Most web developers take all of this for granted. You can easily apply CSS rules and not even think twice about what needs to be done to make a hover event trigger correctly on both a square and circular button.

Trigonometry is everywhere

While it could have been my application of the canvas, I found that I was constantly having to write code that required trigonometry. I kept finding cases where I would have to plot some visualization along some arc. I would constantly find myself pulling out trigonometry to create certain visualization effects.

After spending so much time doing traditional web development, it is easy to forget just how much math is done in the layers of abstraction you are building on.

Reflecting on the process

I found this to be radically different from what I had been doing for so long and had a blast doing it. However, I never quite accomplished the effect I was after. One aspect of this was that I didn't really have good images for the stars or the planets. I think with some better images or spritesheets I could produce a much better effect. I also didn't really consider adding the aspect of a camera with a viewpoint. I think adding this element would have allowed for some more interesting searching experiences when looking at some of the scenes.

That being said, while I had a lot of fun doing it, I doubt I would ever use a canvas for a commercial-grade project. While I learned a lot, using semantic HTML has benefits. The most notable being accessibility. I do not even begin to pretend to know how you could make a site that only uses a canvas usable for someone who couldn't see.

And so it ends

If you have read this blog and all the blogs before it, thank you. While I doubt I have provided any additional insights for anyone who may have read this, I found that some of what I was doing was a touch out of the norm. I had a lot of fun doing this, but there are other interests I have wanted to explore for some time now.

Forgive me for a moment while I get on my soapbox. I have not explicitly stated this up to now, but I would like to emphasize a choice I made during this project. During this project, I limited my use of AI to general research and as a springboard to more official documentation. I find that while using AI for your day job is a great idea, to truly learn new things it is beneficial to go back to the basics and avoid letting AI do everything for you. I would definitely encourage others who are writing code to learn new concepts to be diserning about how they are incorporating AI.

I do not know how long this project will be hosted; as such, I will finish by providing some images in case I take down the site due to cost, lack of interest, or something else.

Basic Search

Interactive Search

Figuring out Page Rank

Sun, 21 Sep 2025 00:00:00 GMT

Figuring out what is important

By now, we have built a functioning web crawler and even generated some administrative tooling to make crawling easier. Now the problem is we don't really know what data is important. This is not a new problem, and research has been conducted on this. The most notable research in this field would be Bringing Order to the Web. In this paper, the concept of a page ranking system was proposed. This is where my curiosity took me next, so I began to fumble through figuring it out.

A brief aside

This is the part where I am glad I am writing a series of blog posts despite the fact that I will likely be the only one to read them. Going into this blog entry, I assumed I had taken notes on implementing page rank. I also assumed I had implemented it correctly. Over the past few days, I found that I was very wrong on this, and I only found that out because I wanted to create this entry.

Bringing Order to the Web

Back in 1998, Sergey Brin and Larry Page wrote a paper on how to address this issue. Why I didn't read this on my first attempt to implement, I have no idea. This paper not only explains how to determine which pages are important, but it also covers some of the challenges of crawling the web, many of which I encountered. In this paper, they discuss how to approach the problem and elaborate on some applications of the algorithm in two different search engines.

The key insight behind the PageRank algorithm is that any given pages importance can be derived from the importance of all page's that link to it.

The difficulty with links

The primary focus of the algorithm is on so-called backlinks. The problem with backlinks is that it's impossible to know if you have all the backlinks to a page. Instead, you can only tell the forward links for a page. If you could effectively gather every page, then you could produce a proper graph with the correct backlinks. However, since the content on the internet is constantly changing, you can only ever work on a partial subset of the internet. Because of this, certain considerations had been made that could make the algorithm more effective.

The algorithm

I will spare you the formal notation of the algorithm; if you are interested in that, I suggest you read the paper. Let's walk through the algorithm. The first step is to build an adjacency matrix of all pages and their edges. In this paper, they suggested building an N by N matrix where all values are either 0 or 1. This bit would correlate to the existence of an edge. Also, in this matrix, the N value is all of the known pages on the internet. After this is done, we need to normalize the matrix such that each page is sending an equal weight to all of its links. Once this is done, we generate a random vector of size N as our starting point. We then repeatedly multiply the vector by the normalized adjacency matrix until it converges.

The property of convergence comes from the fact that this is an eigenvector problem. If you are more interested in the mathematics of this problem, I suggest reading more about eigenvectors and Markov chains.

Additional considerations

This paper explores additional considerations for this algorithm. The most notable ones are dangling references and the personalization vector.

Dangling references

One problem with this algorithm appears when a page has no outbound links. In this case, the page ends up being a bit of a rank sink. Generally, when this happens, it's not because the page doesn't link to anything, but rather that the page has not been processed yet. To manage this, the suggestion is to remove dangling references from the original calculation and then add them back in later with some initial value.

Personalization Vector

Another issue with search is contextualization. The example given in this paper is someone searching for "wolverine". The additional context here is that wolverine is or maybe was an administrative tool used by students at Michigan University. When the typical person searches for wolverine, they would expect to find more details about the animal. However, when a student at Michigan University does the same search, they might be interested in the tool. The suggestion proposed in this paper is that we could create personalization vectors that get added to the equation to make one of the search contexts more likely for the given person.

Looking at an example

Now that I have covered the algorithm, I find it's best to see it applied on a smaller scale. Below is an animation that shows how a small internet with 4 pages and 5 links would propagate rank. This only shows one transformation, and many more of the same operation would happen in practice.

Examining some implementations

I also find it helpful to see how others have implemented the solution. The following are a couple of implementations publicly shared on Github that I used to better understand how others have implemented this algorithm.

Memgraph

Memgraph is a graph-based database written in C++. It is common for graph databases to provide functionality like PageRank out of the box because it is a clear graph algorithm. While I will not cover its implementation, another popular graph database that provides this functionality would be Neo4j.

The majority of the interesting parts of the code can be found here. In this implementation, a few optimizations are added to the original algorithm. The first optimization is building a coordinate list instead of a square matrix of all nodes. So instead of having a data structure like this

[
    [0, 1, 0, 0],
    [1, 0, 0, 0],
    [1, 1, 0, 0],
    [0, 0, 1, 0]
]

we could instead have this

[[0, 1], [1, 0], [2, 0], [2, 1], [3, 2]]

The second optimization that was applied was the use of parallel execution. In the Memgraph implementation, each cycle of the vector by matrix multiplication step is broken up amongst the cores available for execution. Meaning the multiplication is split into batches. Those batches are executed in parallel and then finally merged back together in a final pass.

NetworkX

NetworkX is a Python library for handling network-related algorithms. This appears to align a bit more with the academic side of this topic. The implementation of the algorithm can be found here. This algorithm can run on SciPy or NumPy. My research mainly focused on the SciPy implementation. For the most part, this was very similar to the Memgraph implementation with some notable differences. One of the differences was that there was no explicit multithreading. All operations were done using matrix operations on SciPy classes. Since I have not investigated how SciPy handles aspects like the @ operator, it is entirely possible that under the covers, these matrix operations happen in parallel.

One thing I found interesting was how true to the paper this algorithm was in how it was expressed. It worked on matrix multiplication of adjacency graphs. The bulk of the operation can be summed up in this line of Python.

x = alpha * (x @ A + sum(x[is_dangling]) * dangling_weights) + (1 - alpha) * p

In this case, A is the adjacency matrix, but it is actually a coordinate graph in its storage. We will also notice that in this implementation, dangling nodes are handled, and they introduce the personalization vector p.

Trying it out myself

Even though modern search engines have moved far beyond just using a page rank algorithm, it was a lot of fun for me to explore this. This was one of a few times when I actually got to read a white paper. Since so much of this for me had been for learning, I actually created my own implementation of PageRank. My goal was not to be efficient but to fully understand it, so I even hand-rolled the matrix multiplication logic. I have been timing my implementation, and at the current scale I am testing with, the bottleneck is certainly the time to crawl all pages. Also, in my implementation, I do a crazy inefficient scan of pages to build the initial adjaceny matrix. I suspect I will optimize that as needed instead of reaching for one of these libraries to do the trick.

By now, we have covered basically all of the back-end processing and collection I planned on covering. Going forward, I will take some time to talk about some of the searching UI's and visualizations I put on top of this data.

Build admin site for crawler

Sun, 24 Aug 2025 00:00:00 GMT

So you want an intro?

If you have been following along by now, you know that I have been working on a search project and have covered what it took to make a crawler. If you are interested, stop now and read the "Indexing the web" blog post.

Building an admin app

By now, I had built a Python script that would crawl the web and put some data into a SQLite database. The problem was that the amount of command line arguments was growing. Also, I was often modifying the database directly between runs.

To deal with this, I setteled on creating a very basic administrative style application. This application would handle setting the domain policy, setting the download policy, and would show some insights on the data crawled so far. Eventually, this was extended to trigger runs of the crawling script. Since I have no plans of writing much about that, what ended up happening there was a minor refactor to the Python script to also run as a flask application which allowed you to trigger and view pervious runs of the crawler.

As was the theme with this project, I wanted to learn something new. I could have done something comfortable and built this web application in a front-end framework I use often, like React or Angular. However, instead of doing that, I decided it was time to give HTMX a go. Also on the back-end, I could have done the same using something like Node.js, Java, or .Net core, but instead I decided to pick up Rust again.

Picking a front-end framework

One thing I was paying close attention to while working on this project was the amount of JavaScript that was on the web. I knew that for a very boring administrative app with no major features, there would be no reason to have a lot of JavaScript, as the amount of interactive features would be low. I also have noticed that even "libraries", because some might get angry if I call it a framework, like React, seem to be growing in bundle size without end.

The sales pitch

I had heard the pitch for HTMX in the past. The following is a quick summary, as best as I could remember it. Back before AJAX and Web 2.0, we got along just fine. There was just one problem: fetching net new data required a full page load. To deal with this, a lot of work was done to optimize performance by effectively mangaging DNS, HTTP headers, and document size. After Web 2.0, we had a new shiny tool for handling this, and we used it for everything. Along the way, we lost sight of what hypertext media should have been. Basically, we started off strong with the anchor and form tags, but hypertext media failed to deliver more features. The web didn't have to be this way, so the creators of HTMX set out to make hypertext media what it should have been. Somewhat paradoxically, they did this by creating a JavaScript library. One might hope that if their project is successful that the hypertext media primatives that HTMX attempts to establish just become part of the browser implementation.

But what the heck does that even mean?

So, apparently, Web 1.0 and hypertext media are a good thing, but what does it mean to expand that feature set? I would summarize it as making better use of DOM attributes to control document behavior. Outside of loading a JavaScript file, most of what HTMX does is add new special attributes to DOM nodes. All of these attributes are prefixed with hx-. The types of things that these new attributes allow you to do are attach behaviors to DOM nodes. For example, you might want to say, when a user clicks a button, make a GET request and put the body in some other area. In a framework like Angular, this might look something like this

@Component({
   'selector': 'my-super-cool-search',
   'template': `
       <input type="text" [(ngModel])="searchText" [keyup.enter]="doSearch()" />
       <div>
         Resulting data: {{randomData()}}
       </div>
    `
})
export class MySuperCoolComponent {
    private httpClient: HttpClient = inject(HttpClient);
    public randomData = signal<string[]>();

    public searchText = signal<string>();

    public doSearch() {
        this.httpClient.get(`/api/so-cool/search?q={this.searchText()}`).subscribe(res => this.randomData.set(res));
    }
}

What we will notice is that it is just a small subset of the total code required to make this happen. There is a whole lot of setup and other components that may be required just to be able to use this component. Also, a compile step is required to do this very simple task. In the realm of HTMX, this might instead look something like this.

<html>
  <body>
    <input type="text" hx-get="search" name="q" hx-trigger="keyup[key=='Enter']" hx-target="#result-area" hx-swap="innerHtml" />
    <div id="result-area"></div>
    <script src="path/to/htmx.min.js"></script>
  </body>
</html>

As you can see, using HTMX basically makes all the JavaScript disappear and gets rid of a build setup. The downside is that it comes with the requirement that you learn a new set of attributes and their behavior. The one thing here that took me a while to fully come to terms with was the target and swap. The target is the DOM node that will be impacted by some action, and the swap is what will change. So, in this case, when you press enter you set the innerHtml of the result-area div with the contents from the API call.

But doesn't that change my back-end?

It is true that a front-end that uses HTMX is not a rip-and-replace framework. HTMX works a lot like how JSP and Ruby on Rails work. JSON is dead, and HTML is back. This means that in order to effectively use HTMX, you will either have to write your application with this in mind or create some sort of proxy API server or translation layer on your existing back-end.

I would argue that this comes with some benefits and some drawbacks. The benefit is in some ways, debugging the DOM gets a bit easier. No longer will you have to look at how the mess of JavaScript you have created to understand how it converts JSON into DOM nodes. On top of this, HTTP headers around cache control can be even more impactful. On the drawback side, one thing that JSON did was make it easier to have multiple clients for the same API. Going all in on HTML as a transport can impact non-web clients, for example, a mobile application that uses native rendering.

There are some weird things in the browser specification

One of HTMX's greatest strengths is also one of its weaknesses. One thing modern frameworks do very well is paper over some of the rough edges of the browser specification. While the browser specification is getting better, there are still some rough areas in there. In my case, the one that threw me for a loop was posting a form with a checkbox in it. As it turns out, I am not the only person who encountered this issue, and there is an issue that documents it well in HTMX here.

Jumping over to the back-end

I have used Rust in the past for a couple of things, but I have never used it for a web server. While I know it is often considered the "systems programming language," I figured there had to be a few more people out there who wanted to use it for a web server. After doing a bit of research,I stumbled on Actix, which appears to be a popular Rust framework for doing web development.

But why Rust?

As will always be the theme with this project, it was to learn more. While it is true I had used Rust in the past, it was never for anything of substantial scale. Since all of the previous uses I had for it had been so small, I hadn't had the opportunity to learn as much about it as I wanted. Also, strangely, while it can be a rather infuriating language to work with, I do quite enjoy using it.

Doubling up on front-end frameworks

To say I was just using HTMX is a bit of a lie. On the back-end I was also using the rendering framework tera to do some basic HTML templating before sending the HTML off to the client. Tera is a Jinja2 and Django-compliant templating framework. In hindsight, I am not sure that I really needed this framework as my template substitutions had been so small.

One issue I ran into pretty early on was that my templates had been cached. Turns out, when using tera you have to restart your server every time you make a change to any HTML template. This caused the turn around time to debug issues to be longer than I am accustom to. Also, since I had never really used Django, I found myself struggling with some of the syntax and concepts. For example, checking the length of an array being | length was something that took me some time to figure out.

Crates, crates everywhere

Package managers are a blessing and a curse. What I found is that Rust, in some ways, has the same issue as Node.js. Since there is a module system and it's very easy to use, there are a ton of modules of varying degrees of quality. Also, because there are so many modules you can kind of cheat on learning the language by cobbling together a ton of modules that solve the hard problems. I used quite a few modules on this project, and I don't think I could have done it all without some of them. That being said, I do think I would learn even more about the language if I would try to further limit my crate usage in the future.

Documentation is hard

I am not sure if this is a personal issue or if this is a general issue in Rust. In other languages, I have been pretty easily able to understand a majority of the concepts a package provides just by reading the documentation. Yes, I would inevitably pull open the source to understand a few things, but that was rare. However, in Rust, I constantly found myself looking at the documentation, getting confused, trying something out, reading the documentation again, just to ultimately end up looking at the source code to figure out what the heck is going on.

I am not sure if others in Rust have found the same thing or if I just haven't figured out the art of reading the Rust documentation sites. An alternative possibility is that I do not have enough experience in the ecosystem to know that I am choosing some questionable crates. Regardless of the reason this was, this was the single thing that I found to be the hardest about working in Rust.

Now what about Docker?

Towards the later stages of what I was working on, I found it a bit annoying to have to start three apps and maintain them locally. Because of this, I switched over to using docker-compose to run my applications. However, in order to use docker-compose, you have to put your applications in a docker container. Doing this took a bit more time than I was used to. In many cases, I would just extend from some base language specific image like Go or Python and layer on the stuff I needed. In the case of Rust, I found this harder to do. In the end, I had to use two FROM statements, which I didn't even realize was an option. The first FROM would pull a Rust builder and build the asset. In the second FROM it would actually create the final image with the runtime and the program.

Looking forward

By this point, we have created a functional crawler in Python and an app to administer it using HTMX and Actix. This is a good start, but at this point, there is nothing to really play with. All of the data is in tables on a database, and there are no searching or visualization methods for this data. While we could talk about that next, we are going to take a breif detour to talk about page rank in the next post.

Indexing the web

Fri, 22 Aug 2025 00:00:00 GMT

Taking stock

If you are just jumping into this blog post, I would highly recommend reading the "Journey into search" blog I posted before this one. This helps set the tone for what I am talking about here. If you don't have time for that and want a quick recap, here it is. Basically, I ended up starting a "little" side project to get used to using a terminal-based editor and fell down the deep rabbit hole that is search. Moving forward, I am digging into the first topic, indexing the web.

Indexing the web

The internet is full of interesting data; you just have to go out and find it. Now, Most of us don't think much about that because we let companies with lots of money find the interesting data for us. This turns into companies like Microsoft, Google, Meta, and at one point in time, Yahoo and AltaVista having a lot of indexed data. Now, I am sure most are probably aware of that, but what does it actually take to build and maintain these indexes?

This became a decent part of what I spent my time doing when working on my little search project.

Compiling assumptions

Often, when going after projects like this one, I start by setting up some assumptions. I like to see how far off my assumptions are from the ground truth.

Setting the scope

Before listing the assumptions, the following are some basic plans going into this project. I was going to write a web crawler. This web crawler was going to be written in Python. I chose Python because I had used beautifulsoup4 in the past and found it rather effective for parsing html documents. Then I would use SQLite to store the data because that would be the fastest option to bootstrap this.

Assumptions going in

SQLite will be insufficient for the volume and scale of data.
Most time will be spent waiting on HTTP responses.
5% or less of the traffic I will encounter will be SPAs.
The Average main asset (HTML file) will be between 15 and 50kb.
The initial attempt will miss many links because other linking strategies than anchor tags will be used.
99% of sites will have more than 1kb of Javascript.
The Average size of css will exceed 3kb.
The total execution time will be less than a day to start.
The total db size will be over 1gb on disk (.db file).
The total number of unique domains will be between 100 and 500.
The most linked site will be Wikipedia.
90% of all pages will use compression.

Constraining the problem

Now I knew going into this that I was just going to run a Python program on my local computer. I also knew there was way too much data on the internet to possibly get a program to complete. Because of this, I had to constrain the problem a bit. What I landed on was that I would point my crawler at a domain and then restrict the crawler activity to only that domain.

Since many of my assumptions had not been limited to HTML, I also knew that I would have to consider all types of assets, not just HTML pages.

To deal with this, I landed on using different policies for different domains. Since these metrics by themselves are only interesting if you can get a couple of different domains involved, the domains would either be crawled, read, or blocked.

Site selection

Now it was time to start writing some code and running some tests. The problem was I wasn't really sure what to crawl. I knew if I wanted diverse content, I would want to crawl Wikipedia, but there was no way I was going to be able to crawl all of Wikipedia. Since I lack originality and spend a good amount of my time listening to podcasts about software, I figured that would be a good place to start.

A majority of the podcasts I listen to often have their own websites, and I figured that would be a relatively smaller dataset. I eventually came to find out that not all podcasts are treated equally, and some have a shocking amount of content. While I indexed a handful of sites in my testing, I will primarily focus on my findings when indexing the changelog.

And so the learning begins

To get started, I figured I would focus on crawling the changelog; this ended up being a mistake. Going into this, I didn't realize just how much content was on the changelog. I also didn't realize just how many different linking strategies they used and just how many domains they owned. Let's take a journey through all the dumb mistakes and self-inflicted challenges I had during this time.

Href be crazy

Even if you just constrain links to href attributes on an anchor tag, it turns out that one string has a ton of possibilities. If you get this wrong, you might hit a bunch of dead links because you forgot that hrefs can be relative or absolute. Also, you may forget that you can link to a different part of a page using a # in the link and either fail to process that page, or even worse, process the same page a ton of times.

Assets are hiding everywhere

Given the original assumptions, I needed to figure out things like the total JavaScript used. One way to measure this would be to just look at the script tags with a source and measure those. However, that wouldn't be very accurate because JavaScript on a page might be coming from an external source or embedded in the document itself. For example, you could have a script tag with a body or with a src attribute. Likewise, you have similar problems with CSS.

To deal with this, you end up having to consider all sorts of fun ways that JavaScript and CSS can get loaded and used on a page.

Now this could have been the end but instead I started to think about the other supporting assets. What about the images, JSON files, XML files, and audio files? I might as well download and inspect those too, right? In the end, I think the only file types I didn't actively look for were video files. To support all of this content type, I ended up having to do massive refactors to my program, but it was worth it or at least that is what I kept telling myself.

You wanted that data compressed?

Just because a site supports compression doesn't mean you are going to get compressed content. One of the things I was noticing while looking at my data was that the bytes just didn't add up. I would go to a site in a browser and use the network inspector, and come up with a completely different number than my crawler. For some reason, my crawler had always downloaded more bytes. How could that be? Turns out that those HTTP headers you often forgot about are very important. Also, Python's default HTTP client doesn't help you fall into the pit of success. It is on you to make sure you set that Accept-Encoding header and also handle any encodings you claim to handle. From what I have seen, the most popular types of encoding are brotli and gzip, so I added support for those, and the numbers started tying out.

Did I forget to mention performance?

What I have neglected to tell you is that through all of these learnings, my crawler was crashing after very long runs. You know how I mentioned the dataset was large, well, that large dataset was causing me to crash a lot, but very slowly. I would start up a run, do something else, and come back to a crashed app hours later. Then I would have to play detective to see what went wrong. Most of the time, it was silly design decisions I made.

You don't always have to query small data

While this did not end up being a substantial issue, I quickly found out that I was constantly reading the same data from the SQLite database. This data was largely asking the question "Have I seen this URL yet?". In these sorts of crawl patterns, you will often end up finding links back to sites you have already processed. Because of this, I ended up adding a little cache of previously processed URLs and saw a small speed boost.

Massive columns are bad

I knew this was a bad idea going into the first database design; I just didn't know how bad. In the first version of the database schema, I dumped the entire site HTML into a single column as a raw string. No compression, no special treatment. This was causing me to get databases as big as 3GB before crashing. Doing the sensible thing and saving that content to a file and having a column that stored the file path made a massive diffrence.

Writes don't always have to be small

By this point, I had made some notable improvements to the performance, but things had still been way slower than I wanted them to be. One thing I was not paying attention to initially was just how often I was writing the database. Turns out that when you are constantly writing to the database as you process each page, you have a lot of small writes. Doing some batching of these writes made the process much quicker.

Absurd memory usage

By this point, I had solved many of the obvious database issues, but now a new issue was starting to crop up. After running for a while, the process would start to slow down a lot, and the memory was growing. At one point, I got the process up to 30GB of RAM in use before the program crashed. At first, I noticed that I wasn't quite using the http client correctly, but even addressing that was still causing an absurd amount of memory to be used. After figuring out how to profile in Python, I found that almost all of the memory was being used up by beautifulsoup4.

The process I had been using up to this point was a very naive approach. First, I would get a page and process it. Then I would find all of its links. Then I would repeat the process on the newly found pages that had not already been processed. These pages would be classes that had a property for the beautifulsoup4 content. You could imagine the code as something like the following

def crawl(pending_links):
    new_links = []
    for link in pending_links
        # Process link
        # Add to new_links
    
    if len(new_links) > 0:
        crawl(new_links)

As you may be able to imagine, after a certain point, you get very large lists of pages to process that all have to be completed before getting reclaimed. What I ended up doing to address this was as simple as doing an iteration by popping off a list instead of doing a foreach style enumeration. Creating something closer to this.

def crawl(pending_links):
    new_links = []
    while len(pending_links) > 0:
        link = pending_links.pop()
        # Process link
        # Add to new_links

    if len(new_links) > 0:
        crawl(new_links)

Python has threads

By this point, I had finally had a successful run, but processing one of the sites I wanted to ended up taking 4 days to complete. This was not optimal. Up until this point, I was doing everything in a single thread on a single process. As you can imagine, reading all of this data this way takes a very long time. In order to speed this process up, I started using Python's thread pool feature. This made things much faster, but I found Pythons memory management cross-thread/process to be quite frustrating to deal with.

Evaluating the results

At this point, I had crawled a handful of sites and had enough data to do some evaluations based on my original considerations.

SQLite is solid

Outside of mistakes of my own making with the database, I never really ran into a case where SQLite was an insufficient choice. Eventually, I added support for Postgres as well, but that was mostly for hosting reasons later on. By the end, with the largest amount of data I had crawled, the biggest my database size got to was 260mb.

This is a network-bound problem

Not so surprisingly, the biggest bottleneck in the end was network time. The amount of time spent waiting for HTTP responses was overwhelmingly larger than any processing time or database time.

There might not have been enough variance in my testing

Most, if not all, of my crawling was on podcast websites. The limit diversity even more, almost all of these had been tech podcasts. As such, I found that the data was pretty similar. During this time, I didn't encounter a single SPA. I also did not find a single site that didn't support some form of compression.

I was way off on the domains

Throughout my attempts, I encountered thousands of domains, not just hundreds of domains. That being said, Wikipedia was linked quite a few times, so maybe I get partial credit.

Asset sizes

I had assumed that most sites, especially sites that are not SPAs, would try to limit the amount of JavaScript, HTML, and CSS used. What I ended up finding was what appeared to be quite the opposite to me. The average JavaScript used on the pages I crawled was nearly 1mb, and the average CSS was 150kb. Compare this with the average HTML size, which was only 12Kb. It really does seem like JavaScript has eaten the world and a large chunk of compute on consumer devices.

Additional considerations

As I was going through this and expanding my crawls, I started to run into quite a few sites that have made crawling much harder. I assume this is largely due to AI crawlers. Also, much of the time, I couldn't help but notice that people are constantly visiting websites and processing this same data on their local hardware. This made me realize that if I were a browser vendor like Google with Chrome, I might be tempted to send "telemetry" data to help me build my index on someone else's hardware. Hopefully, none of the browsers do that, but I can understand the temptation.

Moving on

If you made it this far, maybe you are invested enough to read more about my silly little project. The next topic I plan on writing about would be the admin app that I created using Actix and htmx.

Journey into search

Thu, 14 Aug 2025 00:00:00 GMT

Falling down the rabbit hole

When I started all of this madness, I had two very simple goals in mind. First, I was going to start using a terminal-based editor. The second was that I was going to crawl a couple of pages and measure the time it took. Now, some two years later, I have a small three service system that can be run locally or in AWS. It turns out that once you start crawling the web, you are reminded that the Memex is vast and there are links in every content type imaginable. Also, you start to realize that you end up with a fair amount of semi-structured data that can be fun to play around with.

A warning to the reader

At this point, I find it relevant to warn the reader that I am not an expert in search engines or crawlers. I am just some dude playing around, having a good time, and sharing my findings.

Original scope

As I mentioned before, my original scope for this "little" project was to create a web crawler. This crawler was going to take an entry point and restrict its crawling to a single domain. While doing that, I would collect the time it took to load the assets from the network and parse those assets using beautifulsoup4.

Also, since I don't use Python a whole lot, it was going to be a chance to dig a little deeper on something I touch every once in a while.

Since I knew what I was going to collect, I took a bit of time before writing any code and made some assumptions about what I might find. I will go over these assumptions and how the crawler evolved in a later blog post.

And so the scope expands

Once I had run a couple of tests and built up a small index or two of the websites I was interested in, I hit my first scope expansion. I decided it would be nice to save a couple of the queries I was running into an admin style web app that would show a handful of specific performance metrics I cared about.

Now I could have done something sensible and written this application in something I was familiar with, but instead I decided to do something completely different. There was this semi-new quasi-front-end framework that had recently hit the spotlight, and I wanted to check it out. Apparently, picking up htmx wasn't enough; no, I had to go all out and write my backend in Rust.

Now with this new found admin app in hand, I started to find other things I wanted. I started to consider things like, "wouldn't it be nice if I controlled what kinds of assets get downloaded?", and "I really should be able to crawl and manage multiple domains". This led to a slew of features to control the crawler.

But hold on, what about UX?

Armed with data and hours of playing around with crawlers, I decided it was time to do something else. Those more inclined by profits might suggest figuring out how to train a neural network, but apparently, that is not for me. You see, I prefer the simpler things in life. Things like visualizing data into a planetary map.

I was about to set out to build an interactive canvas that would visualize a search as a collection of solar entities and make those different sizes based on the entities' perceived importance. This led me to realize that I needed to change my crawler again. I ended up taking another detour back into the crawler to build some edges between URLs and stored those in an edge table. Then, since the new goal was to compare the importance of the URLs, I took another side quest to learn how to implement a crude version of PageRank.

Now that my crawler and database finally matched my aspirations, I could finally set out on building this interactive canvas. In this process, I was reminded that there are a lot of things you learn to take for granted when doing UI in a framework of any kind. All of this I hope to cover in yet another blog post.

Why not just try a typical search page?

Around this time, I started to think, why not just build a typical search page? I have some data, and there is no way it will be as hard as building an interactive canvas. So I pulled back out my htmx and rust and got back to work.

As is the theme with this whole adventure, I decide well, that is good and fine, but maybe I should add a twist. Since the fascination I was finding was with how data was connected and general metadata, I decided it would be neat to interpret some of this data. In this search page, I would have additional options to see what the page links to and general information about the web primitives the page used without visting it.

This, of course, lead to figuring out how to parse html, javascript, and css in rust. Which, as you can imagine, was another adventure.

What more could you possibly want to do?

Now with three services and two different UI's it was time to make the finishing touch, deploying the thing. Once again, this came with its own set of challenges, most of which came from my choice to use rust.

After cutting my teeth for a while, mucking with cdk and rewriting parts of my service to optionally use s3 as a storage layer, I finally had something that could be deployed by anyone brave enough to take a stranger's CDK project and deploy it into their own AWS account.

Still ideas persist

Admittedly, I do not know what it was about this project that became so consuming. Many times, while I was working on this, I would think of something new I wanted to learn about, but I wouldn't be able to pull myself to do that until I "finished" whatever this was becoming.

What is maybe even more shocking for me is that I still have a long list of things I didn't yet do that I would potentially like to one day do. These are some of the things I have still been fighting the urge to act on.

Vector search

Currently, the search behavior of my site is pretty primitive and rather crappy. While apparently no one uses search anymore, those who do expect a certain amount of flexibility in their textual query. As I understand, most deal with this by using a vector search approach. This being done by generating embeddings on the content and the user input, then you follow that up with some sweet cosine similarity magic and use that to pop out better results.

Robots.txt

I suspect I should feel some shame that my crawler does not respect robots.txt. However, I take solice in the fact that I am not running an entire datacenter of crawl compute and network IO. That being said, it would be nice for my crawler to be a better actor in the crawling space.

Why not a shared library?

While I have no good reason to do it, I have considered making parts of my crawler a shareable Python module. I don't really think the world needs yet another crawler, but it would be a nice opportunity to refactor some of my code.

Sub Indexes

One concept I had considered was the idea of being able to have sub-indices of the main index. Since I have no interest in gathering user data to create more relevant queries, the use of different indexes could fill that void. These indexes would be a way for users of the system to define a more constrained index, so they would just be searching the things they care about. This would be done by having the user choose to either blacklist or whitelist domains of interest to them. These indices could then be shared between users of the system.

Snow Crash Librarian

In the novel Snow Crash, there was an interesting bit in which the main protagonist started working with a piece of software called the librarian. This software would find information for the protagonist and summarize the information he has gathered. While not a profound realization, I realize that now with modern LLMs, you could create something quite similar. This would be done by first doing a search query to "gather pages". Then you could present this in a UI and have an interface with an LLM summarize the search results in an interactive dialog.

What's next?

While I am not sure if I will continue to let this project take up my free time, I do plan on at least wrapping up part of what I set out to do. That being write a series of blog posts about the lessons I have learned along the way. This will start with a blog about crawling the web and building an index. After that, I am considering writing about the following topics

htmx and actix
page rank
using a 2d canvas

If any of the additional ideas end up getting done, I may very well write about those topics as well. If any of this work sounds interesting, feel free to check out the code on Github. Alternatively, if you just want to see what I have built for the time being, I am hosting it here. I make no guarantees for how long I will keep it up. Also, the index will mostly be based on random software engineering podcasts and other similar resources.

IDE shake-up

Mon, 17 Jun 2024 00:00:00 GMT

What's Old is New (to me)

It’s been a while since I’ve altered my development workflow, but I decided it was time for a change. Latetly I have noticed constant switching between keyboard and mouse can cause me some wrist pain. To try and reduce the possibility of future issues I decided it was time to give some other setup a try.

My Usual Workflow

My journey with coding environments has evolved over the years. I started with Windows Notepad and the default terminal for compiling. Then, I discovered Notepad++ in high school, which I used until college. In college, we primarily used Java, and it was time to upgrade to a fully-fleged IDE. In the war of Netbeans vs Eclipse, I landed in the Eclipse camp.

After college, I joined a .NET shop and switched to Visual Studio, incorporating PowerShell into my toolkit. Later, when I moved into web development, I found Visual Studio and Eclipse lacking for JavaScript, HTML, and CSS, so I tried Sublime Text. It didn’t fully click, leading me to Atom, which I enjoyed until I discovered VSCode. Despite initial skepticism, I’ve been hooked on VSCode for years.

Choosing right editor

Armed with a "simple" program idea, I set out to find a terminal-based editor. From experience, I knew Nano was not going to work. This left either emacs or some variant of vi. Since I have seen more community representation in vi-based editors I decided this was the route to go down given my experience with poor community support (Aurelia). Immediately I knew that vi would probably not be the right choice because it was not as rich of an environment as vim and Neovim. Ultimately, I landed on Neovim because it appears to be the latest incarnation of vi.

Setting Expectations

As I set out on my journey to use a new editor I did what I usually do in these situations, and laid out some expectations. In this, I assumed it would be slower at first to edit in neovim than just using the tools I had been using. I also assumed that figuring out the keybindings would take a while. Another thing I assumed was that pretty early on I would have to add plugins and might even have to use tmux. In the end, I assumed I would enjoy the result and might even use it full-time.

Getting started

I began on an Ubuntu machine and installed Neovim with apt-get. The default settings weren’t suitable, so I adjusted the numbering scheme and color settings. Then it came time for the hard part, sifting through all too many resources to figure out what plugins I needed and how to set them up.

Configuring Neovim

The first thing I found was that I just wanted to set some editor properties and move on with my coding; However, it appeared there were two ways to set configuration. You can either use a vim.init file in your config space or write some Lua code to configure your editor. Most online tutorials had been all in on Lua but I just wanted something quick, simple, and easy for me to understand. For this, I decided to proceed with the vim.init pattern.

First Round of Plugins

After getting Neovim set up the way I wanted, I installed my first plugins: airline and nerdtree. I installed Airline because it helped me distinguish between edit and view modes. As for NerdTree, I figured all IDEs have a project tree so of course I would need that right?

Keybindings don't have to have a modifier?

Understanding Neovim’s keybindings took time. Unlike traditional editors, Neovim’s different modes eliminate the need for complex keybindings. For instance, jumping to the next word or end of the line doesn’t require a modifier key. This approach reduced finger strain and improved my workflow. The Vim Cheat Sheet was invaluable during this period.

The hardest part was memorizing the cut command. While copying (yank) was intuitive, using d for cut/delete took some getting used to.

Spoiled by Sensible Defaults

Through this experience I learned having sensible formatting etc defaults baked into the editor is something I take for granted. In the past I had seen defaults break down a bit when writing javascript in Visual Studio, at least circa 2015, or Eclipse. However, normally these standard IDEs handled the default formatting for the language they were designed for very well. VSCode took that to the next level by having different default formats for different languages and even detecting different tools that you should install based on the language you are writing in.

The behavior of VSCode reduces the cognitive load for developers who just want sensible defaults and don't want to spend a lot of time picking out their tools. At times this was something I missed. As it turns out, I do not always enjoy going down a rabbit hole to find a suitable option for the task I am working on.

Finding the Right Plugins

Pretty early on it was clear to me that the original set of plugins I installed wasn't cutting it. Neovim is great but at its core, it is a text editor that gets better with plugins. By default, Neovim had limited or no autocomplete, but with many modern IDEs, it is something you expect. After looking around a bit I found CoC (Conquerer of Completion). This tool worked fine but I never fully enjoyed using it. It was a bit weird to have to deal with a plugin system in one of my plugins. Also, it was a bit strange that one of my plugins was built on Nodejs requiring specific node versions to be present. In the end, this ended up being the right tool to unblock me but I still cannot help but wonder if there is a better way to handle this problem.

As it turns out I like to be able to fuzzy search, specifically for files. I realized that I never interact with my tree structure and exclusively navigate around by fuzzy finding the files. While there might have been a way to get this with CoC I was not finding it. I ended up finding telescope but it didn't seem like it was able to be used on the stable version of Neovim. After some fussing around with versions I ended up dropping that attempt and landed on fzf. This is hands down my favorite plugin I ended up installing. It did exactly what I wanted and didn't involve any special setup on my end.

Now do it again

During this experiment, I ended up getting a new laptop. This put me at having to recreate all of this configuration on a new machine with another operating system. The initial setup was pretty easy but getting all the plugins and init.vim setup correctly was a bit of a pain. Some of this was a difference in reserved keys between operating systems, but a majority of it was recreating the config by hand. I can see the reason why many devs who run with this setup full-time end up putting all of their dotfiles or lua code in a Github repository. While this didn't take an insane amount of time, I could see that repository saving a couple of hours for every new setup.

Enter tmux

As I predicted eventually the time came when I needed to use tmux. For the longest time, I was just working on a Python process that had a slow feedback loop. In these cases opening Neovim, putting it in the background, running the process, waiting for the result, and switching back was not that big of a pain point. However, once I layered in a web interface it became clear that rapid iteration in this fashion was just not fast enough. The ceremony of control/alt+z, run process, control/alt+c, and fg repeat became too much. I was just altering static assets so the run process didn't need to keep being kicked off. Switching to tmux and having both the process run and editor visible at the same time was a big productivity win for me.

What I learned

While I enjoyed working in this way I do not see myself making this my full-time setup. Having to keep on top of what plugins I want for commonly supported IDE workflows does not seem to be how I enjoy spending my time. During this process, I found out which editor features are important to me, if my IDE has decent autocomplete, inline signature help, and fuzzy file finding I am pretty happy. I was a bit surprised to find that I do not use the file tree view much if at all.

Turns out that working on a laptop without access to a mouse is far more enjoyable in Neovim. When working on a laptop in another IDE I am reminded how much I struggle to use a touchpad effectively, working in Neovim gets rid of this classification of issue for me.

As is often the case with my side projects I found that my "simple" program expanded into something far more complicated. This program ended up loosely resembling a search engine. This also ended up using Python, Rust, and standard web primitives (Javascript, CSS, HTML) which led to more and more autocompletion flows I needed to consider. Since I ended up enjoying the semi search engine project I will probably add some additional blog entries about that in the future.

LLMs and TDD

Sat, 04 Nov 2023 00:00:00 GMT

LLMs and TDD

With all the undeniable hype around LLMs and more specifically ChatGPT I decided it was time to give it a fair shot. For the last year or so almost everyone has been talking about how it is going to revolutionize how many of us do our jobs and completely change how software engineers write software. It has also been suggested that those of us who choose not to use the technology will be left in the dust by those who choose to use it.

This sort of hype is so common in our industry that I decided to give it a while before even attempting to use it. When I finally got around to using it at first I was impressed. Visually its presentation style is quite stunning and it provides very promising-looking results to the questions you give it. However, on further inspection, my initial take was it constantly makes mistakes and presents them in a way that sounds factual. Most of my common use cases for a tool like this would be to replace searching documentation on a third-party library I depend on, but alas it would often make up interfaces and function calls for the tool that didn't exist. After this first exposure, I decided to put the tool down and move on.

Over the past several months people in the industry kept singing its praises. This got me thinking maybe I wasn't using the tool the right way and if I could just find the right way to work with it I would become even more productive and might be able to accomplish things I would otherwise pass on due to difficulty.

Understanding how others use it

The first step I figured was to see how other people had been using it and where they seemed to find value with the tool. Through this, I found that the most common use cases had been the following themes

Document this code for me because it's hard to understand.
Write tests for the code I have already authored.
Write my code for me as I describe what it should do.
Refactor some existing code to make it more readable.

Of these cases, the one case I found fascinating was the Write my code for me as I describe what it should do. This seems like an interesting replacement for pair programming in an industry where that sort of activity is becoming a bit rarer. However, most of the other cases just didn't work for me. I suspect it might just be me being stubborn or maybe I have subscribed too much to the dogma of people like Robert C. Martin and Kent Beck but some of these cases seemed a bit irresponsible. I understand documentation is hard to write and tests can be tedious but these seem like part of the rigor that comes along with any Job containing the title "Engineer".

Thinking about how I would want to apply it

At this point, I decided to take some time and reflect. If I was going to use a tool like this how would I want to use it? Upon a fair amount of thinking I landed on the best way to use the tool and get the desired outcomes would be some sort of TDD loop. In this loop, I decided that the human would be in control of the inputs and outputs by writing the tests. This would in turn leave the implementation up to the AI.

At this point, I was starting to get excited and even thought a bit more about how transformative this kind of workflow could be. I even started to consider how it might start to draw parallels between other industries. For example, it's interesting that we design and build code as programmers but other industries like housing have architects that design the house but never have to do the construction.

Starting off simple

At first, I decided to start off simple. In this case, I was going to use a language I use a lot and try something rather trivial. In this case, I decided something trivial would be a simple partitioned log that is backed by localStorage. Going into this I had already built up a few assumptions

ChatGPT would not produce a working result faster
ChatGPT would not write documentation by default
ChatGPT would default to writing typescript when given a test with no types
ChatGPT would write more code than was required.

After this test was completed I found the following. Much to my surprise, ChatGPT did not assume typescript by default. Also since I was in such a rush to set a time record to beat I made a mistake that assumed you would always append a log before reading the log. I also found that ChatGPT did produce a result faster however I had already written the tests in advance so that might account for the time difference.

Expectation breakdowns

One thing I found rather quickly was that ChatGPT was not picking up on the expectations that my tests had been subtly implying. For example, in my test code, I was mocking out getItem and setItem in local storage. However, ChatGPT really wanted to index all items directly from local storage. For example instead of writing some code like this

const value = localStorage.getItem(partition);

it really wanted to do this

const value = localStorage[partition];

Another assumption I had built up was that if I just fed it the errors from execution it would be able to deduce what was wrong and at least attempt a change that would fix the failure. What I found instead was that often the change it made would actually produce more failures and might not even fix the originally failing test. In the end, I found that I had to directly state my expectations around why I thought its implementation was failing tests.

Result

In the end, we worked out a solution that got all tests passing but still wasn't quite what I wanted. Ideally, I wanted it to use generics for values on a log but due to how I wrote the tests string was sufficient. A bit tired from the conversation I gave up and accepted the result.

class LocalStorageLog {
    private logs: Record<string, string[]> = {};

    constructor() {}

    private getLogFromLocalStorage(partition: string): string[] {
        const logJSON = localStorage.getItem(partition);
        if (logJSON) {
            return JSON.parse(logJSON);
        }
        return [];
    }

    private updateLocalStorage(partition: string): void {
        const logArray = this.logs[partition];
        if (logArray) {
            const logJSON = JSON.stringify(logArray);
            localStorage.setItem(partition, logJSON);
        }
    }

    readLog(partition: string): string[] {
        if (!this.logs.hasOwnProperty(partition)) {
            this.logs[partition] = this.getLogFromLocalStorage(partition);
        }
        return this.logs[partition];
    }

    appendLog(partition: string, value: string): void {
        if (!this.logs.hasOwnProperty(partition)) {
            this.logs[partition] = this.getLogFromLocalStorage(partition);
        }
        this.logs[partition].push(value);
        this.updateLocalStorage(partition);
    }
}

export { LocalStorageLog };

The result of this test can be found here.

At this point, I was feeling a bit defeated. I had hoped for so much and found that the result was not what I envisioned. However, I had come up with a plan that involved multiple tests and I was determined to keep on trying.

Trying something a bit less comfortable

Moving on another test I really wanted to try was using a language I was less comfortable with. I suspected in this case maybe I would have fewer opinions and might even be able to learn something about the language I had not picked up on before. In this case, I decided to use rust and write a very similar bit of code. The twist was this time I would be using files to store data.

Observations about my own coding experience

What I found while trying to implement this first by myself was that it took me a significantly longer time to execute than the previous example. Another observation I made was that I ended up doing a lot of context-switching between my IDE and the standard library. In the end, I got a working result.

Handing it over to ChatGPT

Much in the same fashion as before I decided to then take my tests and hand them off to ChatGPT and see what I could do. In this case, I found some familiar issues but to my surprise in the end it produced working code without me having to explain directly how to fix the code. There however two new issues I did not encounter before. The first one was quite minor originally it failed to import something that was needed. The second issue for whatever reason struck a nerve with me. In this case, I was writing tests and wanted only the implementation written however ChatGPT took it upon itself to rewrite my tests. In this case, it was minor it decided that rust should be written in snake_case instead of camelCase, but I didn't like the idea of changing the tests this was supposed to be the human part of the job.

In the end, I will say it did produce a more correct result and even added some documentation I was not expecting.

// Non GPT comment had to manually add File import
use std::fs::{self, File};
use std::fs::OpenOptions;
use std::io::{self, Read, Write};
use std::path::Path;

struct LogManager {
    log_dir: String,
}

impl LogManager {
    fn new(log_dir: &str) -> Self {
        // Create the log directory if it doesn't exist
        fs::create_dir_all(log_dir).expect("Failed to create log directory");
        LogManager { log_dir: log_dir.to_string() }
    }

    fn appendLog(&self, log_name: &str, message: &str) {
        let log_path = format!("{}/{}", self.log_dir, log_name);
        let mut file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(&log_path)
            .expect("Failed to open log file for appending");

        // Append the message to the log file
        writeln!(file, "{}", message).expect("Failed to write to log file");
    }

    fn readLog(&self, log_name: &str) -> Vec<String> {
        let log_path = format!("{}/{}", self.log_dir, log_name);
        if let Ok(mut file) = File::open(&log_path) {
            let mut contents = String::new();
            file.read_to_string(&mut contents)
                .expect("Failed to read log file");
            contents.lines().map(String::from).collect()
        } else {
            vec![]
        }
    }
}

fn main() {

}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_empty_log() {
        let manager = LogManager::new("./unittestdir");
        assert_eq!(manager.readLog("empty"), Vec::<String>::new()); // Gave up and changed test
    }

    #[test]
    fn test_append_log() {
        let manager = LogManager::new("./unittestdir");
        manager.appendLog("append", "My new Message");
        assert_eq!(manager.readLog("append"), vec!["My new Message"]); // Gave up and changed test
    }

    #[test]
    fn test_append_many_log() {
        let manager = LogManager::new("./unittestdir");
        manager.appendLog("appendMany", "My new Message");
        manager.appendLog("appendMany", "My Message 2");
        assert_eq!(manager.readLog("appendMany"), vec!["My new Message", "My Message 2"]); // Gave up and changed test
    }

    #[test]
    fn test_file_creation() {
        let manager = LogManager::new("./unittestdir");
        manager.appendLog("creation", "My creation message");
        assert_eq!(Path::new("./unittestdir/creation").exists(), true);
        assert_eq!(fs::read_to_string("./unittestdir/creation").unwrap(), "My creation message\n");
    }
}

The result of this test can be found here.

Trying something a bit more complicated

With the previous tests out of the way, it was time to attempt my final test. In this case, I was going to give it something a bit more complex that would require implementing multiple files and a common type. This expanded on my first test but added an eventing system in front of the log on top of this we would also have both a local storage and filesystem backed option for this eventing system.

In this case, I had learned from my past experiences and was ready to do a couple of things differently. First I was not going to try to time myself on the initial implementation. I wanted to take it slow and write what I would have wanted to produce. Second, at this point, I was ready to have to be a bit more interactive with ChatGPT. Instead of just throwing failures at it I was ready to give it subtle hints to get to the end goal with my sanity intact.

What I found in this case was that again it took ChatGPT a lot longer than I had anticipated to complete this task and again it did in fact require a lot of direct suggestions to get a working result. In this case, since there were multiple files there was an opportunity to produce more typescript errors and it certainly did produce a fair amount of those which required me to debug the code and make suggestions on a type that would work. Most of the time during this exercise it felt like I was pair programming with a junior developer. That is not to say there is anything wrong with working with junior developers just I would rather use those opportunities to teach another engineer instead of fighting with a machine to just have to do it all over again in the future.

In the end, there were a couple of positives to this experience. Firstly I found that it exposed some assumptions I had about my filesystem-backed implementation that could have been better. I also found that the more files we worked on the better it got at producing what I wanted. For example, it produced the final file with no issue the first time.

The result of this test can be found here.

Unexpected surprise

In between these tests, I decided to play around with ChatGPT a bit and decided to flip this problem on its head. Instead, I would have ChatGPT write some tests and I would write the implementation for those tests. In the end, I found this to be a lot more enjoyable and saw this as a good opportunity to use the tool as more of a generalized hone-your-skills type tool similar to Robert C. Martin's idea of a code kata.

Conclusions

I understand that this is not the most conclusive test and that there are other tools I could have explored. I am sure I would have seen slightly different results using something like GPT-4 from openai or copilot from GitHub. However, after this experience, I was not really willing to invest money to see if I would get a better result.

In the end, many of the assumptions I had about the tool had been wrong in ways I didn't expect. However, instead of being completely disappointed I did find one workflow I quite enjoyed with this tool and it did not end up being a complete waste of time. ChatGPT may be useful for many but for the way I like to work with my tools, it doesn't quite cut it for me.

Book Note - The Mythical Man-Month

Wed, 09 Jun 2021 00:00:00 GMT

The Mythical Man-Month

The Mythical Man-Month is a book authored by Frederick Brooks.

High level overview

This book is a series of essays from Frederick Brooks. In these essays the problems of software engineering and management are explored. Much of this book is polled from Fredricks own experiences while working on IBM System/360.

Notes

The Tar Pit

This is a good introduction essay. In this chapter, Frederick does a great job distinguishing between software that works and production-grade software. The fact that you can get something that works together quickly often gives false hope for moving that to production. In this move, there is a lot of work to make the application stable for many users on many machines with few bugs in new features.

Another aspect of this chapter I found quite interesting was the description of why so many people enjoy building software and how that can become a trap. The comparison of programming and programming projects to a tar pit is a very interesting metaphor that I think often fits.

Original notes


A Program	A Programming System
A Programming Product	A Programming Systems Product

Large system programming is a tar pit and many powerful beasts have thrashed in it.
- many have made it out with running systems.
- less have met goals, schedules, and budgets.
- No one can escape this problem no matter how big or small the team is.
- Most are surprised how sticky it is
If this is such an issue why do small teams working in garages seem to be able to pull it off?
- A functioning product is easy to build but a hardened tested product is much harder.
- See chart above for next points.
- Generally speaking, these teams create something in the top left quadrant.
- To make a program more useful but more costly moving down or to the right adds 3 times the effort and expense
- A programming product
  - Has Generalization
  - Has Testing
  - Has Documentation
  - Is Maintained
  - can be run, tested, and extended by anybody.
  - Is usable in many different operating systems.
  - This means a ton of testing
- A programming system
  - Collection of interacting components.
  - coordinated in function and disciplined in format.
  - every input and output must conform to syntax and semantics with precisely defined interfaces.
  - must only use the prescribed amount of resources (RAM, CPU, etc)
  - must be tested with other components
- A Programming Systems Product
  - cost 9 times as much as a simple program
  - Is a mix of a programming product and a programming system.
Programming is fun
- This is partly due to the joy of making things in general.
- It is pleasurable to make something that works for other people.
- It is puzzle-like which can be quite fun.
- There is a joy to learning and programming is all about learning.
- Programming is not generally bound by physical mediums. It is pure thought. Very few forms of media are as flexible when it comes to creation.
Woes
- The implementation must be perfect. Computers are quite dumb and only handle simple instructions. As humans, we often lack the discipline to define something perfectly.
- The programmer rarely controls what they are working on. This often comes under the direction of a customer or third party.
- Depending on others programs can be painful. It is very annoying to find a system you depend on has a bug in it and its not meeting it's required specifications.
- Designing is fun but fixing bugs painstaking laborious work
- Debugging gets harder as more bugs are solved.
- In many cases, the product you have been laboring over is obsolete by the time you are ready to release it. This can lead to massive frustration.

The Mythical Man-Month

In this essay, Fredrick explores the complexity of getting a software project done on time and correctly. One major theme is it is very hard to estimate a software task. Currently, any estimation techniques we have are somewhat poor and these poor estimations make it hard for managers to give good timelines. One interesting concept that is brought up is that developer estimates are often optimistic. It is argued that software engineers are eternal optimists and this can lead to estimates being very off. Another issue with estimations is they are often ignored or pressured to meet a customer's demand. There is a call for managers with stronger backbones to avoid this sort of temptation.

Another topic that is talked about a lot is the trade-off between men and months. In many other industries if you add more people to a project it will get done faster. Software is a bit of an exception. In the case of software often adding more people will have no effect or a negative impact on timelines. Bringing more people onboard requires more communication from already productive developers.

As far as getting the product out the door there is a surprising amount of time spent doing things other than writing the code. It is argued that really coding is only one-sixth of the total effort. The rest of the effort is spent planning and testing.

In the last section of this chapter, there is some guidance provided on what you can try to do if a project is falling behind. Some advice includes push the task out, drop scope on the task, and if all else fails you can try to add heads but smartly. Adding more developers is a serious risk and should be done early in the process.

Original notes

Techniques of estimating are poorly developed
estimating techniques confuse effort with progress. (This hides the assumption that people and months are interchangeable)
since we are unsure of our estimates it makes software managers lives difficult
progress is poorly monitored.
when slipage is seen often the idea of adding more manpower is assumed to be a solution.
All programmers are optimists
- Often we assume things like "The time time this will surely run"
- Often we assume all will go well.
- Reference to The Mind of the Maker
- In many creative activities (not programming) there are physical limitations that can be expressed. The difficulty in these cases is often implementation
- Since programming is pure thought we expect very few issues in implementation because it is all thought matter.
- Since our ideas are faulty we end up with bugs. This invalidates our optimism.
Man-Month
- Cost varies as the number of men or months are tweaked. However, progress does not.
- A man-month is a bad unit for measuring the size of a job.
- Men and months are only interchangeable in tasks that do not require communication. Programming is not one of these tasks.
- If a task cannot be partitioned due to sequential blockers more effort will have no effect on the schedule.
- In the case that a task can be partitioned it often still requires a lot of communication. This adds to the amount of time required to complete the task.
- There is a burden to adding more developers
  - training, every new member must be brought up to speed and trained accordingly.
  - intercommunication, as you add more people there is even more communication that needs to happen.
- Often adding more developers lengthens instead of shortens a project.
System test
- Debugging of a component and testing of a system are very dependant on sequential constraints.
- Time required is dependant on the subtlety and number of errors found.
- Breakdown of a software task
  - 1/3 Planning
  - 1/6 coding
  - 1/4 component test and early systems test
  - 1/4 system test
- The time devoted to planning is larger than normal
- half of time schedule is debugging
- the easy part to estimate only makes up for 1/6th of the work.
- Failure to plan enough time for a system test is dangerous. This is often found out later into the project and the impact is usually severe.
Estimating
- It is easy to promise something that cannot be delivered. I can promise you an omelet in two minutes but when that time is up the customer must choose to wait or eat it raw.
- False scheduling to meet customer demands is far more common in this industry.
- We need to publicize productivity, bug incidents figures. The whole industry would benefit from this. (Maybe open source and github has helped with this)
- Managers need stronger backbones until our estimation techniques get better.
What to do when a task is falling behind
- You can add more heads but do it smartly. There are limits to how much and to what effect adding more heads will do.
- reschedule, push the task out.
- Trim the task, drop scope.
- Brooks Law simplified "Adding manpower to a late software project makes it later"

The Surgical Team

In this essay, an examination of how to structure software teams is done. In general, it seems that small teams work better. However, small teams cannot produce the large systems that are often required. To deal with this a collection of surgical teams can be created to have small effective teams while still producing large systems.

These surgical teams are made of surgeons, copilots, administrators, editors, secretaries, program clerks toolsmiths, testers, and language lawyers.

A surgeon in this context is your lead developer. The surgeon is responsible for designing code, writing documentation, and testing code. Generally, this person should have 10 years of experience. In a lot of ways, this role is similar to the master described in the clean coder book.

A copilot can do everything a surgeon can but does not quite have the same experience as the surgeon. The biggest part of this role is to work with the surgeon. In this role, the copilot can question the surgeon's choices but the surgeon always gets the final say in the end design. In many ways, this is like clean coders journeyman.

The administrator is the surgeon's boss and tends to handle non-technical aspects of work. These might include things like raises.

The editor primarily works on documentation. The editor will review and edit documentation as needed.

Secretaries in this context handle correspondence for the editor and the administrator.

The program clerk manages all records. This includes both human and machine-generated records. This role would have to aggregate things like developer documentation and build output information.

The toolsmith creates and maintains any specialized tools the surgeon may need.

The tester creates system tests and plans testing sequences.

Lastly, the language lawyer is a programming language expert. In this role, the language expert can help surgeons find ways to do things that might be difficult to accomplish in the language of choice.

Original notes

People favor small teams but small teams cannot produce large systems.
There are wide productivity differences between good and bad programmers. (I assume this is about the controversial 10x programmer)
There is no correlation between years of experience and productivity.
a small team should not exceed 10 people
Harlan Mills has a solution
- A segment of a large job should be tackled by a team.
- This team should be organized like a surgical team.
- In this model, each team member has a specific role to play.
- The Surgeon
  - Also known as the chief programmer.
  - defines functional and performance specifications.
  - designs code
  - tests code
  - writes documentation
  - should have 10 years experience (reminds me of clean coders master role)
- The copilot
  - able to do anything the surgeon can.
  - is less experienced than the surgeon.
  - main role in design is as a thinker, discussant, and evaluator.
  - the surgeon works with them to form ideas but is not bound to their suggestions.
  - They may write code but are not responsible for the code.
- The administrator
  - This is the surgeon's boss.
  - manages personal raises etc.
- The editor
  - This role reviews the documentation generated by the surgeon.
  - Provides references in documentation
  - criticizes and reworks the documentation.
- The secretaries
  - handle correspondence for the administrator and editor.
- The program clerk
  - Maintains all records of the team in a programming-product library
  - Is a secretary for but machine and human-readable files.
  - Makes all computer runs visible
  - aggregates program output/build information. (I suspect this role is largely un-needed now with advanced CI pipelines)
- The toolsmith
  - Creates and maintains special tools for the surgeon
  - each team has their own toolsmith
  - only produces tools wanted or needed by their surgeon.
- The tester
  - creates system tests from functional specs
  - Plans testing sequences
- The language lawyer
  - Understand the programming language in use very well.
  - Helps find ways to do things that are difficult in the language of choice.
  - Can aid many surgeons

Aristocracy, Democracy, and System Design

This chapter focused a lot on architecture implementation and consistent design. It is important to have consistency in design. To make design consistent sometimes it is important to know what features to omit to keep the design consistent. A large portion of this chapter focuses on the difference between architecture and implementation.

It is important to have architecture and implementation be separate concerns. Only by doing this can you make your system consistent. It is important to note that neither responsibility is any more creative than the other. Creativity is just on different dimensions.

An architect's primary responsibility is to define what happens. Architects are focused on the end-user and know how to make the system feel the same. To help keep the system consistent there should be few architects.

On the other hand, an implementor defines how it happens. While this might not sound like a creative process the creativity here lies in cost and performance optimizations. Sometimes implementors have good ideas, sadly if it is not consistent with the system it really should not be done.

Original notes

Conceptual integrity is the most important consideration in system design.
it's better to omit some features if it makes design ideas consistent.
Programming systems are intended to make a computer easy to use.
Ease of use is only enhanced if time gained in functional specification exceeds time lost in learning, remembering, and searching manuals.
Every part of a system should reflect the same philosophies and use the same techniques in syntax.
There are two ways to keep the design consistent with multiple people working on a project
- Careful division of labor between architecture and implementation.
- Structured teams discussed in the surgical team
Separation of architecture and implementation is a powerful tool for conceptual integrity.
Architect of the system
- job is to bring professional and technical knowledge to bear in the interest of the user. This is as opposed to the interested of the salesman
- must carefully be distinguished from implementation.
Architecture is what happens implementation is how it happens.
Just because implementers or users have good ideas does not mean they are the right ideas. It is more important to keep the system consistent and that is the job of the architect.
There should be few architects
Architecture work is not more creative than implementation. Just because you are setting the external boundaries does not mean you have made all the creative choices.
The cost-performance ratio is often where the implementors creativity lies.
Form can be liberating. Sometimes having additional constraints makes your work more creative. this is not just seen in software as it has been seen in other practices like art.
Common concerns for implementors that can have from an architecture team.
- Specifications can be too rich in function and not reflect practical cost considerations. This concern is valid.
- Architects will get all the creative fun. This is argued to not be true.
- They will have to sit by idly while waiting on specifications to arrive. Don't hire implementors until you have the specs.
Implementors can start as soon as a vague idea of the specification exists.
The Implementor has tons of work to do even before the specification exists.

The Second-System Effect

At some point in every software project comes a time in which a second system must be considered. In these cases, extreme caution must be made. Oftentimes additional function, extrapolation of function, and over-design are the issues the architect must battle. Another common issue is matching functionality correctly. It often happens that the second system omits some functionality that the first system had.

On top of these factors friction between the architect and the implementors can come up. The architect needs to remember that it is the implementor's job to deal with implementation. While this is true for some cases the architect must be ready to suggest a way to implement his design.

In general, the best practice is to have an architect who is experienced with second systems working on the second system. These architects should be able to spot the pitfalls before it is too late.

Original notes

When a cost is too high the software architect has two options
1. Cut the design
2. challenge the estimate by suggesting cheaper impelementations.
The second option gets in the way of the implementer doing their job correctly.
To do this right the architect must
- remember that the implementor has inventive and creative control over implementation.
- always be prepared to suggest a way of implementing what he specifies.
- deal with these suggestions
- be ready to forego credit for suggested improvements.
When an architect designs a system they often keep track of embellishments that could be used in a second system.
second systems are dangerous
- tendency to over-design the second system
- tends to change the system a bit. Fundamental aspects of the first system might be lost
- aspects of the old system can become obsolete.
when starting on a second system
- avoid functional ornamentation
- avoid extrapolation of functions
- discipline is key
If a second system is required consider having an architect who has already done two systems take on the job of the lead architect as they will have learned from their mistakes in their second system.

Passing the Word

Keeping a manual or good documentation is key to keeping everyone working on a software project in sync. Manuals should be consistent in style and they should have an appropriate level of precision.

Using implementation as a formal definition can be a useful tool. When this is done questions are answered unambiguously, a debate is never needed and the precision is always appropriate.

Another helpful tool for keeping everyone in sync is consistent meetings where changes get discussed and impacts to other components can be identified or known.

Original notes

a manual is a necessary tool
a manual is an external specification of the product.
manuals should not only include what a user sees but it should also refrain from describing everything a user does not see.
The style of a manual should be precise.
Writers of manuals must strain the language they use to create the level of precision required for a manual.
Merits of formal definitions
- They are precise
- They tend to be complete
Problems with formal definitions
- They lack comprehensibility
many tools exist for formal definitions
care should be made in distinguishing externals in a formal definition
formal definition is an implementation.
implementation can also serve as a formal definition
using implementation as a definition has these advantages
- All questions can be answered unambiguously with experimentation.
- debate is never needed
- answers are always a precise as one needs and are always correct
implementation may over-prescribe
In an unpoliced system side effects can occur.
implementation as formal definition can lead to confusion on if it is in fact standard.
meetings help keep everyone on board. This can prevent hardware and software specifications from drifting
telephone logs should be kept by architects.
- These logs record all questions and answers.
- each week this log is concatenated amongst all architects and sent out to implementors.

Why Did the Tower of Babel Fail?

Most of the time project failures are the result of people not communicating. This is especially easy to end up happening in larger organizations. What tends to happen in these cases is teams build up assumptions about other teams. These assumptions are not always true and can lead to failures to integrate.

There are some key tools to dealing with this. The first is identifying and informing teams of their intergroup dependencies. The second is using project meetings. Finally the third is to use a project workbook.

A project workbook is an imposed standard on documents. In this workbook things like objectives, external specifications, interface specifications, technical standards, internal specifications, and administrative memoranda should be included.

One last issue that tends to get in the way with large projects is tree organization structures. While these might be good for organizations they are terrible for communication. To work around this you can assign people roles to help with this cross tree issue.

Original notes

Many project failures occur because two groups of people in a large organization are not talking
- Each team often has assumptions that have not been verified about the other team.
How to communicate cross team
- Informally. Keeping everyone informed and having a clear understanding of intergroup dependencies helps.
- Meetings. Have project meetings in which team after team gives a technical briefing.
- Workbook. Create a formal workbook.
Project workbook
- It is a structure imposed on documents.
- All documents of a project should follow this structure
- Things that should be included: objectives, external specifications, interface specifications, technical standards, internal specifications, and administrative memoranda.
- This is important because technical prose lasts a long time.
- It is important to get the structure right so that memos can be turned into product-quality manuals.
- This also serves to control the distribution of information. This is to not restrict information but instead to get it into all the hands that need it.
- To keep every programmer in the loop every programmer should have a workbook and that workbook should be current.
- Maintenance of a physical workbook can get very difficult over a long period with many contributors.
  - In this example, they solved the problem with microfiche
  - Later goes on to reflect on how it might be done today (when the book was published several decades ago). In this, we imagine a direct access file that is LIFO and everyone can see it. No one could directly change the document but they could provide updates that later get aggregated and put back into this read-only LIFO document.
  - I imagine today something like wikis or confluence is serving a similar purpose.
Organization in a large programming project
- Tree-like structures are good for authority and responsibility but are terrible for communication.
- A tree-like programming organization must have these roles to be successful
  - a mission
  - a producer
  - a technical director or architect
  - a schedule
  - a division of labor
  - interface definitions
- What is a producer?
  - The produce assembles a team, divides work, and creates a schedule.
  - They require the necessary resources for the team.
  - A major role is communication outside of the team.
  - ensures the schedule is met.
- What is a technical director?
  - thinks up the design to be built
  - identifies parts of the design
  - specifies how things will look from the outside.
  - sketches the internal structure.
  - brings unity and conceptual integrity to the design.
  - limits technical complexity.
  - invents solutions or redesigns the system as problems arise.
- The producer and the technical director can be the same person.
  - Can often work on small teams but gets harder to maintain on larger teams.
  - A person with strong management and technical skills is super rare.
- In some cases, the producer can be the boss and the technical direction his right-hand man.
  - the producer can do subtle things to help the technical director to avoid power struggles. Examples include office size, carpet, furnishing, carbon copies, etc.
- Just like the producer can be the boss the roles could be inverted. In some cases, the technical director is the boss and the producer is the right-hand man.

Calling the Shot

This essay is mostly the evaluation of a lot of studies around estimation and effort of different programming tasks.

One of the key arguments here is that not all programming tasks are equal. Inevitably some programming is harder to do than other types of programming.

The other key argument is there is more than coding that goes into software development. This means that estimates should not only account for coding time but should consider other tasks such as debugging time.

Original notes

Estimations should not be based on coding alone since that is only 1/6th the effort
Charles Portman conducted a study on productivity.
- Found that tasks had been taking twice as long as estimated.
- 50% of the time was found to be debugging time and programming time
- The remaining time was split between unrelated jobs, meetings, paperwork, company business, sickness, personal time, etc.
- Data suggested that there are unrealistic assumptions around technical work hours in a man-year.
Joel Anon studied programmer productivity.
- In this study, the relationship of interactions amongst developers and instructions written per year had been used.
- With few interactions, 10,000 instructions should be able to be completed in a man-year
- With some interactions, 5,000 instructions should be able to be completed in a man-year
- With many interactions, 1,500 instructions should be able to be completed in a man-year
John Harr did another study on this topic
- productivity was measured in debugged words per man-year

	Prog units	Number of programs	Years	Man-years	Program words	Words/man-year
Operational	50	83	4	101	52,000	515
Maintenance	36	60	4	81	51,00	630
Compiler	13	9	2.25	17	38,000	2230
Translator	15	13	2.5	11	25,000	2270

Data from IBM OS/360 experience, Aron and Harr confirm that productivity is related to the complexity and difficulty of a task.
According to this data, compilers are three times as bad as batch application programs, and operating systems are three times as bad as compilers.
Corbto data looks at lines of PL/I per man-year instead of words per man-year. In this study, the lines of debugged PL/I code was found to be 1,200.
- According to this data, productivity is constant in terms of statements
- According to this data, productivity may increase as much as five times with a high-level programming language.

Ten Pounds in a Five-Pound Sack

Most of this chapter focuses on size constraints. This focuses a lot on the cost of resources on older machines. While most of the lessons seem a bit out of date there is likely some equivalent that can be made in modern web and mobile development. Some of the key lessons to take away are the following. If you give users fine-grain control over options more space will be required. Breaking functionality into smaller modules can have an impact on performance and space. This might have been true for native binaries but I am not fully convinced it applies to web development. That is assuming you are not shipping duplicated code in your bundles. The last piece of information we are left with is to think about your data structure. In many cases, space optimization can happen at the data structure level.

Original notes

Much of this is based on the memory consumption of applications on hardware. At the time this was authored renting even 1KB of data a month was not cheap.
builder of software must set size targets, control size and devise size-reduction techniques.
hardware builder sets component-count targets, controls component count, and devises count-reduction techniques.
Size is not a bad thing but unnecessary size is
Size control
- This is partially a technical job and partly a managerial one for a product manager.
- users and applications need to be studied to get sizes right
- Setting targets is not all that needs to be done. In addition, all aspects of size must be budgeted.
- disk access is slow. At this time magnetic tape was common and round trips to the disk had been very expensive.
- A good way to help avoid issues around this kind of thing is to define exactly what a module must do.
- During implementation, architects must be vigilant in ensuring system integrity.
Space techniques
- Making a program small requires invention and craftsmanship.
- More function means more space.
- The first area of craftmanship is trading function for size.
  - The more fine-grained the user's options are the more space the program will take up.
  - Breaking functions into small modules costs performance and space.
- The second area of craftmanship is space-time trade-offs.
  - To do this make sure your team is trained in programming techniques and not just reliant on previous experience.
  - Each program should have access to two different library sets for the common functionality. The quick one and the squeezed one.
Often time gains are made via strategic breakthrough instead of tactical cleverness.
Most breakthroughs will come from redoing the representation of data.

The Documentary Hypothesis

This chapter talks a lot about documentation requirements and how different jobs need different documents. Some recommended documentation for a manager would include documentation around objectives, goals, specifications, schedule, budget, and organization. Managers also need appropriate documentation to handle the forecasting and pricing of their products. Software projects require very similar documentation to that of managers.

One of the bigger takeaways from this chapter is the need for formal documents. We need formal documents without these documents we cannot check our assumptions. Many times gaps in understanding and knowledge can only be found when documentation is created.

Original notes

What are the critical documents for managers?
- Objectives define the need to be met and the goals.
- Specifications are the computer manual plus performance specifications.
- Schedule
- Budget is a very useful document for managers.
- Organization chart
- Space allocations
- Estimate, forecast, and prices.
  - determine the success or failure of a project
  - To generate a forecast you need performance specifications and postulated prices.
  - Forecasts determine a per-unit share of development and fixed costs.
  - Forecasts allow you to determine prices.
  - If prices are below the postulated value then a success spiral occurs.
    - unit costs can drop
    - prices can drop
  - If prices are above the postulated value then a disastrous spiral begins
    - Performance must be squeezed
    - costs must be squeezed down.
Documents for University departments
- Objectives
- Course descriptions
- Degree requirements
- Research proposals
- Class schedule and teaching assignments
- Budget
- Space allocation
- Assignment of staff and graduate students.
Documents for a Software Project
- What: Objectives. These define the needs, goals, desiderata, constraints, and priorities
- What: Product specifications. Starts as a proposal but ends up a manual/internal document.
- When: schedule
- How much: budget
- Where: space allocation
- Who: organization chart
  - If the system is free to change the organization must be prepared to change as well.
Need for formal documents
- Gaps can only be found once inconsistencies arise in written documents.
- They communicate decisions to others.
- Provides a database and checklist to help managers figure out where to put their efforts.
The task of a manager is to develop a plan and then to realize it. A written plan is precise and communicable.

Plan to Throw One Away

A common issue in software development is the first thing you build is what will be used. This is not common in other industries and appears to be an issue in the software industry in general. No matter how good your planning is it will never be good enough hence the need for a first throw-away system.

Since you are planning to throw away you need to accept the fact that change is inevitable. Since change is inevitable there are things you can do to help your system change over time. Some things that help with change over time are the following.

Use careful modularization
Have a complete and precise definition of inter-module interfaces.
Have complete documentation for modules
Use a high-level language.
Use self-documenting techniques.
Use compile-time operations to enforce standard declarations.
Use module versioning
Have good version releasing practices.

Another form of change you need to be ready for is organizational change. The best way to handle this is to make sure that your senior management and engineers are technically and emotionally stable. Another key aspect is to generate sound documentation.

There is a long tail end of software. Even after the product is released there is a maintenance cycle that continues to change the software over time. This maintenance generally costs 40% more than the software had cost to produce. Also with every defect that is fixed in this phase, there is a 20-50% chance a new one will come up as a result of the change. Because of this, it is important to have appropriate regression testing.

Original notes

Chemical engineers build a pilot plant before making a fully operational factory since what works in a lab does not necessarily work in a factory.
Programmers do not seem to have this process and this is a constantly repeated mistake.
In many projects, the first version is barely usable.
Even the best planning will not be sufficient and eventually, all code gets rewritten.
This is an interesting dilemma for managers. Do you plan to build a pilot to throw away?
- You should because you are going to do it anyway
You need to accept change is an inevitable fact of life.
- It's better to be prepared for it than have to comes to terms with it later.
Planning a system for change
- Use careful modularization
- Have a complete and precise definition of inter-module interfaces.
- Have complete documentation for modules
- Use a high-level language.
- Use self-documenting techniques.
- Use compile-time operations to enforce standard declarations.
- Use module versioning
- Have good version releasing practices.
Planning for Organizational changes.
- All plans should be treated as tentative.
- reluctance to generate documentation does not come from laziness. Instead, it comes from a designer's unwillingness to commit themselves to the defense of decisions known to be tentative.
  - "By documenting a design, the designer exposes himself to the criticisms of everyone, and he must be able to defend everything he writes."
- Structuring an organization for change is harder than designing a system for change.
  - Every developer must be given tasks to broaden their skill set.
  - The manager needs to keep two or three top programmers to jump in on the toughest issues.
  - Managers and technical people need to be interchangeable as talents allow.
  - Often has sociological barriers
  - A common trap is for managers to think of senior people as "too valuable" to use for actual programming.
    - Bell labs overcame this issue by getting rid of job titles.
  - It's easy to increase salary with job tiers, it's harder to give higher job tiers more prestige.
  - Switching from a technical path to a manager path should never result in a raise.
    - This is a reassignment, not a promotion.
  - A manager should have technical refresher courses.
  - Senior developers should have management training.
  - Senior people must be kept technically and emotionally ready to manage groups or to build programs.
Programs do not stop changing once they are delivered to a customer.
Hardware and Software have very different maintenance requirements.
Software Maintenance
- Requires constant fixing of defects.
- The cost of maintenance is typically 40% more than developing it in the first place.
- The more users you have the more bugs that are found.
- Old bugs found and solved strangely seem to come back in future releases.
- New functions of a new release often have defects.
- Bugs begin to plateau over time but some of this might be customers becoming complacent with the bugs that are in the system.
- Fixing a defect has a 20-50% chance of introducing another.
  - Unless the program is well documented it is too easy to make a local change at a minimum effort instead of looking at the wider-reaching impact of the change.
  - The repairer is often not the developer that wrote the code. This is often done by a junior programmer or trainee.
  - Results in far more system testing
  - Regression testing must be done but it can be very costly.
- The total number of modules increases linearly with each release number. Repairs often destroy the structure and cause disorder to the system.
- Over time less effort is spent on fixing the original design flaws and more time is spent fixing flaws introduced by earlier fixes.
- Things are always best at their beginning from a statistical model.

Sharp Tools

Most of this chapter feels a bit outdated. Much of the discussion in this chapter is centered around tooling and environments.

In this chapter, there is an argument that tools are secrets held close by developers and not shared. Thankfully the proliferation of open source over the years has almost turned this around completely. At this point, it feels like there are too many tools to choose from. There are also big players in the game backing important tooling projects with real money. There is a lot more that could be said about open source and the pros and cons but that is not related to this chapter.

Another topic that is discussed a ton is environments. This is another thing I think has changed over time. Anymore much of consumer software runs in virtual machines. Often higher-level languages like Java, C#, Javascript, Go, etc are running in virtual machines and do not have to worry much if at all about the target machine.

The last thing is a call for better debugging and tooling. I think there is always room for improvement here but it seems we have come a long way from having to evaluate heaps. Anymore it is not uncommon to be able to have a fully debuggable environment with near real-time feedback to the developer.

Original notes

Many programmers have their own personal set of tools that they acquire over time and do not share
- I do not think this is true anymore, if anything we have too many shared tools.
Individualized tools hamper instead of aiding communication
There are not enough general-purpose tools
- Again not sure I agree with this in modern times.
Target machines
- machine support can be broken down into target machine and vehicle machines
  - The target machine is the one that the software is being written for.
  - The vehicle machines are the ones that provide the services used in the build system.
- Teams building new supervisors or core system software need machines of their own.
  - This comes with the overhead of operators and a system programmer or two to make sure the machines in use are serviceable.
- Debugging machines need to be instrumented so that counts and measurements can be made on all kinds of program parameters.
- Scheduling can be a real issue when there is a limited supply of the target machine.
  - Generally speaking, usage starts low but as teams start to get ready to debug their features demand for time on the system is very high. Generally, all teams hit this point at the same time.
  - In the S/360 project no matter how they tried to divvy up the time it was always an issue and in the end, they need to allocate large segments of time to teams for debugging.
  - Giving large chunks of time on a target machine tends to work out better for productivity.
Vehicle Machines and Data Services
- If the target computer is new you need a simulator.
  - Needs to exist long before the target does.
  - Gives access to dependable debugging. This is not to be confused with accurate debugging as only the target machine will give that to you.
  - Anymore hardware works correctly. This means if you have a bug it would be wise to start looking at the software first.
    - This is not true for new machines. In the case of new machines often there are hardware issues.
    - Simulators help when these hardware issues do crop up.
- Compiler and assemblers
  - Compilers and assemblers should work on vehicle machines but compile down to the target machine.
  - With high-level languages, a lot more testing can be done on the vehicle machine before moving to the target machine.
- Program Libraries
  - OS/360 used vehicle machines to maintenance program libraries.
    - All code tested was kept in the library
    - Libraries were sub-libraries with different access rules.
    - Programmers had playpen areas with no restrictions. These had been used for test cases and component scaffolding.
    - When a program was ready for integration, this code was passed off to a manager that put the code into a system integration sub-library.
      - At this point, the programmer could no longer change the code with permission from the integration manager.
    - When the system version was ready for larger use it would be promoted to the current version sub-library.
      - This version never changes unless there are crippling bugs.
    - What this did right were control and formal separation. In this case, changes went through different stages.
      - This reminds me a bit of dev, test stage, prod cycles that are common in the industry now.
- Program tools
  - Developers need dumps, source-file editors, snapshot dumps, and traces.
  - Developers also need utilities for putting decks on disks, making tape copies, printing files, and changing catalogs.
    - Not sure I believe this is relevant anymore.
- Documentation system
  - It is better to over document than to under document a system.
- Performance simulators are a must-have.
- High-Level Language and Interactive Programming
  - These are powerful tools that had not gotten enough inertia at the time. I think this has changed over time. Anymore I suspect it is harder to find a developer that will work in anything other than a high level-language (assuming you consider c++, rust, etc high level)
  - High-Level languages
    - Are far more productive
    - Have better debugging speeds.
    - There tend to be fewer bugs.
      - This is because many semantic bugs are no longer possible. An example would be misusing registers.
      - The compiler often finds your bugs for you at compile time.
    - Common objections to high-level languages
      - It doesn't let me do what I want
      - Object code is too big
      - Object code is too slow
    - Objections are no longer valid. Compilers are doing a much better job at optimization.
    - At the time of authoring the recommended high level-language was PL/I
  - Interactive programming
    - The author believes interactive programming will never replace batch systems for more applications.
    - There is no evidence to prove these tools are useful.

The Whole and the Parts

In this chapter reducing bugs, testing applications, and debugging are discussed.

For the most part, bugs are the result of mismatched expectations between component authors. To avoid this it is important to have detailed specifications on the component boundaries. The better the documentation is on the components the less likely there are the be bugs.

When it comes to testing developers mustn't be the ones testing their code. It is too easy for developers to invent ways to fix the problem in testing instead of fixing their code.

The top-down design methodology is proposed as a solution to avoid bugs. In this case system building is broken down into architecture, implementation, and realization. At each stage in this process, refinement can happen to make sure the scope is appropriate and those modules are defined well. This process avoids bugs by having a clear structure and representation. This process also produces smaller modules.

Original notes

Designing the bugs out
- Bug proof definitions
  - Most subtle bugs come from mismatched assumptions between component authors.
  - The crucial task is to get the product defined.
  - Most failures happen around things not specified.
  - Bugs can be reduced by careful function definition, careful specification, and disciplined exorcism of frills of function.
- Testing application
  - Before code exists specification should be handled to an outside testing group.
  - developers invent ways around gaps
- Top-down design
  - System building can be divided into architecture, implementation, and realization.
  - This can be done by top-down methods
  - refinement steps
    - sketch rough task definition and solution.
    - example the definition more closely to see how results differ from what is desired.
    - take large steps and break them down into smaller steps.
    - In this process identification of modules of solution or data that can be refined independently are found.
  - This process avoids bugs by
    - having clarify of structure and representation.
    - partitioning modules
    - suppression of detail makes flaws more apparent.
    - design can be tested in refinement steps
  - reduces the temptation to patch over bad design.
  - Argued to be the most important new programming formalization of the decade.
- Structured Programming
  - Design programs whose control structures consist only of loops and conditionals.
  - This is an alternative to using GO TO which leads to many logical errors
  - We should think of the control structures of a system as control structures not as branch statements.
- Component Debugging
  - When debugging on a machine, it is a bad idea to start without planning what you want to test.
  - Memory dumps
    - Sometimes access to a machine is scarce and taking up two-plus hour segments to debug is not always a good option.
    - Allows for postmortem debugging.
  - Snapshots allowed for insertion into a program without reassembly or recompilation
  - Interactive debugging
    - Efficient editing facilities make snapshots easy
    - provides instant turnaround capability.
    - preplanning is not as necessary.
    - The author suggests two hours of desk time for every two hours in the terminal.
      - Forces you to stop and plan.
  - Test cases are important.
- System Debugging
  - System Debugging will take longer than one expects.
  - Only debug after pieces seem to work.
  - The sooner pieces are put together the sooner bugs will emerge.
  - Less scaffolding is required when pieces test each other.
    - This is a bit of a trap and the time spent scaffolding is often worth it.
  - The "documented bug" approach promotes finding bugs before fixing them.
    - This is a wishful thinking approach used to rationalize away slipped schedules.
  - Build scaffolding
    - It is not unreasonable to have half as much code in scaffolding as in the product.
    - A form of scaffolding can be the dummy component. This component exists only of interfaces and fake data.
    - miniature files can be used to make sure formats are correct.
    - auxiliary programs include
      - generators for test data.
      - analysis printouts.
      - cross-reference table analyzers.
  - Tight control should happen.
  - All changes should be logged.
  - Only add one component at a time.
  - Though test cases are needed.
  - Tests should be run after each component is added.
  - Systematic testing procedures should happen for all updates to existing components.
  - Be careful about release time, changes can disrupt others who are expecting a stable testbed.

Hatching a Catastrope

In this chapter projects and deadlines are discussed in depth. A large focus of this chapter is how to detect when a project is falling behind. In many cases, a project does not fall behind over one major thing. Instead, over time a lot of little things add up and cause major deadline issues.

To help mitigate this a schedule is needed with concrete, specific, measurable, and well-defined milestones. The more specific the milestone is the harder it is for anyone to deceive themselves into believing that an unfinished project could be counted as a complete milestone.

In some cases, developers will need to try harder than necessary to cope with mishaps. This is a talent that should be used sparingly but is a talent that needs to be able to be utilized. Since this should be used sparingly a PERT chart should be used to identify when this is required.

Sometimes slippages get hidden from managers. This is often not a deliberate act but instead a shield to not burden upper management with too much information. To avoid this becoming a real issue there are a couple of suggestions made in this chapter for appropriate information transfer in these cases. In many cases, it is an issue of providing the boss with the correct amount of actionable and status information. As a boss, it is your job not to act not status information. In some cases, a boss just needs more information in these cases a PERT chart acts as a perfect document to get them the scheduling information they need.

Original notes

Disaster in projects is often a bunch of little things instead of one big thing
slippages are hard to recognize, harder to prevent, and even harder to make up.
You need a schedule
- Broken down into milestones with fixed dates
  - picking the right date is an estimation problem
  - milestones must be concrete, specific, measurable, and defined with knife-edge sharpness.
    - This is a departure from coding which is usually 90% complete and debugging which is usually 99% complete.
  - milestones must also be unambiguous in the sense that they should be easily verifiable by a boss.
  - If the milestone is defined well enough it is hard for someone to deceive themselves into believing an effort is complete.
  - Sharp milestones are to the benefit of the team.
hustle is a talent that is used in sports and good programming teams. In this case, hustle means trying harder than necessary to cope with routine mishaps.
You need to pay attention to a one-day slip it is a big deal.
PERT technique acts as a guide for when to apply hustle.
- A PERT chart is an elaboration of critical-path scheduling in which one estimates three times for every event. This was covered in an earlier chapter.
- The most valuable use is the preparation of the PERT chart. This helps identify dependencies, lays out the network, and estimates all legs of a project.
- The first chart is always terrible.
It is common for slippages to accidentally get swept under the rug
- All bosses need two kinds of information
  - Exceptions to the plan that requires action.
  - Status picture
- This can cause issues between first-line managers and bosses. The first line manager might want to warn of slippage but there may be no action required.
- To help address this
  - Role conflict must be reduced.
    - The boss needs to be able to distinguish between action and status information.
    - The boss must not act on problems the manager can solve.
    - The boss must never explicitly act on problems when reviewing status.
    - The boss should label meetings as status review or problem action meetings.
  - A boss should occasionally yank the rug back.
    - There need to be review techniques in which true status is known.
    - PERT charts are often the answer
- Preparation of PERT chart is a function of the boss and managers reporting to them.

The Other Face

This is another chapter that talks about the importance of good documentation. This chapter focuses on a couple of different types of documentation.

The first type of documentation that is mentioned is the documentation required to use a program. The common issue with this kind of documentation is it often lacks sufficient explanation. In many cases, the big picture is documented but the finer details are omitted from the documentation. Good usage documentation addresses the following.

Purpose. What is the main function and reason for the program?
Environment. What machines, hardware, and operating systems does it run on?
Domain and Range. What is the type of input? What kinds of output can you expect?
Algorithms used. What does the program do?
Input-Output formats. Complete and precise details on these formats.
Operating Instructions. What are the normal and abnormal end operations?
Options. What choices does the user have? How can the user specify those?
Running time. How long should the program take to run?
Accuracy. How precise are the answers intended to be?

Another type of documentation is the documentation required to believe a program works. In general, this documentation comes in the form of small tests a user can perform against the program to verify it is working as documented.

The last type of documentation covered in this chapter is documentation to modify a program. This documentation gives a clear overview of the internal structure of the product. In this documentation, you might expect to find flow charts, descriptions of algorithms used, explanations of file layouts, and discussions around modifications that had been considered but not implemented.

The last section of this chapter discussed self-documenting programs. In this case, the idea is the source code acts as documentation. In the form proposed in this essay code comments are used but it does mention having descriptive variable names. This call to action is a little bit less strict than the self-documenting approach mentioned in clean code.

Original notes

There are two faces to software
- One is it is a message from man to machine.
- The other is human to human. Memory fades ownership changes and we still need to understand what a program does
Documentation becomes even more important if the user does not have direct access to the author
Documentation and the practices around it are wrong despite efforts to make it better.
The most important aspect is the "how" of documentation.
What documentation is required?
- Different levels of documentation are required for different users.
- Documentation to use a program.
  - Most documentation fails to give a sufficient overview. Big picture stuff is not covered but all the little details are.
  - This documentation should have the following
    - Purpose. What is the main function and reason for the program?
    - Environment. What machines, hardware, and operating systems does it run on?
    - Domain and Range. What is the type of input? What kinds of output can you expect?
    - Algorithms used. What does the program do?
    - Input-Output formats. Complete and precise details on these formats.
    - Operating Instructions. What are the normal and abnormal end operations?
    - Options. What choices does the user have? How can the user specify those?
    - Running time. How long should the program take to run?
    - Accuracy. How precise are the answers intended to be?
  - In general, this documentation should fit into 3 or 4 pages.
- Documentation to believe a program
  - To believe it works you need proof. Proof comes in the form of test cases.
  - Every copy of a program should include some small test cases the user can use to verify the program is correctly loaded on their machine.
  - Types of tests
    - Mainline cases that test key functionality for commonly encountered data.
    - Barely legitimate cases that test the edge cases in input data.
    - Barely illegitimate cases that test inputs that raise diagnostic messages.
- Documentation to modify a program.
  - Adapting or fixing a program requires more information.
  - Needs to have a clear and sharp overview of the internal structure.
  - Components of the overview.
    - A flow chart or subprogram structure graph.
    - Complete descriptions of algorithms used.
    - Explanation of layout of files.
    - Overview of how data is loaded from disk.
    - Discussions of modifications considered but not implemented. These observations are critical.
Flow charts
- Is the most oversold piece of program documentation.
- Many do not need flow charts.
- Only show one aspect of a program's structure.
- Break down when they span more than one page.
- blow-by-blow flow charts are an obsolete nuisance whose only purpose is to initiate beginners into algorithmic thinking.
- flowcharting is more preached than practiced.
Self-documenting programs
- It's better to keep all data in one place instead of spreading it out across multiple locations
- We break this rule all the time trying to maintain a human and machine-readable version of the code.
- Program documentation is notoriously poor and its maintenance is even worse.
- Documentation should be in the source of the program.
- High-level languages make this more possible.
- Approach to minimizing the burden of documentation.
  - Use parts of the program that have to be there anyway. This would include labels, declaration statements, and symbolic names.
  - Use space and format to your advantage.
  - Insert necessary documentation into comments.
- Why not do this?
  - Increased size of source code stored.
  - Requires more keystrokes.
  - In general the benefits out-weight the negatives.

No Silver Bullet - Essence and Accident

This is the longest of the essays in the book and quite possibly the most quoted title in the industry. In this chapter, the silver bullet refers to some possible technique, technology, or management style that would radically improve the productivity of software projects.

The argument posed is that it is very likely that developers would like to find some solution that could drastically improve productivity. This comes in part due to software's counterpart hardware. In the case of hardware, with the introduction of the transistor, we saw very large productivity gains. It is argued that no technology in the history of man has ever grown as fast or effectively as hardware. The issue here is software is built on hardware so there is the hope that we could find the transistor equivalent for software.

In this chapter the argument is that no such technique can be found in software and instead we should strive for incremental improvement. This argument is made by evaluating essential and accidental complexity in software development. To give a loose definition accidental complexity is any complexity that does not need to exist to get the job of software development done. Some examples of accidental complexity would be compile times, machine access, and language. On the other hand, essential complexity is the complexity that must be handled and is the main purpose of software. Essential complexity is defining what it is we want the program to do. It is argued that the only real drastic gains that can be had in software ever is for the accidental complexity. It is also argued that most of the challenges with accidental complexity are mostly solved and there are only minor incremental gains that can be had at this point. Since all that is left is essential complexity there are no more major improvements to be had since all that is left is the main part of the job which is essentially difficult.

Original notes

Software construction is made up of essential tasks and accidental tasks.
- Essential tasks are the fashioning of complex conceptual structures.
- Accidental tasks are the representations of abstract entities in programming languages. This includes the management of space and speed constraints.
Most gains in software productivity have come from removing artificial barriers that have made accidental tasks harder. Examples include severe hardware constraints, awkward programming languages, and lack of machine time.
Ways to address the essential complexity
- Exploit mass market. Do not build what can be bought.
- Use rapid prototyping
- Grow software organically, add function over time as needed.
- Develop great conceptual designers in the next generation of software developers.
Everyone wants to see software costs drop as fast as hardware costs did.
No development or management technique offers even one order of magnitude improvement of productivity in software development.
Progress must be made stepwise. It is time to stop expecting some magical solution to this problem.
We should never expect the type of gains we have seen in hardware in any field. Hardware has progressed faster than any other technology.
Essence includes the interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions.
Essence is abstract but at the same time highly precise and richly detailed.
The hard part of building software is the specification, design, and testing, not the labor of representing it.
Building software will always be hard.
Complexity
- Software is more complex for its size than any other human construct.
- No two parts are alike and if they are we make them one (DRY).
- Digital computers have a very large number of states. This makes conceiving, describing, and testing hard.
- Software systems have orders of magnitude more states than computers.
- The complexity of the software system does not grow linearly.
- The complexity of software is essential, not accidental.
- This complexity leads to difficulty communicating between team members. This then leads to bugs, cost overruns, and schedule delays.
- The complexity of functions is invoking those functions.
- With the complexity of structure comes extending programs to use new functions without creating side effects.
- With the complexity of structure comes unvisualized states that become trapdoors.
- The complexity creates management burdens as well as learning, and understanding burdens.
Conformity
- Unlike in other fields, the complexity that software engineers must face is arbitrary.
- A lot of complexity comes from conformation to other interfaces which cannot be simplified out.
Changeability
- Modifications in most manufactured goods are hard but software modifications are rather easy.
- Software can be changed more easily because it is "pure thought-stuff" and therefore infinitely malleable.
- All successful software gets changed.
  - As software becomes more popular more people use it and those people will continue to press the edge cases.
  - All software survives beyond the normal life of the vehicle for which it was written.
Invisibility
- Software is invisible and unvisualizable.
- Geometrics abstractions are a powerful tool other fields get to use but it does not quite work the same way for software. For instance, an architect can use a floor plan.
- Software does not exist in physical space.
Past Breakthroughs solved accidental difficulties.
- High-level languages
  - Is believed to have created a productivity increase of at least 5 times.
  - Also has gains in reliability, simplicity, and comprehensibility.
  - Eliminates a whole level of complexity that was never inherent in programming.
  - At some point, the elaboration of high-level languages becomes a burden and increases, not reduces, complexity.
  - No additional gains can be found here.
- Time-sharing
  - Had a large increase in productivity just not as much as high-level languages.
  - This tackled a problem of immediacy.
  - Slow turn-around is accidental instead of essential.
  - Turn arounds are getting fast enough that they do not cause a cognitive disconnect.
- Unified programming environments
  - Unix and Interlisp changed this space by having integrated libraries, unified file formats, and pipes and filters.
Hopes for silver
- Ada and other high-level languages
  - Ada is a high-level language of the 1980s
  - The philosophy might be more of an advance than the actual language.
  - The philosophy is the modularization of abstract data types of hierarchical structuring.
  - Ada will not be a silver bullet because it is just another high-level language.
- Object-oriented programming.
  - Must be careful to distinguish between two separate ideas under the same name.
    - abstract data types.
      - object's type should be defined by a name, set of values, and a set of proper operations. This does not include its storage structure which should be hidden.
    - hierarchical types also called classes.
      - Allow for the definition of interfaces.
  - This removes more accidental difficulties from the process by allowing the designer to express the essence of design.
  - This does not solve any problems with essential complexity.
- Artificial Intelligence
  - Many people believe this will be revolutionary and will produce orders of magnitude improvement the author does not agree.
  - Common definitions for Artificial intelligence
    - The use of computers to solve problems that previously could only be solved by human intelligence.
    - The use of a specific set of programming techniques known as heuristic or rule-based programming.
  - There is not enough in common amongst problem domains. Problems solved with artificial intelligence in speech recognition have little to no overlap with image recognition.
- Expert Systems
  - An expert system is a program containing a generalized inference engine and a rule base.
  - These systems are designed to take input data and assumptions to explore logical consequences through the inferences derivable from the rule base.
  - The ultimate outcome is conclusions and advice.
  - Inference engines can deal with fuzzy probabilistic data and purely deterministic logic.
  - Inference engine technology is developed in an application-agnostic way. This allows it to be applied to many uses.
  - The changeable parts of the inference engine are encoded in the rule base.
  - The most important advance is the separation of application complexity from the program.
  - How can this be applied?
    - Interface rule suggestions
    - Suggestions for testing strategies
    - Remembering bug-type frequencies.
    - Providing optimization hints
  - May be able to reduce the total effort involved in bringing up test cases.
  - May help lifelong maintenance and modification testing.
  - Problems with this approach
    - It's hard to generate rules.
    - It's hard to find self-analytical experts that know why they do things. It's even harder to get these people to distill that into a rule base.
  - This pays off most for the inexperienced programmer.
- Automatic Programming
  - A program for solving a problem from a statement of the problem specifications.
  - This has been talked about for a long time but never realized.
  - In essence, it is the solution method that needs specification not the problem.
  - There are some exceptions where this does work, but these are already being put to good use.
  - These exceptions seem to have the following in common
    - Problems are characterized by few parameters.
    - There are known methods of solution to provide a library of alternatives.
    - Extensive analysis has lead to explicit rules for selecting solution techniques.
- Graphical Programming
  - Nothing convincing has come out of this field and the author is convinced nothing ever will.
  - The flow chart is a poor representation so why would this be any different?
  - Screen resolution has not progressed far enough to display everything that is needed.
  - Software is difficult to visualize.
- Program Verification
  - There is no silver bullet for proving designs are correct.
  - This is an important effort but it will not save labor.
  - This does not mean error-proof. Even mathematical proofs can be faulty.
  - Can only establish that the program meets its specification.
  - In general, a lot of the problem with software is debugging the specification.
- Environments and tools
  - Many problems have already been solved here
    - uniform file formats
    - uniform program interfaces
    - generalized tools
  - IDEs look promising
  - The biggest gain yet to be realized is the use of integrated database systems to keep track of details that must be recalled accurately by individual programmers.
  - This will help but the gain will only be marginal.
- Workstations
  - Enhancements are welcome but faster machines do not solve the fact that a lot of the problem is thinking time not compiling time. Often when you are compiling even if it is for a long time you are still thinking.
Promising Attacks on Conceptual Essence
- Some of these look promising
- Buy versus build
  - There are more and more vendors so the option to buy is becoming more common.
  - It is always cheaper to buy than to build.
  - A $100,000 purchased piece of software costs about as much as one programmer-year and its delivery is immediate.
  - These products tend to be better documented.
  - The hard part is applicability
    - Often the abstraction is not right or is too specialized. However, this is getting better due to standardized hardware.
- Refinement and rapid prototyping.
  - The hardest part in building a software system is deciding what to build.
  - The most important function of software builders is the iterative extraction and refinement of product requirements.
  - Generally, customers have not thought of their problem in enough detail to form a valid specification.
  - In planning any software activity it is necessary to allow for an extensive iteration between client and designer.
  - Tools for rapid prototyping of systems as part of the iterative specification of requirements is a very promising effort.
  - These prototypes simulate the important interfaces and performs main functions while not being bound to any speed, size, or cost constraints.
  - Prototypes do not handle exceptions
- Incremental development (grow software)
  - Building metaphor for software has outlived its usefulness instead it should be grown.
    - software overtime behaves more organically than a construct would. This is due to how it changes over time.
  - This calls for a top-down design.
    - It's very powerful to see something run and then see more features get added over time.
- Great designers
  - Good design practices can be taught.
  - New curricula, new literature, and new organizations need to come into being to help raise the bar.
  - Software construction is a creative process.
  - Great designers produce structures that are faster, smaller simpler, cleaner, and are produced with less effort.
  - We need to grow great designers
  - Great designers and great managers are both rare.
  - Every software organization should proclaim that great designers are as important to its success as great managers.
  - How to grow great designers
    - Identify top designers as early as possible, best are often not the most experienced.
    - Assign a career mentor.
    - Provide opportunities for growing designers to interact with one another.

"No Silver Bullet" Refined

For the most part, this is just minor clarifications on the previous chapter and defenses against papers that try to counter prove the author.

Insights

This might be one of the most referenced books of software I can think of. While this book was written quite some time ago, many of its concepts are still applicable today. It is also clear that a lot of the issues talked about have already been addressed and are somewhat obsolete since the authoring of this book. Some examples include machine access and machine speeds. We now live in a time where for the most part stable machines are a given. There is also very rarely software that is built for a specific machine. With these hardware changes, some of the tales and concerns become less relevant.

One thing that was interesting to read about in this was the talk of high-level languages. It was clear that the author had considered high-level languages to be the future and the direction the industry needed to head in. In general, it would appear the author was right and high-level languages do dominate the space now. What is interesting however is that the languages considered are not used today and if they are they would now be considered lower-level languages.

While reading this book I often wondered if there was going to be anything new talked about. What I have come to realize is that this book likely seemed so familiar because all of its teachings are indoctrinated into the industry's teachings. It appears to me that this book has been read by many and its suggestions have made it into industry.

Book Note - Clean Coder

Wed, 14 Apr 2021 00:00:00 GMT

The Clean Coder

The clean coder is a book authored by Robert C. Martin. This is one book of many in Roberts clean series.

High level overview

This book is about uncle bob's take on what it means to be a professional programmer. Some of the key skills this book tries to take on are

What it means to behave as a software craftsman.
How to deal with tight schedules and demanding managers.
How to get into good coding habits.
How to avoid burnout.
How to manage your time.
How to foster an environment where programmers can thrive.
How and when to say no.
How and when to say yes.

Notes

Forward

This forward contains an interesting anecdote in which Robert worked with another manager. At this time he pushed the team hard to meet a deadline. By the description of pushing hard, it seemed like the conduct to his fellow developers was poor. When all was said and done the developers meet their goal but due to a legal team with a different priority structure, their efforts had been in vain. The interesting takeaway is that the legal team in this story was treated as professionals while the software engineers had not been treated the same way.

Preface

This preface while outlining the book's contents also contains another interesting anecdote. This anecdote is 3rd hand and is about the engineers and the failure of engineering and management on the challenger project. This failure ultimately leads to the death of seven individuals and the slowdown in space exploration in the United States.

Professionalism

Professionalism is all about acting as a professional might in any other industry. As with many industries being a professional is something to be proud of. A major part of being a professional is increased accountability. This means you are responsible for knowing your code works and knowing what you can and cannot do in a given time. It also means respecting your QA. It is not your QA engineer's job to find bugs you knew existed.

A professional software engineer understands the code they are writing and follows good design practices when creating features.

Another major aspect of being a professional is taking responsibility for your own career and career direction. It is not your employer's job to make sure you are keeping up to date in the world of software. This means they do not owe you trips to conferences or training materials. Software is an ever-changing industry and you must spend time outside of work to stay up to date and practiced in this field.

Original notes

Being a professional is something to be proud of, but it means that you need to take on more accountability.
Professionals do not cut corners to meet a deliverable target.
Call for a hippocratic oath for software.
Sending broken code to QA is unprofessional. QA should not be finding known bugs.
professionals know code works. They know this because they use rigorous testing practices.
professionals do not compromise structure to deliver a feature. Doing so sets future work up for failure.
- mention of the boy scout rule and various other topics from clean code.
- code should always be changing (constantly refactored)
it is not your employer's job to handle your future
- employers should not have to pay for your training
- employers should not have to send you to conferences
- employers do not have to give you work time to learn something new
The only thing you owe your employer is an amount of time and effort (40 hours a week)
Professionals should spend 60 hours a week on programming (that's right 20 hours of personal time a week)
- Uncle Bob argues that since you can focus on whatever topic you like this should not lead to burnout. In personal experience, I pretty much agree.
Professionals know their field.
- You should be aware of what is happening in your field
- Just because a publication or idea is old does not mean that it is irrelevant now. (read old publications)
- Every software professional should know the following
  - Design patterns (GOF)
  - Design principles (SOLID)
  - Methods (XP, Scrum, Waterfall, etc)
  - Disciplines (TDD, OOP, etc)
  - Artifacts (UML, DFD, etc)
This is a field of continuous learning.
Professional practice often.
Collaboration is a key skill
Mentoring is a great way to learn.
Know your domain. If you are entering a field you should read up on the domain you are going to be working in.
Understand that the employer's problems are your problems.
Professionals should be humble, not arrogant.
Statement I am not sure I agree with. If you caused a bug that cost the company 10k you should be willing to pay for it out of your paycheck.

Saying No

A professional knows when and how to say no. Part of this is assessing a situation and making sure you can commit to something. If your boss asks you to mean a deadline and you agree to it knowing you cannot make the deadline you are acting unprofessionally.

Often when something is at stake others might ask you to compromise your initial estimate, or to attempt to "try harder". In these cases, you must stand your ground. Agreeing to "try harder" implies that you are not already giving it your all and that there is still some reserved effort you can exert.

The cost of saying yes can lead to an utter mess. In these cases, we might think we are meeting deadlines and giving the employer what they want but ultimately what often ends up happening is we create an unmaintainable mess.

Original notes

starts with a tale of Bob being pressured to complete a project too quickly.
- His manager would not budge on the date
- Bob and the team did not have the spine to stand up to the manager.
- the result was a disaster
  - an intermittent bug caused terminals to lock up at a rate of 1 time per hour.
  - they could not roll back the change so the pressure to fix it faster was mounted (4 weeks was unacceptable)
Professionals know how to say no.
Saying no to your boss is not unprofessional, agreeing and falling short is.
Adversity is a good thing. Developers should stand their ground that is doing their job. Likewise, managers should pressure for quicker turnaround while understanding the developers. It is also their job to see things done on time.
Never agree to try. That implies you can do more and you have been holding something in reserve.
Do not compromise estimates, stand your ground. If timelines are an issue provide a reasonable feature expectation (we can have feature x but it will be missing y is that alright?)
Be direct with your estimates.
If you hit a snag that could compromise an estimate let management know as soon as possible.
Be a team player.
Do not be passive-aggressive. An example is keeping a log of memos to use against someone later instead of trying to resolve the issue.
cost of saying yes. This is an interesting article about a developer that agreed to an unreasonable project with shifting requirements under a ridiculous time frame. In the end, this project died and the developer felt remorse about the terrible code they created.
- Bob argues that the developer or his employer is at fault.

Saying Yes

To be a professional you need to know the right way to say yes to something. Language is everything in this case. There are several statements that you can make that sound non-committal. For example words like need, should, hope, wish and let's imply a lack of commitment. When you commit you need to

say you will do it in a way that sounds like you mean it
mean that you will do it
actually do it

A strong commitment often sounds like this, "I will ... by ...".

There are cases in which life happens and a commitment cannot be made. In these cases, it is imperative to raise a red flag as soon as possible.

Original notes

starts with an interesting story about how bob had and lost a patent for voicemail and wanted to start a project around it not knowing the patent was not renewed.
parts of a commitment: say you will do it, mean you will do it, actually do it.
words that indicate a lack of commitment: need/should, hope/wish, let's. These imply a lack of concrete commitment.
an example of a strong commitment sounds more like this: I will ... by ...
Good commitments are binary (either you did it or you did not do it)
You can only commit to things you have full control of, to help mitigate this some suggestions are offered.
- work with external parties to understand your dependencies
- create an interface that mirrors the dependency.
- work with the build team to make sure your work will build.
- use integration tests.
Just because it cannot be done does not mean you cannot commit to things that move you in the right direction.
life happens and sometimes commitments cannot be made. The responsible thing to do in these cases is raise a red flag as soon as possible.
Don't use try also do not agree to try.
professionals know how much overtime they can handle and require people to respect their time. For example at the end of this chapter overtime needs to be done. The programmer agrees to work the weekend with the pre-requisite that their family is alright with it and the understanding that they will take Tuesday off to make up for lost time on the weekend.

Coding

This is a topic Bob, has a lot to say on. He has written a different book on this very topic Clean Code.

A major part of coding is being in the right mindset. You should not code when you are tired or distracted. In some cases, life events can serve as a distraction to good code. In these cases, you should try to resolve your life events before coding. Failing to do this can lead to code you regret later on.

One common topic that most developers talk about is getting into the zone. Often developers claim that being in the zone is a desirable thing because you feel more productive. In this book, Bob argues that the zone is a dangerous thing. The zone is a meditative state that lacks collaboration. In Bob's eyes, this is a problem is programming is a collaborative effort.

When coding one should strive to reduce debugging time. To do this we should follow strong TDD practices. This should assure us the code works and only does exactly what it is specified to do.

At some point, we all run into issues. In these cases, pairing is the best thing we can do. This is something that quite a few books echo. Although it may be comfortable to work solo, two heads are often better than one and coding is no exception.

Original notes

mentions the clean code book.
talks about the unique challenge of code.
- it must work
- it must solve a customer problem
- it has to fit well in an existing system.
- it has to be readable to other programmers.
Don't write code while tired or distracted, it will end up having to be redone.
- Bob tells a story about some 3 am code he regretted for a long time to come.
life events can impact the quality of code since your focus is not on the code.
- find a way to "shut down the background process" (life event)
- take time to deal with your personal issue but not at the expense of your employer.
Bob comments on the "flow state" or the "Zone".
- coding in the zone is a bad thing.
- more of a form of meditation than a mechanism for good.
- the zone is only where you want to be when practicing.
- it is a non-communicative state.
Music while programming in Bob's opinion is bad and is just a tool to enter the "zone".
handling interruptions.
- when in the zone the natural response might be to respond rudely.
- pair programming and TDD can help since the pair programmer or test can keep your context.
- remember to be professional next time you might be the one causing an interruption.
To deal with writer's block consider pair programming.
coding is a creative process so focusing on creativity is a good thing (just not while coding).
- read books
- watch videos (unless you are bob)
- listen to music, etc.
contains an interesting story about bob debugging a terminal lockup problem he mentioned earlier in the book.
Professional programmers should strive to spend less time debugging (use TDD to help with this).
Debugging is part of programming.
Software development is a marathon, not a sprint, pace yourself.
When stuck on a problem disengage from it and come back to it later. You just might find yourself solving the problem in the shower, etc.
Inevitably you will be late on some work, the goal is to be transparent about it. (I feel like this has been talked about before)
Hope is not a strategy hope is a project killer. Do not hope you will get something done in 10 days.
Do not rush, rushing never solves problems faster. Do not give in to pressure and rush through something
Only take overtime if you can handle it. The following are recommended.
- you can personally afford it.
- it is short-term.
- your boss has a fallback plan.
The most unprofessional behavior a programmer can engage in is changing the definition of "done".
- don't say something is done when it is not.
- This is an overt lie.
- It is too easy to rationalize.
- This practice is contagious, if someone sees you doing it they are likely to do it themselves.
- This leads to bad communication between managers and developers.
- Example calling a task done with a failing build.
- The best way to avoid this is to have business analysts and testers create automated acceptance tests.
No matter how skilled you are you can and should still ask for help.
- when being help accept the help do not "protect your turf"
- Do not push away help. Give it time.
- It is unprofessional to remain stuck on an issue when help is available.
- In general, programmers need to work on collaboration. Bob uses the example of the stereotype that often plays out in software (arrogant, self-absorbed introverts)
- More experienced programmers have the responsibility to mentor less experienced programmers.
- Mentoring is a good way to learn.

Test Driven Development

Test Driven Development or TDD is a way of writing code. In this methodology, one writes a test before any code is written. Only enough test code is written to cause a failure. Once this happens no more test code is to be written. After this production code must be written until the test is passing. Once the test is passing no more production code is to be written.

TDD is the professional option. There are very few cases in which this is not the right thing to do.

Original notes

TDD was part of the XP (extreme programming wave)
TDD helps reduce cycle time.
TDD works and everyone should accept that.
Interesting section from the chapter (I like this). "How can you consider yourself to be a professional if you do not know that all your code works? How can you know all your code works if you don't test it every time you make a change? How can you test it every time you make a change if you don't have automated unit tests with very high coverage? How can you get automated unit tests with very high coverage without practicing TDD?"
Laws of TDD
- you cannot write any production code until you have a failing test.
- you are not allowed to write more of a unit test than is sufficient to fail.
- you are not allowed to write more production code than is sufficient to pass the currently failing test.
Benefits
- Certainty, you know the code works so you can release it at any time.
- The defect rate is very low.
- You have the courage to fix bugs because tests will let you know when you mess up another use case.
- Tests act as documentation.
TDD is the professional option.
Warning what TDD is not
- You can still write bad code even if your tests pass.
- There are rare situations where TDD is not appropriate and being dogmatic about it can be wrong to do.

Practicing

All professionals practice programmers should be no exception. There are several different ways you can practice.

coding kata
wasa
Contributing to open source

The important thing is that you practice and practice often to keep your skills sharp.

Original notes

All professionals practice (examples: doctors, musicians, sports players, etc)
practicing is not a new concept consider "Hello, World!"
cycle times on development have gotten better due to Moore's law.
There are things called coding dojo's in which you can solve a known issue with other developers in a room.
A coding kata is a good way to sharpen your skills. Solve a known issue practicing TDD.
A wasa is like a kata but done as a team.
- For example, one person might write a test then the next might get it to pass and write the next test, and so on until the issue is solved.
Open source is a good way to practice and build a resume.
Professional programmers practice on their own time.
become a polyglot
"Practicing is what you do when you are not getting paid. You do it so that you will be paid, and paid well"

Acceptance Testing

Acceptance tests are a great way to make sure the business needs are understood and tested when a feature is created. These tests serve as the best form of documentation of requirements for developers. Acceptance tests must be written in a collaborative effort by both programmers and stakeholders.

While acceptance tests are different than integration and unit tests they should be automated much the same as a unit or integration test would be. In a perfect world, developers would not write these tests instead QA or business analysts would write the tests.

Original notes

A professional developer is a communications role as well as a development role.
Starts with an interesting story about how Bob wrote a feature from start to finish with a coworker and gained insights on how features are communicated.
premature precision is an issue on both sides.
- Business people what you know what they are going to get before they ok a project
- developers want to know what they are supposed to deliver before they estimate the project.
Things appear differently on a piece of paper than they do in a working system.
Even with perfect information developer estimates will have huge variance.
requirements will change making perfect information impossible
professional developers know that estimates can, and should be made based on low precision requirements.
a solution to premature precision is to defer as long as possible.
- This can lead to late ambiguity.
  - This can lead to incorrect features being developed.
Acceptance tests are the answer. In this case, an acceptance test is "tests written by a collaboration of stakeholders and the programmers to define when a requirement is done."
done is when all tests pass, QA and stakeholders have accepted.
Professional developers work with stakeholders and QA to ensure that automated tests are a complete specification of done.
The purpose of acceptance tests is communication, clarity, and precision.
Acceptance tests should always be automated. The reason for doing this is simple manual tests cost more money.
While it does appear that it is more work in the long run it is less work than having manual tests.
In a perfect world, QA and stakeholders would collaborate to write tests and developers would review for consistency.
If developers write tests then the developer who wrote the feature must not also write the test. (interesting)
Business analysts tend to write happy path tests.
QA tends to write unhappy path tests.
developers should negotiate with test authors for better tests when the test does not make sense.
Don't be passive-aggressive "Well, that's what the test says, so that's what I'm going to do." should never be the answer.
Acceptance tests are not unit tests.
- unit tests are written by developers for developers.
- acceptance tests are written by business for business.
GUIs are hard to acceptance test.
GUIs should follow SRP (single responsibility principle)
Test using the correct interface. It's often better to test via an API instead of via the GUI.
Acceptance tests should be run often in CI.
If tests are broken in CI this should be viewed as an emergency.

Testing Strategies

As a professional developer, it is important to have a good testing strategy. Generally, a testing strategy should have a mix of unit, component, integration, system, and manual tests.

Unit tests should cover most if not all code paths. These unit tests are tests for developers not for business. Unit tests should be run in CI.

Component tests are tests against individual components of a system. These are generally acceptance tests defined by the business. Since these only test a single component mocking strategies should be used. Ideally, these tests cover 50% of your code and are written by QA.

Integration tests are tests that run against multiple components. These tests make sure the connections between components are working properly. Component tests should be written by architects or designers and should not be run in CI. While these do not run in CI they should be automated and they should run periodically.

System tests are tests that run against the entire integrated system. These tests are written by architects or technical leads.

Manual tests are exploratory tests conducted by humans. These are important tests but they should have the least coverage. The important thing is humans are creative and will be able to find new ways to break things or notice subtle bad behaviors.

Original notes

Every professional developer has a good testing strategy.
Bug hunts can be a testing strategy.
QA should find nothing wrong. This will rarely happen but when QA finds a bug this should cause alarm for the development team (how did we let this happen)
QA and development should work together and not be adversarial.
QA should create automated acceptance tests.
QA should use the discipline of exploratory testing.
Mention of the test automation pyramid.
- ~100% unit test (xunit, junit, jest, etc)
- ~50% component tests (api)
- ~20% integration tests (api)
- ~10% system tests (gui)
- ~5% manual exploratory tests
unit tests
- written by programmers for programmers
- in the programming language of the system.
- Are executed in CI
- Should provide as close to 100% coverage as possible
Component tests
- are written against individual components of a system.
- these are acceptance tests for the business rules of the component.
- appropriate mocking should be used.
- written by QA.
- cover about 50% of the system.
Integration tests.
- test a collection of components
- basically a connections test.
- Often written by system architects or lead designers of the system.
- might include performance or throughput tests.
- These are not part of CI
- Run periodically (weekly or nightly)
System tests
- execute against the entire integrated system.
- do not test business rules directly.
- written by system architects and technical leads.
Manual Exploratory tests
- are not automated or scripted.
- intended to find unexpected behaviors.
- it helps to have human brains and human creativity to find issues with the system.
- There should be no written test plan for these.
- The goal is not coverage instead it is to make sure the system works correctly.

Time Management

Part of being a professional developer is managing your time appropriately. A major key to this is good meeting etiquette. While meetings are critical they can also be major time wasters. As a professional, you need to balance what meetings you should and should not be attending. Some things to consider when accepting a meeting. It is important to remember, the person inviting you to a meeting is not responsible for your time. One of the most important duties of your manager is to keep you out of meetings. Since meetings are large time sinks it is important to keep them on topic. In general, you should never accept a meeting request if no agenda is present. If at any time a meeting strays from the agenda it is important to stop and consider if a new meeting needs to be created.

Disagreements are common in software engineering. The best way to deal with these is to provide data. Brute force is never the right approach to resolving a disagreement. If there is insufficient data for either side of a disagreement sometimes the best option is a coin flip.

Original Notes

8 hours is not a lot of time?
meetings
- According to bob meetings cost around 200$ per attendee
- meetings are necessary
- meetings are huge time wasters.
- some attendees may find a meeting to be invaluable while others can find it redundant or useless.
- You don't have to go to every meeting you are invited to, doing so would be unprofessional?
- be careful about which meetings to politely refuse.
- the person inviting you to the meeting is not responsible for your time you are.
- examples that are choices you must make
  - something you are interested in but is not immediately necessary.
  - something you can contribute to but is not immediately significant.
  - In these cases, you can work with your manager to see if you should participate.
  - One of the most important duties of your manager is to keep you out of meetings.
- Leaving is an option when meetings get boring leave?
  - Do this politely by asking if your presence is still required.
- Meetings should have an agenda and a goal.
- things you should know going into the meeting
  - what discussions are on the table?
  - how much time is allotted for them?
  - what is the goal to be achieved?
- If the meeting agenda has been high-jacked or abandoned, request the new topic be tabled and the agenda forwarded.
- Stand-up meetings.
  - part of the Agile cannon.
  - The only thing that should be discussed are these three topics.
    - What did I do yesterday?
    - What am I going to do today?
    - What's in my way?
  - each question should require no more than 20 seconds.
- Iteration planning meetings
  - Also part of agile cannon.
  - It takes skill for these meetings to go well.
  - Point is to select backlog items to work on next iteration.
  - Estimates should be done for candidate items.
  - In an exceptionally good organization, acceptance/component tests will already be written or at least sketched out.
  - No more than 5 or 10 minutes should be spent on any one item.
    - If more discussion is needed it should be scheduled at a different time with a smaller subset of people.
  - Bob's rule of thumb this meeting should take no longer than 5% of the total time in the iteration. (ex: 1-week iteration should be over within two hours.)
Iteration Retrospective and demo
- Discuss what went right and wrong.
- stakeholders see a demo.
- These meetings are often abused.
- Bob's rule of thumb these should take 45 minutes. 20 for retrospective and 25 for demo.
Disagreements
- "Any argument that can't be settled in five minutes can't be settled by arguing." Kent Beck.
- Technical disagreements are rarely grounded in data.
- Force does not settle disagreements data does.
- Being passive-aggressive in an argument just to sabotage it later is the worst kind of unprofessional.
  - If you agree you must engage.
- How to settle a disagreement.
  - run experiments.
  - sometimes flipping a coin is the best option.
- It is a good idea to agree to a time and set of criteria for abandoning a path.
- Avoid meetings that exist just to vent a disagreement.
- If an argument must be settled ask each arguer to present their case to the team in five minutes or less.
Programming is an intellectual exercise that requires extended periods of concentration and focus. After you have used up all your focus you will have to recharge by doing unfocused activities for an hour or more.
Managing focus is a key skill for professional developers.
Sleep is key to good focus.
Caffeine can help with focus but too much can cause jitter focus
Good ways to recharge focus
- take a walk
- spend some time looking out the window.
- talk with friends
- meditate
- get a power nap.
- listen to a podcast
physical focus can help mental focus (workout)
A good way to manage time and focus is to use the Pomodoro technique (kindly reject all distractions over a 25 minute period). Repeat this process with 5-30 minute breaks in between.
priority inversion is when you convince yourself something is more important or urgent than a task you do not want to do (for whatever reason). Professionals should not engage in this behavior.
Avoid blind alleys it's easy to get too far down a technical pathway that will go nowhere. Keep an open mind to other ideas so when you hit a dead-end you still have options.
Messes are worse than blind alleys. A mess can be worked by sheer brute force but are major progress impeders
Messes are often the result of a bad design choice realized late into the product.
Once you realize you have a mess you hit an inflection point: you can go back and fix the design or you can continue to go forward.
- Going back means reworking existing code.
- Going back is never easy.
- If you go forward you may never escape.
- Professionals are always on the lookout for messes and will expend all necessary effort to escape from them as early as possible.

Estimation

Estimation is an important part of feature development. In general, the business views an estimate as a commitment while a developer views an estimate as a guess. A professional developer mustn't commit unless they know they can achieve it. As a professional, it is your job to make a clear distinction between an estimate and a commitment.

Various strategies can be used to generate an estimate. One interesting but somewhat involved estimation process is PERT. In PERT you end up generating an estimate by considering the best possible case, worse possible case, and average case.

Original notes

Businesses view estimates as commitments
developers view estimates as guesses.
Professionals do not make commitments unless they know they can achieve them.
- never commit to something you are uncertain about.
- missing a commitment is an act of dishonesty.
An estimate is a guess, no commitment is implied.
developers are bad at estimates.
estimates are a probability distribution.
professionals draw a clear distinction between estimates and commitments.
PERT
- a simple but effective way to convert estimates into probability distributions.
- when estimating a task provide 3 numbers (trivariate analysis)
  - O: Optimistic Estimate. A wildly optimistic number. (less than a 1% chance of happening)
  - N: Nominal Estimate. The most likely estimate to be true.
  - P: Pessimistic Estimate. A wildly pessimistic number. (less than a 1% chance of happening)
- Distribution based on these numbers. µ = O+4N+P/6
- µ is the expected duration of the task
- To calculate the standard deviate one can use this equation. σ = P-O/6 (how uncertain is the task)
- PERT is a good way to prevent overly optimistic estimates.
Estimating should be done with teammates, the group can estimate better than the individual.
wideband delphi is a technique that aims at a consensus in estimation.
Flying fingers is a strategy where everyone sits at a table, discusses the issue carefully, and puts up some fingers 1-5. If an agreement is not reached the topic is discussed further (this seems like the only way I have known estimation to work)
Planning poker (what I have used) for the most part is a software refinement on the flying fingers strategy.
Affinity estimation
- all cards of stories are laid out.
- No talking is to happen.
- Cards are moved left to right (left being easiest relative to others and right being hardest)
- Anyone can move a card at any time.
- If a card is moved N times then it is set aside for discussion.
you can use these same strategies to generate your optimistic and pessimistic values.
To mitigate error you can take advantage of the Law of Large Numbers.

Pressure

A professional software engineer understands how to manage pressure while still behaving professionally. A large portion of this is understanding how to avoid and manage pressure.

As far as avoidance goes it is important to avoid making commitments unless you know they can be met. It is also important to reject unrealistic commitments as it is a disservice to everyone.

An interesting thing is under pressure you find what your disciplines are. If you find that in a bind you stop following TDD then you know that TDD is not something you value. In these cases, it is important to stick with what you believe in. If you start doubling down on practices you do not find value in it will only increase pressure.

When you are under pressure it is important to manage it as best as you can. One great way to help is to make sure you are getting enough sleep. Even though you are under pressure it is very important not to rush. Rushing will lead to problems later on and will likely cause more stress.

When you are under stress communication is a great tool. In these cases make sure your manager is aware and see if there is anything they can do to help. Another great trick is to pair program. This can help reduce your stress while keeping code quality high. Your peer might spot issues you are making while under stress. It is also important that when you see others in these situations you offer your help.

Original notes

starts with a comparison of a surgeon to a typical developer. The surgeon is cool and collected under pressure the developer is not.
goes on to explain his time at what sounds like a toxic startup with insane time pressures.
a good way to stay calm under pressure is to avoid situations that cause pressure.
avoidance is not always an option so limit the required high-pressure situations.
avoid making commitments unless you are sure they can be met.
accepting unrealistic commitments is a disservice to developers and the business.
sometimes commitments are made for us. In those cases, we must help the business find a way to meet those commitments. That being said we do not have to accept the commitments. (Interesting)
- By not accepting the commitment we hold those who committed responsible instead of taking the fall.
You can always hunt for a new job.
It is important to stay clean to meet deadlines.
"Quick and dirty" is an oxymoron, dirty is always slow.
A crisis is a good measure of what you believe. If you follow your disciplines in a crisis then you believe in those disciplines.
- For example, if you use TDD in noncrisis times but abandon it in a crisis you do not think TDD is helpful.
Handling Pressure
- Sometimes you cannot avoid it.
- Manage stress, get sleep.
- Resist temptation and do not rush. Rushing only makes things worse.
- Let your team and manager know you are in trouble.
  - Let them know your plans for getting out of that trouble.
  - ask for input/guidance.
  - avoid surprises.
- trust your disciplines. you have them for a reason now is not the time to abandon them.
- Instead of looking for a solution to get it done faster buckle down on what you know and your disciplines.
- Do pair programming. Partners help you find things you missed and often have helpful ideas.
- When someone else is under pressure offer them help.

Collaboration

Software development is all about working with others. There is a stereotype that is sadly often true in this industry that all programmers are arrogant introverts. It is important for professionals not to act in such ways.

Outside of working well with other developers, a professional software engineer must also be able to work with a variety of other roles in the company with varying levels of technical skill. To help these communications you must understand the business's problems, not just technical solutions.

Original notes

It is unprofessional to be a loner or a recluse on a team.
Starts with an interesting tale of Bob and Tim working together to speed up a searching algorithm for their codebase.
Most programmers are introverts that would rather work alone (based on stereotypes that are often true)
It's good to be passionate about what we do, but it's also good to keep your eyes on the goals of the people who pay you.
It is your job to meet the needs of your employer
- This means you will have to work with other people: testers, business analysts, managers, etc.
- You need to understand why you are writing the code you are writing and the benefit of that code.
- The worst thing you can do is get wrapped up in the technology while the business is burning all around you.
- take time to understand the business. Talk to people to understand it better
- pay attention to the companies health.
- Bob goes on to tell the story of the time he was fired from a job for not understanding his employer's needs and work culture.
Programmer conflict
- a symptom of a dysfunctional team is when each programmer safeguards their code and does not let anyone else touch it.
  - This often leads to massive code duplication.
  - It can also lead to incorrect interfaces.
- It's important to break down walls of code ownership (silos are bad)
- All team members should be allowed to make changes to any module.
- Professionals do not prevent others from working in their code.
- Professionals learn from working with each other.
Pair-programming is the most efficient way to solve a problem(?)
Pair programming is a great way to share knowledge.
All team members have a position to play.
Pairing is the best way to review code.
Working alone might be better for you but it is not better for the team.
While we might not have gotten into this industry to work with people, programming is all about working with people.

Teams and Projects

A good team is something to be envied in software development. The best way to create a good team is to keep that team consistent. It often takes a long time for people to get used to working with one another and creating ad-hoc teams constantly does not help.

By keeping a consistent team it becomes easy to change priorities on a whim since a new team does not have to be formed and the team members are already used to working with one another.

Original notes

There is no such thing as a half-person. Often we do not fully resource a developer to a task.
It does not make sense to have a developer devoted to two different projects especially when there are different project managers, business analysts, developers, and testers on each project.
It takes time for a team to form/gel
A team that has "gelled" is very special as they know how to work with each other very well. In these cases, things get done much better.
A team should consist of: programmers, testers, analysts, and a project manager.
analysts develop the requirements and write automated acceptance tests.
testers write those acceptance tests.
the project manager tracks the team's progress and makes sure the team understands schedules and priorities.
It takes a while for team members to work out their differences. This can take up to a year, but once it's done it's worth it.
Once a team has gelled it would be crazy to break that team up.
Professional development organizations allocate projects to existing teams, they do not create ad-hoc teams.
To measure these teams look at their velocities.
Reallocating priorities in an emergency is virtually impossible with ad-hoc teams.
This allows the business to change priorities on a whim.
- This can intimidate project owners(?) however it is part of their job to make a case of their project.
Teams are harder to build than projects so keep functioning teams together.

Mentoring, Apprenticeship, and Craftmanship

Mentoring is a major gap in the software industry currently. Often college graduates are expected to be able to go straight into the industry and be productive. This mentality is crazy. There are plenty of other fields in which this would never happen and software is becoming too critical to be built by un-prepared individuals.

One idea as to how to handle this issue is to introduce the concept of apprenticeships into the software industry. In this program, there would be three major roles the master, the journeyman, and the apprentice.

Masters are veterans in the industry and have worked in many different programming languages and systems. The master needs to stay up to date on the technology. Part of the responsibilities of the master is working with journeymen and testing apprentices to see if they are ready to become journeymen.

Journeymen are individuals that are knowledgeable but have not quite yet proven themselves to be masters. They are often very skilled at one language and one system. For the most part, it is the responsibility of the journeyman to work closely with the apprentices and mentor them.

The apprentice is where everyone should start their career. Apprentices should be under close supervision by journeymen and should spend a good amount of time peer programming without actually directly authoring any code.

While this might not be the best approach it certainly seems better than the current approach of trial by fire on production-grade software.

Original Notes

cs graduates have not been taught what programming is all about
nearly all cs graduates taught themselves to code before college and continued to teach themselves during college.
- The system can teach you but it can also act as a diploma farm
cs programs often do not prepare people for what they will find in the industry.
- This is apparently common in most disciplines?
Mentoring can happen just through observation
Mentoring can happen from a manual that is well written.
Mentoring can come in many forms
Doctors do not just start on the field. The medical field has a discipline of intense mentoring.
The medical profession oversees universities to make sure graduates have the best education.
Even with a degree doctors are required to send a year under supervision (internship). After this internship, they go on to do a residency for 3 to 5 years. Once all of this is done they can finally take an exam to become board certified.
If doctors have this intense process why don't we do it for software? While deaths are uncommon from software there are possibilities for major monetary loss.
It is insane to expect to hire someone right out of college and ask them to work with a team to build a critical system.
- This practice is seen basically nowhere: plumbers, painters, and electricians do not even do this.
This topic does matter software is far more important than we may realize basically our whole civilization runs on it anymore.
Apprenticeship
- Masters
  - Have taken lead on more than one significant project.
  - typically have 10+ years experience
  - have worked in many languages, systems, and operating systems.
  - know how to lead and coordinate multiple teams.
  - are proficient designers and architects.
  - can code circles around everyone else
  - Generally have been offered management positions
  - maintain a technical role by reading, studying, practicing, doing, and teaching.
- Journeymen
  - Programmers that are trained, competent and energetic
  - Learns to work well with a team and how to become a team leader.
  - Knowledge about technology but lack experience with diverse systems.
  - Tend to know one language, one system, and one platform.
  - Average is around 5 years experience
  - Inbetween Master and Apprentice stages.
  - supervised by masters
  - seldom allowed autonomy
  - code is scrutinized.
- Apprentices/Interns
  - Starting point
  - have no autonomy
  - closely supervised by journeymen
  - start off taking no tasks at all and instead just assist journeymen
  - Intense pair-programming stage.
  - foundational period for values
  - journeymen teach them: design principles, design patterns, disciplines, and rituals. For example, things like TDD, refactoring, and estimation would be taught this way.
  - Should last at least a year
  - masters should evaluate a deemed ready apprentice to see if they should become a journeyman.
- In reality
  - Graduates are generally supervised by young team leads.
  - supervision is not technical
  - programmers get raises and promotions because that is just what you do
  - the key difference between reality and ideal (other points before this) is to focus on technical teaching, training, supervision, and review.
  - What is missing today is the responsibility of elders to teach the young.
A craftsman is someone who works quickly, but without rushing, who provides reasonable estimates, and meets commitments.
Craftsmanship is a mindset held by craftsmen (duh)
Craftsmanship is handed from one person to another and is taught by elders to young.
You can't convince people to be craftsmen. To get others to emulate craftsmanship you must become a craftsman first and let your craftsmanship show.
It is time for apprenticeship to be a part of software.

Tooling

As time has gone on tooling has become better. This chapter is mostly about Bob's preferences for tooling.

Original notes

best source control tooling comes from open source. ones not made this way are generally tools that are sold to managers, not developers.
pessimistic locking is dead. tooling has become more advanced and merging issues are now manageable.
The power of git is its branching strategy.
git and tools like it are the future of source control.
editors
- vi is tried and true it may be old but it maintains its presence due to its speed and ease of use.
- emacs is one of the most powerful editors around.
- Eclipse and Intellij, these editors allow for much more sophisticated manipulation of source code. They have powerful tools for renaming variables, class manipulation, and other similar features.
- TextMate is powerful and lightweight
common tools for issue tracking: Pivotal Tracker, Lighthouse, and Wikis. The best place to start is often manual then find a tool that matches your style.
Having thousands of bugs or features often means you have done something wrong.
CI
- Jenkins is a solid tool.
- CI should run automatically on every code check-in
- The build must be working at all times.
- Failure should never be allowed to persist for over a day or two.
Unit testing
- Bob's favorites: JUnit, RSPEC, NUnit, Midje, CppUTest
- features all testing frameworks should have.
  - should be quick and easy to run tests.
  - should give a clear pass-fail indication.
  - should give a clear visual indicator of progress.
  - should discourage individual tests for communicating with one another.
  - should make it easy to write tests.
Component testing
- API testing tools
- FitNesse, is Bob's own creation and he likes to use it for component testing.
- Other tools in this camp: RobotFx, Green Pepper, Cucumber, and JBehave.
Good UI testing tools are selenium and Watir.
UML has not proven to be a sufficient tool to express all of the details of a program. This is why we still write code instead of UML documents.

Insights

Coming into this book I did not know if I was going to like it or not. This book is highly focused on the soft skills of programming that are rarely focused on in our industry. While I do not agree with Robert on every point I found myself in agreement a good amount of the time.

There is a common thread I did notice while reading this book. Robert loves to tell a story. What is interesting about this is while some of these stories and annecdotes are amusing they often contain a nugget of wisdom. While I do not fully understand how someone can make some of the mistakes Robert has in his career I did enjoy reading them. I often find the best way to learn is to make a mistake and its even better when you can learn from someone elses mistakes. I am assured that some mistakes shown in this book are not just Robert being a stupid kid and are mistakes even would be professionals could make.

A lot of this book seems to talk about dealing with pressure. Maybe the industry has changed or I am a personally that is not easily impacted by unrealistic requests, but I do not often feel the same pressures as described in this book. I understand that I get pressure from product owners, product managers and business analysts but I am always sure to let them know what is at stake. In general I see it as me doing my job and them doing their job.

The concept of practicing and continued studying seems like a no brainer to me but I could see where others might not be willing to devote time into this practice and it is a good call out. Not everything you learn in this industry is on the job.

I think one of the most impactful parts of this book for me was the apprenticeship discussion. I have always felt like the industry has been missing this. I have often heard of people mixing bootcamps and college but that still doesn't quite feel right to me. Personally I prefer the idealized apprenticeship model discussed in this book. This is a practice that is done in many trades and professions and it still boggels my mind that this is not a common or documented practice in software.

Book Notes

Thu, 25 Mar 2021 00:00:00 GMT

Book Notes

Recently I have found that some of the books on the topic of software engineering and other business endeavors have somewhat left me as time goes on. Ideally, I should be taking notes or at least capturing some insight on these topics as I read these books.

Every once in a while I will be adding a post about some book I have read. It might not be well thought out and the format is probably going to read like notes with some reflection.

Since this site is mostly for me and I doubt I have an audience I think this is acceptable. If you have an issue with these posts I am sorry but they are not really for you. If you get value out of them great.

If you are not interested in this topic I will be sure to prefix all articles with the Title Book Note.

TCP Transport

Sat, 07 Nov 2020 00:00:00 GMT

TCP Transport

As a continuation of the transport layer series, in this blog, we will be looking at our second transport layer protocol. Now that we have taken a look at UDP and understand its reliability issues it's time to look at its more reliable alternative. This protocol is the transmission control protocol or TCP for short. TCP is meant to be a layered protocol architecture. In this case, TCP is a layer that builds on IP much like UDP, but TCP is also meant to be extended by other protocols like telnet, FTP, and HTTP. The philosophy of TCP emphasizes a couple of different roles that TCP plays.

Basic Data Transfer

TCP should be able to transfer a continuous stream of octets in each direction by packaging some number of octets into segments for transmission. The availability of a push function should be defined for when users need to be sure all data they have sent has been transmitted.

Reliability

TCP must be able to recover from data that is damaged, lost, duplicated, or delivered out of order. This could happen for multiple reasons such as a faulty network connection or switch. The reason for the loss is not particularly important what is important is that TCP is resilient to such failures. This is one way in which TCP is drastically different from UDP as UDP had no protection of any such problem. To detect damaged segments a checksum is used.

Flow Control

TCP provides a window. This window acts as a sizing mechanism that allows the receiver of data to control the amount of data the sender is allowed to send at a time.

Multiplexing

Since TCP wants to allow multiple processes on a single host to use the protocol, ports are used. In this case, the host is responsible for binding the ports to the processes. The combination of network and host address form a socket. This socket acts as the identifier for one end of a TCP connection. A pair of sockets makes up a unique connection. While a socket identifies one end of a connection it does not have to be used in just one connection. A socket can be reused for multiple different connections at the same time.

Connections

A connection is made up of the combination of sockets, sequence numbers, and window sizes. This connection is uniquely identified by a pair of sockets. The connection must first be successfully established before two processes can communicate over TCP. This is done using a handshaking mechanism with clock-based sequence numbers to avoid erroneous initializations of connections. When the communication between the two processes is completed the connection is terminated or closed.

Precedence and Security

TCP includes mechanisms for setting connection precedence and security. In the case of TCP, users can indicate the security and precedence their communication will use. If this is not honored by both sides of the connection then a successful connection cannot be established.

One other important thing to note before moving on is that TCP introduces the concept of a Transmission Control Block or TCB for short. The TCB stores information about the connection on the host. In a Linux based environment, you can see this information by running the command netstat -at in a terminal.

Reviewing the network protocol

Fields

Source Port

16-bit source port number.

Destination Port

16-bit destination port number.

Sequence Number

32-bit sequence number of the first data octet in the segment. If SYN is present the sequence number is the initial sequence number (ISN) and the first data octet is ISN+1.

Acknowledgment Number

32-bit field that contains the value of the next sequence number the sender of the segments is expecting to receive.

Data Offset

4-bit number that specifies where the data begins. This number represents the number of 32-bit words in the TCP header itself.

Reserved

6 bits that are reserved for future use.

Control bits

6 bits that take on any of the following

Value	Meaning
URG	Urgent pointer field
ACK	Acknowledgment field
PSH	Push function
RST	Reset the function
SYN	Synchronize sequence numbers
FIN	No more data to transfer

Window

16-bit field that specifies the number of data octets starting with the one indicated in the acknowledgment field which the sender is willing to accept.

Checksum

16-bit field which is the one's complement of the one's complement sum of all 16-bit words in the header and text. In the UDP blog, I talked more about calculating the checksum, a very similar calculation would happen for TCP.

Urgent Pointer

16-bit field that communicates the current value of the urgent pointer. This pointer points to the sequence number of the octet following the urgent data.

Options

Options are variable in size but must be a multiple of 8 bits in length. All options are included in the checksum. All options have an option-kind and may have an option length.

End of Option List

The end of the options list has a Kind of 0. This option indicates the end of the options list. This option is only needed if the end of the options would not match up with the end of the TCP header. That is to say, if there are extra bytes after the options this option must be provided.

No-Operation

The no-operation has a Kind of 1. This can be used between options; However, there is no guarantee that senders will use this option.

Maximum Segment Size

The maximum segment size has a Kind of 2 and a length of 4 bytes. This option communicates the maximum receive segment size to the TCP that sends this segment. This option can only be used in the initial connection request.

Padding

Padding is a variable-sized field used to ensure that the header ends and data begins on a 32-bit boundary.

Data

The data that is being sent.

Terminology

Send Sequence Variables

Variable	Meaning
SND.UNA	send unacknowledged
SND.NXT	send next
SND.WND	send window
SND.UP	send urgent pointer
SND.WL1	segment sequence number used for last window update
SND.WL2	segment acknowledgment number used for last window update
ISS	initial send sequence number

Recieve Sequence Variables

Variable	Meaning
RCV.NXT	receive next
RCV.WND	receive window
RCV.UP	receive urgent pointer
IRS	initial receive sequence number

Segment Variables

Variable	Meaning
SEG.SEQ	segment sequence number
SEG.ACK	segment acknowledgment number
SEG.LEN	segment length
SEG.WND	segment window
SEG.UP	segment urgent pointer
SEG.PRC	segment precedence value

Connection states

TCP is driven by different connection states. A connection moves from one connection state to another in response to events. The events a user can call to influence connection state are: OPEN, SEND, RECEIVE, CLOSE, ABORT, and STATUS. Information about the connection state can be found on the TCB.

LISTEN

In the listen state socket is waiting for a connection request from any remote TCP and port.

SYN-SENT

In the SYN-SENT state, the socket is waiting for a matching connection request after having sent a connection request of its own.

SYN-RECEIVED

In the SYN-RECEIVED state, the socket is waiting for a confirmation connection request acknowledgment after having both sent and received a connection request.

ESTABLISHED

In the ESTABLISHED state, the socket is an open connection. This is the normal data transfer phase of the connection.

FIN-WAIT-1

In the FIN-WAIT-1 state, the socket is waiting for a connection termination request from the remote TCP or an acknowledgment of the connection termination request that had previously been sent.

FIN-WAIT-2

In the FIN-WAIT-2 state, the socket is waiting for a connection termination request from the remote TCP.

CLOSE-WAIT

In the CLOSE-WAIT state, the socket is waiting for a connection termination request from the local user.

CLOSING

In the CLOSING state the socket is waiting for a connection termination request acknowledgment from the remote TCP.

LAST-ACK

In the LAST-ACK state, the socket is waiting for an acknowledgment of the connection termination request that had previously been sent to the remote TCP.

TIME-WAIT

In the TIME-WAIT state, the socket is waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.

CLOSED

In the CLOSED state, the socket has no connection.

Sequence Numbers

Sequence numbers are a key part of how TCP ensures no packet loss or duplication. In TCP every octet of data sent on the wire has a sequence number. Each of these sequence numbers can be acknowledged by the remote TCP. In the case of TCP acknowledgment is a cumulative mechanism. In the case of acknowledgment, if you acknowledge sequence N that means the receiving TCP has acknowledged all data octets up to but not including N. Sequence numbers occupy a finite space, however that space is rather large 0 - 2³² - 1. Since a sequence number occupies a finite amount of space, all arithmetic dealing with sequence numbers must be performed with modulo 2³².

Some examples of arithmetic required for dealing with sequence numbers include the following.

Determining that an acknowledgment refers to some sequence number sent but not acknowledged.
Determining that all sequence numbers occupied by a segment have been acknowledged.
Determining that an incoming segment contains sequence numbers that are expected.

When it comes to processing acknowledgments there are a couple of different comparisons that are required.

Variable	Meaning
SND:UNA	The oldest unacknowledged sequence number.
SND.NXT	The next sequence number to be sent.
SEG.ACK	An acknowledgment from the receiving TCP.
SEG.SEQ	The first sequence number of a segment.
SEG.LEN	The number of octets occupied by the data in the segment.

With these variables, you can combine values to produce additional information. For example, SEG.SEQ+SEG.LEN-1 represents the last sequence number of a segment. Likewise, you can tell if a segment on the retransmission queue is fully acknowledged. In this case, if the sum of its sequence number and length is less or equal to the acknowledgment value in the incoming request. This could be represented in this way SND.UNA < SEG.ACK =< SND.NXT.

When it comes to processing received data a different set of comparisons is required.

Variable	Meaning
RCV.NXT	The next sequence number expected on an incoming segment. This is the left or lower edge of the receive window
RVC.NXT+RCV.WND-1	The last sequence number expected on an incoming segment. This is the right upper edge of the receive window.
SEG.SEQ	The first sequence number occupied by the incoming segment.
SEG.SEQ+SEG.LEN-1	The last sequence number occupied by the incoming segment.

TCP can determine if a segment occupies a portion of a valid receive sequence using either RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND or RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND. In the case of RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND this operation is checking to see if the beginning of the segment falls in a window. In the case of RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND this operation is checking to see if the end of the segment falls in a window. One thing that can complicate this logic is if the window size is zero or a zero segment exists. This is possible but a bit of an edge case.

There are a couple of special cases when it comes to sequence numbers. In the case of TCP SYN and FIN are not subject to sequence numbers. SYN is always expected to occur before the first data octet while FIN is always expected to occur after the last data octet.

Initial Sequence number

The initial sequence number or ISN for short is used to help TCP identify duplicate segments from previous connections. This kind of problem and typically happen when a connection is opened and closed in quick succession or when a connection breaks with loss of memory and is reestablished. Even in the case of a crash with a total loss of knowledge of sequence numbers TCP must prevent segment reuse. To prevent segment reuse an ISN generator is used to create a rather unique number. This generator selects a 32-bit ISN based on a 32-bit clock. The 32-bit clock is incremented around every 4 microseconds giving the ISN cycle time a 4.55-hour window. Since 4.55 hours is not unique enough due to the long-lived nature of some TCP connections, the addition of a Maximum Segment Lifetime or MSL is required. The MSL is used to make sure TCP does not create a segment that carries a duplicated sequence number. TCP must keep quiet for the MSL before it is allowed to assign any sequence numbers after recovering from a crash in which memory loss of sequence numbers in use occurred. When the MSL is less than 4.55 hours we can assume that ISN's are unique. Every connection has a send sequence number and a receive sequence number. The initial send sequence number or ISS is decided by the data sending TCP. The initial receive sequence number or IRS is learned during the connection establishing procedure. For a TCP connection to be established or initialized both TCPs must synchronize based on each other's ISN. This exchange of segments is done with a SYN and the sequence numbers. The process of synchronization requires each TCP to send its ISN and receive confirmation of it in an ACK.

Example flow

A --> B SYN sequence number is X
A <-- B ACK sequence number is X
A <-- B SYN sequence number is Y
A --> B ACK sequence number is Y

steps 2 and 3 can be combined in a single message. This the three-way handshake.

Establishing a connection

Since TCP is not able to be tied to a global clock for ISN generation, the process of exchanging ISNs is done through a three-way handshake. Below are some examples of how this handshake works.

Simple Three-way handshake

TCP A	TCP A Direction	Data	TCP B Direction	TCP B
CLOSED				LISTEN
SYN-SENT	-->	<SEQ=100><CTL=SYN>	-->	SYN-RECEIVED
ESTABLISHED	<--	<SEQ=300><ACK=101><CTL=SYN,ACK>	<--	SYN-RECEIVED
ESTABLISHED	-->	<SEQ=101><ACK=301><CTL=ACK>	-->	ESTABLISHED
ESTABLISHED	-->	<SEQ=101><ACK=301><CTL=ACK>	-->	ESTABLISHED

Simultaneous Three-way handshake

TCP A	TCP A Direction	Data	TCP B Direction	TCP B
CLOSED				CLOSED
SYN-SENT	-->	<SEQ=100><CTL=SYN>	In transit
SYN-RECEIVED	<--	<SEQ=300><CTL=SYN>	<--	SYN-SENT
	In transit	<SEQ=100><CTL=SYN>	-->	SYN-RECIEVED
SYN-RECIEVED	-->	<SEQ=100><ACK=301><CTL=SYN,ACK>	In transit
ESTABLISHED	<--	<SEQ=300><ACK=101><CTL=SYN,ACK>	<--	SYN-RECEIVED
	In transit	<SEQ=101><ACK=301><CTL=ACK>	-->	ESTABLISHED

The three-way handshake allows for recovery from duplicate messages by using the reset function.

Recovery from a crash (half-open)

TCP A	TCP A Direction	Data	TCP B Direction	TCP B
(Crash)
CLOSED
SYN-SENT	-->	<SEQ=400><CTL=SYN>	-->	invalid
bad response	<--	<SEQ=300><ACK=100><CTL=ACK>	<--	ESTABLISHED
SYN-SENT	-->	<SEQ=100><CTL=RST>	-->	Abort
SYN-SENT				CLOSED
SYN-SENT	-->	<SEQ=400><CTL=SYN>	-->

Resets

Resets are used to reset the connection in the case of invalid or unexpected data. For example, if a segment arrives which is not intended for the current connection, or the connection does not exist then a reset must be sent. Another case in which a reset should be sent would be when the connection gets into a non-synchronized state. An example of this state would be a TCP receiving an acknowledgment for a segment that has not been sent yet. To validate a reset the sequence fields are used. If the sequence number is in the window then the reset is considered valid. Once a receiver validates a reset request it then changes state. In the case that the receiver was in the LISTEN state the reset request is ignored. In the case that the receiver is in the SYN-RECEIVED state they will return to the LISTEN state. In all other cases, the receiver will abort the connection and go to the CLOSED state.

Closing connection

In the case of TCP, the CLOSE operations signal that "I have no more data to send." In this case, the issuer of the close request may continue to receive data until they are told that the other side has closed the connection as well. By doing this TCP ensures that all data buffers sent before the connection was closed had been delivered. There are a couple of cases in which a connection can be closed.

The first case is when the local user initiates the close. In this case, a FIN segment is created and placed on the outgoing segment queue. Past this point no further SENDs from the user will be accepted, only receives are allowed from this state. When the other TCP has acknowledged the FIN and sent a FIN of its own the closing TCP can ACK the FIN and end the connection.

The second case is when a TCP receives a FIN from the network. This is FIN is considered unsolicited. In this case, the TCP that received the FIN will respond with a CLOSE. If an ACK does not come within a timeout then the connection is aborted.

The third case is when both sides of the connection close at the same time. In this case, both sides exchange FIN segments. Once both sides have received ACKs the connection will be deleted.

Precedence and security

Precedence and security are used to make sure the connection between ports are operating on the same security and compartment values. Any connection attempt with mismatched security or compartment values must be rejected by sending a reset.

Data Communication

TCP is a data communication protocol, so the exchange of data is very important. In the case of TCP data is communicated by the exchange of segments. Unlike with UDP TCP uses retransmission to handle the loss of segments due to errors or network congestion. Since TCP does use retransmission duplicate segments can arrive but this is not an issue due to the way segments are acknowledged. In these cases, the sender keeps track of the next expected sequence number in the SND.NXT while the receiver keeps track of the next sequence number in the RCV.NXT. The sender also keeps track of the oldest unacknowledged sequence number in the SND.UNA. One important thing to note is that SND,NXT and RCV.NXT is incremented by the length of data in the segment.

When sending data over TCP an important thing to consider is the window size. In the case of a large window transmissions are encouraged, but in the case of a small window transmission of data can be restricted to the point of introducing a round trip delay.

Interfaces

User Commands
- Open - opens the TCP connection.
- Send - sends data in a buffer. Send is considered an error if it occurs on a connection that is not open. Multiple sends can be called before closing the connection.
- Receive - Waits for data and returns to the program once the buffer is filled.
- Close - causes a connection to be closed.
- Status - Information from the TCB about the connection.
  - foreign socket
  - local connection name
  - receive window
  - send window
  - connection state
  - number of buffers awaiting ack
  - number of buffers pending receipt
  - urgent state
  - precedence
  - security
  - transmission timeout
- Abort - cancels all pending SENDs and RECEIVEs
TCP to user messages
- When TCP signals a user program it should provide the following information
  - Local Connection Name
  - Response String
  - Buffer Address
  - Byte count
  - Push flag
  - Urgent flag

Event Processing

The activity of TCP can be defined as responding to events. In the case of TCP, there are 3 different kinds of events.

User Calls
- OPEN
- SEND
- RECEIVE
- CLOSE
- ABORT
- STATUS
Arriving Segments
- SEGMENT ARRIVES
Timeouts
- USER TIMEOUT
- RETRANSMISSION TIMEOUT
- TIME-WAIT TIMEOUT

Example of a standard library

Much like UDP, TCP is a standard protocol and it is relatively easy to find support for it in most languages standard library. Below are just a few examples of programming languages and their associated TCP library.

nodejs - https://nodejs.org/api/net.html
Java - https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html
Go - https://golang.org/pkg/net/
Python - https://docs.python.org/3/library/socket.html
Rust - https://doc.rust-lang.org/beta/std/net/struct.TcpStream.html

Example of a client-server application using the protocol

https://github.com/JeffreyRiggle/tcp-example-app

This application is very similar to the one covered in the last blog about UDP. The notable differences in this example are: there is a disconnect button, and the application uses TCP instead of UDP.

Look at the results in a network capture

Similar to the UDP analysis we will be using Wireshark to look at the network captures. For more information on install and setup see the previous blog.

Initial connection

In this case, we can see a simple three-way handshake occurs between the connecting client at port 50174 and the server on port 7000.

Source Port	Destination Port	Flags	Notes	Content
50174	7000	SYN	Initial connection request from the client with an ISN of 1047673890	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 3c ad db 40 00 40 06 8e de 7f 00 00 01 7f 00 .<.Û@.@..Þ...... 0020 00 01 c3 fe 1b 58 3e 72 3c 22 00 00 00 00 a0 02 ..Ãþ.X>r<".... . 0030 ff d7 fe 30 00 00 02 04 ff d7 04 02 08 0a 66 e1 ÿ×þ0....ÿ×....fá 0040 a0 f1 00 00 00 00 01 03 03 07 ñ........
7000	50174	SYN, ACK	Acknowledge the connection and provide ISN 3883755340	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 3c 00 00 40 00 40 06 3c ba 7f 00 00 01 7f 00 .<..@.@.<º...... 0020 00 01 1b 58 c3 fe e7 7d 67 4c 3e 72 3c 23 a0 12 ...XÃþç}gL>r<# . 0030 ff cb fe 30 00 00 02 04 ff d7 04 02 08 0a 66 e1 ÿËþ0....ÿ×....fá 0040 a0 f2 66 e1 a0 f1 01 03 03 07 òfá ñ....
50174	7000	ACK	Acknowledge the ISN	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 34 ad dc 40 00 40 06 8e e5 7f 00 00 01 7f 00 .4.Ü@.@..å...... 0020 00 01 c3 fe 1b 58 3e 72 3c 23 e7 7d 67 4d 80 10 ..Ãþ.X>r<#ç}gM.. 0030 02 00 fe 28 00 00 01 01 08 0a 66 e1 a0 f2 66 e1 ..þ(......fá òfá 0040 a0 f2 ò

Client sending a message

In this case, we send a simple hello world message to the server. Just like in the UDP example the client sends a custom byte structure while the server sends a JSON response.

Source Port	Destination Port	Flags	Notes	Content
50174	7000	PSH,ACK	The client is sending a hello world message to the server.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 41 ad df 40 00 40 06 8e d5 7f 00 00 01 7f 00 .A.ß@.@..Õ...... 0020 00 01 c3 fe 1b 58 3e 72 3c 24 e7 7d 67 65 80 18 ..Ãþ.X>r<$ç}ge.. 0030 02 00 fe 35 00 00 01 01 08 0a 66 f1 44 8d 66 e1 ..þ5......fñD.fá 0040 a1 52 02 48 65 6c 6c 6f 20 57 6f 72 6c 64 21 ¡R.Hello World!
7000	50174	PSH,ACK	The server acknowledges the message and sends a message of its own.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 7b 8c 49 40 00 40 06 b0 31 7f 00 00 01 7f 00 .{.I@.@.°1...... 0020 00 01 1b 58 c3 fe e7 7d 67 65 3e 72 3c 31 80 18 ...XÃþç}ge>r<1.. 0030 02 00 fe 6f 00 00 01 01 08 0a 66 f1 44 8e 66 f1 ..þo......fñD.fñ 0040 44 8d 7b 22 69 74 65 6d 73 22 3a 5b 7b 22 74 69 D.{"items":[{"ti 0050 6d 65 22 3a 31 36 30 34 36 35 39 35 31 36 36 30 me":160465951660 0060 30 2c 22 6d 65 73 73 61 67 65 22 3a 22 48 65 6c 0,"message":"Hel 0070 6c 6f 20 57 6f 72 6c 64 21 22 7d 5d 2c 22 73 65 lo World!"}],"se 0080 67 6d 65 6e 74 22 3a 31 7d gment":1}
50174	7000	ACK	The client acknowledges the last message from the server.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 34 ad e0 40 00 40 06 8e e1 7f 00 00 01 7f 00 .4.à@.@..á...... 0020 00 01 c3 fe 1b 58 3e 72 3c 31 e7 7d 67 ac 80 10 ..Ãþ.X>r<1ç}g¬.. 0030 02 00 fe 28 00 00 01 01 08 0a 66 f1 44 8e 66 f1 ..þ(......fñD.fñ 0040 44 8e D.

Client disconnect

Source Port	Destination Port	Flags	Notes	Content
50174	7000	FIN,ACK	Request to close connection from the client.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 34 ad e1 40 00 40 06 8e e0 7f 00 00 01 7f 00 .4.á@.@..à...... 0020 00 01 c3 fe 1b 58 3e 72 3c 31 e7 7d 67 ac 80 11 ..Ãþ.X>r<1ç}g¬.. 0030 02 00 fe 28 00 00 01 01 08 0a 66 f6 ea 0e 66 f1 ..þ(......föê.fñ 0040 44 8e D.
7000	50174	FIN,ACK	Acknowledge close and request close from the server.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 34 8c 4a 40 00 40 06 b0 77 7f 00 00 01 7f 00 .4.J@.@.°w...... 0020 00 01 1b 58 c3 fe e7 7d 67 ac 3e 72 3c 32 80 11 ...XÃþç}g¬>r<2.. 0030 02 00 fe 28 00 00 01 01 08 0a 66 f6 ea 0e 66 f6 ..þ(......föê.fö 0040 ea 0e ê.
50174	7000	ACK	Acknowledge the close connection from the server.	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 34 ad e2 40 00 40 06 8e df 7f 00 00 01 7f 00 .4.â@.@..ß...... 0020 00 01 c3 fe 1b 58 3e 72 3c 32 e7 7d 67 ad 80 10 ..Ãþ.X>r<2ç}g... 0030 02 00 fe 28 00 00 01 01 08 0a 66 f6 ea 0e 66 f6 ..þ(......föê.fö 0040 ea 0e ê.

Interesting finds

While looking into this I did find a couple of interesting finds. One find that I was remotely aware of is that many variants of HTTP are built on TCP. One notable exception to that rule is the upcoming HTTP3 or QUICK which is built on a multiplexed UDP connection. Another interesting find was that many online games use TCP as the transmission protocol. The thing that was a bit surprising about this is that TCP adds a lot of overhead with all of the acknowledgments so I was unsure that an online game could perform at the speed expected with all that additional checking.

References

https://tools.ietf.org/html/rfc793

UDP Transport

Tue, 02 Jun 2020 00:00:00 GMT

UDP Transport

This is the first in the transport layer series. In this blog, we will look into the User Datagram Protocol or UDP for short. UDP is part of the Internet Protocol (IP) suite and resides in the application layer. UDP is an unreliable protocol, that is not to say it is bad it just does not guarantee packet delivery like some other protocols.

Reviewing the network protocol

UDP is made up of 5 parts: Source Port, Destination Port, Length, Checksum, and data octets.

Source Port

The source port is an optional field. This field indicates the port that the sending process is using to communicate.

Destination Port

This is the port that has a meaning within the context of a particular internet destination address.

Length

Length is the number in octets of the user datagram including the header and the data. The minimum possible value for length is 8.

Checksum

The checksum is the 16-bit one's complement of the ones complement sum of a pseudo-header using the IP header, the UDP header, and the data. This is padded with zero octets and the end to make a multiple of two octets.

Example

Source IP address: 192.168.0.15 Destination IP address: 192.168.0.23 Source Port: 35 Destination Port: 40 Length: 10 Message: Hi

value	Decimal	Binary	Hex
Source IP	192.168 0.15	1100 0000 1010 1000 0000 0000 0000 1111	C0 A8 00 0F
Destination IP	192.168 0.23	1100 0000 1010 1000 0000 0000 0001 0111	C0 A8 00 17
Reserved/UDP	0/17	0000 0000 0001 0001	00 11
Padding	0/10	0000 0000 0000 1010	00 0A
Source Port	35	0000 0000 0010 0011	00 23
Destination Port	40	0000 0000 0010 1000	00 28
Length	10	0000 0000 0000 1010	00 0A
Data	Hi	0100 1000 0110 1001	48 69

Add up all numbers to generate pseudo-header
1100 0000 1010 1000
+
0000 0000 0000 1111
+
1100 0000 1010 1000
+
0000 0000 0001 0111
+
0000 0000 0001 0001
+
0000 0000 0000 1010
+
0000 0000 0010 0011
+
0000 0000 0010 1000
+
0000 0000 0000 1010
+
0100 1000 0110 1001
=
1 1100 1010 0110 1111

Since this number overflows split into two 16bit address spaces and add.
0000 0000 0000 0001
+
1100 1010 0110 1111
=
1100 1010 0111 0000

Take ones compliment to get checksum value.
0011 0101 1000 1111

Data Octets

The actual data you want to send in the packet.

field	bit range
Source Port	0-15
Destination Port	16-31
Length	32-47
Checksum	48-63
Data octets	remaning

Example of a standard library

UDP is a standard protocol and it is relatively easy to find support for it in most languages standard library. Below are just a few examples of programming languages and their associated UDP library.

nodejs - https://nodejs.org/api/dgram.html
Java - https://docs.oracle.com/javase/7/docs/api/java/net/DatagramSocket.html
Go - https://golang.org/pkg/net/
Python - https://docs.python.org/3/library/socket.html
Rust - https://doc.rust-lang.org/std/net/struct.UdpSocket.html

Example of a client-server application using the protocol

https://github.com/JeffreyRiggle/udp-example-app

The following Github page shows a simple example of a UDP based client-server. In this example, we have a UDP server that listens on port 4000.

The client in this case is an electron-based application. To reduce complexity the UI does not use any modern framework and just interacts directly with the DOM API.

Overview of the architecture

For now to show that UDP does not enforce any structure on the payload the messages sent to the server use a binary like structure. However, the response from the UDP server is in JSON and can be parsed.

Look at the results in a network capture

To take a look at how this application works at the network layer Wireshark will be used. This application can be rather easily installed on most operating systems.

Once Wireshark has installed a capture of network traffic can be issued. Since I am running this on my local machine I need to capture network on my loopback interface.

Now that the capture is running using the very simple example application we should be able to see UDP traffic flowing from the electron application to the UDP server.

Seq	Info	packet
1	Request existing messages	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ....... .......E. 0010 00 1d 5a 29 40 00 40 11 e2 a4 7f 00 00 01 7f 00 ..Z)@.@.â¤...... 0020 00 01 e3 77 0f a0 00 09 fe 1c 01 ..ãw. ..þ..
2	Respond with existing	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ........... ...E. 0010 00 34 5a 2a 40 00 40 11 e2 8c 7f 00 00 01 7f 00 .4Z*@.@.â....... 0020 00 01 0f a0 e3 77 00 20 fe 33 7b 22 69 74 65 6d ... ãw. þ3{"item 0030 73 22 3a 5b 5d 2c 22 73 65 67 6d 65 6e 74 22 3a s":[],"segment": 0040 30 7d 0}
3	Send new message	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 29 5a c0 40 00 40 11 e2 01 7f 00 00 01 7f 00 .)ZÀ@.@.â....... 0020 00 01 e3 77 0f a0 00 15 fe 28 02 48 65 6c 6c 6f ..ãw. ..þ(.Hello 0030 20 57 6f 72 6c 64 21 World!
4	Ack message	0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00 ..............E. 0010 00 63 5a c1 40 00 40 11 e1 c6 7f 00 00 01 7f 00 .cZÁ@.@.áÆ...... 0020 00 01 0f a0 e3 77 00 4f fe 62 7b 22 69 74 65 6d ... ãw.Oþb{"item 0030 73 22 3a 5b 7b 22 74 69 6d 65 22 3a 31 35 39 31 s":[{"time":1591 0040 30 30 39 32 36 37 32 39 33 2c 22 6d 65 73 73 61 009267293,"messa 0050 67 65 22 3a 22 48 65 6c 6c 6f 20 57 6f 72 6c 64 ge":"Hello World 0060 21 22 7d 5d 2c 22 73 65 67 6d 65 6e 74 22 3a 31 !"}],"segment":1 0070 7d }

Looking at these messages you will notice that all messages going to the server are buffers and all messages coming from the server are buffers containing JSON based strings. As mentioned earlier this was done to show that UDP does not enforce structure on message content.

To better understand what is going on here lets take a closer look at sequence packets 3 and 4.

In packet 3 the client (electron application) is sending a new message to the server. 0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00
0010 00 29 5a c0 40 00 40 11 e2 01 7f 00 00 01 7f 00
0020 00 01 e3 77 0f a0 00 15 fe 28 02 48 65 6c 6c 6f
0030 20 57 6f 72 6c 64 21

Byte range	Bytes	Infomation	Value
0 - 5	00 00 00 00 00 00	Destination IP Address	0.0.0.0
6 - 11	00 00 00 00 00 00	Source IP Address	0.0.0.0
12 - 13	08 00	Type	IPv4
14	45	Header Length	20 bytes
15	00	Explicit Congestion Notification	0
16 - 17	00 29	Total Length	41
18 - 19	5A C0	Identification	23232
20 - 21	40 00	Fragment offset	0
22	40	Time to live	64
23	11	Protocol (start of UDP protocol)	UDP (17)
24 - 25	E2 01	Header checksum	0xE201
26 - 29	7F 00 00 01	Source	127.0.0.1
30 - 33	7F 00 00 01	Destination	127.0.0.1
34 - 35	E3 77	Source Port	58231
36 - 37	0F A0	Destination Port	4000
38 - 39	00 15	Length	21
40 - 41	FE 28	Checksum	0xFE28
42 - 54	02 48 65 6C 6C 6F 20 57 6F 72 6C 64 21	Data	.Hello World!

If we break down the data we will notice that the message data is comprised of two parts a type and a message. In this case, 02 is the type and for the application in question 02 is a new message. The remainder of the message 48 65 6C 6C 6F 20 57 6F 72 6C 64 21, is Hello World! as a UTF-8 byte array.

Moving on to packet 4 the server has seen and processed packet 3 and is ready to respond.

0000 00 00 00 00 00 00 00 00 00 00 00 00 08 00 45 00
0010 00 63 5a c1 40 00 40 11 e1 c6 7f 00 00 01 7f 00
0020 00 01 0f a0 e3 77 00 4f fe 62 7b 22 69 74 65 6d
0030 73 22 3a 5b 7b 22 74 69 6d 65 22 3a 31 35 39 31
0040 30 30 39 32 36 37 32 39 33 2c 22 6d 65 73 73 61
0050 67 65 22 3a 22 48 65 6c 6c 6f 20 57 6f 72 6c 64
0060 21 22 7d 5d 2c 22 73 65 67 6d 65 6e 74 22 3a 31
0070 7d

Byte range	Bytes	Infomation	Value
0 - 5	00 00 00 00 00 00	Destination IP Address	0.0.0.0
6 - 11	00 00 00 00 00 00	Source IP Address	0.0.0.0
12 - 13	08 00	Type	IPv4
14	45	Header Length	20 bytes
15	00	Explicit Congestion Notification	0
16 - 17	00 63	Total Length	99
18 - 19	5A C1	Identification	23233
20 - 21	40 00	Fragment offset	0
22	40	Time to live	64
23	11	Protocol (start of UDP protocol)	UDP (17)
24 - 25	E1 C6	Header checksum	0xE1C6
26 - 29	7F 00 00 01	Source	127.0.0.1
30 - 33	7F 00 00 01	Destination	127.0.0.1
34 - 35	0F A0	Source Port	4000
36 - 37	E3 77	Destination Port	58231
38 - 39	00 4F	Length	79
40 - 41	FE 62	Checksum	0xFE62
42 - 112	7b 22 69 74 65 6d 73 22 3a 5b 7b 22 74 69 6d 65 22 3a 31 35 39 31 30 30 39 32 36 37 32 39 33 2c 22 6d 65 73 73 61 67 65 22 3a 22 48 65 6c 6c 6f 20 57 6f 72 6c 64 21 22 7d 5d 2c 22 73 65 67 6d 65 6e 74 22 3a 31 7d	Data	.{"item s":[{"time":1591 009267293,"messa ge":"Hello World !"}],"segment":1 }

In this case, we can see that the server has acknowledged the message and added it to the messages list. In our acknowledgment, we can see that Hello World! is segment 1 and the response from the server contained 1 segment. We can also see that the server used a JSON format to send the data back to the client.

Interesting finds

QUIC (HTTP/3)

HTTP/3 is backed by UDP due to its increased flexibility. In previous HTTP protocol versions, TCP was used as a base transport.

BISON

While working on this blog I got a bit carried away wondering if I could compress JSON into a smaller binary stream. I am not the first person who thought to do it but I had a lot of fun creating https://github.com/JeffreyRiggle/bison.

References

Transport Layer

Wed, 06 May 2020 00:00:00 GMT

Transport Layer

For the first time, I am going to dedicate a series of blogs to a single topic. In this case, I am going to dive into network protocols. In this series I am hoping to dive into a couple of different protocols starting with UDP and ending with HTTP, possibly including HTTP/3.

Why do this?

I usually enjoy looking at the lower levels of client-server applications, which in many cases is the network layer between the client and server. In the past, I have used TCP and HTTP in client server-based applications, but I have never really gotten to dig in as far as I would like to. I will also be using this as an opportunity to learn more about the network protocols that I use often.

General Format

In general, these next few blogs should follow a consistent pattern.

Reviewing the network protocol

At this point, we will look at the network protocol and help break down how the protocol is supposed to work. This will mostly be a recap of related RFC's maybe with some diagrams or other visuals to help aid in explaining how the protocol is supposed to work.

Building an example of the protocol

In this section, I hope to create a very basic implementation of the protocol in either Java or Javascript. This will simply talk about building it and layout some highlights but most of the work will be done in a Github repository and referenced in this section.

Look at the results in a network capture

Take a bit to show how you could use a wire capture application like Wireshark to look at the bytes in transit over the wire.

Example of a standard library

Since creating a unique implementation of a wire protocol is probably not the best idea and probably not the most secure, at this point, I will give examples of more reliable libraries that do the same sort of transportation.

Example of a client server application using the protocol

In this section, we will use the protocol to create a simple client-server application. This might be something simple like a chat system or something like that.

Interesting finds

In this section, I will highlight any interesting finds I maybe have stumbled upon during my research.

References

What I referenced to create this blog post.

What I will not be covering

While I am hoping to take a good look at network protocols in these blogs, I will not be looking at network security. In my opinion network security (UDPS, TLS, HTTPS) fall into a slightly different realm than what I am looking at right now. At some later point if I enjoy this process I might attempt another series on cryptography in which I would look at things like https.

Aurelia

Tue, 18 Feb 2020 00:00:00 GMT

Coming back to Aurelia

A while back I was comparing a couple of different javascript front-end frameworks. In hindsight, I probably could have saved some time using Real World for my comparisons.

During this time I decided to give Aurelia a shot. At first, it seemed like an ideal framework. At the time I had mostly been using Angularjs (1.x era) and anything without scope was starting to sound like a better framework.

Why Aurelia?

If you happened to look at the comparison you would notice that I evaluated two rather popular choices and one not so well known choice. At the time React still seemed like a questionable choice to me. I was having a hard time believing that the virutual dom would not cause so much memory pressure on the heap.

As for Aurelia, it was similar to Angular in the sense that it used an MVC pattern to present the view. The one thing I had liked was the syntax and oftentimes lack of it. I realize that starting in Angular 2+ the syntax got better, but still confusing bindings like [src], *ngFor, and (click) still bothered me. From a readability standpoint it's hard to remember the difference between those. Aurelia, on the other hand, chose to be a bit more explicit about its binding to the template. Instead, you would have bindings like src.bind, repeat.for and click.delegate. To me, this seemed a lot more readable and produced less mysterious-looking code.

What I learned

While I fully enjoyed using Aurelia, I grew to find certain aspects of the framework to be painstaking. What I had come to realize is that like in many frameworks I would hit a snag in my work and have to turn to documentation or sources like stack overflow to get moving in the right direction again. However, this time I found out that finding the answers to my questions was a lot harder. Documentation was sparse on the issues I was running into and the examples I had found ended up being years out of date.

I had also found that access to different plugins and wrappers for the language was limited. Where in other popular frameworks I would have a slew of choices for plugins, libraries, and tools, in Aurelia most of these did not exist or appeared to be years out of date.

Even finding the right combination of tools to use for something like testing was hard. Some examples showed and recommended jest while others used karma and jasmine. While the tool was not available at the time, eventually Aurelia did create a CLI which might have addressed many of the pain points I had encountered getting setup.

The hardest challenge for Aurelia is the lack of a strong community. While this is not the best metric in the world, it seems pretty clear to me using Github stars that Aurelia is not a super popular choice.

Project	Stars
Aurelia	653
Angular	57.9k
React	144k
Vue	157k

I had never really known how much of a difference a large community makes on a project, but after this experience, it has become clear to me that it makes a rather large difference. I am personally glad I learn this lesson in a way that the maintenance burden only effects me.

Easy CI/CD with github actions

Tue, 07 Jan 2020 00:00:00 GMT

Getting into it

Recently I had decided I wanted to start creating a CI/CD (Continuous Integration / Continuous Delivery) pipeline for some of the code I have up on Github. I have dealt with CI/CD in the past and generally have enjoyed the consistent feedback loop on the changes I push. One thing I did want with this CI was some kind of feedback loop in Github to show the current state of the project (failing, passing, etc).

Normally in these cases, I would reach for Jenkins, but I tend to like to try something different when I am working on non-work related projects. Looking around the big competitors I noticed had been Travis and Circle. However, I had heard of Github Actions and that sounded like a perfect fit for what I wanted to do.

It just so happened that the public release of Github's actions timed very well with me wanting to create some CI/CD for my projects.

Enter Github actions

Recently Github released Github Actions to the general public. Github actions is a CI/CD system for code hosted on Github. Actions can be added to any Github project by creating a YAML file in .github/workflows.

In a workflow there are a couple of important properties to know about.

name: The name property gives the workflow a name.
on: This can be one of a couple of different values. This is the trigger condition, when can this action be automatically executed. For me, the most valuable actions are triggered on push or pull_request but there are plenty of other triggers listed here.
jobs: This is the key to the workflow. A workflow has at least 1 job and a job is the main body of work to be done in a workflow.
- runs-on: Controls what the job runs on. So far I have only had to use ubuntu-latest, but Windows and Mac OS's are supported. For more information see this page.
- steps: These are the individual actions that a job will take.
- run: This is the script that will run in the step. In the case of ubuntu, this should be a bash script.

Simple example workflow

name: My CI
on: [push, pull_request]
jobs:
    greet:
        name: Greet
        runs-on: ubuntu-latest
        steps:
            - name: Greet
              run: echo 'Hello world'

Putting actions to use

After a bit of research, I started getting to work on creating a CI pipeline for one of my go based projects.

In this process, I created two jobs a build job and an integration test job. In the build job, a couple of things had been done.

Install go on the ubuntu image.
Checkout related go code from Github.
Install dependencies for go code.
Build the go application.
Run the unit tests.

In the integration test job, a couple more things would be done.

Install go on the ubuntu image.
Checkout related go code from Github
Install dependencies and start the go application.
Install NodeJS
Build and start a simple web application. This was used to test Webhooks with my go application.
Install python 2.
Run integration tests that were written in python to test the go server.

The end result for all of this is the following yml file.

name: Go
on: [push, pull_request]
jobs:

  build:
    name: Build
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.13
      uses: actions/setup-go@v1
      with:
        go-version: 1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v1

    - name: Get dependencies
      run: |
        go get -v -d -t ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi
    - name: Build
      run: go build -v .
      working-directory: ./src

    - name: Test
      run: go test
      working-directory: ./src

  Test:
    runs-on: ubuntu-latest
    steps:

      - name: Set up Go 1.13
        uses: actions/setup-go@v1
        with:
          go-version: 1.13
        id: go

      - name: Check out code into the Go module directory
        uses: actions/checkout@v1

      - name: Get dependencies
        run: |
          go get -v -d -t ./...
          if [ -f Gopkg.toml ]; then
              curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
              dep ensure
          fi
      - name: Start
        run: |
          go build -v .  
          ./src ../test/RuntimeConfig.json &
        working-directory: ./src
    
      - name: Install node
        uses: actions/setup-node@v1
        with:
          node-version: '10.x'

      - name: Start web hook server
        run: |
          npm install
          npm run start &
        working-directory: ./test/webhooktester
          
      - name: Install python
        uses: actions/setup-python@v1
        with:
          python-version: '2.x'
      
      - name: Install dependencies
        run: |
          pip install requests
          pip install assertpy
      
      - name: Run E2E Tests
        run: python apitest.py
        working-directory: ./test

UI

Run output

Commit feedback

Final thoughts

Github actions are really easy to get up and running. All you have to do is add a yml file to your source control and you are up and running. There is no concern around managing infrastructure like you would have to do with Jenkins.

One big difference that I was worried about was the loss of a scripting language in my pipeline. Again I am used to Jenkins in which I could define my workflow and have the power of groovy. At no point did I really feel limited by only being able to execute bash scripts.

It is also incredibly nice to have a decent UI to show my Job status as well as immediate feedback on my commit or PR around CI status.

Revisiting java

Mon, 12 Aug 2019 00:00:00 GMT

Why java?

As part of creating my website, I decided it would be a good idea to go back and update some projects I haven't touched in a while. If you haven't read my last post the idea is that my projects are listed on my website and those pull from my GitHub repositories to create pages.

While going through these I stumbled upon some of the first projects I put into Github. Most of these projects had been java based projects. Looking back on these projects I wanted to do two things: package javadocs, and upgrade to java 11. All of my projects had been written in java 8 before jigsaw, so I knew this might be a bit of a challenge.

Getting down to it.

Almost immediately I ran into issues. The first library I decided to work on was my core library. The first change I noticed was that JavaFX was no longer in the standard library. It appears that this had a relatively short run under the jdk justified by release cadence.

So after figuring out my JavaFX issue the process went rather smoothly. Figuring out jigsaw was not as hard as I imagined it might be. My toolchain had little to no issue handling java 11. Maven worked fine and my testing frameworks continued to work great.

However, at some point I got around to working more with JavaFX and started to run into more issues. I think I might just be an edge case, but in java 8 I was abusing some internal/protected features of JavaFX to solve problems I had. Since jigsaw did a good job of encapsulating protected features, I immediately had issues to solve. The big problem I had was JavaFX seems to be a rather mature UI toolkit but it is still missing some features.

The big feature I was having issues around was JavaFX's Tab. From what I can tell my use case might have been a bit of an edge case and there is an open issue with the jdk this issue. In the end, I had to duplicate logic from the library itself to accomplish the same outcome.

Take aways

Java is not my first choice for UI anymore.

I have spent a decent portion of my time writing software working with front-end technologies. I have used many frameworks and languages including: WinForms, WPF, Silverlight, swing, JavaFX and qt. In my experience, the frameworks that seem to get it right separate the view logic from the interaction logic (sorry WinForms and swing). One thing I do really like about JavaFX is that it separates the view logic (fxml), the interaction logic (java) and the style logic (css).

I have, however, spent a good amount of time working in html/javascript based UI's. In this time I have noticed that UI development goes quite a bit faster when every tweak does not require a full recompile of your application. I have also noticed that it is rather nice to not have to switch between a bunch of optional tooling to debug your issues.

	JavaFX	web
Code Debugging	Console output or ide of your choice (eclipse/netbeans)	Browser of your choice
Style inspection	Scenic View	Browser of your choice
Toolchain	Maven or Gradle	Too many options, but none are actually required

Outside of that, the things that java might be better able to handle are interacting with the OS and maybe performance.

Performance

I have not done any real performance testing and I might do this in the future. However, in both JavaFX and web-based technologies you are interacting with the OS via a virtual machine. The only framework above that would not be limited to a virtual machine would be qt. Because of this, I am likely to believe that performance should be relatively equal between JavaFX and web-based UI's.

OS access

This one is a bit tricky. While web-based technologies cannot and should not be able to access the host OS from the browser, there is a way to work around the restrictions of the browser. In many cases, this comes in the way of Electron or NW.js.

There are plenty of resources for Electron, but I will give a quick recap. Electron allows you to distribute your web UI as a standalone application. This application ships its own version of chromium and uses NodeJS to interact with the host OS.

In the case of required OS access, there might be a good argument for using something like JavaFX instead of something like Electron.

Final thoughts

Java isn't a terrible language nor is JavaFX a bad technology. However, lately, I am finding it hard to justify spending the extra time to write a side project in something like JavaFX versus a web-based technology.

Using Gatsbyjs

Mon, 15 Jul 2019 00:00:00 GMT

The idea

Recently I decided it was time to make my own personal website. You might be on this site right now. I knew to go into this that I was going to want to make a static site of some sort but I was not sure what I would settle on.

The requirements

I wanted this site to combine data about all the little projects I work on in my free time into a site. I also wanted to get this data as dynamically as possible so that when I made changes in github I would have to do very little to update this site.

First attempt

Initially, I thought I would just cobble together some kind of nodejs application that takes data from multiple sources and transform data and markdown files into html files. However, I quickly found that doing this was leading to difficult to maintain spaghetti code. At this point, I was starting to realize that maybe a framework would help me accomplish my goals.

After a bit of research

After some research, I found a couple of different frameworks that might help. During this period I looked at jekyll, Eleventy, and Gatsby.

Jekyll

From what I was able to gather this has been around for a while, is stable and has been used by quite a few. To be fair I did not give this an honest try. While I know ruby I prefer to avoid using it where I can.

Eleventy

This seemed similar to Jekyll but build on nodejs instead. While I gave this an initial try the big hang-up I had was is was pretty hard to get sources from github and get them into a format that worked. It required a lot of manual pulling of files and attempting transforms on existing markdown files. While I found this to be an interesting choice, the process just didn't work well enough for my use case.

Gasbyjs

As you might be able to tell by the title of this post, this is the framework I eventually landed on. It seemed like Gatsby was built to handle the type of issue I was trying to solve, combining data from multiple sources into components. It also helped that it has a setup that feels like a rather modern web development experience. These are the major reasons I ended up sticking with this choice.

Graphql
Reactjs
webpack like system.
community

Thoughts

Graphql

While working on with Gatsby early on it became apparent that I was going to have to learn graphql. I have heard a lot about graphql but I had never had a reason to use it up until this point.

The way I understood graphql was that it was a query language for API's that allows you to join multiple or complex requests into one request. What I found is that is true but I expected graphql to do more that it does.

The query I wanted to create would pull all repositories I owned and then would grab the content using a different request. My initial approach did not work because you cannot pass results from one query into another. So in my case, I was trying to make the following requests.

GET users/JeffreyRiggle/repos

GET repos/JeffreyRiggle/{resultn}/readme

This is not able to be expressed in graphql so I had to go with a different approach. that just used the repo api.

{
  viewer {
    name
    repositories(last: 100) {
      nodes {
        name
        description
        id
        descriptionHTML
        url
        first: object(expression: "master:README.md") {
          id
          ... on Blob {
            text
          }
        }
        second: object(expression: "master:doc/doc.json") {
          id
          ... on Blob {
            text
          }
        }
      }
    }
  }
}

The only issue I have with this approach is the last 100 bit. While it's minor I ended up having to maintain a basic knowledge of how many repositories I have.

Reactjs

This is a rather familiar framework for me by now. This made creating components easy for me to figure out in Gatsbyjs.

Webpack

Gatsby uses a Webpack like configuration system. Since webpack is becoming a standard these days it was pretty easy to reason about how the configuration would work in Gatsbyjs.

Community

This is really what sold me on Gatsbyjs. It just has a really large community. I did not struggle to find any documentation or examples of how others have been using this tool. This was a really big win for me since I have a limited amount of free time to figure things out.

How the generation actually came together

In the end, I created a simple website that has information about the side projects I work on in github and a blog.

The github part queries github using the earlier mentioned graphql statement. Then it creates pages for each project. However, if the project has a doc.json file it will do some extra work to create a more complex page with additional child pages. This allows me to create different pages for Installation, Usage, development or whatever else I might need.

After all this is done it scraps the current directory structure to generate my blog pages.

Final thoughts

All in all this was a pretty fun project and I got to learn a lot about static site generation. Since I had never had a reason to do static site generation before I had a lot to learn. Gastbyjs made this process easy for me and I would recommend it to anyone familiar with a more modern frontend web stack.

Hello World

Thu, 11 Jul 2019 00:00:00 GMT

It only seems fitting that I start this like I would any other programming adventure, with a hello world.

I have decided its time to create a personal site and attach a blog while I am at it. I have no idea how consistent I will be about making posts or updating this site but I figure its worth a shot.

I am hoping to use this blog to track the things I have learned and any frustrations I have encountered along the way. If nothing else I hope this will serve as documentation for my future self.