The boy scout rule and git in practice

There’s a dichotomy when applying the boy scout rule to programming: cleaning up code that you happen to come across ‘pollutes’ your merge-/pull-requests, making it harder to review and therefor more unlikely to be accepted.

One way to cope with this is to submit the ‘clean up’ and the feature/task related changes separately, and merge them back into upstream in separate steps. But often times, it is much easier to just fix a small problem right away instead of switching back to your main branch and doing it there. In fact, it might prevent the developer from doing the improvement, which I want to avoid. Quite the opposite, I want to encourage my fellow developers to do improvements.

So one thing that we do about this is to mark the changes that are unrelated (or tangentially related) to the task with their own commit and a special prefix in the commit message like:

BSR: More consistent function signatures

As you might have guessed, BSR stands for boy scout rule. This does not solve the fact that the diffs get larger than necessary, but it makes it possible to ‘filter out’ the pure refactorings. In some cases, these commits can later be cherry-picked onto the main branch before doing the review. Of course, this only works for small refactorings, but this is where the boy scout rule applies.

The four stages of automation – Part II

One of the core concepts of software development and IT in general is “automation”. By delegating work to machines, we hope to reduce costs and save time while maintaining the quality of results. But automation is not an all-or-nothing endeavor, there are at least four different stages of automation that can be distinguished.

In the first part of this blog series, we looked at the first two stages, namely “documentation” and “recurring reminders”. Both approaches are low tech, but high effect. Machines only played a minor role – this will change with this blog series part. Let’s look at the remaining two stages of automation:

Stage 3: Semi-automatic

If you have a process that is properly documented and you are reminded in a regular fashion, like once a month, you’ll soon find that some steps of the process could be done by a machine, while you as the “human in duty” still pull all the strings that orchestrate the whole thing.

If you know the term “semi-automatic” from firearms, a semi-automatic firearm doesn’t aim or shoot itself, it just reloads automatically after each shot. The shooter still has to pull (and release) the trigger for each single shot. The shooter is in full control of the weapon, it just automates the mundane and repetitive task of chambering the next round.

This is the kind of automation we are taking about for stage 3. It is the most common type of automation. We know it from our cars, our coffee machines and other consumer electronics. The car manages a lot of different tasks under the hood while we are still in control of the overall task of driving from A to B.

How does it look like for business processes? One class of stage 3 utilities are reporting tools that gather and aggregate data from different sources and present the result in a suitable manner. In our company, these tools make up the majority of stage 3 services. There are reporting tools for the most important numbers (the key performance indices – KPI) and even some for less important, but cumbersome to acquire data. Most tools just present a nice website with the latest results while others send e-mails or create pages in our wiki. If you need a report, just press a button or visit an URL and the machine comes up with the answer. I tend to call this class of tools “sensors”, because they acquire data and process it, but don’t decide on the results.

The other class of stage 3 utilities that are common are “actuators” in the sense that they perform tasks on command. We have scripts in place to shut down whole clusters of computers, clear wiki spaces or reset custom fields on important data objects, but those scripts are only triggered by humans.

A stage 3 actuator could even be something small as a mailto link. Let’s say you have to send a standardized e-mail to a known recipient as part of a monthly process. Sure, you can save a draft in your e-mail application, but you can also prepare the whole mail in an URL directly in the documentation of the process:

mailto:nobody@softwareschneiderei.de?subject=The%20schneide%20blog%20rocks!&body=I%20read%20your%20blog%20post%20about%20automation%20and%20tried%20the%20mailto%20link.%20This%20thing%20is%20awesome%2C%20thank%20you!

If you click the link above, your e-mail application will prompt you to send an e-mail to us. You don’t need to follow through – we won’t read it on that address.

You can read about the format of mailto links here, but you probably want to create working mailto links right away, which is possible with this nifty stage 3 service utility written by Michael McKeever (buy him a beer!).

Be aware that this is a classic example of chaining stage 3 tools together: You use a tool to create the mailto link that you use subsequently to write, but not send, e-mails. You, as the human coordinator, decide when to write the e-mail, if you want to adapt it to current circumstances and when to send it. The tools only speed you up, but don’t act or decide on their own.

An important aspect of this type of automation is the human duty of orchestration (which service does its thing when) and the possibility of inspection and adaption. The mailto link doesn’t send an e-mail, it just prepares it for you to send. You have the final word on the things that happen.

If you require this level of control, stage 3 automation is where your automation journey ends. It still needs the competent human operator (what, when, why) – but given a decent documentation (as outlined in stage 1), this competence can be delegated quickly. It is also the first automation stage that enables higher effectiveness through speedup and error reduction. The speedup is capped by the maximum speed of the human operator, though.

Stage 4: Full automation

The last stage of automation is “full automation”, which means that a machine gathers the data on its own, comes up with a decision based on the data and acts on its own. This is a powerful tool, but a dangerous one, too.

It is powerful because you just employed an additional worker. Not a human worker, but a machine. It doesn’t go on holiday, it doesn’t lose interest and won’t ask for a bonus.

It is dangerous, because your additional worker does exactly what it is told (programmed) to do, even if it doesn’t make sense or needs just the slightest adaption to circumstances.

Another peril lies in the fact that the investment to reach the fully automated stage is maximized. As with nearly everything related to IT, there is a relevant xkcd comic for this:

https://xkcd.com/1319/

The problem is that machines are not aware of their context. They don’t deal well with slight deviations (like “1,02” instead of “1.02”) and cannot weigh the consequences of task failure. All these things are done by a competent human operator, even without specific training. You need to train a machine for every eventuality, down to the dots.

This means that you can’t just program the happy path, as you do in stage 3, when a human operator will notice the error and act accordingly. You have to implement special case behaviour, failure detection, failure handling and problem reporting. You have to adapt the program to changes in the environment in a timely manner (this work is also present in stage 3, but can be delayed more often).

If the process contains mostly routine and is recurring often enough to warrant full automation, it is a rewarding investment that pays off quickly. It will take your human-based work on a new level: designing and maintaining an automation platform that is cost-efficient, scalable and adjustable. The main problem will be time-critical adjustments and their overall effects on the whole system. You don’t need routine workers anymore, but you’ll need competent technicians on stand-by.

Examples of fully automated processes in our company are data backups, operating system upgrades, server monitoring and the recurring reminder system that creates the issues for our stage 2 automation. All of these processes have increased reporting capabilities that highlight problems or just anomalies in a direct manner. They all have one thing in common: They are small, work on only one thing and try to do so with minimal dependencies and interaction.

Conclusion

There are four distinguishable stages of automation:

  1. documentation
  2. recurring reminders
  3. semi-automatic
  4. full automation

The amount of human work for the actual process decreases with each stage, while the amount of human work for the automation increases. For most processes in an organization, there will be a sweet spot between process cost and automation cost somewhere on that spectrum. Our job as automaters is to find the sweet spot and don’t apply too much automation.

If you have a good story about not enough automation or too much automation or even about automation being just right – tell us in the comments!

The four stages of automation – Part I

One of the core concepts of software development and IT in general is “automation”, the “creation and application of technologies to produce and deliver goods and services with minimal human intervention” (definition from techopedia).

The problem is that “minimal human intervention” is often misunderstood as “no human intervention”, which is the most laborious and expensive stage of automation that might not have the most economic return on investment. It might be more efficient to have some degree of intervention left while investing only a fraction of the automation work and duration.

In order to decide “how much” automation is the most profitable for the foreseeable future, I’ve established a model with four stages of automation that I can quickly check against the circumstances. In this blog post, I describe the first two stages and give some ideas how to implement them.

Stage 1: Documentation

The first step to automation is to just describe the process in a manner that can be repeated. The documentation itself does nothing, but it enables repetition and scalability, two fundamental aspects of automation.

Think about baking a pie. If you just mix some ingredients and put it in the oven for an arbitrary amount of time, you might produce the most delicious pie ever, but you cannot do it again if you don’t remember all details and, even more tragic, nobody else can bake your pie. In order to give others the secret to your special pie, you have to give them the recipe – the documentation of its production process. Once the recipe is written down (and published), it can be read by many bakers in parallel and enables all of them to recreate your invention (to some degree at least, there are probably still some tricks and secrets left out of the recipe).

While the pie baking process still needs human intervention (the bakers that read the recipe and transform it into a series of actions), it is automated in the sense that it can be repeated with roughly the same result and these repetitions, given enough bakers and ovens, can be performed in parallel.

The economic evaluation of documentation shows that it is really easy to create, fast to change and, given some quality of content, nearly universally understood. If you don’t want to invest a lot of time and money, documenting your processes is the first and most important step towards automation. For a lot of your processes, it will also be the last possible stage of automation, at least until artificial intelligence learns your tricks and interpretations.

Documenting your processes is (no surprises here) the foundation of most quality assurance standards. But it is surprisingly hard to start with. This is not a matter of tools – pen and paper will do in the beginning. It is a shift in your mindset. The goal is no longer to bake a pie. It is to write a recipe while you bake the pie as a reference piece for it. If you want to start documenting your processes, here are three tips that might help you:

  • Choose a digital tool that doesn’t obstruct you. It should be digital because this facilitates distribution and collaboration. It should not hinder you because every time you need to think about the tool, you lose the focus on your process. I’m using a Wiki that lets me type the things I want to say without interference. In my case, that’s Confluence, but Obsidian or other tools are just as good.
  • Try to adopt a narrative structure to describe your processes. Think about the established structure of a baking recipe. For example, there is an ingredients list separate from the preparation instructions. If you find a structure that works for you, repeat and evolve it. It helps you and your readers to stay on track and don’t scatter the information all over the place. In my case, the structure consists of four paragraphs:
    1. Event/Trigger – The circumstance(s) that should be present at the beginning of the process
    2. Actions/Steps – The things you have to do, described in the necessary details for the target audience. This is often the paragraph with the most content.
    3. Result – Description of the circumstance(s) that should be present once you’ve done all steps. In recipes, this is often a photo of the meal/pastry. For first-time performers, this description is important to be able to declare success.
    4. Report – Who needs to be informed? This paragraph is often missing in descriptions, but crucial for collaboration. If nobody knows there is a fresh and delicious pie in the kitchen, it will not be eaten. Ok, that’s a bad example: Pies in the oven announce themselves with their smell. Digital products often have no smell – inform your peers!
  • Iterate over your documentation any chance you get. It is easy to bake your signature pie from memory. But is the recipe still accurate? Are there details that are important, but missing from the description? Your digital tool probably allows immediate modification of your documentation and maybe even informs interested readers about your update. Unchanged documentation is dead documentation. In my case, I always open my process description on a secondary monitor whenever I perform them. Sometimes, I invite others to perform the process for me to review the accuracy and fidelity of the documentation.

If you can open the process description of many of your routine tasks, you have reached the first stage of automation for your work. Of course, there will be lots of things you do that are not “routine” – yet. With good documentation, you can even think about delegation – the art of maximizing the amount of work done by others – without sacrificing essential quality.

In later stages, the delegation target (the “others”) will be machines.

Stage 2: Recurring reminders

If you’ve documented a process with a structure similar to mine, you specified a trigger or event that requires the process to be performed. Perhaps its the first day of the month and you need to update your timesheet or send out the appointment overview for the next weeks. Maybe your office plants silently thirst for some water. Whatever it is, if your process is recurrent, you might think about recurring reminders.

This will not automate the performance of the process, but unburden you of thinking about the triggering event. The machines will now remind you about certain tasks. This can be a simple series of reminders in your schedular app or, like in my case, the automated creation of issues (or todo items, tickets) in your work planning application.

For example, once every few weeks, a friendly machine creates an issue for me to write a blog entry on this blog. It does the same for my colleagues and even sets a “due date” (The due date for this post is today). With this simple construct, some discipline and coordination, we’ve managed to write one blog post every week for more than ten years now.

The machine that creates the issues doesn’t check them. It doesn’t supervise their progress and isn’t offended if we “won’t fix” issues because we are on holiday or the plants are still wet. It will just create the next issue according to the rhythm. It is our duty as humans to check if that rhythm fits or if it should be sped up or slowed down.

If you want to employ really elaborate triggers for your reminders, a platform like “If this then that (IFTTT)” might be the right choice. Just keep in mind that with complexity, there often comes rigidity, which isn’t always desired.

By automating the aspect of reminding us about the routine tasks, we can concentrate on doing them. We don’t forget to write blog posts or to water the plants because the machine doesn’t forget. Another improvement is that this clearly distinguishes between routine (has a recurring reminder) and anomaly. If the special one-time task occurs again, we give it a recurring reminder and adopt it as a new routine task. If a reminder about a routine task is “won’t fixed” often enough without any inclination that it will be required again, we delete the reminder.

Conclusion for part I

If you combine automated recurring reminders with structured documentation, you already gain a lot of advantages and can free your mind from the mundane details and intervals of your routine tasks. You haven’t automated any aspect of your real work yet, which means that these two stages can be applied to most if not all workplaces.

In the next part of this series, we will look at the two stages that become integrated with your actual work. Stay tuned!