The four archetypes of cloud users – part 1 of 2

You probably know people that avoid cloud services as if they are poisonous and others that jump onto every cloud bandwagon. You can categorize them in four archetypes. Let’s start with the tinfoil hat and the clipboard.

In the occupational field of accounting, the strong trend towards cloud services is noticeable. Everything needs to be digital, and with digital, they mean online, and with online, they mean in the cloud. Every expense voucher needs to be scanned and uploaded, because in many cases, it can be booked automatically. In the new era of accounting, human intervention is only needed for special cases.

I see this as a good example of how digital online services can transform the world. Every step in the process would have technically been possible for the last twenty years, but only the cloud could unify the different participants enough so that a streamlined end-to-end process is marketable to the masses. And in this marketing ecstasy, the stakeholders that profit the most (the accountants) often forget that their benefits are just a part of the whole picture. In order to assess the perceived and actual benefits of all stakeholders, you at least need to apply an archetype to each participant.

The four archetypes

In my opinion, there are four different archetypes of cloud users. Let’s have a look at them and then assess the risks and potentials when selling a digital online service to them. I’ll list the archetypes in the order from biggest risk to biggest potential.

Archetype 1: The tinfoil hat

A person that could be identified as a “tinfoil hat” doesn’t need to be a conspiracy weirdo or paranoid maniac. In fact, the person probably has deep and broad knowledge about technology and examines new technologies in detail. The one distinctive feature of the tinfoil hats is that they take security, including IT security, very seriously. They don’t take security for granted, don’t trust asseverations and demand proof. You can’t convince a tinfoil hat by saying that the data transfer is “encrypted”, you need to specify the actual encryption algorithm. Using RC4 ciphers for the SSL protocol isn’t good enough for the tinfoil. You need at least proof that you understood the last sentence and took actions to mitigate the problem. Even then, the tinfoil will hesitate to give any data out of hands and often choose the cumbersome way in order to stay safe. “Better safe than sorry” is his everyday motto.

Tinfoil hats always search for scenarios that could compromise their data or infrastructure. They are paranoid by default and actively invest in security. “On premises” is the only way they deploy their own services, and “on premises” is how they prefer to keep their data.

Typical signs of a tinfoil hat archetype include:

  • self-hosted applications
  • physical servers
  • lack of (open) wireless network
  • physically separated networks
  • signed and encrypted e-mails

Trying to sell a cloud service to a tinfoil hat is like trying to sell a flight to an aviophobian (somebody with fear of flying). There is always another way to get from A to B, seemingly safer and more controllable. If you are selling cloud services, tinfoil hats are your worst nightmare. If you can convince a tinfoil hat, your product is probably made of fairy dust and employs lots of unicorns.

Archetype 2: The clipboard

Clipboard people are wary of new technologies, but assess them in the context of usability. They demand high security, but will compromise if the potential of the new technology far exceeds the risk. Other than the tinfoil hat, the clipboard sees his role as an enabler, but will not rest to increase the perceived or actual safety of the product. You can appease a clipboard by giving evidence of security audits from a third party. They will trust known authorities, because it means that they can always deflect blame in case of an accident to these authorities.

Clipboards run on checklists, safety protocols and recurring audits. They don’t try to avert every possibility of a security breach, but will examine each incident in detail and update their checklists. They don’t care about “on premises” or “off premises” as long as the service is reachable, safe enough and reliable. If a cloud service has an higher availability than the local counter-part, the clipboard will think about a migration.

Typical signs of a clipboard archetype include:

  • Virtual Private Networks (VPN)
  • Two-Factor Authentication
  • Token-Based Authentication
  • Strong Encryption

The clipboard will listen if you pitch your cloud service and can be enticed by the new or better capabilities. But in the very next sentence, he will ask about security and be insistent until you provide proof – first-hand or by credible third parties. You can convince a clipboard if your product is designed with safety in mind. As long as the safety is state-of-the-art, you’ll close the deal.

Outlook on the second part

In the second part of this blog entry, we will look at the remaining two archetypes, namely the “combination lock” and the “smartphone”. Stay tuned.

Did you identify with one of the archetypes? What are your most important aspects of cloud services? I would love to hear from you.

Book review: Clean Architecture by Robert C. Martin

A personal review of the book “Clean Architecture” by Robert C. Martin. Spoiler: you might want to read it.

In 2008, a book changed the way software developers around the globe talked (and hopefully) acted about their code. Robert C. Martin’s “Clean Code” was and still is a cornerstone of modern software development. The book itself is remarkably weak in its code examples, but has strong and effective messages on the level of practices and principles. Even today, ten years later, this is the one book that most of my students read and are passionate about. It’s a book that speaks reason to them, albeit with some contortion because of high volume. Robert C. Martin has the tendency to preach 200 percent in order to still get the half-convinced to an acceptable level.

So when a new book from him, called “Clean Architecture”, appeared on the horizon, I was thrilled. Would it be groundbreaking like “Clean Code” or a dud like “The Clean Coder” (sorry, my opinion – this is a personal review, not an academic evaluation)? I’ve read some very good books about software development (like “The Pragmatic Programmer”), fantastic books about programming (“Refactoring” and “Working Effectively With Legacy Code” come to mind) and even some mind-blowing pieces about design and emerging architecture (my first read of “Growing Object-Oriented Software, Guided by Tests” felt like a personal audience with Steve Freeman and Nat Pryce). But all these books dealt with tactics, with the immediacies of software development. Don’t get me wrong! This is the most important part and it helped me tremendously. But there are parts “above” the footwork that needs to be addressed in bigger systems, too.

And there, the literature got thin or stale. Books about software architecture talked about large-scale architectures (so-called “enterprise scale” systems that span from horizon to horizon, like in “Patterns of Enterprise Application Architecture”) or had the taste of dry plywood because it was clear that the findings were from another era and would translate badly into modern software development.

“Clean Architecture” begins with a quick and focused overview over the current programming paradigms and a conclusion that there are no different “eras”. We didn’t get better in designing systems, we just changed the aroma and color of our failures. Future generations will look at our code and architectures as scornful as we looked at the ruins of the systems of our ancestors. And make no mistake – the ruins are still in production today! We cannot place our hope on another new and liberating programming paradigm because there probably won’t be one. We have to make do with what we have.

This is the first six chapters of “Clean Archicture”. The chapters are short and on point and I loved every line of it. It probably isn’t the most comprehensive and balanced description of structured, object-oriented and functional programming, but it provides a narrative that is intuitive and convincing – your mileage may vary, I was hooked.

In the next five chapters, Robert C. Martin reiterates the known SOLID design principles. I rolled my eyes when I glanced the content because I’ve read it like a hundred times in maybe as many books. But I decided to read it once more and I’m glad I did. The principles are known, but the underlying revelation is woven into the text like a good thriller. I hesitate to give away too much, because I really think this book can be spoiled – just like a good thriller. I was sold. Robert C. Martin can explain the same old SOLID to me and I still learn something and have fun.

Then, the part about components. It feels like an intermezzo to an even better thriller, because suddenly there is math and formulas. Its interesting and noteworthy, but if you followed the metrics discussion in the last fifteen years, the excitement of this part will be dampened.

But wait, there is more! Starting with page 133 of 321 (yeah, the Appendix is interesting, but more in the “The Clean Coder” way of things), there is the central question: “What is Architecture?”. There it was again, the thrill that in every line, there could be insights that are worth weeks of thoughts. I read this part in the train from south to north germany and I stared out of the window often, following my own train of thoughts.

Again, no spoilers, but the way the answers are given is so refreshing and the answer itself is so simple that I’m surprised that it took me this long to not come to the same conclusion. Software architecture lost some of its mysticism, but gained a lot of applicability for me. I was spent (in a good way).

And then, on page 200, finally, “The Clean Archicture”. Well, I watched all the trailers on this topic, so my surprise wasn’t really there, but with all the knowledge and insights from the first 200 pages, I could have “invented” the Clean Architecture by myself then and there. It’s more or less the logical next step from the prerequisites. I applaud this masterwork of storytelling, because it doesn’t overwhelm the reader with the genius of the narrator, it drives him to connect the dots himself.

The rest of the book, like the title of part VI, are just “Details”. The central message  – The Dependency Rule, this little spoiler should be allowed – is simple, convincing and deduced from the beginning. I’ve seen the heart of software architecture and it is beautiful.

I even forgive the many typos and grammatical errors (far more than usual) and the bulky appendix for this ride. This book is definitely up there with “Clean Code”. It is accessible, has a clear message and profound effects. And it refrains from preaching most of the time. No need to turn it up to 200 percent when your message is so convincing in itself.

Conclusion: If you are interested in software development with a structure, go grab this book as soon as possible. We’ve waited long enough!

Implementation visibility – Part III

In the thirf part of our series about implementation visibility, we look at the high ground of our visibility scale and peek even higher. The series ends with a recap of the concept and its implications.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

We examined a simple code example in both preceding articles. The level 0, 0+ and 0++ were covered in the first article, while the second article talked about level 1, 2 and 2+. You might want to read them first if you want to follow the progression through the ranks. In this article, we look at the example at level 3, have a short outlook on further levels and then recap the concept.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 3: Domain drive all the things!

We’ve introduced a new class for our requirement in visibility level 2 and made it a domain type. This is mostly another name for the concept of Entities or Value Objects from Domain Driven Design (DDD). If you aren’t familiar with Domain Driven Design, I recommend you grab the original book or its worthy successor and read about it. It is a way to look at requirements and code that will transform the way you develop software. To give a short spoiler, DDD Entities and DDD Value Objects are named core domain concepts that form the foundation of every DDD application. They are found by learning about the problem domain your software is used in. DDD Entities have an own identity, while DDD Value Objects just exist to indicate a certain value. Every DDD Entity and most DDD Value Objects are part of an DDD Aggregate. To load and store DDD Aggregates, a DDD Repository is put into place. The DDD Repository encapsulates all the technical stuff that has to happen when the application wants to access an DDD Aggregate through its DDD Root Entity. Sorry for all the “DDD” prefixes, but the terms are overloaded with many different meanings in our profession and I want to be clear what I mean when I use the terms “Repository” or “Aggregate”. Be very careful not to mistake the DDD meanings of the terms for any other meaning out there. Please read the books if you are unsure.

So, in Domain Driven Design, our BruttoPrice type is really a DDD Value Object. It represents a certain value in our currency of choice (Euro in our example), but has no life cycle on its own. Two BruttoPrices can be considered “the same” if their values are equal. This raises the question what the DDD Root Entity of the corresponding DDD Aggregate might be. Just imagine what happens in the domain (in real life, on paper) if you calculate a brutto price from a given netto price: You determine the value added tax category of your taxable product, look up its current percentage and multiply your netto price with the percentage. The DDD Root Entity is the value added tax category, as it can be introduced and revoked by your government and therefor has a life cycle on its own. The tax percentage, the netto price and the brutto price are just DDD Value Objects in its vicinity.

To bring DDD into our code and raise the implementation visibility level, we need to introduce a lot of new types with lots of lines of code:

  • NettoPrice is a DDD Value Object representing the concept of a monetary value without taxes.
  • BruttoPrice is a DDD Value Object representing the concept of a monetary value including taxes.
  • ValueAddedTaxCategory is a DDD Root Entity standing for the concept of different VAT percentages for different product groups.
  • ValueAddedTaxPercentage might be a DDD Value Object representing the concept of a percentage being applied to a NettoPrice to get a BruttoPrice. We will omit this explicit concept and let the ValueAddedTaxCategory deal with the calculation internally.
  • ValueAddedTaxRepository is a DDD Repository providing the ability to retrieve a ValueAddedTaxCategory for a known Taxable.
  • Taxable might be a DDD Entity. For us, it will remain an abstraction to decouple our taxes from other concrete types like Product.

The most surprising new class is probably the ValueAddedTaxRepository. It lingered in our code in nearly all previous levels, but wasn’t prominent, not visible enough to be explicit. Remember lines like this?

final BigDecimal taxFactor = <gets the right tax factor from somewhere> 

Now we know where to retrieve our ValueAddedTaxCategory from! And we don’t even know that the VAT is calculated using a percentage or factor anymore. That’s a detail of the ValueAddedTaxCategory given to us from the ValueAddedTaxRepository. If one day, for example at April 1th, 2020, the VAT for bottled water is decreed to be a fixed amount per bottle, we might need to change the internals of our VAT DDD Aggregate, but the netto and brutto prices and the rest of the application won’t even notice.

We’ve given our different reasons of change different places in our code. We have separated our concerns. This separation requires a lot of work to be spelled out. Let’s look at the code of our example at implementation visibility level 3:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    final ValueAddedTaxRepository vatProvider = givenVatRepository();
    for (Product each : inCart) {
      final ValueAddedTaxCategory vat = vatProvider.forType(each);
      final BruttoPrice bruttoPrice = vat.applyTo(each.nettoPrice());
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

There are now three lines of code responsible for calculating the brutto prices. It gets ridiculous! First we obtain the DDD Repository from somewhere. Somebody probably gave us the reference in the constructor or something. Just to remind you: The class is named ShowShoppingCart and now needs to know about a class that calls itself ValueAddedTaxRepository. Then, we obtain the corresponding ValueAddedTaxCategory for each Product or Taxable in our shopping cart. We apply this VAT to the NettoPrice of the Product/Taxable and pass the resulting BruttoPrice side by side with the NettoPrice in the addProductLine() method. Notice how we changed the signature of the method to differentiate between NettoPrice and BruttoPrice instead of using just to Euro parameters. Those domain types are now our level of abstraction. We don’t really care about Euro anymore. The prices might be expressed in mussle shells or bottle caps and we still could use our code without modification.

The ValueAddedTaxCategory we obtain from the DDD Repository isn’t a class with a concrete implementation. Instead, it is an interface:


/**
* AN-17: Calculates the brutto price (netto price with value added tax (VAT))
* for the given netto price.
*/
public interface ValueAddedTaxCategory {
  public BruttoPrice applyTo(NettoPrice nettoPrice);
}

Now we could nearly get rid of the comment above. It just repeats what the signature of the single method in this type says, too. We keep it for the reference to the requirement (AN-17).

Right now, the interface has only one implementation in the class PercentageValueAddedTaxCategory:


public class PercentageValueAddedTaxCategory implements ValueAddedTaxCategory {
  private final BigDecimal percentage;

  public PercentageValueAddedTaxCategory(final BigDecimal percentage) {
    this.percentage = percentage;
  }

  @Override
  public BruttoPrice applyTo(NettoPrice nettoPrice) {
    final Euro value = nettoPrice.multiplyWith(this.percentage).inEuro();
    return new BruttoPrice(value);
  }
}

You might notice that the concrete code of applyTo still has knowledge about the Euro. As long as we don’t ingrain the relationship between NettoPrice and BruttoPrice in these types, somebody has to do the conversion externally – and needs to know about implementation details of these types. That’s an observation that you should at least note down in your domain crunching documents. It isn’t necessarily bad code, but a spot that will require modification once the currency changes to cola bottle caps.

This is a good moment to reconsider what we’ve done to our ShowShoppingCart class. Let’s refactor the code a bit and move the responsibility for value added taxes where it belongs: in the Product type.


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            each.bruttoPrice());
    }
    return result;
  }

}

Now we have made a full circle: Our code looks like it began without the brutto prices, but with one additional line that delivers the brutto prices to the product line in the ShoppingCartRenderModel. The whole infrastructure that we’ve built is hidden behind the Product/Taxable type interface. We still use all of the domain types from above, we’ve just changed the location where we use them. The whole concept complex of different price types, value added taxes and tax categories is a top level construct in our application now. It shows up in the domain model and in the vocabulatory of our project. It isn’t a quick fix, it’s the introduction of a whole set of new ideas and our code now reflects that.

The code at implementation visibility level 3 might seem bloated and over-engineered to some. There is probably truth in this judgement. We’ve introduced far more code seams in the form of abstractions and indirections than we can utilize in the moment. We’ve prepared for an uncertain future. That might turn out to be unnecessary and would then be waste.

So let’s look at our journey as an example of what could be done. There is no need to walk all the way all the time. But you should be able to walk it in case it proves necessary.

Visibility level 4 and above: To infinity and beyond!

Remember that there are implementation visibility levels above 3! If you choose such a level, there will be even more code, more classes and types, more indirection and more abstraction. Suddenly, your new code will show up on system architecture diagrams and be deployed independently. Maybe you’ll need a dedicated server for it or scale it all the way up to its own server farm. Our example doesn’t match those criterias, so I stop here and just say that visibility level 3 isn’t the end of the journey. But you probably got the idea and can continue on your own now.

Recap: Rising through the visibility levels

We’ve come a long way since level 0 in terms of implementation visibility. The code still does the same thing, it just accumulates structure (some may call it cruft) and fletches out the relationships between concepts. In doing so, different axis of change emerge in different locations instead of entangled in one place. Our development effort rises, but we hope for a return on investment in the future.

I’ve found it easier to elevate the implementation visibility level of some code later than to decrease it. You might experience it the other way around. In the end, it doesn’t matter which way we choose – we have to match the importance of the requirement in the code. And as the requirements and their importance change, our code has to adjust to it in order to stay relevant. It isn’t the visibility level you choose now that will decide if your code is visible enough, it is the necessary visibility level you cannot reach for one reason or the other that will doom your code. Because it “feels bloated” and gets replaced, because it wasn’t found in time and is duplicated somewhere else, because it fused together with unrelated code and cannot be separated. Because of a plethora of reasons. By choosing and changing the implementation visibility level of your code deliberately, you at least take the responsibility to minimize the effects of those reasons. And that will empower you even if not all your decisions turn out profitable.

Conclusion

With the end of this third part, our series about the concept of implementation visibility comes to an end. I hope you’ve enjoyed the journey and gained some insights. If you happen to identify an example where this concept could help you, I’d love to hear from you! And if you know about a book or some other source where this concept is explained, too – please comment with a link below.

Implementation visibility – Part II

In the second part of our series about implementation visibility, we examine the middle ground of our visibility scale and discover that most requirements end up in this range. The maintainability is good, but we still have to check for architectural sins.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

The article then introduced a simple example and examined how the level 0, 0+ and 0++ would appear within the example code. You may want to read the first article before we carry on with level 1 and 2 in this article.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable&lt;Product&gt; inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 1: Extracted code lives longer

After all the (rather depressing) level 0 implementations of our brutto price calculation, the separated method is the first visibility level to result in code that can be discussed and tested separately:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPriceFor(each));
    }
    return result;
  }

  /**
  * AN-17: Calculates the brutto price for the given product.
  */
  private Euro bruttoPriceFor(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor);
  }
}

The new code is in lines 8 and 13 onwards. The new method was introduced to separate the calculation code from the rendering code. It still lives in the wrong class, but can be tested on its own if you make it public or package accessible. The comment now has a natural scope. And, most important: This implementation is the first where the notion of “brutto price” appears in the JavaDoc and the IDE.

Methods are the smallest parts of our object-oriented code. If you would have one method per requirement, you would just need one extra method of glue code to tie everything together. If one requirement needs to change or becomes obsolete, you know where to cut.

Methods are the primary focus of unit tests. You prepare the parameters for the method you want to test, call it and check the result. This is the AAA or triple-A normal form of unit testing: Arrange, Act, Assert. If several methods or even several objects need to be tested in conjunction, the testing effort rises.

We can conclude that with its own method, the VAT calculation now has its own home. Future readers can grasp the scope of our implementation easily and hopefully make changes under direct test coverage. This is the first visibility level that starts to feel like we meant it.

Visibility level 2: Make it a top-level affair

There is one part in object-oriented code that is even more basic than a method: the class. In Java, each class strives to have its own text file. Before you can write a method in Java, you need to define a class to contain it. Classes are the primary granularity level we navigate our code. Every IDE will show classes as the default elements in our “project explorers”. So what if we introduce a new class for our VAT calculation and move all our code there?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            CalculateBruttoPrice.forProduct(each));
    }
    return result;
  }
}
/**
 * AN-17: Calculates the brutto price with value added tax (VAT) for the given product.
 */
public class CalculateBruttoPrice {
  public static Euro forProduct(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor); 
  }
}

The new code is in line 8 and the full new class file. This implementation might not look a lot different from level 1 (separate method), but it really is on another level. The brutto price calculation now isn’t tied to rendering shopping carts anymore. It is not tied to anything other than a given product. It is a top-level concept of our application now. Anybody with a product can call the method and receive the brutto price, from anywhere in our application (hopefully respecting our architecture boundaries).

Our unit test class now reads as if we had written it only for the new requirement: CalculateBruttoPriceTest. We still need to invent test products in our test, but the whole notion of render models and shopping carts is gone. In essence, we freed the concept of price calculation from its “evolutionary” ties.

Implementing the new requirement in a separate class, if feasible, adheres to the Single Responsibility Principle (SRP), that requires each class of a system to only have one reason to change. In our case, the CalculateBruttoPrice class only changes if the brutto prices needs adjustment. For all previous visibility levels, that wasn’t true. The ShowShoppingCart class would need modifications if the brutto prices or the shopping cart rendering were to be changed. This improvement is reason enough to elevate our implementation visibility past level 1.

In short, a good heuristics for new requirements (as opposed to change requests for existing requirements) is to start with a new class. If you are unsure, start lower, but keep in mind that classes are the main navigation layer of object-oriented code.

Visibility level 2+: Inviting the requirement to be part of the project’s language

Introducing a new class for our requirement felt good, but something still feels off. When we review the interface of the CalculateBruttoPrice, two things stick out immediately: The class is named as a service (CalculateXYZ as in “do XYZ for me”) and can only calculate brutto prices for products. Our customer was serious with his requirement, so it’s safe to assume that brutto prices will stay in the application and play a key role. We should reflect this seriousness by lifting the implementation visibility level once more and make the BruttoPrice a top level concept of our project’s domain:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            BruttoPrice.of(each).inEuro());
    }
    return result;
  }
}

The ShowShoppingCart code doesn’t look very different from the level 2 code beforehands. The new code is in line 8, too. The new class isn’t named like a service anymore, but like a concept or domain type. The named constructor of() returns a BruttoPrice instance and not just a Euro object:

/**
 * AN-17: Represents the brutto price with value added tax (VAT) for the given Taxable.
 */
public final class BruttoPrice {
  public static BruttoPrice of(Taxable item) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return new BruttoPrice(item.nettoPrice().multiplyWith(taxFactor));
  }
  
  private final Euro asValue;

  private BruttoPrice(Euro value) {
    this.asValue = value;
  }

  public Euro inEuro() {
    return this.asValue;
  }
}

Now, we can accumulate additional behaviour in the new BruttoPrice type if the need arises. With the service class of level 2, we probably wouldn’t have risen above the Euro abstraction and mixed up netto and brutto prices somewhere in the future.  If we model our NettoPrice and BruttoPrice as domain types, the compiler will help us keeping them separate – even if both contain Euros as their value.

With this visibility elevation, we discovered another abstraction: We can create brutto prices for virtually anything that can be taxed. It doesn’t have to be a product, it just needs a netto price and a tax factor. The new (abstract) domain type is named Taxable. Of course, Product is an implementation of Taxable.

This makes us even more independent from any webshop, shopping cart or product. We can now write unit tests for our BruttoPrice without being coupled to the Product class at all. We have successfully decoupled the cart/product part of our application from the prices part. Recognizing and implementing the independence of concepts is an important step towards even higher visibility levels. It is also the groundwork of a low coupled, high cohesive code base where most things fall into their place naturally.

The step from level 2 (separate class) to level 2+ (new domain type) wasn’t just syntactic sugar, it was driven by the insight that separation of concerns is the fundamental principle to achieve maintainability, as long as the abstractions aren’t overwhelming. A good indicator that you’ve taken it too far is when your domain expert (in our example our client) raises her eyebrows in surprise when you talk about your abstract domain types because the names sound outlandish and far-fetched.

But you can take your implementation visibility even further and should really consider doing so given the circumstances. We will learn about visibility level 3 (separate aggregate) in the next blog post of this series. Stay tuned!

Implementation visibility – Part I

The concept of implementation visibility is very powerful. If you choose the wrong visibility for your requirements, your code will end up unmaintainable.

Somewhere in my take on programming, there lingers the concept of “implementation visibility”, that I’m not quite sure to be able to express clearly, but I’ll try.

Let’s say you are writing an academic text like a bachelor thesis and your professor makes it clear that she regards the list of literature a very important part of your work. What are you going to do? Concentrate on your cool topic and treat the literature as a secondary task? Or will you shift your focus and emphasize your extensive literature research, highlighting promising cross-references in your text? You’ll probably adjust your resources to make your list of literature more prominent, more visible. You respond to the priorities of your stakeholders.

Now imagine that your customer wants you to program a web application, but has one big requirement: All actions of the users need to be reassured with a confirmation question (as in “do you really want to delete this?”). He makes it clear that this is a mandatory feature that needs to be implemented with utmost care and precision. What would you do? We responded by adjusting our system’s architecture to incorporate the requirement into the API. You can read about our approach in this blog post from 2015. The gist of it is that every possible client of the system will be immediately aware of the requirement and has a much easier time conforming to it. It is harder to ignore or forget the requirement than to adhere to it because the architecture pushes you into the right direction.

The implementation of the customer’s requirement in the example above is very visible. You’ll take one look at the API and know about it. It isn’t hidden into well-meaning but out-dated developer documentation or implicitly stated because every existing action has a confirmation step and you should be sentient enough to know that this means your new one needs one, too. The implementation visibility of the customer’s requirement is maximized with our approach.

Stages of visibility

I have identified some typical stages (or levels) of implementation visibility that I want to present in this blog post series. That doesn’t mean that there won’t or can’t be others. I’m not even sure if the level system is as one-dimensional as I’m claiming here. I invite you to think about the concept, make your own observations and evolve from there. This is a starting point, not an absolute truth.

The following stages typically appear in my projects:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

In my day-to-day work, the levels 1 to 3 are the most relevant, but that’s probably not universally applicable. Our example above with the requirement-centered API isn’t even located on this list. I suggest it’s at level 6 and called separate concept or something similar.

An example to explain the visibility levels

Let’s assume a customer wants us to program a generic webshop. We are not very versed in commerce or e-commerce things and just start implementing requirements one after one.

After the first few iterations with demonstrated and usable artifacts, our customer calls us and explains that all prices in the webshop are netto prices and that there need to be some kind of brutto price calculation. You, being accustomed to prices that don’t change once you put products into your shopping cart, ask a few questions and can finally grasp the concept of value added taxes. Now you want to implement it into the webshop.

The first approach to the whole complex is to show the brutto prices right besides the netto prices when the user views his shopping cart. You can then validate the results with your customer and discuss problems or misconceptions that are now visible and therefore tangible.

The original code of your shopping cart renderer might look like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
               each.description(),
               each.nettoPrice());
      }
      return result;
  }
}

A quick explanation of the code: The class ShowShoppingCart takes some products and converts them into a ShoppingCartRenderModel that contains the shopping cart data in a presentable form so the GUI just needs to take the render model and paste it into some kind of template. For each product, there is one line with a description and the (already renamed) netto price on the page.

Visibility level 0: It’s just code anyway

Let’s start with the lowest and most straight-forward implementation visibility level: The inline implementation.

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5 and 9. As you can see, the programmer chose to implement exactly what he understood from the discussion about netto and brutto prices with the customer. A brutto price is a netto price with value added tax. The VAT rate is 19 percent at the time of writing, so a multiplication with 1.19 is a valid implementation.

Our problem with this approach isn’t the usage of floating point numbers in the calculations or that calculations even exist in a method that should do nothing more than render some products, but that the visibility of the requirement is minimal. If you, I or somebody else doesn’t know exactly where this code hides, we will have a hard time finding it once the VAT is changed or anything else should be done with brutto prices or VATs.

Technically, the customer’s requirement is implemented and the brutto prices will show up. But because the concept of taxes (or VAT) is important for the customer, we likely made the code too invisible to be maintainable.

Visibility level 0+: Hey, I even wrote a comment

To make some part of the code stick out of the mess, we have the tool of inline code comments. Let’s apply them to our example and raise our visibility level from 0 to 0+:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      // AN-17: calculating the brutto price from the netto price
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5, 6 and 10. You can see that the programmer chose the same approach as before, but realized that the code would be buried if not marked. Given that the requirement identifier is “AN-17”, the code can be found by a text search of this number. And if you happen to stumble upon this part of the application, you can deduct meaning about what you see from the comment.

Except that you cannot really be sure what the AN-17 code really is. Is the result.addProductLine() part of AN-17 or not? Would you expect the calculation of taxes and prices in a method called render() in a class named ShowShoppingCart? Is this implementation really correct? Aren’t there different tax rates for different products? Did the original author think about that? Is the customer content with this functionality?

Note that you cannot really test the brutto price calculation. You have to invent some products, render them and then scrape the brutto prices from the render model. That’s tedious at best and a clear sign that the implementation visibility is still too low. On to the next level

Visibility level 0++: This sucks, but I’ve got to go now

This level tries to make you a partner in crime by explicitly stating what’s obviously wrong with the code at hand. Now it’s your responsibility to fix it. You wouldn’t leave a broken window be, would you?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
  for (Product each : inCart) {
    // AN-17: calculating the brutto price from the netto price
    // TODO: take different tax factors into account
    final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
    result.addProductLine(
          each.description(),
          each.nettoPrice(),
          bruttoPrice);
  }
  return result;
  }
}

The new code is in lines 5, 6, 7 and 11. The new comment line 6 is typical for this visibility level: The original programmer knew that his implementation isn’t adequate but couldn’t be bothered with improving it. Perhaps he had external circumstances force him to do it. Whatever it was, this code is the equivalent to a soiled public toilet. The difference is, this time we can determine who made the mess.

The apologetic “I know I made a mess” comment often begins with TODO or FIXME. This isn’t directed towards the original author, it’s pointed at you, the person that happens to read the comment. Now, what are you going to do? Pretend you didn’t read the comment? Leave the toilet soiled? Clean up the mess of your predecessor? You probably have work to do, too. And doesn’t it work the way it is? Never change a running system!

We will see how you can improve the implementation visibility of the requirement in the next blog post of this series. Stay tuned!

Recap of the Schneide Dev Brunch 2017-10-08

If you couldn’t attend the Schneide Dev Brunch at 08th of October 2017, here is a summary of the main topics.

brunch64-borderedLast sunday, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was smaller, which enabled us to use the meeting table with some comfort. Sometimes, with many attendees and bad weather, this table can get a little bit crowded. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were a small group, we discussed without an agenda. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Area of training

We shared some stories about top-notch video game players and how they keep up with the demand to stay competitive. Similar stories can be told about every topic imagineable: What did the “king of the hill” do to rise to such levels? The answer is always: training. Excessive, brutal training. They are first in the gym and lock the door late at night because they are the last, too. The use every waking second for practice and repetition. They are obsessed with being “the best”. If you want to follow such a story in movie format, you might enjoy “Whiplash”, a movie about an aspiring expert drummer that also highlights the delicate trainer/trainee dynamics. If you are more interested in the strategies of obtaining mastery, the book “Mastery” by Robert Greene will give you a lot of insights.

With this background, we asked around what our area of training (not expertise, not mastery – just training) is. The answers varied wildly, from the obvious “programming” to “whisky” (as in whisky tasting and collecting whisky). It’s an interesting question: what goal are you actively pursuing at the moment?

Hacking challenges

Evolving from the first topic, we talked about coding challenges and “capture the flag” hacking contests. If you aren’t the grandmaster in the area of the contest, you’ll get the most out of it by following the participating teams and trying to understand their approaches.  The local security capture the flag team of the KIT is especially open with their approach, their failures and successes. You might want to check out their website.

One challenge included trying to break a whitebox encryption, which is an interesting topic in itself. Maybe somebody can read up on this topic and give a little presentation in the future. Another challenge seemed to lead to an elaborate buffer overflow attack, when in reality, it could be solved with a “simple” use-after-free attack.

An useful starting point for aspiring security hackers might be the CTF (capture the flag) field guide. There are also some online challenges for basic training purposes, like the cryptopals or the bandit wargame. Thanks Tobias for the links!

If you are more interested in playful challenges and don’t want to show up on somebody’s radar, programming/hacking games like TIS-100 are perfect for you. Our game night with TIS-100 is still in vivid memory.

Software Architecture training

There are a lot of programming contests and hacking challenges out there. But what about dry-run training for software architects? On a related scale, there are hundreds of training simulators for the foot soldier (called ego-shooters), but little games for the aspiring officer/general. The website armchair general lists a few and even has some contests, but they lack the depth of real experience. Similarly, the training for software architects will probably be clean-room exercises, when in reality, the customer needs, the team mood, the latest fad in technology and even the weather will influence the architecture just as well as textbook knowledge.

We couldn’t discuss this topic to its full potential, so it will re-appear on the agenda of the next Dev Brunch. And its open for discussion in the comments: What are good books and trainings for software architects?

Thomas pointed us to the Architectural Katas by Ted Neward. Perhaps we should schedule a Schneide Event to try them out?

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in December. We even have some topics still on the agenda (like a report about first-hand experiences with the programming language Rust). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

The project jugglers

This blog entry describes some aspects that help us to work on multiple projects at once.

By Usien (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia CommonsIn this blog post, I will shine some light on a feature of our company that is often met with disbelief: How five developers can work on twenty projects at once without being stressed. I try to use the metaphor of a juggler, though I know nothing about juggling other than it can be done. I cannot hold more than one object in the air at any given time and even that is an optimistic estimate. But I’ve seen jugglers keep six to eight objects flying with seemingly no effort. We do this with software development projects.

A layman theory of juggling

A good metaphor can be applied from start to finish. I’ve probably chosen a bad metaphor, but it gives the right initial impression: Every developer at our company leads several projects at once. He (or she) keeps the projects alive and in the “green zone”, the ratio of remaining budget, time and scope (read: work left to be done) that promises little to no trouble in the foreseeable future.

In order to juggle without visible effort, you probably need to practice a lot. You probably drop your objects a lot. You probably need to watch the objects fly in the beginning.

In our case, we needed a lot of practice to reach our level of confidence. We lead development projects for up to 17 years now. Each developer finishes between four and seven projects per year. That’s up to a hundred projects to gain experience from. But we couldn’t drop (read: fail) a lot of projects, because it directly hurt our bottom line. Just imagine you want to learn how to juggle, but all you got are expensive ming vases that you bought from your own money, without insurance. That’s how it feels to “experiment” with projects. So we play it safe and only accept projects we know we can handle. And we watch our projects fly, very very closely. In fact, we have a dedicated position, the “project manager”, with the one duty to periodically ask a bunch of questions to assure that the project is still in the green zone.

Draw the trajectory

Every object that you can juggle has its own characteristics of how it behaves once it is in the air. A good juggler can feel its trajectory and grab it in just the right moment before it would fall out of reach. The trick to keep a software development project in the green zone is the same: Get a hold on it before it ventures too far in an unfortunate direction, which is the natural tendency of all projects. The project lead has to periodically apply effort to keep the project afloat. But when is the right time to invest this effort? Spend it too soon and it has only minimal effect. Spend it too often and you’ll exhaust your power and your budget. But because every project has its own trajectory, too, and you can’t afford to let it slip, you should make the trajectory as visible as possible. You should draw it!

We’ve experimented with a lot of tools and visualizations. The one setup that works best for us is a low technology, high visibility approach. The project lead takes one half of a whiteboard and draws a classic burn-down chart or a variation (we often use a simple vertical progress bar). This chart is hand-drawn and rather crude, but big enough that everybody can see it. It is updated at least every time the project manager comes around to ask his or her questions. One of the questions actually is: “Is this chart up to date?”. The remaining budget of your project needs to be available at a glance, from across the hall. The project lead needs to “feel” this budget. And if it makes him or her nervous, it’s high time to determine the remaining scope and calendar time of the project once again.

In doing this positioning by triangulation on a regular schedule, the project lead draws the trajectory of the project for all three axes and can probably interpolate its future course. He can then apply effort to nudge (or yank, if things got worse fast) the project back on track.

Without the visibly drawn trajectory, your project lead is like a juggler in the dark, tossing unknown objects in the air and hoping that they’ll fall in place somehow.

Know your limit

As the complete noob to juggling that I am, I imagine that jugglers have a secret dress code like martial artists (watch their belts!), where other jugglers can read how many objects they can hold up at once. Something like buttons on the vest or the length of a scarf. So the beginning 3-objects juggler bows in awe to the master 12-juggler, who himself is star-struck by the mighty 18-juggler that happens to attend the same meet-up.

In our company, this “dress code” would be based on the number of projects you are leading. And just like with the jugglers, it is important that you know your current limit. There is no use in over-extending yourself, if you accidentally let one project slip, the impact is big enough that you’ll fail your other projects, too. Just like the juggler loses his or her rhythm, you’ll lose your “flow”.

The most important part of juggling many projects is that you always juggle one less than you are capable of. You need reaction time if one of them topples over. A good juggler can “rescue” the situation with subtle speedup or extra movements because the delay between necessary actions allows for it. A good project lead has emergency reserves to spend without compromising other projects.

There is nothing wrong to start with two projects and add more later on when you are more confident. But don’t start with only one. You can only form habits of resource sharing if you share from the beginning. Even I can pose as a competent 1-juggler, but the lowest bar to juggling has to be two objects.

Box your time

Again, I know nothing about juggling. But from a mathematical viewpoint, juggling is “just” an exercise in timeboxing. If you have four objects in the air, in an arc that requires one second to go around, you’ll be able to spend a quarter second (250 ms) of attention to each object on each rotation. The master 12-juggler from above can only afford 1/12th or 80 milliseconds for each object. If he takes longer for one object, the next one will suffer. If he has no time reserves, a jam will build up and ultimately break the routine.

So, as a project lead, you need to apply timeboxes on all of your projects. They don’t need to be of equal size, but small enough that you can multiplex between your projects fast and often enough. A time box is a fixed-size amount of time that you allot to a specific task. The juggler uses a time box to put the next falling object flying back up. In our lunch break, we allot 60 minutes to food, beverages and some amount of walking. And if the process of eating takes longer than usual (I’m a chronic slow-eater, I can’t help it)? Then I have to go back partially hungry because I’ll end my lunch break on time. That’s the most important characteristic of a time box: You either succeed “in time” or interrupt or even cancel the task. The juggler will drop the one iffy object instead of risking a complete breakdown of the arc. You need to let your problematic task go (for the moment) instead of spreading the problem onto your other projects, too.

We’ve found that the amount of “one workday” is the most natural and easiest to manage time box. So we try our best to partition our week in the granularity of days and not our days in the granularity of hours. One aspect that helps tremendously is to have different physical locations for different projects. So you can be physically present “in the project” or “too far away at the moment” from the project. You plan your work week in locations as much as you plan it in project time boxes. The correlation of workdays, locations and projects is so strong that it doesn’t even seem to be timeboxing or project multiplexing. You just happen to be in the right place to work on project X for today. This is how you can juggle up to five projects without having to compromise all that much (provided you have a five-day work week).

If you can’t physically relocate your work, at least try to have a fixed schedule for your projects, like the “project A monday” or the “project X friday”. This might also mean to postpone emerging issues with project X until next friday. You need to build up skills to negotiate these delays with your customers. If your customers can dictate your schedule, you’ll get torn to shreds in no time. It’s friday or no day for issues on project X – at least as a good start for heavy bargaining. But that’s a whole topic for another blog post. Please leave a comment if you are interested to hear more about it.

Stuff your box

The “one workday” time box has a strong implication: Every little thing you do for a project takes one day. That doesn’t mean you should work for five minutes on project A, completing the task, and then stare into the air and twiddle your thumbs. It means that you should accumulate enough tasks for project A that you can spend the better half of the day on the known tasks and the remaining time on the unknown problems that arise on the way. In the evening, you should be able to finish your work for project A with a feeling of closure. You can put project A aside until next week (or whenever your next cycle is). You can concentrate on project B tomorrow and project C the next day. Both projects didn’t bother you today (well, perhaps a bit, but you only acknowledged some e-mails and deferred any real thoughts on it until you enter their timebox).

Perpetual closure

The feeling of closure at the end of a successful work day is the most important thing that keeps you composed. You’ve done your thinking for project A this week and will think of project B tomorrow. But now, you can rest.

This must be the feeling that the juggler experiences with each object that goes up again. It is out of sight and only needs attention after it has nearly completed its arc again. And now for the next object, one at a time…

 

On the usefulness of volatile memory

Observing ants is a wonderful source of insights. Here are some thoughts about memory and information storage gathered from ants.

I have a strange hobby: Raising ants as pets. Not literally pets, because I don’t give them names and they probably don’t know I even exist, but as tended animals in a controlled and restricted environment, the formicarium. This doesn’t sound too strange if you know three additional things about me: I was fascinated by ants since kindergarten, I play dwarf fortress for fun and I’m interested in computers.

My main colony is a lasius niger nest with about 1500 individuals. Lasius niger or black garden ant is a very common and easy to raise ant that is small enough to not require too much space but big enough to be observed and tracked.

I could tell you hours and hours of ant facts, but let’s concentrate on the topic of this blog post: An ant colony can be perceived as one big organism. Each individual ant has a clear role in the hierarchy:

  • The queen ant is the sole egg layer. In an established colony, she won’t do anything else. She will be fed, cleaned and protected by her workers. She was born a queen and cannot be replaced. If she dies, the whole colony will slowly fade away because nobody produces new ants anymore. Worker ants don’t know if they still have a queen and don’t care anyway. If a queen dies, she will be fed to the remaining larvas as soon as her scent disappears.
  • The male ants are born, fed and kicked out of the nest, preferably when the young queens fly out to find a mate. They won’t do anything else and have a lifespan of days.
  • The worker ants are all sterile females and do all the work. Literally all the work. They are born workers and spend every moment of their lives contributing to the hive or just sitting around scrounging food. Other workers are too busy to judge them.

But keep in mind that ants only have short-range communication means (scent, antennae drumming and ground vibrations), so no single ant has complete overview about the situation. It wouldn’t have the brains to process that information anyway. Ants have limited capability to remember things but no long-time memory. It isn’t necessary for the single worker ant to store any information. The information is stored in the hive – literally.

If you abstract some details away, you can also perceive and describe an ant colony as a computer or at least as a massive parallel problem solver. The algorithms are ingrained in the worker ants and are executed ruthlessly. The problems are food, water, enemies, cleaning and nurture. If the environment (the problem space) is suitable for the algorithms, the colony will thrive. Else the colony will ultimately fail. An ant colony at the side of a busy road will lose many workers in sudden “enemy” attacks, while a colony in the middle of a meadow might find less insects killed by cars. Not a single ant in each colony is aware of these relations. But they have found a way to share information: They mark their position (and therefore their way) by special scent fluid. They use their scents to label the environment for other ants.

If you ever encountered an avid user of printed sticker labels, this is what ants do, too: They put a sticker label on everything they come in contact with. If you seem to be food, they label you as food and rush back to the hive, laying out a “food in this direction” lane. If you seem dead and inedible, they mark you as garbage and them or some other ant will pick you up and follow the garbage line (finding the garbage line often requires them to return near their hive so it seems they mistook the garbage for food). The garbage area is marked with its own scent, preferably at a cliff. My ants love to throw things down a cliff! If you didn’t get the relationship to dwarf fortress yet, it should be clear by now. There are countless more examples, but you get the idea. The environment is not only an area, it is the long-term memory of the colony. The colony’s “memory brain” is scent on dirt, stones and sticks. The ant colony has outsourced its complete knowledge about the surrounding into the surrounding itself.

The ants have invented something I would call an “inverse cloud”. The cloud in IT is a concept of data storage that exists mostly independent of physical location and provides access to that data from virtually anywhere. The ants’ inverse cloud is a concept of data storage that is tightly coupled to a physical location and provides access to that data only if you are in the immediate vincinity. If you remove a physical data storage part in the IT cloud, it gets replaced by other parts that contain mirrored copies of the data. The cloud never forgets. If you remove a physical data storage part in the ants’ inverse cloud, it is forgotten immediately. Ants accept their environment as it is right now and never look back.

Now think about what happens when it rains and all the scents are washed away.

After every rain, the ant colony enters a “new level”, a fresh environment to be discovered and labeled. They probably never grow old to rediscover the same cliff again and again. This is what happens to a computer when the power is lost. It loses its working memory. But keep in mind that the colony retains some memories: the hive is underground and often rain-proof by overlaying stones or plants. So the colony remembers that it is currently in the hive, because it always smells like hive. The colony remembers the queen chamber because it always smells like queen. The colony remembers the brood chambers because they always smell like teen spirit (SCNR).

In my formicarium, it never rains. The ants get enough water by drinking troughs, but the marked lanes are never erased. This leads to all sorts of silly situations that the ants don’t even recognize as such. For example, a strong “food here” scent lane exists long after the food is gone. So a lot of enthusiastic workers run around searching in the target area. And remember that the hive smell never fades? My ants have assimilated area after area as “hive” after enough ants have marked it. So now they react excessively to disturbances because they think they are defending the hive – and therefore the queen! – but are ant miles away from the colony. Even better, a lot of young ants that usually never leave the hive until they are older wander out into the open (“it’s still the hive, just less dark”) and panic as soon as they encounter something that shouldn’t be in a hive. The panic spreads by, you’ve guessed it already, alarm scent and soon hundreds of battle-ready ants are running around frantically without any one of them knowing why.

So, to speak in computer terms, the main memory never gets erased and the caches never flushed. A little error can spread like a wildfire (like the panic example above) and cause disadvantages like energy consumption without any real gain (no enemy to defend against) or a lot of delicate ants sitting around in plain sight of their predators. The whole system is fragile and erratic and probably wouldn’t survive in the wild. A good measure of rain would remove the odd memories and probably ease the ants because their hive, the area that needs to be defended at all costs, would shrink again.

I’ve raised neurotic ants in need of a cold shower.

How does this give us any insight into modern computing? Well, my train of thoughts is this: If modern systems are unable to forget, because memory is cheap and permanent, we might be prone to design software that acts neurotic and hyped up. The ability to forget, to really don’t remember at all, might be crucial in designing resilient parallel systems. There is the cost of losing valueable information, but the benefit of losing all results of errors seems to match it. So volatile memory might be a nuisance for us programmers, but it also provides a “blank slate” every time the system starts and is the reason for the most important question in IT: “Have you tried turning it off and on again?”

Systems that rely on “place oriented programming”  seem to have the need of regular reset phases where the working memory is cleared and the system goes into the next cycle fresh and rested. We might even call it sleep. And in case you wonder: The sleep of ants is an ongoing topic for research.

Disclaimer: I know that not all ants are as dumb as lasius niger. Some ants even teach each other facts about their environment. The wikipedia article mentions some wonderful examples. I had ant colonies with more complex ants and they were wonderful. But right now, as I’m typing this, there is a lasius niger worker that heaves a wasp husk part to the top corner of the formicarium, throws it down (did I mention they love throwing things?) and runs back down to heave it up again, probably because the corner is somehow marked as a garbage dump zone. It has repeated this process at least half a dozen times now. This is how a biological infinite loop looks like. Some ants even parallelize such a loop to exhaustion:

Learning about Class Literals after twenty years of Java

The story about a customer request that led to my discovery of a cool feature of Java: Class Literals (you’ve probably already heard about them).

I’ve programmed in Java nearly every day for twenty years now. At the beginning of my computer science studies, I was introduced to Java 1.0.x and have since accompanied every version of Java. Our professor made us buy the Java Language Specification on paper (it was quite a large book even back then) and I occassionally read it like you would read an encyclopedia – wading through a lot of already known facts just to discover something surprising and interesting, no matter how small.

With the end of my studies came the end of random research in old books – new books had to be read and understood. It was no longer efficient enough to randomly spend time with something, everything needed to have a clear goal, an outcome that improved my current position. This made me very efficient and quite effective, but I only uncover surprising facts and finds now if work forces me to go there.

An odd customer request

Recently, my work required me to re-visit an old acquaintance in the Java API that I’ve never grew fond of: The Runtime.exec() method. One of my customer had an recurring hardware problem that could only be solved by rebooting the machine. My software could already detect the symptoms of the problem and notify the operator, but the next logical step was to enable the software to perform the reboot on its own. The customer was very aware of the risks for such a functionality – I consider it a “sabotage feature”, but asked for it anyway. Because the software is written in Java, the reboot should be written in Java, too. And because the target machines are exclusively running on Windows, it was a viable option to implement the feature for that specific platform. Which brings me to Runtime.exec().

A simple solution for the reboot functionality in Java on Windows looks like this:


Runtime.exec("shutdown /r");

With this solution, the user is informed of the imminent reboot and has some time to make a decision. In my case, the reboot needed to be performed as fast as possible to minimize the loss of sensor data. So the reboot command needs to be expanded by a parameter:


Runtime.exec("shutdown /r /t 0");

And this is when the command stops working and politely tells you that you messed up the command line by printing the usage information. Which, of course, you can only see if you drain the output stream of the Process instance that performs the command in the background:


final Process process = Runtime.exec("shutdown /r /t 0");
try (final Scanner output = new Scanner(process.getInputStream())) {
    while (output.hasNextLine()) {
        System.out.println(output.nextLine());
    }
}

The output draining is good practice anyway, because the Process will just stop once the buffer is filled up. Which you will never see in development, but definitely in production – in the middle of the night on a weekend when you are on vacaction.

Modern thinking

In Java 5 and improved in Java 7, the Runtime.exec() method got less attractive by the introduction of the ProcessBuilder, a class that improves the experience of creating a correct command line and a lot of other things. So let’s switch to the ProcessBuilder:


final ProcessBuilder builder = new ProcessBuilder(
        "shutdown",
        "/r",
        "/t 0");
final Process process = builder.start();

Didn’t change a thing. The shutdown command still informs us that we don’t have our command line under control. And that’s true: The whole API is notorious of not telling me what is really going on in the background. The ProcessBuilder could be nice and offer a method that returns a String as it is issued to the operating system, but all we got is the ProcessBuilder.command() method that gives us the same command line parts we gave it. The mystery begins with our call of ProcessBuilder.start(), because it delegates to a class called ProcessImpl, and more specific to the static method ProcessImpl.start().

In this method, Java calls the private constructor of ProcessImpl, that performs a lot of black magic on our command line parts and ultimately disappears in a native method called create() with the actual command line (called cmdstr) as the first parameter. That’s the information I was looking for! In newer Java versions (starting with Java 7), the cmdstr is built in a private static method of ProcessImpl: ProcessImpl.createCommandLine(). If I could write a test program that calls this method directly, I would be able to see the actual command line by myself.

Disclaimer: I’m not an advocate of light-hearted use of the reflection API of Java. But for one-off programs, it’s a very powerful tool that gets the job done.

So let’s write the code to retrieve our actual command line directly from the ProcessImpl.createCommandLine() method:


public static void main(final String[] args) throws Exception {
    final String[] cmd = {
            "shutdown.exe",
            "/r",
            "/t 0",
    };
    final String executablePath = new File(cmd[0]).getPath();

    final Class<?> impl = ClassLoader.getSystemClassLoader().loadClass("java.lang.ProcessImpl");
    final Method myMethod = impl.getDeclaredMethod(
            "createCommandLine",
            new Class[] {
                    ????, // <-- Damn, I don't have any clue what should go here.
                    String.class,
                    String[].class
            });
    myMethod.setAccessible(true);

    final Object result = myMethod.invoke(
            null,
            2,
            executablePath,
            cmd);
    System.out.println(result);
}

The discovery

You probably noticed the “????” entry in the code above. That’s the discovery part I want to tell you about. This is when I met Class Literals in the Java Language Specification in chapter 15.8.2 (go and look it up!). The signature of the createCommandLine method is:


private static String createCommandLine(
        int verificationType,
        final String executablePath,
        final String cmd[])

Note: I didn’t remove the final keyword of verificationType, it isn’t there in the original code for unknown reasons.
When I wrote the reflection code above, it occurred to me that I had never attempted to lookup a method that contains a primitive parameter – the int in this case. I didn’t think much about it and went with Integer.class, but that didn’t work. And then, my discovery started:


final Method myMethod = impl.getDeclaredMethod(
        "createCommandLine",
        new Class[] {
                int.class, // <-- Look what I can do!
                String.class,
                String[].class
        });

As stated in the Java Language Specification, every primitive type of Java conceptionally “has” a public static field named “class” that contains the Class object for this primitive. We can even type void.class and gain access to the Class object of void. This is clearly written in the language specification and required knowledge for every earnest usage of Java’s reflection capabilities, but I somehow evaded it for twenty years.

I love when moments like this happen. I always feel dumb and enlightened at the same time and assume that everybody around me knew this fact for years, it is just me that didn’t get the memo.

The solution

Oh, and before I forget it, the solution to the reboot command not working is the odd way in which Java adds quote characters to the command line. The output above is:


shutdown /r "/t 0"

The extra quotes around /t 0 make the shutdown command reject all parameters and print the usage text instead. A working, if not necessarily intuitive solution is to separate the /t parameter and its value in order to never have spaces in the parameters – this is what provokes Java to try to help you by quoting the whole parameter (and is considered a feature rather than a bug):


final String[] cmd = {
        "shutdown",
        "/r",
        "/t",
        "0",
};

This results in the command line I wanted from the start:


shutdown /r /t 0

And reboots the computer instantaneous. Try it!

Your story?

What’s your “damn, I must’ve missed the memo” moment in the programming language you know best?

The personal economics of programming languages

An attempt to answer a student’s question about what programming language to learn.

Recently, one of my students asked a good question about what programming languages I would recommend learning. His ideal language would be “syntactically ugly, but giving insights that are universal to programming”. My first reaction was to answer that he has just described Perl, but that was too easy of an answer. So I tried to define the basics about programming languages, starting with the personal economics.

Economics of programming languages

An organization that wants to produce a piece of software needs to answer a lot of questions like “what programming language will be best suited for the task?”. Often, these questions get diluted and rather sound like “what programming language should we stipulate for all our projects, now and forever?”. That’s when politics and economics overlap and intermingle. We can leave this problem for the organizations to solve themselves. But if we scale the question down to an individual programmer – you, what influences are there to find an answer to the question “what programming languages should I learn?”.

I try to answer with the concept of utility: Learn those languages that, over a reasonable time, yield the most “utility”. There are at least two types of utility in our profession: money and joy. You can learn a programming language because your job requires it (money) or because you are curious and/or dig its particularities (joy). Most of the time, a specific programming language contains a mixture of both utilities for you. How you rate those utilities is up to you and probably varies from situation to situation. If you start a private fun project, picking the boring mainstream language from work might get things get done faster, but when would you want the fun to be over sooner?

Let me give two extreme examples for this concept:

  • If you start to learn COBOL now, chances are high that you will achieve two things: You will be disgusted by the language and the existing codebase, but delighted by the salary and job security. COBOL is a high money-utility programming language. It ranks low in any survey or statistics about programming languages, but is widely used in big business today and tomorrow. You might refer to https://blog.hackerrank.com/the-inevitable-return-of-cobol/ for more information.
  • If you start to learn Esterel now, you might experience two things over time: an epiphany about how flawed our concept of time is in most programming languages and an existential crisis because your brain isn’t capable to wrap itself around most sourcecode. Whatever comes first will define your learning success. There are virtually no jobs that require Esterel (even if some might benefit from it) and you can only program and build so many bicycle computers in your spare time (this is a typical introduction project to Esterel). Esterel is a pure joy-utility programming language. You can claim to be proficient in synchronous programming afterwards, but nobody will know what that even is.

A third type of utility

But I think that there might be a third type of utility for personal learning choices based on economics: The stirrup iron utility. Knowledge of some programming languages isn’t useful from a money-driven viewpoint and may lack enjoyability, but it serves as a door-opener to more enjoyable or sellable languages. It serves as an interim utility because it doesn’t have value in itself, but serves as a multiplier for either the money or joy utility. To rate the value of this utility to your career, you need to be clear about your career goals, especially your anticipated skill portfolio.

Skill portfolio shapes

Modern recruitment differentiates between several skill portfolio shapes, most noteably the “I” and “T” shape:

  • Programmers with “I”-shaped skill portfolios are experts in one specific field of programming. They might, for example, be the best C# programmer you’ve ever met. But they flop around like a fish out of water once they need to use another programming language. They will choose their familiar tools for every problem that needs to be solved and will solve it fast if possible or
  • Programmers with “T”-shaped skill portfolios have knowledge across all fields of programming, albeit limited, and drilled down into one field specifically. Why they chose to master their field can mostly be explained with the money or joy utility. They probably gained their broad knowledge base by using stirrup irons.

If you happen to know what’s expected from you until your retirement (let’s say you chose to program in COBOL), the “I”-shape is a viable and efficient strategy to manage your skill portfolio. There is nothing wrong with this approach (as long as it works).

If you have a hunch that you don’t have the capability to invest in broad knowledge, the “I”-shaped skill portfolio is your logical choice. It takes a lot to be able to come to a self-assessment that shows your limitations. It’s a good thing to know your limits and build a career within them. A lot of programmers don’t know their limits and burn out, because not meeting the requirements produce a lot of stress (on both sides). Better be yourself than over-promise and under-deliver constantly.

The “T”-shape means that you need to invest your time wisely. And we are not talking “work time” only, but “life time”, because you’ll probably need to spend your spare time working on your portfolio, too. Becoming a “jack of all trades” programmer is an endeavour of at least ten years without any possibility to shortcut. You need to select your jobs in accordance to your learning strategy and always be receptive to opportunities. You need to improve your learning abilities. You need to do so much at once that I suggest you start by watching Cory House’s talk about “Becoming an Outlier”. He’s spot on with so many things.

Stirrup iron programming languages

There are some programming languages that can be seen as the archetypes of a whole class of languages. Most knowledge of these archetypes can be directly applied or transfered to each language in the class. It’s the language’s concepts that are the real benefit. If you understand the synchronous programming aspect in Esterel, you’ll recognize it straight away in languages like LabView or SIGNAL. It may even just be a part of the other language (like in many multi-paradigm programming languages), but it will be familiar to you.

So what are some stirrup iron languages?

That’s a tough question and I want to place it out there. Can you drop a comment and name the programming language that had the most peculiar influence on your knowledge? I would like to refer to the book Seven Languages In Seven Weeks from the Pragmatic Bookshelf. It covers Ruby, Io, Prolog, Scala, Erlang, Clojure and Haskell. Do you agree with that selection? I would like to hear from you.

There are some ideas about this topic already: The talk “The Future of Programming” from Bret Victor (if you don’t know this guy already, please watch his legendary “Inventing on Principle” too). Richard Astbury presents three “new” hot programming languages (with matching outfits) in his talk “The State of the Art”. And Robert C. Martin is sure to have found “The Last Programming Language”.

One thing is sure: We should train the next generations of programmers in those stirrup iron languages, so they can quickly grasp the language flavour of the year. This is mostly done already, of course, but the students inevitably complain about the “weird” choices. So we need to explain upfront the economics of programming languages.

And, in a lighter tone at the end, there is always the ongoing competition for the worst programming language ever.