Ten books that shaped me as a software developer – Part I (Books 0 to 4)

I was asked what books shaped me as a software developer the most. Here is my answer, or the first part of it.

Last week, I’ve done a question and answers event with students when the question came up what the most influential books were that I have read as a software developer. I couldn’t answer the question right away but promised to compile the list with short descriptions of the book’s influence. And here it is – my list of books that left a big mark in my day-to-day work. Others have done the list of books thing before me, and most lists contain the same books over and over again. I take it as an indicator that my list isn’t too far off.

Prologue

Before I start the list, I want to say a few things. The list isn’t ordered or ranked. I describe the effects of each book from my current standpoint, sometimes 20 years after the fact. I read a lot more good, interesting and inspiring books in the last 20 years and they all added to my work personality. But with all the books on my list, I felt enlightened and vibrant with new ideas. They didn’t just inspire me, they elevated my thinking. And because of this criteria of immediate improvement, one book is missing from the list. It’s the first “serious” software development book I’ve ever read in 1998: “Design Patterns. The book was just too much for me (and my study group peers) to handle such early in our careers. We were in our first year of study and had a lot of other battles to fight. I crossed it from my reading list and moved on. Years later, I re-read it and saw so much insight I plainly missed the first time, but gathered elsewhere since. If you want to read this classic, don’t hesitate! If you “only” want to know about design patterns, there’s a better book for that: “Head First Design Patterns“.

The Pragmatic Programmer

https://images-na.ssl-images-amazon.com/images/I/41BKx1AxQWL._SX396_BO1,204,203,200_.jpgMore by chance, my co-founder stumbled upon “The Pragmatic Programmer” in 1999 and devoured it. Then he gave the book to me and it shattered me to my core. I thought I was a decent software developer and here are Dave Thomas and Andy Hunt and talk about things I didn’t even knew existed. A healthy dose of Dunning-Kruger effect is crucial in everybody’s upbringing, but this book ended my overestimation once and for all and gave my studies a focus and direction I wouldn’t have thought to be possible before. I own my whole career to this book, at least in terms of work ethics. I cannot fathom how my professional life would have played out otherwise.

Refactoring

https://images-na.ssl-images-amazon.com/images/I/51K-M5hR8qL._SX392_BO1,204,203,200_.jpg

Also in 1999, Martin Fowler wrote his instant classic “Refactoring“. We bought this book at the first chance we got and raced through the pages. I was a Java developer back then and with most of the examples being in Java, the book needed no explanation nor translation. It was directly applicable knowledge that gave me years of experience virtually for free. This book is a must-read even 20 years later, and has just recently had the second edition announced, this time with code examples in JavaScript. I thought it was a joke first, but I guess it makes sense.

Working Effectively With Legacy Code

https://images-na.ssl-images-amazon.com/images/I/51EgCCLOWxL._SX376_BO1,204,203,200_.jpgIn 2004, Michael Feathers wrote a book that contains his 20+ years of experience with software development and named it “Working Effectively With Legacy Code“. Well, joke’s on you – I don’t write legacy code, my code is perfect. That wasn’t my attitude since 1999 (see list entry #1) and I took this book everywhere. It’s a heavy one, but I read it in the tram, right before the movie starts in cinema, during breakfast, lunch and dinner and virtually any other circumstance. I realized that reading this book will gain me experience a lot faster than actually writing code, so I just stopped for a few weeks. This book answered a lot of mysteries in the form of “is there really no better way to do this?” for me. And it introduced the concept of code seams for me that permeates my work ever since. I can clearly remember the day when I looked at my existing code again and saw the seams for the first time. It was truly eye-opening for me.

Analysis Patterns

https://images-na.ssl-images-amazon.com/images/I/41uNHkTq8NL._SX378_BO1,204,203,200_.jpgMartin Fowler was a very productive author in the late nineties. I’ve read most of his books from this period, if maybe with a few years delay. “Analysis Patterns” from 1996 arrived in my bookshelf in the early 2000’s and was my wake-up call to seeing models instead of actualities. I’ve given this book to many peers, but haven’t received the reactions that I had with this book: Being taught a language (with a graphical notation) that can express actual problems in terms of an overarching solution. Since then, I’ve seen the same solutions applied in many different forms, with many different names and a lot of different special requirements. But they all derive from the same model. This effect was promised by the “Design Patterns” book, but for me, delivered by “Analysis Patterns”. Even Martin Fowler admits that the book is showing its age, but for me, its timeless.

Peopleware

https://images-na.ssl-images-amazon.com/images/I/51MlUgcSICL._SX331_BO1,204,203,200_.jpgSince the late 80’s, Tom DeMarco and Timothy Lister wrote one book after the other. Each book describes a common business-oriented problem and at least one working solution for it. And yet, the very same problems still persist in the business world. It’s as if nobody reads books. “Peopleware” was written in 1987, 30 years ago, and discovered by me and my peers in the late 1990’s. We talked about this book a lot, as it described a (business) world where we didn’t want to work in. We wanted to do better. In a way, this book was a spark to found our own company and don’t repeat the mistakes that seemed to be prevalent in our industry. If you’ve ever shaken your head about “the management”, do yourself a favor and read this book. It will pinpoint the precise problem you’ve felt and give you the words to describe it. And if you’ve read “Peopleware”, liked it and want more, there is good news: There is a whole series waiting for you (not just Vienna).

Epilogue to Part I

These are the first five books from my list, with the last entry being more of a catch-all for a whole series. Remember that this isn’t a generic “go and read these books if you want to call yourself a professional software developer” list. I’m not gatekeeping and it would be useless to even try to do so. These books helped me further my career in the last 20 years, they won’t necessarily help you for the next 20 years. Good books are published every year, you just have to read them.

I’m looking forward to share the second part of my list in the next blog post of this series. Stay tuned!

Recap of the Schneide Dev Brunch 2018-02-11

If you couldn’t attend the Schneide Dev Brunch at 11 February 2018, here is a summary of the main topics.

brunch64-borderedOn Sunday, February 11th, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was well-attended, with two new guests that seemed to feel comfortable after just a few minutes. The table provided just enough space for us. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were a larger group, we discussed with an agenda. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Asciidoctor

Our first topic was a presentation of the asciidoc syntax and the asciidoctor converter. The asciidoc syntax can be used to describe structured textual content in a concise manner with a few funny special characters. It looks like markdown at the first glance, but has the benefit of being fully standardized and extensible instead of one of several competing dialects.

The asciidoctor is an active rewrite of the first asciidoc converter. Given the right set of formatters, you can generate a PDF, a self-contained interactive HTML presentation and a static web page from one single source. This follows the “one true source, many derived artifacts”-approach that every software developer should know by heart (Don’t Repeat Yourself!).

Because setting up a productive asciidoctor environment is still some manual work, one of our attendees has published a github repository that automates the manual work as much as possible: asciidoc presentation.

If you need an alternative to markdown or even TeX/LaTeX, have a look at asciidoc. It seems specifically aimed at software developers and is probably already integrated in your favorite IDE (the integration in IntelliJ is seamless).

Yarn

We discussed the two extreme approaches to handle dependencies for your project. The first extreme is to only include links to other projects/repositories that need to be fetched manually or automatically. Most modern build tools orientate towards this approach, even if there are some disadvantages like the recent Go/Github disturbance.

The second extreme is to include everything that’s needed in your repository. For a Javascript project that means that you provide your own, probably out-dated version of leftpad and thousand other libraries. You need a way to deal with transitive dependencies and keep an eye on all the versions to mitigate the risk of long-fixed vulnerabilities.

The second extreme is extremely helpful if you don’t have internet access but want to develop.

A good compromise is the local offline mirror, something that build tools/dependency managers like maven have for over a decade. This local repository is filled with all the leftpads and apache-commons that your projects need. If you checkout a new project, remember to make the build tool download the dependencies to your local repository before you go offline.

for Javascript, this concept seems a bit foreign. Who would develop for the web without the web, anyways? Yarn seems to provide a working offline mirror functionality for npm packages, though. Perhaps it is worth a look.

Opt-Out explained with groceries

During out dependency management discussion, we also compared downloadable installers with malware droppers. But that’s not where our comparisons stopped. We also came up with a good metaphor for Opt-In vs. Opt-Out methods.

If you enter a grocery store and grab a shopping cart, only to find that it already contains two or three packages of sweets and some overpriced milk, you chose an Opt-Out store. Your responsibility is to return the goods to their aisle or to buy them.

You’re probably used to Opt-In type grocery stores.

Book review: Functional Programming in Java

We took a look at Pierre-Yves Saumont’s book “Functional Programming in Java”. This book is a little bit odd in that you shouldn’t read it, you are meant to program it. Or at least try to solve the numerous training exercises and riddles. This makes it hard to read the paper version of the book, because it’s a pick-two situation of keyboard, mouse and book on your desk.

The book explains real functional programming and not the functional additions of Java 8. It explains it on top of the JVM, using Java’s language constructs. But, you will learn it from the origins and develop abstractions like Function oder Supplier yourself. Imagine you had all compiler magic of Java 8 but no JDK classes to leverage it – this book tells you how to use it.

It’s a good book, but unique in its style. It grounds on exercises and your own understanding of the material. It isn’t spoon-fed, you have to work for it yourself. It didn’t chose any existing pure functional language, but plain Java for this. So you have no excuse about weird syntax or unfamiliar ecosystems. It’s boring old Java turned in an exciting new way.

And if you are lazy and don’t feel like writing your own functional groundwork toolkit, you might want to look at vavr, a functional programming library for Java.

Polyglot language idioms

We discussed the portability of language idioms and highlighted the Curiously Recurring Template Pattern (CRTP) from C++. Then we spent some time explaining and understanding the CRTP and finally comparing it to similar things like Java’s Enum<E extends Enum<E>>. It can get wicked complex fast with those constructs.

Laser printer identification

Since 2011, we know that every single page of a color laser printer can be individually identified and traced back to your printer. This is common knowledge as stated on Wikipedia, but it still was a surprise to some of us. Why do we need such tracking? On request of many goverments.

Spectre and Meltdown

We didn’t repeat the fresh common knowledge about the nearly universal CPU security vulnaribilities Meltdown and Spectre. But we noted that it got eerily quiet, as if everybody holds their breath and waits for the morning clock to wake them up.

Some rumors has it that the current prototypes of ARM and Intel CPUs are not vulnerable, as if the manufacturers changed their speculative code execution unit long before the exploits came to light. Maybe they circumvented the problem by pure luck?

We hope to hit snooze soon.

Planned obsolescence

We discussed the notion of planned obsolescence. Typical consumer products have a flaw or weakness that is bound to break soon after manufacturer guarantee is void. Or it is deliberately incorporated into the product like page counters, waste tanks with limited capacity or the infamous short-lived light bulb.

A good start on the topic is the documentary “buy it for the waste” or “Kaufen für die Müllhalde” on german.

Given the recent noise around Apple battery life, we are now in an era where planned obsolescence is sold like a feature. Twenty-five years ago, this was Science Fiction. The author of this blog entry remembers a science fiction story by Robert Sheckley (“Utopia mit kleinen Fehlern” or “A Ticket to Tranai” in english). The protagonist reaches a planet that seems to be perfect. It is so perfect that nothing breaks anymore. The industry is desperate and sees the protagonist as a genius when he invents “planned obsolescence” and “designed discomfort” as means to raise sales. The planet has several other flaws as well. The story and the whole book is worthwhile and right on topic.

Book review (again): Clean Architecture

At last, we spoke about Robert C. Martin’s (Uncle Bob’s) new book “Clean Architecture”. I’ve already published my book review on our blog, but added some impressions and context after thinking about the book for some more time. Summary: The book is good, even if nearly half the pages might qualify as filler material and there are only two main messages. If the announcement of Uncle Bob on the last page in the Appendix becomes true, you might want to skip his next book, though.

Another book review for the future might be the new Effective Java, 3rd edition.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in April. We even have some topics still on the agenda (like a report about first-hand experiences with the programming language Rust). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

The four archetypes of cloud users – part 1 of 2

You probably know people that avoid cloud services as if they are poisonous and others that jump onto every cloud bandwagon. You can categorize them in four archetypes. Let’s start with the tinfoil hat and the clipboard.

In the occupational field of accounting, the strong trend towards cloud services is noticeable. Everything needs to be digital, and with digital, they mean online, and with online, they mean in the cloud. Every expense voucher needs to be scanned and uploaded, because in many cases, it can be booked automatically. In the new era of accounting, human intervention is only needed for special cases.

I see this as a good example of how digital online services can transform the world. Every step in the process would have technically been possible for the last twenty years, but only the cloud could unify the different participants enough so that a streamlined end-to-end process is marketable to the masses. And in this marketing ecstasy, the stakeholders that profit the most (the accountants) often forget that their benefits are just a part of the whole picture. In order to assess the perceived and actual benefits of all stakeholders, you at least need to apply an archetype to each participant.

The four archetypes

In my opinion, there are four different archetypes of cloud users. Let’s have a look at them and then assess the risks and potentials when selling a digital online service to them. I’ll list the archetypes in the order from biggest risk to biggest potential.

Archetype 1: The tinfoil hat

A person that could be identified as a “tinfoil hat” doesn’t need to be a conspiracy weirdo or paranoid maniac. In fact, the person probably has deep and broad knowledge about technology and examines new technologies in detail. The one distinctive feature of the tinfoil hats is that they take security, including IT security, very seriously. They don’t take security for granted, don’t trust asseverations and demand proof. You can’t convince a tinfoil hat by saying that the data transfer is “encrypted”, you need to specify the actual encryption algorithm. Using RC4 ciphers for the SSL protocol isn’t good enough for the tinfoil. You need at least proof that you understood the last sentence and took actions to mitigate the problem. Even then, the tinfoil will hesitate to give any data out of hands and often choose the cumbersome way in order to stay safe. “Better safe than sorry” is his everyday motto.

Tinfoil hats always search for scenarios that could compromise their data or infrastructure. They are paranoid by default and actively invest in security. “On premises” is the only way they deploy their own services, and “on premises” is how they prefer to keep their data.

Typical signs of a tinfoil hat archetype include:

  • self-hosted applications
  • physical servers
  • lack of (open) wireless network
  • physically separated networks
  • signed and encrypted e-mails

Trying to sell a cloud service to a tinfoil hat is like trying to sell a flight to an aviophobian (somebody with fear of flying). There is always another way to get from A to B, seemingly safer and more controllable. If you are selling cloud services, tinfoil hats are your worst nightmare. If you can convince a tinfoil hat, your product is probably made of fairy dust and employs lots of unicorns.

Archetype 2: The clipboard

Clipboard people are wary of new technologies, but assess them in the context of usability. They demand high security, but will compromise if the potential of the new technology far exceeds the risk. Other than the tinfoil hat, the clipboard sees his role as an enabler, but will not rest to increase the perceived or actual safety of the product. You can appease a clipboard by giving evidence of security audits from a third party. They will trust known authorities, because it means that they can always deflect blame in case of an accident to these authorities.

Clipboards run on checklists, safety protocols and recurring audits. They don’t try to avert every possibility of a security breach, but will examine each incident in detail and update their checklists. They don’t care about “on premises” or “off premises” as long as the service is reachable, safe enough and reliable. If a cloud service has an higher availability than the local counter-part, the clipboard will think about a migration.

Typical signs of a clipboard archetype include:

  • Virtual Private Networks (VPN)
  • Two-Factor Authentication
  • Token-Based Authentication
  • Strong Encryption

The clipboard will listen if you pitch your cloud service and can be enticed by the new or better capabilities. But in the very next sentence, he will ask about security and be insistent until you provide proof – first-hand or by credible third parties. You can convince a clipboard if your product is designed with safety in mind. As long as the safety is state-of-the-art, you’ll close the deal.

Outlook on the second part

In the second part of this blog entry, we will look at the remaining two archetypes, namely the “combination lock” and the “smartphone”. Stay tuned.

Did you identify with one of the archetypes? What are your most important aspects of cloud services? I would love to hear from you.

Book review: Clean Architecture by Robert C. Martin

A personal review of the book “Clean Architecture” by Robert C. Martin. Spoiler: you might want to read it.

In 2008, a book changed the way software developers around the globe talked (and hopefully) acted about their code. Robert C. Martin’s “Clean Code” was and still is a cornerstone of modern software development. The book itself is remarkably weak in its code examples, but has strong and effective messages on the level of practices and principles. Even today, ten years later, this is the one book that most of my students read and are passionate about. It’s a book that speaks reason to them, albeit with some contortion because of high volume. Robert C. Martin has the tendency to preach 200 percent in order to still get the half-convinced to an acceptable level.

So when a new book from him, called “Clean Architecture”, appeared on the horizon, I was thrilled. Would it be groundbreaking like “Clean Code” or a dud like “The Clean Coder” (sorry, my opinion – this is a personal review, not an academic evaluation)? I’ve read some very good books about software development (like “The Pragmatic Programmer”), fantastic books about programming (“Refactoring” and “Working Effectively With Legacy Code” come to mind) and even some mind-blowing pieces about design and emerging architecture (my first read of “Growing Object-Oriented Software, Guided by Tests” felt like a personal audience with Steve Freeman and Nat Pryce). But all these books dealt with tactics, with the immediacies of software development. Don’t get me wrong! This is the most important part and it helped me tremendously. But there are parts “above” the footwork that needs to be addressed in bigger systems, too.

And there, the literature got thin or stale. Books about software architecture talked about large-scale architectures (so-called “enterprise scale” systems that span from horizon to horizon, like in “Patterns of Enterprise Application Architecture”) or had the taste of dry plywood because it was clear that the findings were from another era and would translate badly into modern software development.

“Clean Architecture” begins with a quick and focused overview over the current programming paradigms and a conclusion that there are no different “eras”. We didn’t get better in designing systems, we just changed the aroma and color of our failures. Future generations will look at our code and architectures as scornful as we looked at the ruins of the systems of our ancestors. And make no mistake – the ruins are still in production today! We cannot place our hope on another new and liberating programming paradigm because there probably won’t be one. We have to make do with what we have.

This is the first six chapters of “Clean Archicture”. The chapters are short and on point and I loved every line of it. It probably isn’t the most comprehensive and balanced description of structured, object-oriented and functional programming, but it provides a narrative that is intuitive and convincing – your mileage may vary, I was hooked.

In the next five chapters, Robert C. Martin reiterates the known SOLID design principles. I rolled my eyes when I glanced the content because I’ve read it like a hundred times in maybe as many books. But I decided to read it once more and I’m glad I did. The principles are known, but the underlying revelation is woven into the text like a good thriller. I hesitate to give away too much, because I really think this book can be spoiled – just like a good thriller. I was sold. Robert C. Martin can explain the same old SOLID to me and I still learn something and have fun.

Then, the part about components. It feels like an intermezzo to an even better thriller, because suddenly there is math and formulas. Its interesting and noteworthy, but if you followed the metrics discussion in the last fifteen years, the excitement of this part will be dampened.

But wait, there is more! Starting with page 133 of 321 (yeah, the Appendix is interesting, but more in the “The Clean Coder” way of things), there is the central question: “What is Architecture?”. There it was again, the thrill that in every line, there could be insights that are worth weeks of thoughts. I read this part in the train from south to north germany and I stared out of the window often, following my own train of thoughts.

Again, no spoilers, but the way the answers are given is so refreshing and the answer itself is so simple that I’m surprised that it took me this long to not come to the same conclusion. Software architecture lost some of its mysticism, but gained a lot of applicability for me. I was spent (in a good way).

And then, on page 200, finally, “The Clean Archicture”. Well, I watched all the trailers on this topic, so my surprise wasn’t really there, but with all the knowledge and insights from the first 200 pages, I could have “invented” the Clean Architecture by myself then and there. It’s more or less the logical next step from the prerequisites. I applaud this masterwork of storytelling, because it doesn’t overwhelm the reader with the genius of the narrator, it drives him to connect the dots himself.

The rest of the book, like the title of part VI, are just “Details”. The central message  – The Dependency Rule, this little spoiler should be allowed – is simple, convincing and deduced from the beginning. I’ve seen the heart of software architecture and it is beautiful.

I even forgive the many typos and grammatical errors (far more than usual) and the bulky appendix for this ride. This book is definitely up there with “Clean Code”. It is accessible, has a clear message and profound effects. And it refrains from preaching most of the time. No need to turn it up to 200 percent when your message is so convincing in itself.

Conclusion: If you are interested in software development with a structure, go grab this book as soon as possible. We’ve waited long enough!

Implementation visibility – Part III

In the thirf part of our series about implementation visibility, we look at the high ground of our visibility scale and peek even higher. The series ends with a recap of the concept and its implications.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

We examined a simple code example in both preceding articles. The level 0, 0+ and 0++ were covered in the first article, while the second article talked about level 1, 2 and 2+. You might want to read them first if you want to follow the progression through the ranks. In this article, we look at the example at level 3, have a short outlook on further levels and then recap the concept.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable&amp;amp;lt;Product&amp;amp;gt; inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 3: Domain drive all the things!

We’ve introduced a new class for our requirement in visibility level 2 and made it a domain type. This is mostly another name for the concept of Entities or Value Objects from Domain Driven Design (DDD). If you aren’t familiar with Domain Driven Design, I recommend you grab the original book or its worthy successor and read about it. It is a way to look at requirements and code that will transform the way you develop software. To give a short spoiler, DDD Entities and DDD Value Objects are named core domain concepts that form the foundation of every DDD application. They are found by learning about the problem domain your software is used in. DDD Entities have an own identity, while DDD Value Objects just exist to indicate a certain value. Every DDD Entity and most DDD Value Objects are part of an DDD Aggregate. To load and store DDD Aggregates, a DDD Repository is put into place. The DDD Repository encapsulates all the technical stuff that has to happen when the application wants to access an DDD Aggregate through its DDD Root Entity. Sorry for all the “DDD” prefixes, but the terms are overloaded with many different meanings in our profession and I want to be clear what I mean when I use the terms “Repository” or “Aggregate”. Be very careful not to mistake the DDD meanings of the terms for any other meaning out there. Please read the books if you are unsure.

So, in Domain Driven Design, our BruttoPrice type is really a DDD Value Object. It represents a certain value in our currency of choice (Euro in our example), but has no life cycle on its own. Two BruttoPrices can be considered “the same” if their values are equal. This raises the question what the DDD Root Entity of the corresponding DDD Aggregate might be. Just imagine what happens in the domain (in real life, on paper) if you calculate a brutto price from a given netto price: You determine the value added tax category of your taxable product, look up its current percentage and multiply your netto price with the percentage. The DDD Root Entity is the value added tax category, as it can be introduced and revoked by your government and therefor has a life cycle on its own. The tax percentage, the netto price and the brutto price are just DDD Value Objects in its vicinity.

To bring DDD into our code and raise the implementation visibility level, we need to introduce a lot of new types with lots of lines of code:

  • NettoPrice is a DDD Value Object representing the concept of a monetary value without taxes.
  • BruttoPrice is a DDD Value Object representing the concept of a monetary value including taxes.
  • ValueAddedTaxCategory is a DDD Root Entity standing for the concept of different VAT percentages for different product groups.
  • ValueAddedTaxPercentage might be a DDD Value Object representing the concept of a percentage being applied to a NettoPrice to get a BruttoPrice. We will omit this explicit concept and let the ValueAddedTaxCategory deal with the calculation internally.
  • ValueAddedTaxRepository is a DDD Repository providing the ability to retrieve a ValueAddedTaxCategory for a known Taxable.
  • Taxable might be a DDD Entity. For us, it will remain an abstraction to decouple our taxes from other concrete types like Product.

The most surprising new class is probably the ValueAddedTaxRepository. It lingered in our code in nearly all previous levels, but wasn’t prominent, not visible enough to be explicit. Remember lines like this?

final BigDecimal taxFactor = <gets the right tax factor from somewhere> 

Now we know where to retrieve our ValueAddedTaxCategory from! And we don’t even know that the VAT is calculated using a percentage or factor anymore. That’s a detail of the ValueAddedTaxCategory given to us from the ValueAddedTaxRepository. If one day, for example at April 1th, 2020, the VAT for bottled water is decreed to be a fixed amount per bottle, we might need to change the internals of our VAT DDD Aggregate, but the netto and brutto prices and the rest of the application won’t even notice.

We’ve given our different reasons of change different places in our code. We have separated our concerns. This separation requires a lot of work to be spelled out. Let’s look at the code of our example at implementation visibility level 3:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    final ValueAddedTaxRepository vatProvider = givenVatRepository();
    for (Product each : inCart) {
      final ValueAddedTaxCategory vat = vatProvider.forType(each);
      final BruttoPrice bruttoPrice = vat.applyTo(each.nettoPrice());
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

There are now three lines of code responsible for calculating the brutto prices. It gets ridiculous! First we obtain the DDD Repository from somewhere. Somebody probably gave us the reference in the constructor or something. Just to remind you: The class is named ShowShoppingCart and now needs to know about a class that calls itself ValueAddedTaxRepository. Then, we obtain the corresponding ValueAddedTaxCategory for each Product or Taxable in our shopping cart. We apply this VAT to the NettoPrice of the Product/Taxable and pass the resulting BruttoPrice side by side with the NettoPrice in the addProductLine() method. Notice how we changed the signature of the method to differentiate between NettoPrice and BruttoPrice instead of using just to Euro parameters. Those domain types are now our level of abstraction. We don’t really care about Euro anymore. The prices might be expressed in mussle shells or bottle caps and we still could use our code without modification.

The ValueAddedTaxCategory we obtain from the DDD Repository isn’t a class with a concrete implementation. Instead, it is an interface:


/**
* AN-17: Calculates the brutto price (netto price with value added tax (VAT))
* for the given netto price.
*/
public interface ValueAddedTaxCategory {
  public BruttoPrice applyTo(NettoPrice nettoPrice);
}

Now we could nearly get rid of the comment above. It just repeats what the signature of the single method in this type says, too. We keep it for the reference to the requirement (AN-17).

Right now, the interface has only one implementation in the class PercentageValueAddedTaxCategory:


public class PercentageValueAddedTaxCategory implements ValueAddedTaxCategory {
  private final BigDecimal percentage;

  public PercentageValueAddedTaxCategory(final BigDecimal percentage) {
    this.percentage = percentage;
  }

  @Override
  public BruttoPrice applyTo(NettoPrice nettoPrice) {
    final Euro value = nettoPrice.multiplyWith(this.percentage).inEuro();
    return new BruttoPrice(value);
  }
}

You might notice that the concrete code of applyTo still has knowledge about the Euro. As long as we don’t ingrain the relationship between NettoPrice and BruttoPrice in these types, somebody has to do the conversion externally – and needs to know about implementation details of these types. That’s an observation that you should at least note down in your domain crunching documents. It isn’t necessarily bad code, but a spot that will require modification once the currency changes to cola bottle caps.

This is a good moment to reconsider what we’ve done to our ShowShoppingCart class. Let’s refactor the code a bit and move the responsibility for value added taxes where it belongs: in the Product type.


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            each.bruttoPrice());
    }
    return result;
  }

}

Now we have made a full circle: Our code looks like it began without the brutto prices, but with one additional line that delivers the brutto prices to the product line in the ShoppingCartRenderModel. The whole infrastructure that we’ve built is hidden behind the Product/Taxable type interface. We still use all of the domain types from above, we’ve just changed the location where we use them. The whole concept complex of different price types, value added taxes and tax categories is a top level construct in our application now. It shows up in the domain model and in the vocabulatory of our project. It isn’t a quick fix, it’s the introduction of a whole set of new ideas and our code now reflects that.

The code at implementation visibility level 3 might seem bloated and over-engineered to some. There is probably truth in this judgement. We’ve introduced far more code seams in the form of abstractions and indirections than we can utilize in the moment. We’ve prepared for an uncertain future. That might turn out to be unnecessary and would then be waste.

So let’s look at our journey as an example of what could be done. There is no need to walk all the way all the time. But you should be able to walk it in case it proves necessary.

Visibility level 4 and above: To infinity and beyond!

Remember that there are implementation visibility levels above 3! If you choose such a level, there will be even more code, more classes and types, more indirection and more abstraction. Suddenly, your new code will show up on system architecture diagrams and be deployed independently. Maybe you’ll need a dedicated server for it or scale it all the way up to its own server farm. Our example doesn’t match those criterias, so I stop here and just say that visibility level 3 isn’t the end of the journey. But you probably got the idea and can continue on your own now.

Recap: Rising through the visibility levels

We’ve come a long way since level 0 in terms of implementation visibility. The code still does the same thing, it just accumulates structure (some may call it cruft) and fletches out the relationships between concepts. In doing so, different axis of change emerge in different locations instead of entangled in one place. Our development effort rises, but we hope for a return on investment in the future.

I’ve found it easier to elevate the implementation visibility level of some code later than to decrease it. You might experience it the other way around. In the end, it doesn’t matter which way we choose – we have to match the importance of the requirement in the code. And as the requirements and their importance change, our code has to adjust to it in order to stay relevant. It isn’t the visibility level you choose now that will decide if your code is visible enough, it is the necessary visibility level you cannot reach for one reason or the other that will doom your code. Because it “feels bloated” and gets replaced, because it wasn’t found in time and is duplicated somewhere else, because it fused together with unrelated code and cannot be separated. Because of a plethora of reasons. By choosing and changing the implementation visibility level of your code deliberately, you at least take the responsibility to minimize the effects of those reasons. And that will empower you even if not all your decisions turn out profitable.

Conclusion

With the end of this third part, our series about the concept of implementation visibility comes to an end. I hope you’ve enjoyed the journey and gained some insights. If you happen to identify an example where this concept could help you, I’d love to hear from you! And if you know about a book or some other source where this concept is explained, too – please comment with a link below.

Implementation visibility – Part II

In the second part of our series about implementation visibility, we examine the middle ground of our visibility scale and discover that most requirements end up in this range. The maintainability is good, but we still have to check for architectural sins.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

The article then introduced a simple example and examined how the level 0, 0+ and 0++ would appear within the example code. You may want to read the first article before we carry on with level 1 and 2 in this article.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable&lt;Product&gt; inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 1: Extracted code lives longer

After all the (rather depressing) level 0 implementations of our brutto price calculation, the separated method is the first visibility level to result in code that can be discussed and tested separately:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPriceFor(each));
    }
    return result;
  }

  /**
  * AN-17: Calculates the brutto price for the given product.
  */
  private Euro bruttoPriceFor(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor);
  }
}

The new code is in lines 8 and 13 onwards. The new method was introduced to separate the calculation code from the rendering code. It still lives in the wrong class, but can be tested on its own if you make it public or package accessible. The comment now has a natural scope. And, most important: This implementation is the first where the notion of “brutto price” appears in the JavaDoc and the IDE.

Methods are the smallest parts of our object-oriented code. If you would have one method per requirement, you would just need one extra method of glue code to tie everything together. If one requirement needs to change or becomes obsolete, you know where to cut.

Methods are the primary focus of unit tests. You prepare the parameters for the method you want to test, call it and check the result. This is the AAA or triple-A normal form of unit testing: Arrange, Act, Assert. If several methods or even several objects need to be tested in conjunction, the testing effort rises.

We can conclude that with its own method, the VAT calculation now has its own home. Future readers can grasp the scope of our implementation easily and hopefully make changes under direct test coverage. This is the first visibility level that starts to feel like we meant it.

Visibility level 2: Make it a top-level affair

There is one part in object-oriented code that is even more basic than a method: the class. In Java, each class strives to have its own text file. Before you can write a method in Java, you need to define a class to contain it. Classes are the primary granularity level we navigate our code. Every IDE will show classes as the default elements in our “project explorers”. So what if we introduce a new class for our VAT calculation and move all our code there?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            CalculateBruttoPrice.forProduct(each));
    }
    return result;
  }
}
/**
 * AN-17: Calculates the brutto price with value added tax (VAT) for the given product.
 */
public class CalculateBruttoPrice {
  public static Euro forProduct(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor); 
  }
}

The new code is in line 8 and the full new class file. This implementation might not look a lot different from level 1 (separate method), but it really is on another level. The brutto price calculation now isn’t tied to rendering shopping carts anymore. It is not tied to anything other than a given product. It is a top-level concept of our application now. Anybody with a product can call the method and receive the brutto price, from anywhere in our application (hopefully respecting our architecture boundaries).

Our unit test class now reads as if we had written it only for the new requirement: CalculateBruttoPriceTest. We still need to invent test products in our test, but the whole notion of render models and shopping carts is gone. In essence, we freed the concept of price calculation from its “evolutionary” ties.

Implementing the new requirement in a separate class, if feasible, adheres to the Single Responsibility Principle (SRP), that requires each class of a system to only have one reason to change. In our case, the CalculateBruttoPrice class only changes if the brutto prices needs adjustment. For all previous visibility levels, that wasn’t true. The ShowShoppingCart class would need modifications if the brutto prices or the shopping cart rendering were to be changed. This improvement is reason enough to elevate our implementation visibility past level 1.

In short, a good heuristics for new requirements (as opposed to change requests for existing requirements) is to start with a new class. If you are unsure, start lower, but keep in mind that classes are the main navigation layer of object-oriented code.

Visibility level 2+: Inviting the requirement to be part of the project’s language

Introducing a new class for our requirement felt good, but something still feels off. When we review the interface of the CalculateBruttoPrice, two things stick out immediately: The class is named as a service (CalculateXYZ as in “do XYZ for me”) and can only calculate brutto prices for products. Our customer was serious with his requirement, so it’s safe to assume that brutto prices will stay in the application and play a key role. We should reflect this seriousness by lifting the implementation visibility level once more and make the BruttoPrice a top level concept of our project’s domain:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            BruttoPrice.of(each).inEuro());
    }
    return result;
  }
}

The ShowShoppingCart code doesn’t look very different from the level 2 code beforehands. The new code is in line 8, too. The new class isn’t named like a service anymore, but like a concept or domain type. The named constructor of() returns a BruttoPrice instance and not just a Euro object:

/**
 * AN-17: Represents the brutto price with value added tax (VAT) for the given Taxable.
 */
public final class BruttoPrice {
  public static BruttoPrice of(Taxable item) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return new BruttoPrice(item.nettoPrice().multiplyWith(taxFactor));
  }
  
  private final Euro asValue;

  private BruttoPrice(Euro value) {
    this.asValue = value;
  }

  public Euro inEuro() {
    return this.asValue;
  }
}

Now, we can accumulate additional behaviour in the new BruttoPrice type if the need arises. With the service class of level 2, we probably wouldn’t have risen above the Euro abstraction and mixed up netto and brutto prices somewhere in the future.  If we model our NettoPrice and BruttoPrice as domain types, the compiler will help us keeping them separate – even if both contain Euros as their value.

With this visibility elevation, we discovered another abstraction: We can create brutto prices for virtually anything that can be taxed. It doesn’t have to be a product, it just needs a netto price and a tax factor. The new (abstract) domain type is named Taxable. Of course, Product is an implementation of Taxable.

This makes us even more independent from any webshop, shopping cart or product. We can now write unit tests for our BruttoPrice without being coupled to the Product class at all. We have successfully decoupled the cart/product part of our application from the prices part. Recognizing and implementing the independence of concepts is an important step towards even higher visibility levels. It is also the groundwork of a low coupled, high cohesive code base where most things fall into their place naturally.

The step from level 2 (separate class) to level 2+ (new domain type) wasn’t just syntactic sugar, it was driven by the insight that separation of concerns is the fundamental principle to achieve maintainability, as long as the abstractions aren’t overwhelming. A good indicator that you’ve taken it too far is when your domain expert (in our example our client) raises her eyebrows in surprise when you talk about your abstract domain types because the names sound outlandish and far-fetched.

But you can take your implementation visibility even further and should really consider doing so given the circumstances. We will learn about visibility level 3 (separate aggregate) in the next blog post of this series. Stay tuned!

Implementation visibility – Part I

The concept of implementation visibility is very powerful. If you choose the wrong visibility for your requirements, your code will end up unmaintainable.

Somewhere in my take on programming, there lingers the concept of “implementation visibility”, that I’m not quite sure to be able to express clearly, but I’ll try.

Let’s say you are writing an academic text like a bachelor thesis and your professor makes it clear that she regards the list of literature a very important part of your work. What are you going to do? Concentrate on your cool topic and treat the literature as a secondary task? Or will you shift your focus and emphasize your extensive literature research, highlighting promising cross-references in your text? You’ll probably adjust your resources to make your list of literature more prominent, more visible. You respond to the priorities of your stakeholders.

Now imagine that your customer wants you to program a web application, but has one big requirement: All actions of the users need to be reassured with a confirmation question (as in “do you really want to delete this?”). He makes it clear that this is a mandatory feature that needs to be implemented with utmost care and precision. What would you do? We responded by adjusting our system’s architecture to incorporate the requirement into the API. You can read about our approach in this blog post from 2015. The gist of it is that every possible client of the system will be immediately aware of the requirement and has a much easier time conforming to it. It is harder to ignore or forget the requirement than to adhere to it because the architecture pushes you into the right direction.

The implementation of the customer’s requirement in the example above is very visible. You’ll take one look at the API and know about it. It isn’t hidden into well-meaning but out-dated developer documentation or implicitly stated because every existing action has a confirmation step and you should be sentient enough to know that this means your new one needs one, too. The implementation visibility of the customer’s requirement is maximized with our approach.

Stages of visibility

I have identified some typical stages (or levels) of implementation visibility that I want to present in this blog post series. That doesn’t mean that there won’t or can’t be others. I’m not even sure if the level system is as one-dimensional as I’m claiming here. I invite you to think about the concept, make your own observations and evolve from there. This is a starting point, not an absolute truth.

The following stages typically appear in my projects:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

In my day-to-day work, the levels 1 to 3 are the most relevant, but that’s probably not universally applicable. Our example above with the requirement-centered API isn’t even located on this list. I suggest it’s at level 6 and called separate concept or something similar.

An example to explain the visibility levels

Let’s assume a customer wants us to program a generic webshop. We are not very versed in commerce or e-commerce things and just start implementing requirements one after one.

After the first few iterations with demonstrated and usable artifacts, our customer calls us and explains that all prices in the webshop are netto prices and that there need to be some kind of brutto price calculation. You, being accustomed to prices that don’t change once you put products into your shopping cart, ask a few questions and can finally grasp the concept of value added taxes. Now you want to implement it into the webshop.

The first approach to the whole complex is to show the brutto prices right besides the netto prices when the user views his shopping cart. You can then validate the results with your customer and discuss problems or misconceptions that are now visible and therefore tangible.

The original code of your shopping cart renderer might look like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
               each.description(),
               each.nettoPrice());
      }
      return result;
  }
}

A quick explanation of the code: The class ShowShoppingCart takes some products and converts them into a ShoppingCartRenderModel that contains the shopping cart data in a presentable form so the GUI just needs to take the render model and paste it into some kind of template. For each product, there is one line with a description and the (already renamed) netto price on the page.

Visibility level 0: It’s just code anyway

Let’s start with the lowest and most straight-forward implementation visibility level: The inline implementation.

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5 and 9. As you can see, the programmer chose to implement exactly what he understood from the discussion about netto and brutto prices with the customer. A brutto price is a netto price with value added tax. The VAT rate is 19 percent at the time of writing, so a multiplication with 1.19 is a valid implementation.

Our problem with this approach isn’t the usage of floating point numbers in the calculations or that calculations even exist in a method that should do nothing more than render some products, but that the visibility of the requirement is minimal. If you, I or somebody else doesn’t know exactly where this code hides, we will have a hard time finding it once the VAT is changed or anything else should be done with brutto prices or VATs.

Technically, the customer’s requirement is implemented and the brutto prices will show up. But because the concept of taxes (or VAT) is important for the customer, we likely made the code too invisible to be maintainable.

Visibility level 0+: Hey, I even wrote a comment

To make some part of the code stick out of the mess, we have the tool of inline code comments. Let’s apply them to our example and raise our visibility level from 0 to 0+:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      // AN-17: calculating the brutto price from the netto price
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5, 6 and 10. You can see that the programmer chose the same approach as before, but realized that the code would be buried if not marked. Given that the requirement identifier is “AN-17”, the code can be found by a text search of this number. And if you happen to stumble upon this part of the application, you can deduct meaning about what you see from the comment.

Except that you cannot really be sure what the AN-17 code really is. Is the result.addProductLine() part of AN-17 or not? Would you expect the calculation of taxes and prices in a method called render() in a class named ShowShoppingCart? Is this implementation really correct? Aren’t there different tax rates for different products? Did the original author think about that? Is the customer content with this functionality?

Note that you cannot really test the brutto price calculation. You have to invent some products, render them and then scrape the brutto prices from the render model. That’s tedious at best and a clear sign that the implementation visibility is still too low. On to the next level

Visibility level 0++: This sucks, but I’ve got to go now

This level tries to make you a partner in crime by explicitly stating what’s obviously wrong with the code at hand. Now it’s your responsibility to fix it. You wouldn’t leave a broken window be, would you?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
  for (Product each : inCart) {
    // AN-17: calculating the brutto price from the netto price
    // TODO: take different tax factors into account
    final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
    result.addProductLine(
          each.description(),
          each.nettoPrice(),
          bruttoPrice);
  }
  return result;
  }
}

The new code is in lines 5, 6, 7 and 11. The new comment line 6 is typical for this visibility level: The original programmer knew that his implementation isn’t adequate but couldn’t be bothered with improving it. Perhaps he had external circumstances force him to do it. Whatever it was, this code is the equivalent to a soiled public toilet. The difference is, this time we can determine who made the mess.

The apologetic “I know I made a mess” comment often begins with TODO or FIXME. This isn’t directed towards the original author, it’s pointed at you, the person that happens to read the comment. Now, what are you going to do? Pretend you didn’t read the comment? Leave the toilet soiled? Clean up the mess of your predecessor? You probably have work to do, too. And doesn’t it work the way it is? Never change a running system!

We will see how you can improve the implementation visibility of the requirement in the next blog post of this series. Stay tuned!

Recap of the Schneide Dev Brunch 2017-10-08

If you couldn’t attend the Schneide Dev Brunch at 08th of October 2017, here is a summary of the main topics.

brunch64-borderedLast sunday, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. This brunch was smaller, which enabled us to use the meeting table with some comfort. Sometimes, with many attendees and bad weather, this table can get a little bit crowded. As usual, the main theme was that if you bring a software-related topic along with your food, everyone has something to share. Because we were a small group, we discussed without an agenda. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Area of training

We shared some stories about top-notch video game players and how they keep up with the demand to stay competitive. Similar stories can be told about every topic imagineable: What did the “king of the hill” do to rise to such levels? The answer is always: training. Excessive, brutal training. They are first in the gym and lock the door late at night because they are the last, too. The use every waking second for practice and repetition. They are obsessed with being “the best”. If you want to follow such a story in movie format, you might enjoy “Whiplash”, a movie about an aspiring expert drummer that also highlights the delicate trainer/trainee dynamics. If you are more interested in the strategies of obtaining mastery, the book “Mastery” by Robert Greene will give you a lot of insights.

With this background, we asked around what our area of training (not expertise, not mastery – just training) is. The answers varied wildly, from the obvious “programming” to “whisky” (as in whisky tasting and collecting whisky). It’s an interesting question: what goal are you actively pursuing at the moment?

Hacking challenges

Evolving from the first topic, we talked about coding challenges and “capture the flag” hacking contests. If you aren’t the grandmaster in the area of the contest, you’ll get the most out of it by following the participating teams and trying to understand their approaches.  The local security capture the flag team of the KIT is especially open with their approach, their failures and successes. You might want to check out their website.

One challenge included trying to break a whitebox encryption, which is an interesting topic in itself. Maybe somebody can read up on this topic and give a little presentation in the future. Another challenge seemed to lead to an elaborate buffer overflow attack, when in reality, it could be solved with a “simple” use-after-free attack.

An useful starting point for aspiring security hackers might be the CTF (capture the flag) field guide. There are also some online challenges for basic training purposes, like the cryptopals or the bandit wargame. Thanks Tobias for the links!

If you are more interested in playful challenges and don’t want to show up on somebody’s radar, programming/hacking games like TIS-100 are perfect for you. Our game night with TIS-100 is still in vivid memory.

Software Architecture training

There are a lot of programming contests and hacking challenges out there. But what about dry-run training for software architects? On a related scale, there are hundreds of training simulators for the foot soldier (called ego-shooters), but little games for the aspiring officer/general. The website armchair general lists a few and even has some contests, but they lack the depth of real experience. Similarly, the training for software architects will probably be clean-room exercises, when in reality, the customer needs, the team mood, the latest fad in technology and even the weather will influence the architecture just as well as textbook knowledge.

We couldn’t discuss this topic to its full potential, so it will re-appear on the agenda of the next Dev Brunch. And its open for discussion in the comments: What are good books and trainings for software architects?

Thomas pointed us to the Architectural Katas by Ted Neward. Perhaps we should schedule a Schneide Event to try them out?

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei in December. We even have some topics still on the agenda (like a report about first-hand experiences with the programming language Rust). And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

The project jugglers

This blog entry describes some aspects that help us to work on multiple projects at once.

By Usien (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia CommonsIn this blog post, I will shine some light on a feature of our company that is often met with disbelief: How five developers can work on twenty projects at once without being stressed. I try to use the metaphor of a juggler, though I know nothing about juggling other than it can be done. I cannot hold more than one object in the air at any given time and even that is an optimistic estimate. But I’ve seen jugglers keep six to eight objects flying with seemingly no effort. We do this with software development projects.

A layman theory of juggling

A good metaphor can be applied from start to finish. I’ve probably chosen a bad metaphor, but it gives the right initial impression: Every developer at our company leads several projects at once. He (or she) keeps the projects alive and in the “green zone”, the ratio of remaining budget, time and scope (read: work left to be done) that promises little to no trouble in the foreseeable future.

In order to juggle without visible effort, you probably need to practice a lot. You probably drop your objects a lot. You probably need to watch the objects fly in the beginning.

In our case, we needed a lot of practice to reach our level of confidence. We lead development projects for up to 17 years now. Each developer finishes between four and seven projects per year. That’s up to a hundred projects to gain experience from. But we couldn’t drop (read: fail) a lot of projects, because it directly hurt our bottom line. Just imagine you want to learn how to juggle, but all you got are expensive ming vases that you bought from your own money, without insurance. That’s how it feels to “experiment” with projects. So we play it safe and only accept projects we know we can handle. And we watch our projects fly, very very closely. In fact, we have a dedicated position, the “project manager”, with the one duty to periodically ask a bunch of questions to assure that the project is still in the green zone.

Draw the trajectory

Every object that you can juggle has its own characteristics of how it behaves once it is in the air. A good juggler can feel its trajectory and grab it in just the right moment before it would fall out of reach. The trick to keep a software development project in the green zone is the same: Get a hold on it before it ventures too far in an unfortunate direction, which is the natural tendency of all projects. The project lead has to periodically apply effort to keep the project afloat. But when is the right time to invest this effort? Spend it too soon and it has only minimal effect. Spend it too often and you’ll exhaust your power and your budget. But because every project has its own trajectory, too, and you can’t afford to let it slip, you should make the trajectory as visible as possible. You should draw it!

We’ve experimented with a lot of tools and visualizations. The one setup that works best for us is a low technology, high visibility approach. The project lead takes one half of a whiteboard and draws a classic burn-down chart or a variation (we often use a simple vertical progress bar). This chart is hand-drawn and rather crude, but big enough that everybody can see it. It is updated at least every time the project manager comes around to ask his or her questions. One of the questions actually is: “Is this chart up to date?”. The remaining budget of your project needs to be available at a glance, from across the hall. The project lead needs to “feel” this budget. And if it makes him or her nervous, it’s high time to determine the remaining scope and calendar time of the project once again.

In doing this positioning by triangulation on a regular schedule, the project lead draws the trajectory of the project for all three axes and can probably interpolate its future course. He can then apply effort to nudge (or yank, if things got worse fast) the project back on track.

Without the visibly drawn trajectory, your project lead is like a juggler in the dark, tossing unknown objects in the air and hoping that they’ll fall in place somehow.

Know your limit

As the complete noob to juggling that I am, I imagine that jugglers have a secret dress code like martial artists (watch their belts!), where other jugglers can read how many objects they can hold up at once. Something like buttons on the vest or the length of a scarf. So the beginning 3-objects juggler bows in awe to the master 12-juggler, who himself is star-struck by the mighty 18-juggler that happens to attend the same meet-up.

In our company, this “dress code” would be based on the number of projects you are leading. And just like with the jugglers, it is important that you know your current limit. There is no use in over-extending yourself, if you accidentally let one project slip, the impact is big enough that you’ll fail your other projects, too. Just like the juggler loses his or her rhythm, you’ll lose your “flow”.

The most important part of juggling many projects is that you always juggle one less than you are capable of. You need reaction time if one of them topples over. A good juggler can “rescue” the situation with subtle speedup or extra movements because the delay between necessary actions allows for it. A good project lead has emergency reserves to spend without compromising other projects.

There is nothing wrong to start with two projects and add more later on when you are more confident. But don’t start with only one. You can only form habits of resource sharing if you share from the beginning. Even I can pose as a competent 1-juggler, but the lowest bar to juggling has to be two objects.

Box your time

Again, I know nothing about juggling. But from a mathematical viewpoint, juggling is “just” an exercise in timeboxing. If you have four objects in the air, in an arc that requires one second to go around, you’ll be able to spend a quarter second (250 ms) of attention to each object on each rotation. The master 12-juggler from above can only afford 1/12th or 80 milliseconds for each object. If he takes longer for one object, the next one will suffer. If he has no time reserves, a jam will build up and ultimately break the routine.

So, as a project lead, you need to apply timeboxes on all of your projects. They don’t need to be of equal size, but small enough that you can multiplex between your projects fast and often enough. A time box is a fixed-size amount of time that you allot to a specific task. The juggler uses a time box to put the next falling object flying back up. In our lunch break, we allot 60 minutes to food, beverages and some amount of walking. And if the process of eating takes longer than usual (I’m a chronic slow-eater, I can’t help it)? Then I have to go back partially hungry because I’ll end my lunch break on time. That’s the most important characteristic of a time box: You either succeed “in time” or interrupt or even cancel the task. The juggler will drop the one iffy object instead of risking a complete breakdown of the arc. You need to let your problematic task go (for the moment) instead of spreading the problem onto your other projects, too.

We’ve found that the amount of “one workday” is the most natural and easiest to manage time box. So we try our best to partition our week in the granularity of days and not our days in the granularity of hours. One aspect that helps tremendously is to have different physical locations for different projects. So you can be physically present “in the project” or “too far away at the moment” from the project. You plan your work week in locations as much as you plan it in project time boxes. The correlation of workdays, locations and projects is so strong that it doesn’t even seem to be timeboxing or project multiplexing. You just happen to be in the right place to work on project X for today. This is how you can juggle up to five projects without having to compromise all that much (provided you have a five-day work week).

If you can’t physically relocate your work, at least try to have a fixed schedule for your projects, like the “project A monday” or the “project X friday”. This might also mean to postpone emerging issues with project X until next friday. You need to build up skills to negotiate these delays with your customers. If your customers can dictate your schedule, you’ll get torn to shreds in no time. It’s friday or no day for issues on project X – at least as a good start for heavy bargaining. But that’s a whole topic for another blog post. Please leave a comment if you are interested to hear more about it.

Stuff your box

The “one workday” time box has a strong implication: Every little thing you do for a project takes one day. That doesn’t mean you should work for five minutes on project A, completing the task, and then stare into the air and twiddle your thumbs. It means that you should accumulate enough tasks for project A that you can spend the better half of the day on the known tasks and the remaining time on the unknown problems that arise on the way. In the evening, you should be able to finish your work for project A with a feeling of closure. You can put project A aside until next week (or whenever your next cycle is). You can concentrate on project B tomorrow and project C the next day. Both projects didn’t bother you today (well, perhaps a bit, but you only acknowledged some e-mails and deferred any real thoughts on it until you enter their timebox).

Perpetual closure

The feeling of closure at the end of a successful work day is the most important thing that keeps you composed. You’ve done your thinking for project A this week and will think of project B tomorrow. But now, you can rest.

This must be the feeling that the juggler experiences with each object that goes up again. It is out of sight and only needs attention after it has nearly completed its arc again. And now for the next object, one at a time…

 

Learning about Class Literals after twenty years of Java

The story about a customer request that led to my discovery of a cool feature of Java: Class Literals (you’ve probably already heard about them).

I’ve programmed in Java nearly every day for twenty years now. At the beginning of my computer science studies, I was introduced to Java 1.0.x and have since accompanied every version of Java. Our professor made us buy the Java Language Specification on paper (it was quite a large book even back then) and I occassionally read it like you would read an encyclopedia – wading through a lot of already known facts just to discover something surprising and interesting, no matter how small.

With the end of my studies came the end of random research in old books – new books had to be read and understood. It was no longer efficient enough to randomly spend time with something, everything needed to have a clear goal, an outcome that improved my current position. This made me very efficient and quite effective, but I only uncover surprising facts and finds now if work forces me to go there.

An odd customer request

Recently, my work required me to re-visit an old acquaintance in the Java API that I’ve never grew fond of: The Runtime.exec() method. One of my customer had an recurring hardware problem that could only be solved by rebooting the machine. My software could already detect the symptoms of the problem and notify the operator, but the next logical step was to enable the software to perform the reboot on its own. The customer was very aware of the risks for such a functionality – I consider it a “sabotage feature”, but asked for it anyway. Because the software is written in Java, the reboot should be written in Java, too. And because the target machines are exclusively running on Windows, it was a viable option to implement the feature for that specific platform. Which brings me to Runtime.exec().

A simple solution for the reboot functionality in Java on Windows looks like this:


Runtime.exec("shutdown /r");

With this solution, the user is informed of the imminent reboot and has some time to make a decision. In my case, the reboot needed to be performed as fast as possible to minimize the loss of sensor data. So the reboot command needs to be expanded by a parameter:


Runtime.exec("shutdown /r /t 0");

And this is when the command stops working and politely tells you that you messed up the command line by printing the usage information. Which, of course, you can only see if you drain the output stream of the Process instance that performs the command in the background:


final Process process = Runtime.exec("shutdown /r /t 0");
try (final Scanner output = new Scanner(process.getInputStream())) {
    while (output.hasNextLine()) {
        System.out.println(output.nextLine());
    }
}

The output draining is good practice anyway, because the Process will just stop once the buffer is filled up. Which you will never see in development, but definitely in production – in the middle of the night on a weekend when you are on vacaction.

Modern thinking

In Java 5 and improved in Java 7, the Runtime.exec() method got less attractive by the introduction of the ProcessBuilder, a class that improves the experience of creating a correct command line and a lot of other things. So let’s switch to the ProcessBuilder:


final ProcessBuilder builder = new ProcessBuilder(
        "shutdown",
        "/r",
        "/t 0");
final Process process = builder.start();

Didn’t change a thing. The shutdown command still informs us that we don’t have our command line under control. And that’s true: The whole API is notorious of not telling me what is really going on in the background. The ProcessBuilder could be nice and offer a method that returns a String as it is issued to the operating system, but all we got is the ProcessBuilder.command() method that gives us the same command line parts we gave it. The mystery begins with our call of ProcessBuilder.start(), because it delegates to a class called ProcessImpl, and more specific to the static method ProcessImpl.start().

In this method, Java calls the private constructor of ProcessImpl, that performs a lot of black magic on our command line parts and ultimately disappears in a native method called create() with the actual command line (called cmdstr) as the first parameter. That’s the information I was looking for! In newer Java versions (starting with Java 7), the cmdstr is built in a private static method of ProcessImpl: ProcessImpl.createCommandLine(). If I could write a test program that calls this method directly, I would be able to see the actual command line by myself.

Disclaimer: I’m not an advocate of light-hearted use of the reflection API of Java. But for one-off programs, it’s a very powerful tool that gets the job done.

So let’s write the code to retrieve our actual command line directly from the ProcessImpl.createCommandLine() method:


public static void main(final String[] args) throws Exception {
    final String[] cmd = {
            "shutdown.exe",
            "/r",
            "/t 0",
    };
    final String executablePath = new File(cmd[0]).getPath();

    final Class<?> impl = ClassLoader.getSystemClassLoader().loadClass("java.lang.ProcessImpl");
    final Method myMethod = impl.getDeclaredMethod(
            "createCommandLine",
            new Class[] {
                    ????, // <-- Damn, I don't have any clue what should go here.
                    String.class,
                    String[].class
            });
    myMethod.setAccessible(true);

    final Object result = myMethod.invoke(
            null,
            2,
            executablePath,
            cmd);
    System.out.println(result);
}

The discovery

You probably noticed the “????” entry in the code above. That’s the discovery part I want to tell you about. This is when I met Class Literals in the Java Language Specification in chapter 15.8.2 (go and look it up!). The signature of the createCommandLine method is:


private static String createCommandLine(
        int verificationType,
        final String executablePath,
        final String cmd[])

Note: I didn’t remove the final keyword of verificationType, it isn’t there in the original code for unknown reasons.
When I wrote the reflection code above, it occurred to me that I had never attempted to lookup a method that contains a primitive parameter – the int in this case. I didn’t think much about it and went with Integer.class, but that didn’t work. And then, my discovery started:


final Method myMethod = impl.getDeclaredMethod(
        "createCommandLine",
        new Class[] {
                int.class, // <-- Look what I can do!
                String.class,
                String[].class
        });

As stated in the Java Language Specification, every primitive type of Java conceptionally “has” a public static field named “class” that contains the Class object for this primitive. We can even type void.class and gain access to the Class object of void. This is clearly written in the language specification and required knowledge for every earnest usage of Java’s reflection capabilities, but I somehow evaded it for twenty years.

I love when moments like this happen. I always feel dumb and enlightened at the same time and assume that everybody around me knew this fact for years, it is just me that didn’t get the memo.

The solution

Oh, and before I forget it, the solution to the reboot command not working is the odd way in which Java adds quote characters to the command line. The output above is:


shutdown /r "/t 0"

The extra quotes around /t 0 make the shutdown command reject all parameters and print the usage text instead. A working, if not necessarily intuitive solution is to separate the /t parameter and its value in order to never have spaces in the parameters – this is what provokes Java to try to help you by quoting the whole parameter (and is considered a feature rather than a bug):


final String[] cmd = {
        "shutdown",
        "/r",
        "/t",
        "0",
};

This results in the command line I wanted from the start:


shutdown /r /t 0

And reboots the computer instantaneous. Try it!

Your story?

What’s your “damn, I must’ve missed the memo” moment in the programming language you know best?