Oversimplified C++ Project FAQ 2018

If you are starting a new C++ project, you’re faced with a few difficult decisions. C++ is not a ‘batteries-included’ language, so you need to pick a few technologies before you can start.
Yet worse, the answer to most of the pressing questions is often ‘it depends’ and changing one of the choices mid-project can be very expensive.
Therefore, I have compiled this list to give totally biased and oversimplified to the most important questions. If you want more nuanced answers, feel free to do your own research.
This is meant to be a somewhat amusing starting point.

FAQ

1. Which OS should I pick?

Linux

Rationale

Usually, not a choice you can make yourself – but if you do: dependency management is easier with a package manager, and it seems to be the most dominant OS in the C++ community. Hence you will get the best support and easiest access to technologies.

2. Which build system should I use?

CMake

Rationale

This is what everyone else is using, and those that are not are a real pain. For better or worse, the market is locked in. With target based properties in modern CMake, it’s not even that bad.

3. Which IDE should I choose?

Visual Studio 2017 on Windows, CLion everywhere else.

Rationale

CLion is getting more robust and feature rich with every release. Native CMake support and really cool refactoring capabilities finally make this a valid contender to Visual Studio’s crown. However, the VS debugger is still the best in the game, so VS still comes out on top on Windows – tho not by a huge margin.

4. Which Language version should you use?

C++14

Rationale

C++17 is not quite there yet with library, tool and platform support. Also, people do not really know how to use it well yet. C++14 builds on the now well-established C++11, which a few rather important “fixes” – and support is ubiquitous.

5. Which GUI toolkit should you use?

Qt

Rationale

No other toolkit comes close in maturity. Qt’s signal/slot system almost seamlessly integrates with C++11 lambdas, making the precompile step needed for SLOTs a non-issue. Barring the license costs for closed-source projects, there is really no reason not to use it.

6. Should you use Boost?

No

Rationale

Boost is a huge and clunky dependency that will explode your build times as soon as you even touch it. And it’s ‘viral’ enough that you can distinguish a Boost project from a non-Boost project. Boost.Optional, Boost.Variant and Boost.Filesystem prepare you for a smooth transition to C++17, but there are other more lightweight alternatives available.

Closing thoughts

There you have my totally biased opinion but hopefully entertaining. YMWV, but I think this is a good starting point if you don’t want to exeriment too much.

OPC-UA Performance and Bulk Reads

In a previous post on OPC on this blog I introduced some basics of OPC. Now we’ll take look at some performance characteristics of OPC-UA. Performance depends both on the used OPC server and the client, of course. But there are general tips to improve performance.

  • to get maximum performance use OPC without security

OPC message signing and encryption adds overhead. Turn off security for maximum performance if your use case allows to use OPC without security.

  • bulk reads increase performance

Bulk reads

A bulk read call reads multiple variables at once, which reduces communication overhead between client and server.

Here’s a code example using Eclipse Milo, an open-source OPC-UA stack implementation for the Java VM.

final String endpointUrl = "opc.tcp://localhost:53530/OPCUA/SimulationServer";
final EndpointDescription[] endpoints = UaTcpStackClient.getEndpoints(endpointUrl).get();
final OpcUaClientConfigBuilder config = new OpcUaClientConfigBuilder();
config.setEndpoint(endpoints[0]);

final OpcUaClient client = new OpcUaClient(config.build());
client.connect().get();

final List<NodeId> nodeIds = IntStream.rangeClosed(1, 50).mapToObj(i -> new NodeId(5, "Counter" + i)).collect(Collectors.toList());
final List<ReadValueId> readValueIds = nodeIds.stream().map(nodeId -> new ReadValueId(nodeId, AttributeId.Value.uid(), null, null)).collect(Collectors.toList());

// Bulk read call
final ReadResponse response = client.read(0, TimestampsToReturn.Both, readValueIds).get();
final DataValue[] results = response.getResults();
if (null != results) {
	final List<Integer> values = Arrays.stream(results).map(result -> (Integer) result.getValue().getValue()).collect(Collectors.toList());
	System.out.println(values.stream().map(String::valueOf).collect(Collectors.joining(",")));
}

client.disconnect().get();

The code performs a bulk read call on 50 integer variables (“Counter1” to “Counter50”). For performance tests you can put the bulk read call in a loop and measure the times. You should, however, connect to the server over the target network, not on localhost.

With a free (however not open-source) OPC UA simulation server by Prosys and Eclipse Milo for the client I measured times around 3.3 ms per bulk read of these 50 integer variables. I got similar results with the UA.NET stack by the OPC Foundation. Of course, you should do your own measurements with your target setup.

Keep also in mind that the preferred way to use OPC UA is not to constantly poll the values of all the variables. OPC UA allows you to monitor variables for changes and to get notified in case of a change, which is a more event-driven approach.

Gradle projects as Debian packages

Gradle is a great tool for setting up and building your Java projects. If you want to deliver them for Ubuntu or other debian-based distributions you should consider building .deb packages. Because of the quite steep learning curve of debian packaging I want to show you a step-by-step guide to get you up to speed.

Prerequisites

You have a project that can be built by gradle using gradle wrapper. In addition you have a debian-based system where you can install and use the packaging utilities used to create the package metadata and the final packages.

To prepare the debian system you have to install some packages:

sudo apt install dh-make debhelper javahelper

Generating packaging infrastructure

First we have to generate all the files necessary to build full fledged debian packages. Fortunately, there is a tool for that called dh_make. To correctly prefill the maintainer name and e-mail address we have to set 2 environment variables. Of course, you could change them later…

export DEBFULLNAME="John Doe"
export DEBEMAIL="john.doe@company.net"
cd $project_root
dh_make --native -p $project_name-$version

Choose “indep binary” (“i”) as type of package because Java is architecture indendepent. This will generate the debian directory containing all the files for creating .deb packages. You can safely ignore all of the files ending with .ex as they are examples features like manpage-generation, additional scripts pre- and post-installation and many other aspects.

We will concentrate on only two files that will allow us to build a nice basic package of our software:

  1. control
  2. rules

Adding metadata for our Java project

In the control file fill all the properties if relevant for your project. They will help your users understand what the package contains and whom to contact in case of problems. You should add the JRE to depends, e.g.:

Depends: openjdk-8-jre, ${misc:Depends}

If you have other dependencies that can be resolved by packages of the distribution add them there, too.

Define the rules for building our Java project

The most important file is the rules makefile which defines how our project is built and what the resulting package contents consist of. For this to work with gradle we use the javahelper dh_make extension and override some targets to tune the results. Key in all this is that the directory debian/$project_name/ contains a directory structure with all our files we want to install on the target machine. In our example we will put everything into the directory /opt/my_project.

#!/usr/bin/make -f
# -*- makefile -*-

# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1

%:
	dh $@ --with javahelper # use the javahelper extension

override_dh_auto_build:
	export GRADLE_USER_HOME="`pwd`/gradle"; \
	export GRADLE_OPTS="-Dorg.gradle.daemon=false -Xmx512m"; \
	./gradlew assemble; \
	./gradlew test

override_dh_auto_install:
	dh_auto_install
# here we can install additional files like an upstart configuration
	export UPSTART_TARGET_DIR=debian/my_project/etc/init/; \
	mkdir -p $${UPSTART_TARGET_DIR}; \
	install -m 644 debian/my_project.conf $${UPSTART_TARGET_DIR};

# additional install target of javahelper
override_jh_installlibs:
	LIB_DIR="debian/my_project/opt/my_project/lib"; \
	mkdir -p $${LIB_DIR}; \
	install lib/*.jar $${LIB_DIR}; \
	install build/libs/*.jar $${LIB_DIR};
	BIN_DIR="debian/my_project/opt/my_project/bin"; \
	mkdir -p $${BIN_DIR}; \
	install build/scripts/my_project_start_script.sh $${BIN_DIR}; \

Most of the above should be self-explanatory. Here some things that cost me some time and I found noteworthy:

  • Newer Gradle version use a lot memory and try to start a daemon which does not help you on your build slaves (if using a continous integration system)
  • The rules file is in GNU make syntax and executes each command separately. So you have to make sure everything is on “one line” if you want to access environment variables for example. This is achieved by \ as continuation character.
  • You have to escape the $ to use shell variables.

Summary

Debian packaging can be daunting at first but using and understanding the tools you can build new packages of your projects in a few minutes. I hope this guide helps you to find a starting point for your gradle-based projects.

For your last project ask yourself: What did the stakeholders learn

For your last project ask yourself: What did the stakeholders learn

At the start of a new project we like to begin with a naive mind, a beginner’s mind. In it we try to avoid our assumptions and start with a blank slate. Our clients do not. They are expert in their respective domain and know a lot. It’s naturally that during the project we learn lot about them and their domain, their work and their daily struggles. We see how they work around the limitations of their tools and cope with software written more than 30 years ago.
But besides us learning something about the domain, the stakeholders learn something about their domain, too. Because to develop the domain, the use cases and the daily work, we have to know details and reasons. Why is this step before that? Is it optional? Are these all the formats which are allowed? How long is the text usually? Why is there an exception to the rule? How often does it happen?
Usually we ask questions which cover the most traveled path, the happy trail. But in order to understand we need to get to the edges as well. The dark edges. Sometimes the number of objects we deal with is so big, nobody has all the answers. Our work, even before we write the software, enables collaboration. People and different departments have to work together. We work with all of them. Our software helps them to reach their common goals. But before that we need to know. And in order to tell us that the stakeholders need to dig deeper in their respective domain. Sometimes we need to look at the history in their domain, their work history, the decisions other stakeholders made in the past. It’s like archeology without the shovels, well, most of the time :).
Luckily the people we work with enjoy getting to know more about their work. They are astonished what depth the details have. How much different types of things, where gaps are. It is not always easy to light up areas that were kept in the dark so long. That were done just the way they were done. No we come and ask sometimes uneasy questions. We need to know. We need to know exactly. We need to know deeply.
This curiosity is not for its own sake. Our clients can confirm that the new software is so much better than the old. Not technical, but most importantly more adapted to their daily work.

That’s what’s important.

The four archetypes of cloud users – part 1 of 2

You probably know people that avoid cloud services as if they are poisonous and others that jump onto every cloud bandwagon. You can categorize them in four archetypes. Let’s start with the tinfoil hat and the clipboard.

In the occupational field of accounting, the strong trend towards cloud services is noticeable. Everything needs to be digital, and with digital, they mean online, and with online, they mean in the cloud. Every expense voucher needs to be scanned and uploaded, because in many cases, it can be booked automatically. In the new era of accounting, human intervention is only needed for special cases.

I see this as a good example of how digital online services can transform the world. Every step in the process would have technically been possible for the last twenty years, but only the cloud could unify the different participants enough so that a streamlined end-to-end process is marketable to the masses. And in this marketing ecstasy, the stakeholders that profit the most (the accountants) often forget that their benefits are just a part of the whole picture. In order to assess the perceived and actual benefits of all stakeholders, you at least need to apply an archetype to each participant.

The four archetypes

In my opinion, there are four different archetypes of cloud users. Let’s have a look at them and then assess the risks and potentials when selling a digital online service to them. I’ll list the archetypes in the order from biggest risk to biggest potential.

Archetype 1: The tinfoil hat

A person that could be identified as a “tinfoil hat” doesn’t need to be a conspiracy weirdo or paranoid maniac. In fact, the person probably has deep and broad knowledge about technology and examines new technologies in detail. The one distinctive feature of the tinfoil hats is that they take security, including IT security, very seriously. They don’t take security for granted, don’t trust asseverations and demand proof. You can’t convince a tinfoil hat by saying that the data transfer is “encrypted”, you need to specify the actual encryption algorithm. Using RC4 ciphers for the SSL protocol isn’t good enough for the tinfoil. You need at least proof that you understood the last sentence and took actions to mitigate the problem. Even then, the tinfoil will hesitate to give any data out of hands and often choose the cumbersome way in order to stay safe. “Better safe than sorry” is his everyday motto.

Tinfoil hats always search for scenarios that could compromise their data or infrastructure. They are paranoid by default and actively invest in security. “On premises” is the only way they deploy their own services, and “on premises” is how they prefer to keep their data.

Typical signs of a tinfoil hat archetype include:

  • self-hosted applications
  • physical servers
  • lack of (open) wireless network
  • physically separated networks
  • signed and encrypted e-mails

Trying to sell a cloud service to a tinfoil hat is like trying to sell a flight to an aviophobian (somebody with fear of flying). There is always another way to get from A to B, seemingly safer and more controllable. If you are selling cloud services, tinfoil hats are your worst nightmare. If you can convince a tinfoil hat, your product is probably made of fairy dust and employs lots of unicorns.

Archetype 2: The clipboard

Clipboard people are wary of new technologies, but assess them in the context of usability. They demand high security, but will compromise if the potential of the new technology far exceeds the risk. Other than the tinfoil hat, the clipboard sees his role as an enabler, but will not rest to increase the perceived or actual safety of the product. You can appease a clipboard by giving evidence of security audits from a third party. They will trust known authorities, because it means that they can always deflect blame in case of an accident to these authorities.

Clipboards run on checklists, safety protocols and recurring audits. They don’t try to avert every possibility of a security breach, but will examine each incident in detail and update their checklists. They don’t care about “on premises” or “off premises” as long as the service is reachable, safe enough and reliable. If a cloud service has an higher availability than the local counter-part, the clipboard will think about a migration.

Typical signs of a clipboard archetype include:

  • Virtual Private Networks (VPN)
  • Two-Factor Authentication
  • Token-Based Authentication
  • Strong Encryption

The clipboard will listen if you pitch your cloud service and can be enticed by the new or better capabilities. But in the very next sentence, he will ask about security and be insistent until you provide proof – first-hand or by credible third parties. You can convince a clipboard if your product is designed with safety in mind. As long as the safety is state-of-the-art, you’ll close the deal.

Outlook on the second part

In the second part of this blog entry, we will look at the remaining two archetypes, namely the “combination lock” and the “smartphone”. Stay tuned.

Did you identify with one of the archetypes? What are your most important aspects of cloud services? I would love to hear from you.

C++17: The two line visitor explained

If you have ever used an “idiomatic” C++ variant datatype like Boost.Variant or the new C++17 std::variant, you probably wished you could assemble a visitor to dispatch on the type by assembling a couple of lambda expressions like this:

auto my_visitor = visitor{
  [&](int value) { /* ... */ },
  [&](std::string const& value) { /* ... */ },
};

The code in question

While reading through the code for lager I stumbled upon a curious way to to make this happen. And it is just two lines of code! Wow, that is cool.

template<class... Ts> struct visitor: Ts... { using Ts::operator()...; };
template<class... Ts> visitor(Ts...) -> visitor<Ts...>;

A comment in the code indicated that the code was copied from cppreference.com where I quickly found the source on the page for std::visit, albeit with the different name “overloaded”. There were, however, no comments as to how this code worked.

Multiple inheritance to the rescue

Lambda expressions in C++ are just syntactic sugar for callables, pretty much like a struct with an operator(). As such, you can derive from them. which is what the first line does.
It uses variadic templates and multiple inheritance to assemble the types of the lambdas into one type. Without the content in the struct body, an instantiation with our example would be roughly equivalent to this:

struct int_visitor {
  void operator()(int value)
  {/* ... */}
};

struct string_visitor {
  void operator()(std::string const& value)
  {/* ... */}
};

struct visitor : int_visitor, string_visitor {
};

Using all of it

Now this cannot yet be called, as overload resolution (by design) does not work across different types. Hence the using in the structs body. It pulls the operator() implementations into the visitor type where overload resolution can work across all of them.
With it, our hypothetical instantiation becomes:

struct visitor : int_visitor, string_visitor {
  using int_visitor::operator();
  using string_visitor::operator();
};

Now an instance of that type can actually be called with both our types, which is what the interface for, e.g. std::visit demands.

Don’t go without a guide

The second line intruiged me. It looks a bit like a function declaration but that is not what it is. The fact that I had to ask in the (very helpful!) C++ slack made me realize that I did not keep up with the new features in C++17 as much as I would have liked.
This is, in fact, a class template argument deducation (CTAD) guide. It is a new feature in C++17 that allows you do deduce template arguments for a type based on constructor parameters. In a way, it supercedes the Object Generator idiom of old.
The syntax is really quite straight-forward. Given a list of constructor parameter types, resolve to a specific template instance based on those.

Constructing

The last piece of the puzzle is how the visitor gets initialized. The real advantage of using lambdas instead of writing the struct yourself is that you can capture variables from your context. Therefore, you cannot just default-initialize most lambdas – you need to transport its values, its bound context.
In our example, this uses another new C++17 feature: extended aggregate initialization. Aggregate initialization is how you initialized structs way back in C with curly-brackets. Previously, it was forbidden to do this with structs that have a base class. The C++17 extension now lifts this restriction, thus making it possible to initialize this visitor with curly brackets.

Edit 2018/04/16: The people on r/cpp rightfully pointed out that using the “other name” in the code snippet was confusing – so the visitor is now called “visitor”.

Book review: Clean Architecture by Robert C. Martin

A personal review of the book “Clean Architecture” by Robert C. Martin. Spoiler: you might want to read it.

In 2008, a book changed the way software developers around the globe talked (and hopefully) acted about their code. Robert C. Martin’s “Clean Code” was and still is a cornerstone of modern software development. The book itself is remarkably weak in its code examples, but has strong and effective messages on the level of practices and principles. Even today, ten years later, this is the one book that most of my students read and are passionate about. It’s a book that speaks reason to them, albeit with some contortion because of high volume. Robert C. Martin has the tendency to preach 200 percent in order to still get the half-convinced to an acceptable level.

So when a new book from him, called “Clean Architecture”, appeared on the horizon, I was thrilled. Would it be groundbreaking like “Clean Code” or a dud like “The Clean Coder” (sorry, my opinion – this is a personal review, not an academic evaluation)? I’ve read some very good books about software development (like “The Pragmatic Programmer”), fantastic books about programming (“Refactoring” and “Working Effectively With Legacy Code” come to mind) and even some mind-blowing pieces about design and emerging architecture (my first read of “Growing Object-Oriented Software, Guided by Tests” felt like a personal audience with Steve Freeman and Nat Pryce). But all these books dealt with tactics, with the immediacies of software development. Don’t get me wrong! This is the most important part and it helped me tremendously. But there are parts “above” the footwork that needs to be addressed in bigger systems, too.

And there, the literature got thin or stale. Books about software architecture talked about large-scale architectures (so-called “enterprise scale” systems that span from horizon to horizon, like in “Patterns of Enterprise Application Architecture”) or had the taste of dry plywood because it was clear that the findings were from another era and would translate badly into modern software development.

“Clean Architecture” begins with a quick and focused overview over the current programming paradigms and a conclusion that there are no different “eras”. We didn’t get better in designing systems, we just changed the aroma and color of our failures. Future generations will look at our code and architectures as scornful as we looked at the ruins of the systems of our ancestors. And make no mistake – the ruins are still in production today! We cannot place our hope on another new and liberating programming paradigm because there probably won’t be one. We have to make do with what we have.

This is the first six chapters of “Clean Archicture”. The chapters are short and on point and I loved every line of it. It probably isn’t the most comprehensive and balanced description of structured, object-oriented and functional programming, but it provides a narrative that is intuitive and convincing – your mileage may vary, I was hooked.

In the next five chapters, Robert C. Martin reiterates the known SOLID design principles. I rolled my eyes when I glanced the content because I’ve read it like a hundred times in maybe as many books. But I decided to read it once more and I’m glad I did. The principles are known, but the underlying revelation is woven into the text like a good thriller. I hesitate to give away too much, because I really think this book can be spoiled – just like a good thriller. I was sold. Robert C. Martin can explain the same old SOLID to me and I still learn something and have fun.

Then, the part about components. It feels like an intermezzo to an even better thriller, because suddenly there is math and formulas. Its interesting and noteworthy, but if you followed the metrics discussion in the last fifteen years, the excitement of this part will be dampened.

But wait, there is more! Starting with page 133 of 321 (yeah, the Appendix is interesting, but more in the “The Clean Coder” way of things), there is the central question: “What is Architecture?”. There it was again, the thrill that in every line, there could be insights that are worth weeks of thoughts. I read this part in the train from south to north germany and I stared out of the window often, following my own train of thoughts.

Again, no spoilers, but the way the answers are given is so refreshing and the answer itself is so simple that I’m surprised that it took me this long to not come to the same conclusion. Software architecture lost some of its mysticism, but gained a lot of applicability for me. I was spent (in a good way).

And then, on page 200, finally, “The Clean Archicture”. Well, I watched all the trailers on this topic, so my surprise wasn’t really there, but with all the knowledge and insights from the first 200 pages, I could have “invented” the Clean Architecture by myself then and there. It’s more or less the logical next step from the prerequisites. I applaud this masterwork of storytelling, because it doesn’t overwhelm the reader with the genius of the narrator, it drives him to connect the dots himself.

The rest of the book, like the title of part VI, are just “Details”. The central message  – The Dependency Rule, this little spoiler should be allowed – is simple, convincing and deduced from the beginning. I’ve seen the heart of software architecture and it is beautiful.

I even forgive the many typos and grammatical errors (far more than usual) and the bulky appendix for this ride. This book is definitely up there with “Clean Code”. It is accessible, has a clear message and profound effects. And it refrains from preaching most of the time. No need to turn it up to 200 percent when your message is so convincing in itself.

Conclusion: If you are interested in software development with a structure, go grab this book as soon as possible. We’ve waited long enough!

Advanced deb-packaging with CMake

CMake has become our C/C++ build tool of choice because it provides good cross-platform support and very reasonable IDE (Visual Studio, CLion, QtCreator) integration. Another very nice feature is the included packaging support using the CPack module. It allows to create native deployable artifacts for a plethora of systems including NSIS-Installer for Windows, RPM and Deb for Linux, DMG for Mac OS X and a couple more.

While all these binary generators share some CPACK-variables there are specific variables for each generator to use exclusive packaging system features or requirements.

Deb-packaging features

The debian package management system used not only by Debian but also by Ubuntu, Raspbian and many other Linux distributions. In addition to dependency handling and versioning packagers can use several other features, namely:

  • Specifying a section for the packaged software, e.g. Development, Games, Science etc.
  • Specifying package priorities like optional, required, important, standard
  • Specifying the relation to other packages like breaks, enhances, conflicts, replaces and so on
  • Using maintainer scripts to customize the installation and removal process like pre- and post-install, pre- and post-removal
  • Dealing with configuration files to protect end user customizations
  • Installing and linking files and much more without writing shell scripts using ${project-name}.{install | links | ...} files

All these make the software easier to package or easier to manage by your end users.

Using deb-features with CMake

Many of the mentioned features are directly available as appropriately named CMake-variables all starting with CPACK_DEBIAN_.  I would like to specifically mention the CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA variable where you can set the maintainer scripts and one of my favorite features: conffiles.

Deb protects files under /etc from accidental overwriting by default. If you want to protect files located somewhere else you specify them in a file called conffiles each on a separate line:

/opt/myproject/myproject.conf
/opt/myproject/myproject.properties

If the user made changes to these files she will be asked what to do when updating the package:

  • keep the own version
  • use the maintainer version
  • review the situation and merge manually.

For extra security files like myproject.conf.dpkg-dist and myproject.conf.dpkg-old are created so no changes are lost.

Unfortunately, I did not get the linking feature working without using maintainer scripts. Nevertheless I advise you to use CMake for your packaging work instead of packaging using the native debhelper way.

It is much more natural for a CMake-based project and you can reuse much of your metadata for other target platforms. It also shields you from a lot of the gory details of debian packaging without removing too much of the power of deb-packages.

Lessons learned developing hybrid web apps (using Apache Cordova)

In the past year we started exploring a new (at leat for us) terrain: hybrid web apps. We already developed mobile web apps and native apps but this year we took a first step into the combination of both worlds. Here are some lessons learned so far.

In the past year we started exploring a new (at leat for us) terrain: hybrid web apps. We already developed mobile web apps and native apps but this year we took a first step into the combination of both worlds. Here are some lessons learned so far.

Just develop a web app

after all the hybrid app is a (mobile) web app at its core, encapsulating the native interactions helped us testing in a browser and iterating much faster. Also clean architecture supports to defer decisions of the environment to the last possible moment.

Chrome remote debugging is a boon

The tools provided by Chrome for remote debugging on Android web views and browser are really great. You can even see and control the remote UI. The app has some redraw problems when the debugger is connected but overall it works great.

Versioning is really important

Developing web apps the user always has the latest version. But since our app can run offline and is installed as a normal Android app you have to have versions. These versions must be visible by the user, so he can tell you what version he runs.

Android app update fails silently

Sometimes updating our app only worked in parts. It seemed that the web view cached some files and didn’t update others. The problem: the updater told the user everything went smoothly. Need to investigate that further…

Cordova plugins helped to speed up

Talking to bluetooth devices? checked. Saving lots of data in a local sqlite? Plugins got you covered. Writing and reading local files? No problemo. There are some great plugins out there covering your needs without going native for yourself.

JavaScript isn’t as bad as you think

Working with JavaScript needs some discipline. But using a clean architecture approach and using our beloved event bus to flatten and exposing all handlers and callbacks makes it a breeze to work with UIs and logic.

SVG is great

Our apps uses a complex visualization which can be edited, changed, moved and zoomed by the user. SVG really helps here and works great with CSS and JavaScript.

Use log files

When your app runs on a mobile device without a connection (to the internet) you need to get information from the device to you. Just a console won’t cut it. You need log files to record the actions and errors the user provokes.

Accessibility is harder than you think

Modern design trends sometimes make it hard to get a good accessibility. Common problems are low contrast, using only icons on buttons, indiscernible touch targets, color as information bearer and touch targets that are too small.

These are just the first lessons we learned tackling hybrid development but we are sure there are more to come.

Implementation visibility – Part III

In the thirf part of our series about implementation visibility, we look at the high ground of our visibility scale and peek even higher. The series ends with a recap of the concept and its implications.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

We examined a simple code example in both preceding articles. The level 0, 0+ and 0++ were covered in the first article, while the second article talked about level 1, 2 and 2+. You might want to read them first if you want to follow the progression through the ranks. In this article, we look at the example at level 3, have a short outlook on further levels and then recap the concept.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable&amp;amp;lt;Product&amp;amp;gt; inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 3: Domain drive all the things!

We’ve introduced a new class for our requirement in visibility level 2 and made it a domain type. This is mostly another name for the concept of Entities or Value Objects from Domain Driven Design (DDD). If you aren’t familiar with Domain Driven Design, I recommend you grab the original book or its worthy successor and read about it. It is a way to look at requirements and code that will transform the way you develop software. To give a short spoiler, DDD Entities and DDD Value Objects are named core domain concepts that form the foundation of every DDD application. They are found by learning about the problem domain your software is used in. DDD Entities have an own identity, while DDD Value Objects just exist to indicate a certain value. Every DDD Entity and most DDD Value Objects are part of an DDD Aggregate. To load and store DDD Aggregates, a DDD Repository is put into place. The DDD Repository encapsulates all the technical stuff that has to happen when the application wants to access an DDD Aggregate through its DDD Root Entity. Sorry for all the “DDD” prefixes, but the terms are overloaded with many different meanings in our profession and I want to be clear what I mean when I use the terms “Repository” or “Aggregate”. Be very careful not to mistake the DDD meanings of the terms for any other meaning out there. Please read the books if you are unsure.

So, in Domain Driven Design, our BruttoPrice type is really a DDD Value Object. It represents a certain value in our currency of choice (Euro in our example), but has no life cycle on its own. Two BruttoPrices can be considered “the same” if their values are equal. This raises the question what the DDD Root Entity of the corresponding DDD Aggregate might be. Just imagine what happens in the domain (in real life, on paper) if you calculate a brutto price from a given netto price: You determine the value added tax category of your taxable product, look up its current percentage and multiply your netto price with the percentage. The DDD Root Entity is the value added tax category, as it can be introduced and revoked by your government and therefor has a life cycle on its own. The tax percentage, the netto price and the brutto price are just DDD Value Objects in its vicinity.

To bring DDD into our code and raise the implementation visibility level, we need to introduce a lot of new types with lots of lines of code:

  • NettoPrice is a DDD Value Object representing the concept of a monetary value without taxes.
  • BruttoPrice is a DDD Value Object representing the concept of a monetary value including taxes.
  • ValueAddedTaxCategory is a DDD Root Entity standing for the concept of different VAT percentages for different product groups.
  • ValueAddedTaxPercentage might be a DDD Value Object representing the concept of a percentage being applied to a NettoPrice to get a BruttoPrice. We will omit this explicit concept and let the ValueAddedTaxCategory deal with the calculation internally.
  • ValueAddedTaxRepository is a DDD Repository providing the ability to retrieve a ValueAddedTaxCategory for a known Taxable.
  • Taxable might be a DDD Entity. For us, it will remain an abstraction to decouple our taxes from other concrete types like Product.

The most surprising new class is probably the ValueAddedTaxRepository. It lingered in our code in nearly all previous levels, but wasn’t prominent, not visible enough to be explicit. Remember lines like this?

final BigDecimal taxFactor = <gets the right tax factor from somewhere> 

Now we know where to retrieve our ValueAddedTaxCategory from! And we don’t even know that the VAT is calculated using a percentage or factor anymore. That’s a detail of the ValueAddedTaxCategory given to us from the ValueAddedTaxRepository. If one day, for example at April 1th, 2020, the VAT for bottled water is decreed to be a fixed amount per bottle, we might need to change the internals of our VAT DDD Aggregate, but the netto and brutto prices and the rest of the application won’t even notice.

We’ve given our different reasons of change different places in our code. We have separated our concerns. This separation requires a lot of work to be spelled out. Let’s look at the code of our example at implementation visibility level 3:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    final ValueAddedTaxRepository vatProvider = givenVatRepository();
    for (Product each : inCart) {
      final ValueAddedTaxCategory vat = vatProvider.forType(each);
      final BruttoPrice bruttoPrice = vat.applyTo(each.nettoPrice());
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

There are now three lines of code responsible for calculating the brutto prices. It gets ridiculous! First we obtain the DDD Repository from somewhere. Somebody probably gave us the reference in the constructor or something. Just to remind you: The class is named ShowShoppingCart and now needs to know about a class that calls itself ValueAddedTaxRepository. Then, we obtain the corresponding ValueAddedTaxCategory for each Product or Taxable in our shopping cart. We apply this VAT to the NettoPrice of the Product/Taxable and pass the resulting BruttoPrice side by side with the NettoPrice in the addProductLine() method. Notice how we changed the signature of the method to differentiate between NettoPrice and BruttoPrice instead of using just to Euro parameters. Those domain types are now our level of abstraction. We don’t really care about Euro anymore. The prices might be expressed in mussle shells or bottle caps and we still could use our code without modification.

The ValueAddedTaxCategory we obtain from the DDD Repository isn’t a class with a concrete implementation. Instead, it is an interface:


/**
* AN-17: Calculates the brutto price (netto price with value added tax (VAT))
* for the given netto price.
*/
public interface ValueAddedTaxCategory {
  public BruttoPrice applyTo(NettoPrice nettoPrice);
}

Now we could nearly get rid of the comment above. It just repeats what the signature of the single method in this type says, too. We keep it for the reference to the requirement (AN-17).

Right now, the interface has only one implementation in the class PercentageValueAddedTaxCategory:


public class PercentageValueAddedTaxCategory implements ValueAddedTaxCategory {
  private final BigDecimal percentage;

  public PercentageValueAddedTaxCategory(final BigDecimal percentage) {
    this.percentage = percentage;
  }

  @Override
  public BruttoPrice applyTo(NettoPrice nettoPrice) {
    final Euro value = nettoPrice.multiplyWith(this.percentage).inEuro();
    return new BruttoPrice(value);
  }
}

You might notice that the concrete code of applyTo still has knowledge about the Euro. As long as we don’t ingrain the relationship between NettoPrice and BruttoPrice in these types, somebody has to do the conversion externally – and needs to know about implementation details of these types. That’s an observation that you should at least note down in your domain crunching documents. It isn’t necessarily bad code, but a spot that will require modification once the currency changes to cola bottle caps.

This is a good moment to reconsider what we’ve done to our ShowShoppingCart class. Let’s refactor the code a bit and move the responsibility for value added taxes where it belongs: in the Product type.


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            each.bruttoPrice());
    }
    return result;
  }

}

Now we have made a full circle: Our code looks like it began without the brutto prices, but with one additional line that delivers the brutto prices to the product line in the ShoppingCartRenderModel. The whole infrastructure that we’ve built is hidden behind the Product/Taxable type interface. We still use all of the domain types from above, we’ve just changed the location where we use them. The whole concept complex of different price types, value added taxes and tax categories is a top level construct in our application now. It shows up in the domain model and in the vocabulatory of our project. It isn’t a quick fix, it’s the introduction of a whole set of new ideas and our code now reflects that.

The code at implementation visibility level 3 might seem bloated and over-engineered to some. There is probably truth in this judgement. We’ve introduced far more code seams in the form of abstractions and indirections than we can utilize in the moment. We’ve prepared for an uncertain future. That might turn out to be unnecessary and would then be waste.

So let’s look at our journey as an example of what could be done. There is no need to walk all the way all the time. But you should be able to walk it in case it proves necessary.

Visibility level 4 and above: To infinity and beyond!

Remember that there are implementation visibility levels above 3! If you choose such a level, there will be even more code, more classes and types, more indirection and more abstraction. Suddenly, your new code will show up on system architecture diagrams and be deployed independently. Maybe you’ll need a dedicated server for it or scale it all the way up to its own server farm. Our example doesn’t match those criterias, so I stop here and just say that visibility level 3 isn’t the end of the journey. But you probably got the idea and can continue on your own now.

Recap: Rising through the visibility levels

We’ve come a long way since level 0 in terms of implementation visibility. The code still does the same thing, it just accumulates structure (some may call it cruft) and fletches out the relationships between concepts. In doing so, different axis of change emerge in different locations instead of entangled in one place. Our development effort rises, but we hope for a return on investment in the future.

I’ve found it easier to elevate the implementation visibility level of some code later than to decrease it. You might experience it the other way around. In the end, it doesn’t matter which way we choose – we have to match the importance of the requirement in the code. And as the requirements and their importance change, our code has to adjust to it in order to stay relevant. It isn’t the visibility level you choose now that will decide if your code is visible enough, it is the necessary visibility level you cannot reach for one reason or the other that will doom your code. Because it “feels bloated” and gets replaced, because it wasn’t found in time and is duplicated somewhere else, because it fused together with unrelated code and cannot be separated. Because of a plethora of reasons. By choosing and changing the implementation visibility level of your code deliberately, you at least take the responsibility to minimize the effects of those reasons. And that will empower you even if not all your decisions turn out profitable.

Conclusion

With the end of this third part, our series about the concept of implementation visibility comes to an end. I hope you’ve enjoyed the journey and gained some insights. If you happen to identify an example where this concept could help you, I’d love to hear from you! And if you know about a book or some other source where this concept is explained, too – please comment with a link below.