Implementation visibility – Part II

In the second part of our series about implementation visibility, we examine the middle ground of our visibility scale and discover that most requirements end up in this range. The maintainability is good, but we still have to check for architectural sins.

In the first article of this series, I presented the concept of “implementation visibility”. Every requirement can be expressed in source code on a scale of how prominent the implementation will be. There are at least five stages (or levels) on the scale:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

The article then introduced a simple example and examined how the level 0, 0+ and 0++ would appear within the example code. You may want to read the first article before we carry on with level 1 and 2 in this article.

A quick reminder

Our example is a webshop that lacks brutto prices. The original code of our shopping cart renderer might looked like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice());
    }
    return result;
  }
}

Visibility level 1: Extracted code lives longer

After all the (rather depressing) level 0 implementations of our brutto price calculation, the separated method is the first visibility level to result in code that can be discussed and tested separately:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPriceFor(each));
    }
    return result;
  }

  /**
  * AN-17: Calculates the brutto price for the given product.
  */
  private Euro bruttoPriceFor(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor);
  }
}

The new code is in lines 8 and 13 onwards. The new method was introduced to separate the calculation code from the rendering code. It still lives in the wrong class, but can be tested on its own if you make it public or package accessible. The comment now has a natural scope. And, most important: This implementation is the first where the notion of “brutto price” appears in the JavaDoc and the IDE.

Methods are the smallest parts of our object-oriented code. If you would have one method per requirement, you would just need one extra method of glue code to tie everything together. If one requirement needs to change or becomes obsolete, you know where to cut.

Methods are the primary focus of unit tests. You prepare the parameters for the method you want to test, call it and check the result. This is the AAA or triple-A normal form of unit testing: Arrange, Act, Assert. If several methods or even several objects need to be tested in conjunction, the testing effort rises.

We can conclude that with its own method, the VAT calculation now has its own home. Future readers can grasp the scope of our implementation easily and hopefully make changes under direct test coverage. This is the first visibility level that starts to feel like we meant it.

Visibility level 2: Make it a top-level affair

There is one part in object-oriented code that is even more basic than a method: the class. In Java, each class strives to have its own text file. Before you can write a method in Java, you need to define a class to contain it. Classes are the primary granularity level we navigate our code. Every IDE will show classes as the default elements in our “project explorers”. So what if we introduce a new class for our VAT calculation and move all our code there?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            CalculateBruttoPrice.forProduct(each));
    }
    return result;
  }
}
/**
 * AN-17: Calculates the brutto price with value added tax (VAT) for the given product.
 */
public class CalculateBruttoPrice {
  public static Euro forProduct(Product product) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return product.nettoPrice().multiplyWith(taxFactor); 
  }
}

The new code is in line 8 and the full new class file. This implementation might not look a lot different from level 1 (separate method), but it really is on another level. The brutto price calculation now isn’t tied to rendering shopping carts anymore. It is not tied to anything other than a given product. It is a top-level concept of our application now. Anybody with a product can call the method and receive the brutto price, from anywhere in our application (hopefully respecting our architecture boundaries).

Our unit test class now reads as if we had written it only for the new requirement: CalculateBruttoPriceTest. We still need to invent test products in our test, but the whole notion of render models and shopping carts is gone. In essence, we freed the concept of price calculation from its “evolutionary” ties.

Implementing the new requirement in a separate class, if feasible, adheres to the Single Responsibility Principle (SRP), that requires each class of a system to only have one reason to change. In our case, the CalculateBruttoPrice class only changes if the brutto prices needs adjustment. For all previous visibility levels, that wasn’t true. The ShowShoppingCart class would need modifications if the brutto prices or the shopping cart rendering were to be changed. This improvement is reason enough to elevate our implementation visibility past level 1.

In short, a good heuristics for new requirements (as opposed to change requests for existing requirements) is to start with a new class. If you are unsure, start lower, but keep in mind that classes are the main navigation layer of object-oriented code.

Visibility level 2+: Inviting the requirement to be part of the project’s language

Introducing a new class for our requirement felt good, but something still feels off. When we review the interface of the CalculateBruttoPrice, two things stick out immediately: The class is named as a service (CalculateXYZ as in “do XYZ for me”) and can only calculate brutto prices for products. Our customer was serious with his requirement, so it’s safe to assume that brutto prices will stay in the application and play a key role. We should reflect this seriousness by lifting the implementation visibility level once more and make the BruttoPrice a top level concept of our project’s domain:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            BruttoPrice.of(each).inEuro());
    }
    return result;
  }
}

The ShowShoppingCart code doesn’t look very different from the level 2 code beforehands. The new code is in line 8, too. The new class isn’t named like a service anymore, but like a concept or domain type. The named constructor of() returns a BruttoPrice instance and not just a Euro object:

/**
 * AN-17: Represents the brutto price with value added tax (VAT) for the given Taxable.
 */
public final class BruttoPrice {
  public static BruttoPrice of(Taxable item) {
    final BigDecimal taxFactor = <gets the right tax factor from somewhere>
    return new BruttoPrice(item.nettoPrice().multiplyWith(taxFactor));
  }
  
  private final Euro asValue;

  private BruttoPrice(Euro value) {
    this.asValue = value;
  }

  public Euro inEuro() {
    return this.asValue;
  }
}

Now, we can accumulate additional behaviour in the new BruttoPrice type if the need arises. With the service class of level 2, we probably wouldn’t have risen above the Euro abstraction and mixed up netto and brutto prices somewhere in the future.  If we model our NettoPrice and BruttoPrice as domain types, the compiler will help us keeping them separate – even if both contain Euros as their value.

With this visibility elevation, we discovered another abstraction: We can create brutto prices for virtually anything that can be taxed. It doesn’t have to be a product, it just needs a netto price and a tax factor. The new (abstract) domain type is named Taxable. Of course, Product is an implementation of Taxable.

This makes us even more independent from any webshop, shopping cart or product. We can now write unit tests for our BruttoPrice without being coupled to the Product class at all. We have successfully decoupled the cart/product part of our application from the prices part. Recognizing and implementing the independence of concepts is an important step towards even higher visibility levels. It is also the groundwork of a low coupled, high cohesive code base where most things fall into their place naturally.

The step from level 2 (separate class) to level 2+ (new domain type) wasn’t just syntactic sugar, it was driven by the insight that separation of concerns is the fundamental principle to achieve maintainability, as long as the abstractions aren’t overwhelming. A good indicator that you’ve taken it too far is when your domain expert (in our example our client) raises her eyebrows in surprise when you talk about your abstract domain types because the names sound outlandish and far-fetched.

But you can take your implementation visibility even further and should really consider doing so given the circumstances. We will learn about visibility level 3 (separate aggregate) in the next blog post of this series. Stay tuned!

Implementation visibility – Part I

The concept of implementation visibility is very powerful. If you choose the wrong visibility for your requirements, your code will end up unmaintainable.

Somewhere in my take on programming, there lingers the concept of “implementation visibility”, that I’m not quite sure to be able to express clearly, but I’ll try.

Let’s say you are writing an academic text like a bachelor thesis and your professor makes it clear that she regards the list of literature a very important part of your work. What are you going to do? Concentrate on your cool topic and treat the literature as a secondary task? Or will you shift your focus and emphasize your extensive literature research, highlighting promising cross-references in your text? You’ll probably adjust your resources to make your list of literature more prominent, more visible. You respond to the priorities of your stakeholders.

Now imagine that your customer wants you to program a web application, but has one big requirement: All actions of the users need to be reassured with a confirmation question (as in “do you really want to delete this?”). He makes it clear that this is a mandatory feature that needs to be implemented with utmost care and precision. What would you do? We responded by adjusting our system’s architecture to incorporate the requirement into the API. You can read about our approach in this blog post from 2015. The gist of it is that every possible client of the system will be immediately aware of the requirement and has a much easier time conforming to it. It is harder to ignore or forget the requirement than to adhere to it because the architecture pushes you into the right direction.

The implementation of the customer’s requirement in the example above is very visible. You’ll take one look at the API and know about it. It isn’t hidden into well-meaning but out-dated developer documentation or implicitly stated because every existing action has a confirmation step and you should be sentient enough to know that this means your new one needs one, too. The implementation visibility of the customer’s requirement is maximized with our approach.

Stages of visibility

I have identified some typical stages (or levels) of implementation visibility that I want to present in this blog post series. That doesn’t mean that there won’t or can’t be others. I’m not even sure if the level system is as one-dimensional as I’m claiming here. I invite you to think about the concept, make your own observations and evolve from there. This is a starting point, not an absolute truth.

The following stages typically appear in my projects:

  • level 0: Inline
    • level 0+: Inline with comment
    • level 0++: Inline with apologetic comment
  • level 1: separate method
  • level 2: separate class
    • level 2+: new type in domain model
  • level 3: separate aggregate
  • level 4: separate package or module
  • level 5: separate application or service

In my day-to-day work, the levels 1 to 3 are the most relevant, but that’s probably not universally applicable. Our example above with the requirement-centered API isn’t even located on this list. I suggest it’s at level 6 and called separate concept or something similar.

An example to explain the visibility levels

Let’s assume a customer wants us to program a generic webshop. We are not very versed in commerce or e-commerce things and just start implementing requirements one after one.

After the first few iterations with demonstrated and usable artifacts, our customer calls us and explains that all prices in the webshop are netto prices and that there need to be some kind of brutto price calculation. You, being accustomed to prices that don’t change once you put products into your shopping cart, ask a few questions and can finally grasp the concept of value added taxes. Now you want to implement it into the webshop.

The first approach to the whole complex is to show the brutto prices right besides the netto prices when the user views his shopping cart. You can then validate the results with your customer and discuss problems or misconceptions that are now visible and therefore tangible.

The original code of your shopping cart renderer might look like this:


public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      result.addProductLine(
               each.description(),
               each.nettoPrice());
      }
      return result;
  }
}

A quick explanation of the code: The class ShowShoppingCart takes some products and converts them into a ShoppingCartRenderModel that contains the shopping cart data in a presentable form so the GUI just needs to take the render model and paste it into some kind of template. For each product, there is one line with a description and the (already renamed) netto price on the page.

Visibility level 0: It’s just code anyway

Let’s start with the lowest and most straight-forward implementation visibility level: The inline implementation.

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5 and 9. As you can see, the programmer chose to implement exactly what he understood from the discussion about netto and brutto prices with the customer. A brutto price is a netto price with value added tax. The VAT rate is 19 percent at the time of writing, so a multiplication with 1.19 is a valid implementation.

Our problem with this approach isn’t the usage of floating point numbers in the calculations or that calculations even exist in a method that should do nothing more than render some products, but that the visibility of the requirement is minimal. If you, I or somebody else doesn’t know exactly where this code hides, we will have a hard time finding it once the VAT is changed or anything else should be done with brutto prices or VATs.

Technically, the customer’s requirement is implemented and the brutto prices will show up. But because the concept of taxes (or VAT) is important for the customer, we likely made the code too invisible to be maintainable.

Visibility level 0+: Hey, I even wrote a comment

To make some part of the code stick out of the mess, we have the tool of inline code comments. Let’s apply them to our example and raise our visibility level from 0 to 0+:

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
    final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
    for (Product each : inCart) {
      // AN-17: calculating the brutto price from the netto price
      final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
      result.addProductLine(
            each.description(),
            each.nettoPrice(),
            bruttoPrice);
    }
    return result;
  }
}

The new code is in lines 5, 6 and 10. You can see that the programmer chose the same approach as before, but realized that the code would be buried if not marked. Given that the requirement identifier is “AN-17”, the code can be found by a text search of this number. And if you happen to stumble upon this part of the application, you can deduct meaning about what you see from the comment.

Except that you cannot really be sure what the AN-17 code really is. Is the result.addProductLine() part of AN-17 or not? Would you expect the calculation of taxes and prices in a method called render() in a class named ShowShoppingCart? Is this implementation really correct? Aren’t there different tax rates for different products? Did the original author think about that? Is the customer content with this functionality?

Note that you cannot really test the brutto price calculation. You have to invent some products, render them and then scrape the brutto prices from the render model. That’s tedious at best and a clear sign that the implementation visibility is still too low. On to the next level

Visibility level 0++: This sucks, but I’ve got to go now

This level tries to make you a partner in crime by explicitly stating what’s obviously wrong with the code at hand. Now it’s your responsibility to fix it. You wouldn’t leave a broken window be, would you?

public class ShowShoppingCart {
  public ShoppingCartRenderModel render(Iterable<Product> inCart) {
  final ShoppingCartRenderModel result = new ShoppingCartRenderModel();
  for (Product each : inCart) {
    // AN-17: calculating the brutto price from the netto price
    // TODO: take different tax factors into account
    final Euro bruttoPrice = each.nettoPrice().multiplyWith(1.19D);
    result.addProductLine(
          each.description(),
          each.nettoPrice(),
          bruttoPrice);
  }
  return result;
  }
}

The new code is in lines 5, 6, 7 and 11. The new comment line 6 is typical for this visibility level: The original programmer knew that his implementation isn’t adequate but couldn’t be bothered with improving it. Perhaps he had external circumstances force him to do it. Whatever it was, this code is the equivalent to a soiled public toilet. The difference is, this time we can determine who made the mess.

The apologetic “I know I made a mess” comment often begins with TODO or FIXME. This isn’t directed towards the original author, it’s pointed at you, the person that happens to read the comment. Now, what are you going to do? Pretend you didn’t read the comment? Leave the toilet soiled? Clean up the mess of your predecessor? You probably have work to do, too. And doesn’t it work the way it is? Never change a running system!

We will see how you can improve the implementation visibility of the requirement in the next blog post of this series. Stay tuned!

4 Tips for better CMake

We are doing one of those list posts again! This time, I will share some tips and insights on better CMake. Number four will surprise you! Let’s hop right in:

Tip #1

model dependencies with target_link_libraries

I have written about this before, and this is still my number one tip on CMake. In short: Do not use the old functions that force properties down the file hierarchy such as include_directories. Instead set properties on the targets via target_link_libraries and its siblings target_compile_definitions, target_include_directories and target_compile_options and “inherit” those properties via target_link_libraries from different modules.

Tip #2

always use find_package with REQUIRED

Sure, having optional dependencies is nice, but skipping on REQUIRED is not the way you want to do it. In the worst case, some of your features will just not work if those packages are not found, with no explanation whatsoever. Instead, use explicit feature-toggles (e.g. using option()) that either skip the find_package call or use it with REQUIRED, so the user will know that another lib is needed for this feature.

Tip #3

follow the physical project structure

You want your build setup to be as straight forward as possible. One way to simplify it is to follow the file system and and the artifact structure of your code. That way, you only have one structure to maintain. Use one “top level” file that does your global configuration, e.g. find_package calls and CPack configuration, and then only defers to subdirectories via add_subdirectory. Only for direct subdirectories though: if you need extra levels, those levels should have their own CMake files. Then build exactly one artifact (e.g. add_executable or add_library) per leaf folder.

Tip #4

make install() an option()

It is often desirable to include other libraries directly into your build process. For example, we usually do this with googletest for our unit test. However, if you do that and use your install target, it will also install the googletest headers. That is usually not what you want! Some libraries handle this automagically by only doing the install() calls when they are the top level project. Similar to the find_package tip above, I like to do this with an option() for explicit user control!

Generating done

That is it for today! I hope this is helps and we will all see better CMake code in the future.

OPC Basics

OPC (Open Platform Communications) is a machine to machine communication protocol for industrial automation. In its simplest form it works like this:

An OPC server defines a set of variables within a directory tree-like hierarchy forming namespaces. Each variable has a data type like integer, boolean, real, string and a default value.

One or many OPC clients connect to the OPC server via a TCP based binary protocol, usually on port 4840. The clients can read and write the OPC variables provided by the server. Clients can also monitor OPC variables for changes so that you don’t have to poll the variables. In code this is usually done by registering a callback function that gets executed when the monitored variable changes.

Handshaking

A simple communication pattern between two OPC clients that we have used in OPC based interfaces is a handshake. This can either be a two-way handshake or a three-way handshake. The two-way handshake is in fact just an acknowledgement: One OPC client sets the value of a variable, the other client reads the variable and resets the value to the default value to confirm that it has read the variable. If you do not want to use the default value to indicate a read confirmation you can also use another variable as a confirmation flag. In a three-way handshake the first client also confirms the confirmation.

OPC UA

The current specification of OPC is OPC UA (Unified Architecture) by the OPC Foundation. It covers a lot more functionality than what is described above. It’s a unified successor to various OPC Classic specifications like OPC DA, A&E and HDA. If you want to get started with OPC UA development you can use one of the many client and server SDKs and toolkits for various programming languages.

Functional tests for Grails with Geb and geckodriver

Previously we had many functional tests using the selenium-rc plugin for Grails. Many were initially recorded using Selenium IDE, then refactored to be more maintainable. These refactorings introduced “driver” objects used to interact with common elements on the pages and runners which improved the API for walking through a multipage process.

Selenium-rc got deprecated quite a while ago and support for Firefox broke every once in a while. Finally we were forced to migrate to the current state-of-the-art in Grails functional testing: Geb.

Generally I can say it is really a major improvement over the old selenium API. The page concept is similar to our own drivers with some nice features:

  • At-Checkers provide a standardized way of checking if we are at the expected page
  • Default and custom per page timeouts using atCheckWaiting
  • Specification of relevant content elements using a JQuery-like syntax and support for CSS-selectors
  • The so-called modules ease the interaction with form elements and the like
  • Much better error messages

While Geb is a real improvement over selenium it comes with some quirks. Here are some advice that may help you in successfully using geb in the context of your (grails) webapplication.

Cross-plattform testing in Grails

Geb (or more specifically the underlying webdriver component) requires a geckodriver-binary to work correctly with Firefox. This binary is naturally platform-dependent. We have a setup with mostly Windows machines for the developers and Linux build slaves and target systems. So we need binaries for all required platforms and have to configure them accordingly. We have simply put them into a folder in our project and added following configuration to the test-environment in Config.groovy:

environments {
  test {
    def basedir = new File(new File('.', 'infrastructure'), 'testing')
    def geckodriver = 'geckodriver'
    if (System.properties['os.name'].toLowerCase().contains('windows')) {
      geckodriver += '.exe'
    }
    System.setProperty('webdriver.gecko.driver', new File(basedir, geckodriver).canonicalPath)
  }
}

Problems with File-Uploads

If you are plagued with file uploads not working it may be a Problem with certain Firefox versions. Even though the fix has landed in Firefox 56 I want to add the workaround if you still experience problems. Add The following to your GebConfig.grooy:

driver = {
  FirefoxProfile profile = new FirefoxProfile()
  // Workaround for issue https://github.com/mozilla/geckodriver/issues/858
  profile.setPreference('dom.file.createInChild', true)
  new FirefoxDriver(profile)
}

Minor drawbacks

While the Geb-DSL is quite readable and allows concise tests the IDE-support is not there. You do not get much code assist when writing the tests and calling functions of the page objects like in our own, code based solution.

Conclusion

After taking the first few hurdles writing new functional tests with Geb really feels good and is a breeze compared to our old selenium tests. Converting them will be a lot work and only happen on a case-by-case basis but our coverage with Geb is ever increasing.

How I start a project – the next steps

The next steps after the initial meetings I use to bring a project on the way to success.

After the initial meetings with the project stakeholders my next step is to get a big picture of the processes the project tries to improve. Every (enterprise) software implements processes or workflows stemming from the business side. Since I want to improve the work for the people using the software I take the user’s perspective and try to understand it and describe it from their perspective.
For this task my go-to-tool is the user journey map. My first draft starts with a handful of steps outlining the main functions performed by the major actors. These are normally just 5 – 7 steps and serve as an overview and communication starter:

Often some users or other systems interact inside one process. Important is to concentrate on the big picture, the big steps of the process. These are the ones you need to get right, details and deviations from the process come after that. The point of all the drawings and documents I use is to foster a shared understanding between the stakeholders (including the users) and the team. We as a whole need to talk about the same things with the same language. If one step has different name we have to rename it to match the name used by the stakeholders. This is crucial. The same applies to the names used in the user interface. These must be the ones used by the users, not some internal words, not even synonyms.
From the big picture I iterate through the assumptions and getting more detailed on the way. Assumptions can make or break a project. Even if I am pretty sure I need to verify which means often ask or observe the user. The method for getting a good answer depends on the kind of assumption. But I need to verify. I usually use a wiki to record assumptions in the form of open questions like

  • which device will be used for this process?
  • will step 3 always be the next step?
  • how does the user hold the device?

These question get more detailed in the course of the project. Even when implementing the solution in code questions will arise. These need to be verified. Sometimes the question can be answered by the defined goals: which solution helps the user reaching his goals. If I cannot answer the question from the collected information, I need to ask questions, test it with prototypes, observe the user or do some research.
Usually along the way I use a prototype, mockup or demo to show how I understand the problem of the users. Again this is a great point for build a shared understanding. Sometimes I need a picture and often a simulation of the steps and actions involved. These interactive prototypes help to spark discussions about details I didn’t think of in the first place. Details that matter. Details that when overseen or seen too late can defer or break the project. When there are still in demo form or in a prototype the stakeholders find it easier to discuss things and the amount of work done is minimal. These demos evolve over time and are the basis for the solution later on. This does not mean that I reuse the code, but I reuse the insights gained from them.
Also prototypes are great for spikes evaluating a solution, the feasibility of a technology or testing assumptions. Prototypes are not only for the start of the project but can be used throughout. Sometimes something quick has to be tested without compromising the application as a whole. Here small and separate prototypes can be used.

Keeping connections alive with libcurl

libcurl is quite a comfortable option to transfer files across a variety of network protocols, e.g. HTTP, FTP and SFTP.

It’s really easy to get started: downloading a single file via http or ftp takes only a couple of lines.

Drip, drip..

But as with most powerful abstractions, it is a bit leaky. While it does an excellent job of hiding such steps as name resolution and authentication, these steps still “leak out” by increasing the overall run-time.

In our case, we had five dozen FTP servers and we needed to repeatedly download small files from all of them. To make matters worse, we only had a small time window of 200ms for each transfer.

Now FTP is not the most simple protocol. Essentially, it requires the client to establish a TCP control connection, that it uses negotiate a second data connection and initiate file transfers.

This initial setup phase needs a lot of back and forth between server and client. Naturally, this is quite slow. Ideally, you would want to do the connection setup once and keep both the control and the data connection open for subsequent transfers.

libcurl does not explicitly expose the concept of an active connection. Hence you cannot explicitly tell the library not to disconnect it. In a naive implementation, you would download multiple files by simply creating an easy session object for each file transfer:

for (auto file : FILE_LIST)
{
  std::vector<uint8_t> buffer;
  auto curl = curl_easy_init();
  if (!curl)
    return -1;
  auto url = (SERVER+file);
  curl_easy_setopt(curl, CURLOPT_URL,
    url.c_str());
  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
    appendToVector);
  curl_easy_setopt(curl, CURLOPT_WRITEDATA,
    &buffer);
  if (curl_easy_perform(curl) != CURLE_OK)
    return -1;

  process(buffer);
  curl_easy_cleanup(curl);
}

That does indeed reset the connection for every single file.

Re-use!

However, libcurl can actually keep the connection open as part of a connection re-use mechanism in the session object. This is documented with the function curl_easy_perform. If you simply hoist the easy session object out of the loop, it will no longer disconnect between file transfers:

auto curl = curl_easy_init();
if (!curl)
  return -1;

for (auto file : FILE_LIST)
{
  std::vector<uint8_t> buffer;
  auto url = (SERVER+file);
  curl_easy_setopt(curl, CURLOPT_URL, 
    url.c_str());
  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, 
    appendToVector);
  curl_easy_setopt(curl, CURLOPT_WRITEDATA, 
    &buffer);
  if (curl_easy_perform(curl) != CURLE_OK)
    return -1;

  process(buffer);
}
curl_easy_cleanup(curl);

libcurl will now cache the active connection in the session object, provided the files are actually on the same server. This improved the download timings of our bulk transfers from 130ms-260ms down to 30ms-40ms, quite the enormous gain. The timings now fit into our 200ms time window comfortably.

Using PostgreSQL with Entity Framework

The most widespread O/R (object-relational) mapper for the .NET platform is the Entity Framework. It is most often used in combination with Microsoft SQL Server as database. But the architecture of the Entity Framework allows to use it with other databases as well. A popular and reliable is open-source SQL database is PostgreSQL. This article shows how to use a PostgreSQL database with the Entity Framework.

Installing the Data Provider

First you need an Entity Framework data provider for PostgreSQL. It is called Npgsql. You can install it via NuGet. If you use Entity Framework 6 the package is called EntityFramework6.Npgsql:

> Install-Package EntityFramework6.Npgsql

If you use Entity Framework Core for the new .NET Core platform, you have to install a different package:

> Install-Package Npgsql.EntityFrameworkCore.PostgreSQL

Configuring the Data Provider

The next step is to configure the data provider and the database connection string in the App.config file of your project, for example:

<configuration>
  <!-- ... -->

  <entityFramework>
    <providers>
      <provider invariantName="Npgsql"
         type="Npgsql.NpgsqlServices, EntityFramework6.Npgsql" />
    </providers>
  </entityFramework>

  <system.data>
    <DbProviderFactories>
      <add name="Npgsql Data Provider"
           invariant="Npgsql"
           description="Data Provider for PostgreSQL"
           type="Npgsql.NpgsqlFactory, Npgsql"
           support="FF" />
    </DbProviderFactories>
  </system.data>

  <connectionStrings>
    <add name="AppDatabaseConnectionString"
         connectionString="Server=localhost;Database=postgres"
         providerName="Npgsql" />
  </connectionStrings>

</configuration>

Possible parameters in the connection string are Server, Port, Database, User Id and Password. Here’s an example connection string using all parameters:

Server=192.168.0.42;Port=5432;Database=mydatabase;User Id=postgres;Password=topsecret

The database context class

To use the configured database you create a database context class in the application code:

class AppDatabase : DbContext
{
  private readonly string schema;

  public AppDatabase(string schema)
    : base("AppDatabaseConnectionString")
  {
    this.schema = schema;
  }

  public DbSet<User> Users { get; set; }

  protected override void OnModelCreating(DbModelBuilder builder)
  {
    builder.HasDefaultSchema(this.schema);
    base.OnModelCreating(builder);
  }
}

The parameter to the super constructor call is the name of the configured connection string in App.config. In this example the method OnModelCreating is overridden to set the name of the used schema. Here the schema name is injected via constructor. For PostgreSQL the default schema is called “public”:

using (var db = new AppDatabase("public"))
{
  var admin = db.Users.First(user => user.UserName == "admin")
  // ...
}

The Entity Framework mapping of entity names and properties are case sensitive. To make the mapping work you have to preserve the case when creating the tables by putting the table and column names in double quotes:

create table public."Users" ("Id" bigserial primary key, "UserName" text not null);

With these basics you’re now set up to use PostgreSQL in combination with the Entity Framework.

 

Evolution of programming languages

Programming languages evolve over time. They get new language features and their standard library is extended. Sounds great, doesn’t it? We all know not going forward means your go backward.

But I observe very different approaches looking at several programming ecosystems we are using.

Featuritis

Java and especially C# added more and more “me too” features release after release making relatively lean languages quite complex multi-paradigm languages. They started object oriented and added generics, functional programming features and declarative programming (LINQ in C#) and different UI toolkits (AWT, Swing, JavaFx in Java; Winforms, WPF in C#) to the mix.

Often the new language features add their own set of quirks because they are an afterthought and not carefully enough designed.

For me, this lack of focus makes said language less attractive than more current approaches like Kotlin or Go.

In addition, deprecation often has no effect (see Java) where 20 year old code and style still works which increases the burden further . While it is great from a business perspektive in that your effort to maintain compatibility is low it does not help your code base. Different styles and old ways of doing something tend to remain forever.

Revolution

In Grails (I know, it is not a programming language, but I has its own ecosystem) we see more of a revolution. The core concept as a full stack framework stays the same but significant components are changed quite rapidly. We have seen many changes in technology like jetty to tomcat, ivy to maven, selenium-rc to geb, gant to gradle and the list goes on.

This causes many, sometimes subtle, changes in behaviour that are a real pain when maintaining larger applications over many years.

Framework updates are often a time-consuming hassle but if you can afford it your code base benefits and will eventually become cleaner.

Clean(er) evolution

I really like the evolution in C++. It was relatively slow – many will argue too slow – in the past but it has picked up pace in the last few years. The goal is clearly stated and only features that support it make it in:

  • Make C++ a better language for systems programming and library building
  • Make C++ easier to teach and learn
  • Zero-Cost abstractions
  • better Tool-support

If you cannot make it zero-cost your chances are slim to get your feature in…

C at its core did not change much at all and remained focused on its merits. The upgrades mostly contained convenience features like line comments, additional data type definitions and multithreading.

Honest evolution – breaking backwards compatibility

In Python we have something I would call “honest evolution”. Python 3 introduced some breaking changes with the goal of a cleaner and more consistent language. Python 2 and 3 are incompatible so the distinction in the version number is fair. I like this approach of moving forward as it clearly communicates what is happening and gets rid of the sins in the past.

The downside is that many systems still come with both, a Python 2 and a Python 3 interpreter and accompanying libraries. Fortunately there are some options and tools for your code to mitigate some of the incompatibilities, like the __future__ module and python-six.

At some point in the future (expected in 2020) there will only support for Python 3. So start making the switch.

Quick and dirty is a skill

This post is a reminder, first and foremost to myself, that it is important to work according to what’s essential right now: quality or speed.

Being clean coders we build our software based on quality and reflect on how we do it. We set internal standards in code and UIs, we write tests, we polish.
But there are times when all this focus on quality is obstructive. Times when we need to learn something. For example: at a start of a project when fundamental questions like is it feasible, how should this interaction work, what’s the right order of steps are unanswered, learning needs to be as cheap as possible.
Here quick and dirty is important. The problem is our ego. We want to polish it, we want to build real software with a sound structure. But quality takes time. The problem is quality is not important when answering the fundamental project questions, learning is. May be a mockup in Powerpoint is enough? (not even writing code? ugh). A simple sketch on a piece of paper. Or maybe just a quick demo hacked together in an afternoon.
I know these suggestions may insult our pride. But we need to focus on what’s important: sometimes that’s quality, sometimes that’s speed.
Decades ago when I started coding, quick and dirty wasn’t a problem. Everything I wrote was quick and dirty. I was learning all the time. Over time I got better at developing software, structuring applications and building robust systems. But quick and dirty was lost along the way.
When you write something for the purpose of learning it can happen that you are wrong and all the code has to be thrown away. If it was just 2 hours patching something together that’s okay, but what if you spent a whole week? Just like writing quality software, quick and dirty is a skill in itself and as with other skills we need to practice it.
But beware this is not only a problem at the start of a project: often as developers we tend to overthink something, we plan for every possible outcome, imagine scenarios with weirdly acting users or systems. This is the time to stop and implement something to learn. To get feedback. Not to overanalyse or overdesign. Just release something and test it with real users, it doesn’t need to be part of the software in production, just use a demo or a staging environment. But if you need to learn something, focus on that, not on quality.