Category: Software Development

Making CherryPy Application WSGI compatible

When choosing a micro web framework evolving it to fit your needs is key. As CherryPy is one of our choices I want to show you how to evolve it in terms of web server. Of course you can use the embedded CherryPy web server in development and for small sites. It is fast enough for many use cases and supports important features like SSL so you may come a long way just using it. There are several reasons to put your CherryPy behind a tried and trusted native web server like Apache or nginx:

  • Consistent production environment using different application servers (e.g. for Java and Python) using a powerful and uniform frontend
  • Many options and possibilites using Apache modules
  • Well known and understood environment for administrators
  • Separation of web-facing http server concerns and your web application
  • Improved performance and security

Making CherryPy a WSGI-compatible

The good news is that CherryPy application objects are already a WSGI-compliant application. So creating a wsgi.py like the following will enable integration with mod_wsgi of Apache:

def application(environ, start_response):
    cherrypy.tree.mount(MyApp(), script_name=None, config=None)
    return cherrypy.tree(environ, start_response)

Integrating with Apache’s mod_wsgi

It is quite easy to integrate a Python WSGI application with apache using mod_wsgi. If the module is present you just need to add some directives telling Apache where to mount the wsgi application defined by your wsgi.py script:

WSGIScriptAlias /my_app /path/to/wsgi.py
# May be required to allow your web app using libraries installed on the system
<Directory /usr/lib/python2.7/site-packages/ >
    Order deny,allow
    Allow from all
    Require all granted
</Directory>

After you have such a setup working properly you can consult the mod_wsgi documentation on how to improve in regards to threading, script reloading etc.

Configuring the WSGI-app

Many web applications need some form of configuration. Your application should not make assumptions on its install location or some directory structure. Generally speaking, an application should never assume that it can use relative path names for accessing the filesystem. Also access to operating system environment variables is dangerous because the application may run in different contexts. But we can specify WSGI-environment variables in the web servers’ configuration. An easy and safe way is to provide the configuration directory and other values using WSGI-environment variables that we can specify in the mod_wsgi configuration:

WSGIScriptAlias /my_app /path/to/wsgi.py
SetEnv configuration_dir /etc/my_shiny_web_app
...

We can access the wsgi-environment in python like so:

def application(environ, start_response):
    configdir = environ['configuration_dir']
    cherrypy.config.update(os.path.join(configdir, 'global.conf'))

    cherrypy.tree.mount(MyApp(), config=os.path.join(configdir, 'my_app.conf'))
    return cherrypy.tree(environ, start_response)

Note: Because your web app can be mounted to other locations than “/” on the the web server your application should not hard-code absolute links and the like. They all will be dead if your app is mounted at a different location.

Modular development of complex UIs with atomic design

Creating user interfaces is traditionally an expensive development effort. Every web page, dialog or screen is hand crafted from scratch. Developers on the one hand write object oriented, modular code in the whole application but as soon as the UI level is reached everything breaks down.

Creating user interfaces is traditionally an expensive development effort. Every web page, dialog or screen is hand crafted from scratch. Developers on the one hand write object oriented, modular code in the whole application, use myriads of frameworks and libraries but as soon as the UI level is reached everything breaks down. Each view is written in isolation.
Designers have a different view of the UI. They see the interface through the lens of style guides and guidelines. The look and feel throughout the interface should be consistent and should be experienced as a whole.

Atomic design

These two worlds can be combined.
Many designers and developers see the need to design and create design systems. Brad Frost is the one who coined and describes a language for structuring user interfaces: atomic design. The names take heavy cues from chemistry but the important part is the containment part.
Atoms are the low level building blocks: e.g. the widgets in native UI kits or the tags in the web world. But also things like colors or type faces are atoms.
Molecules are simple combinations of atoms. A search field which is comprised of a label, a text field and a button is a molecule.
Organisms are more complex UI components. Organisms can be created from atoms, molecules and other organisms. A complete form would be a perfect example.
Combining all these into a full page or window layout is called a template in atomic design. This template is the abstract definition, the blueprint of the complete screen or page.
Filling this template with content results in a page.
All this sounds pretty abstract and the examples found in the web are very basic so let’s dive in and identify the parts in an example UI.

Decomposing a complex UI

Here we take an example from the excellent UI concept by Lennart Ziburski: desktop neo. (If you haven’t seen this, you should take a look).

finder

Our first decomposing task is to identify distinct parts of the user interface and give them names. These would be the organisms.

organisms

Interlude: how to name things

As with every naming endeavor it is hard to decide which name is appropriate. Dan Mall argues in favor of display patterns to be name givers. Display patterns describe the (abstract) visual aspect and can be used with multiple content patterns. Content patterns describe the types of elements and can be rendered in multiple display patterns. Since we want to name an organism which is content agnostic we should take cues from the visual appearance not the content inside it.

Decomposing further

Now we break those organisms further down. Let’s start with the card grid organism. As the name already suggests it organizes cards in a grid or tabular layout. We have different kinds of cards. First take a look at the preview card at the left.

preview_card

The preview card consists of a thumbnail showing a preview of an item, an icon and a label. This is a simple interface element and is therefore a molecule created from the three mentioned atoms. A name for this molecule could be “image with caption”.

Interlude: testing states in the abstract

Our example touches an important and often neglected part of interfaces: you need to test for different content. Here the longer name is cut with an ellipsis. This is a simple case. But what if the name is missing? Or has unusual characters. Or or or. Besides that we need to indicate the current state of the interface as well. Do we have an error? Are we loading something? Interfaces have different states. Five to be exact. The good part is that we can (and should) test them on the abstract level of atoms, molecules and organisms.

A more complex organism

The cards in the right card grid are more complex examples. Every card is an organism with a title (atom) and a content part (molecule/organism).

The weather card has a simple molecule consisting of an icon and two labels.

weather_card

Whereas the schedule card consists of a list organism which itself includes molecules. These molecules have two labels and one or more actions (links or buttons).

schedule_card

The other parts of the interface can be decomposed as well. Charlotte Jackson describes an interesting approach to decomposing your existing interfaces: print them out, cut them to pieces and name these pieces.

Making the jump

Until now we talked about the designer’s view of the interface but the developer has to translate all these definitions into code and hook them up to content. The approach from the atomic design side is largely the same for web or native but in development we have to distinguish between them.

On Rails

Let’s first take a look at the web side of things. We could use a client side component framework like react but here are like to keep it simple.
We just use Rails in our example but every other web framework will work as well. We need to organize our newly defined chemistry lab in three parts: HTML (or views), CSS and JavaScript.
For CSS and JavaScript we use the include mechanism of the asset pipeline or import if you use SASS. Each dimension gets a separate directory inside app/assets/stylesheets or app/assets/javascripts respectively.
We name our directories atoms, molecules and organisms. The same is true for views: a directory named molecules and one named organisms inside app/views/atomic_design. No need for atoms since they are basic HTML tags or helpers. Atomic design’s templates become Rails’ layouts. Via calls to render we can inject content into these abstract organisms:

<%= render layout: '/atomic_design/organisms/card', locals: {title: 'weather in Berlin'} do %>
  <%= render layout: '/atomic_design/molecules/image_with_text', locals: {image_class: 'fa_sunny'} do %>
    <span class="temperature">23 °C</span><span class="condition">Sunny</span>
  <% end %>
<% end %>

Native

On the native side we also need a component and include mechanism. Usually every widget toolkit has a preferred way to create custom components or containers. If you develop for iOS you extend the UIView class in order to create a custom UI component. These custom views would be the molecules and organisms of our design system. To combine them you add them to other views as their subviews. The init* or properties can be used to fill these with content. The actual mechanism is similar for most native UI kits.

Design with benefits

Using atomic design to create a design system seems to be a lot of work at first. And it is.
We already mentioned two benefits: creating a common understanding and a better way to test things in isolation. Design systems help all project participants, not only designers and developers, to share a common language and understand each other. They help new members to hit the ground running. With tools like pattern lab your atomic design can also be used as documentation.
On the testing front the holy grail is to test things in isolation and in integration, atomic design and its strict separation helps immensely. Often only the sunshine or ideal state is tested and maybe a handful of error states. Thinking in isolation of molecules and organisms about the whole five states and the diverse structure of your content creates a manageable endeavor and maps a path through the jungle of our interfaces. The value which atomic design brings to the table is that your efforts to test scale with the number of molecules and organisms and not with the number of pages or screens. The isolation which a design system, and in particular atomic design, creates is comparable to the advent of unit testing in the world of software development. The separation of display patterns and content patterns reminds me of the functional paradigm with its separation of data and functions.

Simple C++11 – Part II – Class declarations

In the previous part, I’ve shown my guidelines for setting up compilation units. When writing simple application code with C++11, either classes or free-functions should be your main building blocks. Therefor, in this part, I will focus on what to look out for when writing class declarations.

While templates can be very useful, they do not scale well as the code base gets larger. Metaprogramming or other niche styles have their places, too, but I like to look at those as a means to create language extensions rather than principal implementation tools.

Avoid inline implementations

…especially in header files. It can be tempting to write classes solely in the header file. In fact, it has almost become a sign of quality for parts of C++ code to be header only. But this scales badly in most cases, and evolving such a code-base will result in a dramatic explosion of compile times. Always splitting classes into a declaration and definition acts as a first-level compile- firewall and dependency-breaker. Users of your class no longer need to worry about changes in the implementation of the member functions of that class. Note that those changes are often indirect: a change only affects a class that is used in the implementation of your class’ member functions. By splitting the declaration and definition, users of your class do not have to be recompiled.

But why stop at the compiler? The same argument holds for programmers. If you start to split interface and implementation on this level, you automatically provide ‘reader-firewalls’ as well. By just providing a clean header file, you are giving readers sort of a manual for your class. No need to look at the implementation at all, if the interface is well-defined.

Inline code definition is also the main reason against excessive use of templates. Yes, they grant a lot of flexibility, but you pay a hefty price which needs to be justified by an enormous reduction of complexity elsewhere. In general, templates are a bit too powerful for their own good, which is why they need extra moderation.

Always declare implicit functions

Implicitly declared functions seem comfortable, but they have a few implications that are hard to understand. First of, if an implicit function gets generated for your class, it will be generated as inline. This means that the implementation becomes a dependency to all users of your class. This can have very subtle effects such as this:

#include <vector>
class Entry;

class EntryManager {
public:
  EntryManager(EntryGenerator& generator);
  int getEntryCount() const;
  std::string getIDForEntry(int index) const;
private:
  std::vector<Entry> mData;
};

On the surface, it looks like there should be no dependency (other than the name) on MyEntry when including this header. But there is!
The destructor is not declared so it will get generated – as inline. Because deletion of a vector requires the held type to be complete, any place that needs to be able to destruct a MyEntryManager also needs to know how to destruct MyEntry, which is not intended at all. Remember there’s a total of six functions that can be implicitly generated! Because of that, there are analogous problems for copy-construction, assignment, move-construction and move-assignment.

To avoid these problems, either delete the function explicitly in the header, default it in the implementation file, or actually implement it. You rarely need to do the latter, so I advise to default all the ones you need, and delete the rest:

#include <vector>
class Entry;

class EntryManager {
public:
  EntryManager(EntryGenerator& generator);
  EntryManager(EntryManager const&)=delete;
  EntryManager& operator=(EntryManager const&)=delete;
  EntryManager(EntryManager&& rhs);
  EntryManager& operator=(EntryManager&& rhs);
  ~EntryManager();
  int getEntryCount() const;
  std::string getIDForEntry(int index) const;
private:
  std::vector<MyEntry> mData;
};

And somewhere in the implementation file:

EntryManager::EntryManager(EntryManager&& rhs) = default;
EntryManager::~EntryManager() = default;
EntryManager& EntryManager::operator=(EntryManager&& rhs) = default;

This has another nice side effect because the vector-template gets instantiated into that object file and does not “bloat” all use-sites.

Exactly one public function and one private data section per class

..starting with the public section. This is where you address the next programmer that has to read your class. And it should be the only place for him to look.

I avoid private member functions because they cannot be tested easily and can add hidden compile-time dependencies to a project. Why should a user of your class recompile if you change an implementation detail? For small and trivial implementation helpers, the unnamed-namespace in the implementation file is a much better place. If those helpers become larger or more complex, it is a better idea to implement them in a collaborating class, which can be tested and reused.

Protected member functions split your interface to two parts, one exclusively for derived classes and one for everyone (including derived classes). This is very rarely needed, and in almost all of those cases, a separate interface will scale better (although it is slightly harder to implement).

Either an interface or an implementation

So far, I have left inheritance out of the picture and only talked about concrete classes. Inheritance is actually rarely needed, composition often suffices. But if it is needed, make sure that a class is either concrete and final (implementations), or has a complete and minimal set of pure-virtual member functions (interfaces). This will result in shallow hierarchies and easily understood interfaces. Remember that inheritance is not a tool for sharing code from the classes you implement, but for the code using those classes – i.e. where the Liskov Substition Principle holds.

Now it gets really easy to implement new classes in the hierarchy: Just implement all the functions in the interface. No more questioning whether to leave the default behaviour or override. You will also automatically tend towards clearer separation of components – things that need to be polymorphic move to the interface, other  functionality merely uses it.

This pattern is useful even when polymorphy is not needed. Such small interfaces devoid of any implementation detail can act as another compiler firewall. Collaborators can work with just the interface and do not have to be recompiled when the implementation changes. Also, the interface can be implemented for mock or fake objects in testing.

Conclusion

This concludes the second part of the series. I originally intended it to be about how to write a whole class, but that would have been too much to digest for one post. I am well aware that some of these guidelines can stir quite the controversy in the C++ community. For example, declaring the implicit functions seems to be in conflict with the recently popular rule of zero. Scott Meyers had similar concerns, but does not quite touch the inline aspect.

For me personally, these guidelines have helped tremendously, especially when scaling to bigger code-bases. But as before, I am curious what others are thinking about this!

Small is beautiful – CherryPy Microframework

Origami Elephant (Image used with kind permission of Anya Midori)

We are developing web applications for our clients using various different frameworks and technologies. The choice depends on several different factors like hosting, the clients IT infrastructure and administration and of course project scope. Some of our successful web projects use the Grails or Ruby on Rails with JRuby full stack frameworks. While they provide a ton of powerful features they also intrinsically carry some complexity. This is not always needed nor wanted, some projects might fare well with much less. Sometimes the scope of a project is not entirely clear so choosing a lean alternative you can gradually extend and refine may be the right fit.

 

Full stack frameworks

A full stack framework usually delivers built-in solutions for persistence/database access, domain modelling, routing, html-templating and so on. If you know for sure that you will need all the provided features from the start and that they are a good fit feel free to choose a full stack framework like Rails, Grails, Play or Django. If you need less or there is no convincing solution for your needs start with a minimal system that delivers value to your clients.

Microframeworks

Microframeworks provide only the basic functionality for routing and handling requests. No templating, no databases, no session management etc. attached. We made really good experiences with microframeworks like Nancy for .NET or CherryPy for Python. You start very simple and are up an running in minutes. If most of your GUI is client-side you do not need templating and you do not have it from the start. You do not need a database, well, you do not have one. Your server side logic revolves around REST, there you go!

The key point is that you are not stuck with this minimal set of support but you can easily extend the framework with features if need be. And usually you have several options per concern to choose from. For templating in Python there are many different solutions like Mako, Jinja2,  Genshi and others. Need only file-based persistence, want to manage your database in SQL or need an object-relational mapper (ORM) – everything is up to you.

CherryPy

Our experience with CherryPy was very positive. It is easy to run standalone using the integrated web server. You can also run it behind any WSGI-compatible web server because a CherryPy application automatically serves as a WSGI application. This is very nice if you need the power of a native web server and the ease and flexibility of a Python microframework. CherryPy is also very flexible when it comes to request routing offering much convenience with the default routing and method annotations to expose actions on the one hand and _cp_dispatch or MethodDispatcher on the other. It feels easy to extend the solution bit by bit as needed, there is seldom something in your way. Configuration is easy and powerful and the documentation gets you up to speed most of the time.

Conclusion

Microframeworks let you choose what you need and which implementation best fits your needs. You do not carry all that complexity with you from the start and easily pull more framework support in as you go choosing the most appropriate for your current situation. Your choices may differ in the next project since every project is different.

Beyond agile: building on the roots

Software development changes. Over a decade ago the agile manifesto adduced evidence that the document heavy processes needed to change.

Software development changes. Over a decade ago the agile manifesto adduced evidence that the document heavy processes needed to change.
Before that progress was measured by producing documents, a big planning and design phase was held at the start of every project to minimize the risk of producing the wrong software. Requirements were carved into stone. Changes later in the processes were shunned.
Agile changed all that. And besides losing some valuable practices (like documentation) recent developments call for a new form and focus of software development.
What changed? Software is now used by all kinds of people. This is not new. But design, user experience (UX), user focus and so on need to be an integral part of software development.

Business people and developers and designers must work
together daily throughout the project.

Nowadays developers need to interact not only with business people but with designers as well. Especially the waterfall like processes from the UX design or product management world struggle with the iterative and highly unplanned nature of agile.

Deliver working software value continuously frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Agile only thinks in iterations or sprints. Even this changed with continuous delivery and lean development. The focus is not so more on a plannable release schedule but on bringing value to the customer.

Our highest priority is to satisfy the customer and the users
through early and continuous delivery of valuable software.

With a user focussed process not only the customer but also the user needs to be satisfied. UX design has this focus.
But UX design needs a significant research (developers call this analysis), design and test phase up front. So in practice this is masked as a big sprint 0.
Unfortunately agile has no concept of planning, design or analysis. The agile philosophy concentrates on execution. This has lead to the notion that planning can be done in every iteration.

Simplicity–the art of maximizing the amount of work not done– Managing and understanding complexity–the art of finding the right places where to invest into details– is essential.

As little planning, design and analysis as possible. The famous pendulum swings the other way. In order to avoid the planning heavy processes from the past, agile ignored and many practitioners even shunned these practices. As many designers can tell us, you need a good foundation to start with. Research is needed. “But these requirements will change” I hear the cries of the past.

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage and the user’s delight.

A dilemma. On the one hand we need an initial investment for a good foundation. On the other hand we know that many things will change during the course of the project. We are stuck.

The best architectures, requirements, and designs solutions emerge from self-organizing teams.

One key principle that the UX design world has to offer for software development is the reasoning. Everything builds upon another thing. There is an incentive to find the cause, to construct a chain of whys. Requirements do not emerge from teams. They are grounded in user needs and business goals. These form the foundation of every user centered project.

At regular intervals All through the project, the team reflects on how to become more effective, then tunes and adjusts its behavior approach accordingly.

The agile philosophy has one core principle which helps to build a bridge between the worlds of development and design: continuous feedback.
In order to build reasoning you need to test your assumptions. UX has many practices to validate assumptions: interviewing, prototyping, analytics.
Feedback is a vital to a project and needs to be included in every action, not only the technical and measurable ones.
This feedback needs to be continuous. Regular intervals won’t cut it anymore.
We need to remove the notion of phases, iterations and sprints from our thinking. We have to define practices that help to build software that meets goals and satisfies needs. These build upon another by reasoning and on demand, not when a time plan demands it. We cannot freeze features for an iteration. Work needs to be organized around streams not iterations. These are not isolated.
Lean thinking tries to foster the notion of bringing constant value to the customers and users. Value streams.
A stream has no defined time frame or regular interval.

Working software is Goals met and satisfied needs are the primary measure of progress.

To bring design into the boat we need move the focus away from requirements and features to goals and needs. Streams of development need to identify, evaluate and build these jobs to be done, the goals. Designers bring the user needs to the table. Developers add the technical constraints. Together all these three factors, business goals, user needs and technical constraints, are balanced by the team.
All this is not new. And in practice many feedback steps or actions are omitted because of time, budget and other constraints. Information and insights are lost because of communication or documentation problems.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Teams should consist of developers and designers. Both need to be present in a kind of pair. Not pair programming. Pair everything. Decisions are made at every step. All three factors need to be considered.
Right now this seems to be pretty abstract and I think we should break it down.
What are the actions during a project?

Listen

A project should start with listening (but it doesn’t need to). And listening should never stop. Listen to the customer, to the users, to the systems, to the code.

Reflect

With all that information you get from listening you constantly need to filter, to reflect, to think. What is important? What not? What is an assumption? What has changed? What needs to be together? What needs to be separated? What forms a whole? What is missing? What is common? What is different? Are we moving towards the goals or away?
Ordering and prioritizing is one of the most important tools we have to bring sense to the mess.

Imagine

How can all this information be translated into software? Into a user interface. Into parts of the software architecture. How will it be deployed? What is the environment? For the user and for the system.
Design systems, guidelines for design and development need to be constructed.

Test

Is my thinking and imaging, the design, right? Does it meet the goal? Is it fast enough? Are the technical constraints met? Does it help the user? All of this needs to be tested. Whether you use the real software, prototypes or other means: you need to remove assumptions and find proof that your translations of information into software works.

Create

This should be obvious. You need to write the software and the documentation.

Ship

Without ever reaching the user, software is worthless. It needs to be shipped. Continuously.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

Every action happens all the time. There’s no right order of steps. This might sound chaotic.
But a project is more like the exploration of unknown terrain. You do not know beforehand what the terrain will be like. You will need to improvise. For this it is better to have practices and principles which help you recognize and react to change.
This change comes from questioning assumptions and furthering your understanding of the goals and needs. Change does not come out of nowhere. Be prepared to react to it.

Reality

One big problem from talking about development and design processes comes from the huge gap between ideal and real world. In the real world there’s not enough time, money, skills, people.

Agile processes promote sustainable development. The sponsors, developers, designers and users should be able to maintain a constant pace indefinitely.

Be pragmatic. You don’t need the newest tools. Go with universal tools: pen and paper, wiki and most importantly your head. In all this you need to think and you need to communicate. Thinking is hard and good communication is even harder. But they stand at the center of a successful project. Do not avoid meetings if they are needed for communication or decision making. Use what you have and build from there.

Continuous attention to technical excellence and good design thinking and communication enhances agility.

It is essential that you know some practices really well. Practices which help you with the actions: listening, reflecting, imagining, testing, creating, shipping.

(The quotes are taken from the principles behind the agile manifesto.)
(The idea for naming the first three actions came from Putting thought into things

Recap of the Schneide Dev Brunch 2015-12-13

If you couldn’t attend the Schneide Dev Brunch at 13th of December 2015, here is a summary of the main topics.

brunch64-borderedTwo weeks ago, we held another Schneide Dev Brunch, a regular brunch on the second sunday of every other (even) month, only that all attendees want to talk about software development and various other topics. So if you bring a software-related topic along with your food, everyone has something to share. The brunch was small this time, but with enough stuff to talk about. As usual, a lot of topics and chatter were exchanged. This recapitulation tries to highlight the main topics of the brunch, but cannot reiterate everything that was spoken. If you were there, you probably find this list inconclusive:

Company strategies

Our first topic was about the changes that happen in company culture once a certain threshold is overstepped. The founders lose touch with their own groundwork and then with their own employees. Compliance frameworks are installed and then enforced, even if the rules make no sense in specific cases. A new hierarchy layer, the middle management, springs into existence and is populated by people that never worked on the topic but make all the decisions. The brightest engineers are promoted to a management position and find themselves helpless and overburdened. Adopting a new technology or tool takes forever now. The whole company stalls technologically.

Sounds familiar? We discussed several cases of this dramaturgy and some ways around it. One possible remedy is to never grow big enough. Stay small, stay fast and stay agile. That’s the Schneide way.

Code analysis with jDeodorant

We devoted a lot of time on getting to know jDeodorant, an eclipse-based code smell detection tool for Java. We grabbed a real project and analyzed it with the tool. Well, this step alone took its time, because the plugin cannot be operated in an intuitive manner. It presents itself as a collection of student thesis work without overarching narrative and a clear disregard of expectation conformity. If several experienced eclipse users cannot figure out how a tool works despite having used similar tools for years, something is afoul. We got past the bad user experience by viewing several screencasts, the most noteworthy being a plain feature demonstration.

Once you figure out the handling, the tool helps you to find code smells or refactoring opportunities. In our case, most of the findings were false alarms or overly picky. But in two cases, the tool provided a clear hint on how to make the code better (both being feature envies). If the project would really benefit from the proposed refactorings is subject for discussion. The tool acts like a very assiduous colleague in a code review when every improvement gets rewarded.

We really don’t know how to rate this tool. It’s hard to learn and provides little value on first sight, but might be useful on larger legacy code bases. We’ll keep it at the back of our minds.

Naming and syntax rules

During the discussion about jDeodorant, we talked about naming schemes and other syntax rules. We remembered horrific conventions like prefixed I and E or suffixed Exception. The last one got some curious looks, because it’s still a convention in the Java SDK and some names won’t make it without, like the beloved IOException. But what about the NullPointerException? Wouldn’t NullPointer describe the problem just as good? Kevlin Henney already talked about this and other ineffective coding habits (if you have audio degradation halfway through, try another video of the talk). It’s a good eye-opener to (some of) the habits we’ve adopted without questioning them. But challenging the status quo is a good thing if done in reasonable doses and with a constructive attitude.

Unit testing

When we played around with jDeodorant and surfed the code of the project that served as our testing ground, the Infinitest widget raised some questions. So we talked about Continuous Testing, unit tests and some pitfalls if your tests aren’t blazing fast. The eclipse plugin for MoreUnit was mentioned soon. Those two plugins really make a difference in working with tests. Especially the unannounced shortcut Ctrl+J is very helpful. I’ve even blogged about the topic back in 2011.

Epilogue

As usual, the Dev Brunch contained a lot more chatter and talk than listed here. The number of attendees makes for an unique experience every time. We are looking forward to the next Dev Brunch at the Softwareschneiderei. And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

Renaming is hard work

Imagine you’ve developed a software solution for one of your customers, for example a custom CRM. Everything in this system revolves around the concept of a customer as the key entity. Of course, this is reflected in your code as well. The code is crowded with variable and class names like customerThis and CustomerThat. But now your customer wants you to change the term customer to client.

The minimal and lazy solution would be to change only those occurrences of the word, which are visible to the users. This includes:

  • Texts displayed in the user interface. If your application is internationalized or at least prepared for internationalization, they are usually neatly contained in translation files and can be easily updated.
  • User documentation like manuals. Don’t forget to update the screenshots.

However, if you do not want to let the code diverge from the language of the domain, you have to rename and update a lot more:

  • identifiers in the code (types, namespaces/packages, variables, constants)
  • source files and directories
  • code comments
  • log output strings
  • internationalization keys
  • text protocol strings (e.g. JSON keys) and HTTP API routes/parameters. Of course you can only do that if you are allowed to break the protocol! Don’t accidentaly break your API or formats, e.g. if JSON keys or XML tags are generated via reflection from code identifiers.
  • IDs of HTML tags in views, CSS class names and selector strings in referencing Javascript code
  • internal documentation (e.g. Wiki)

IDE support for renaming can help a lot. Especially the renaming of identifiers is reliable in statically typed languages. However, local variables, function parameters, and fields usually have to be renamed separately. In dynamically typed languages automated renaming of identifiers is often guesswork. Here you must rely on a good test coverage. Even in statically typed languages identifiers can be referenced via strings if reflection is used.

If you use a grep-like tool or the search feature of your IDE or editor to assist you, keep in mind that composite terms can occur in many different forms like FooBar, fooBar, foo-bar, foo_bar, foo.bar, foo bar or foobar. Don’t forget about irregular plural forms, e.g. technology – pl. technologies. Also think of abbreviations, otherwise a

for (Option opt : options)

might end up as

for (Alternative opt : alternatives)

But this is not the end of the story. A lot of renamings require migration scripts or update scripts to be executed at the time of the next update or deployment:

  • database tables and columns
  • some ORMs like Hibernate store class names in the database for the “table-per-hierarchy” mapping
  • string-based enum values in database entries
  • configuration keys and values
  • keys and values in data formats (JSON, XML, …) of stored data. You probably have to provide backward-compatibility for the old formats if the data files are not stored all in one place and can’t be converted in one go
  • names of data folders

If you don’t rename everything all at once, but decide to use the new term only in newly written code and to leave the old term in the existing code until it eventually gets replaced, you will probably end up with a long transition phase and lots of confusion for new project members who don’t know about the history of the project.

Simple C++11 – Part I – Unit Structure

C++ has long had the stigma of an overlay complex and unproductive language. Lately, with the advent of C++11, things have brightened a bit, but there are still a lot of misconceptions about the language. I think this is mostly because C++ was taught in a wrong way. This series aims to show my, hopefully somewhat simpler, way of using C++11.

Since it is typically the first thing I do when starting a new project, I will start with how I am setting up a new compile unit, e.g. a header and compile unit pair.

Note that I will try not to focus on a specific C++11 paradigm, such as object-oriented or imperative. This structure seems to work well for all kinds of paradigms. But without much further ado, here’s the header file for my imaginary “MyUnit” unit:

MyUnit.hpp

#pragma once

#include <vector>
#include "MyStuff.hpp"

namespace MyModule { namespace MyUnit {

/** Does something only a good bar could.
*/
std::vector<float> bar(int fooCount);

/** Foo is an integral part of any program.
    Be sure to call it frequently.
*/
void foo(MyStuff::BestType somethingGood);

}}

I prefer the .hpp file ending for headers. While I’m perfectly fine with .h, I think it is helpful to differentiate pure C headers from C++ headers.

#pragma once

I’m using #pragma once here instead of include guards. It is not an official part of the standard, but all the big compilers (Visual C++, g++ and clang) support it, making it a de-facto standard. Unlike include guards, you only have to add only one line, which says exactly what you want to achieve with it. You do not have to find a unique identifier for your include guard that will most certainly break if you rename the file/unit. It’s more readable, more resilient to change and easier to set up.

Namespaces

I like to have all the contents of a unit in a single namespace. The actual structure of the namespaces – i.e. per unit or per module or something else entirely depends on the specifics of the project, but filling more than one namespace is a guarantee for chaos. It’s usually a sign that the unit should be broken up into smaller pieces. An exception to this would be the infamous “detail” namespace, as seen in many of the Boost libraries. In that case, the namespace is not used to structure the API, but to explicitly omit things from the API that have to be visible for technical reasons.

Documentation

Documentation goes into the header, not into the implementation. The header describes the API, not only to the compiler, but also to humans. It is by no means an implementation detail, but part of the seam that isolates it from the rest of the code. Note that this part of the documentation concerns the API contract only, never the implementation. That part goes into the .cpp file.

But now to the implementation file:

MyUnit.cpp

#include "MyUnit.hpp"

#include "CoolFunctionality.hpp"

using namespace MyModule;
using namespace MyUnit;

namespace {

int helperFunction(float rhs)
{
  /* ... */
}

}// namespace

std::vector<float> MyUnit::bar(int fooCount)
{
  /* ... */
}

void MyUnit::foo(MyStuff::BestType somethingGood)
{
  /* ... */
}

Own #include first

The only rule I have for includes is that the unit’s own include is always the first. This is to test whether the header is self-sufficient, i.e. that it will compile without being in the context of other headers or, even worse, code from an implementation file. Some people like to order the rest of their includes according to their “origin”, e.g. sections for system headers or library headers. I think imposing any extra order here is not needed. If anything, I prefer not waste time sorting include directives and just append an include when I need it.

Using namespace

I choose using-directives of my unit’s namespaces over explicitly accessing the namespaces each time. Unlike the headers, the implementation file lives in a locally defined context. Therefore, it is not a problem to use a very specific view onto the unit. In fact, it would be a problem to be overly generic. The same argument also holds for other “local” modules that this unit is only using, as long as there are no collisions. I avoid using namespaces from external libraries to mark the library boundary (such as std, boost etc.).

Unnamed namespace

The unnamed namespace contains all the implementation helpers specific to this unit. It is quite common for this to contain a lot of the “meat” of an actual unit, while the unit’s visible functions merely wrap and canonize the functionality implemented here. I try to keep only one unnamed namespace in each file, to have a clear separation of what is supposed to be visible to the outside – and what is not.

Visible implementation

The implementation of the visible API of the module is the most obvious part of the .cpp file. For consistency reasons, the order of the functions should be the same as in the header.

I’d advice against implementing in a file wide open namespace. That means balancing an unnecessary pair of parenthesis over the whole implementation file.  Also, you can not only define functions and types, but also declare them – this leads to a function further down in the implementation to see a different namespace than one before it.

Conclusion

This concludes the first part. I’ve played with the thought of using a 3-piece setup instead, extending the header/implementation with a unit-test file, but have not gathered any sharable experience yet. This setup, however, has worked for me for a long time and with many different projects. Have you had similar – or completely different – setups that worked for you? Do tell!

Synchronizing asynchronous calls in JavaScript

In Node.js all I/O performing operations like HTTP requests, file access etc. are designed to be non-blocking. The functions for these operations usually take a callback function argument as last parameter, which will be called once the operation is finished. The asynchronous nature of these operations often requires special means of synchronization, as exemplified in this post.

Let’s assume we want to download a set of files from a list of URLs:

var urls = [
    'http://example.org/file1',
    'http://example.org/file2',
    'http://example.org/file3',
];

If we were to use a classic, synchronous blocking function for the download operation the implementation might look like this:

urls.forEach(function(url) {
    downloadSync(url);
    console.log('Downloaded ' + url);
});
console.log('Downloads complete.')

Output:

Downloaded http://example.org/file1
Downloaded http://example.org/file2
Downloaded http://example.org/file3
Downloads complete.

The program loops through the list of URLs and downloads one file after the other. Each subsequent download is only started after the previous download has finished. No two downloads are active at the same time. Finally, the program prints a message indicating that all downloads are complete.

Now we want to embrace the asynchronous programming style of Node.js. We have a different function, downloadAsync, which is a non-blocking asynchronous function. The function uses the callback convention of Node.js to let the programmer handle the completion of the asynchronous operation:

urls.forEach(function(url) {
    downloadAsync(url, function() {
        console.log('Downloaded ' + url);
    });
});
console.log('Downloads complete.')

Output:

Downloads complete.
Downloaded http://example.org/file2
Downloaded http://example.org/file3
Downloaded http://example.org/file1

The download operations are now started at the same time and finish in a different order, depending on how long it takes each file to download. The quickest download finishes first, the slowest last. The total time for all downloads to finish is probably shorter than before, because slower connections do not make faster connections wait for them to finish.

However, there is one problem with this code: the message “Downloads complete.” is shown before the downloads are actually completed, which is obviously not the intended behavior. The reason why this happens is because the control flow immediately continues after the forEach loop has started all the downloads.

The ‘async’ module

What we need to correct this behavior is a way to wait for all asynchronous download operations started by the loop to finish before we print the “Downloads complete” message.

The async module for JavaScript provides various synchronization mechanisms for exactly these kinds of situations. In our case we’re going to use async.each:

var async = require('async');

async.each(urls, function(url, callback) {
    downloadAsync(url, function() {
        console.log('Downloaded ' + url);
        callback();
    });
}, function done() {
    console.log('Downloads complete.')
});

The async.each function is similar to JavaScript’s forEach function. However, the function called for each iteration has an additional callback parameter, which must be called when each iteration is considered to be completed. The third parameter to async.each is also a callback function. This function is called after all iterations reported their completion. This is where we place the output of the “Downloads complete” message.

Conclusion

The async module provides a rich toolset for the synchronization of asynchronous operations in JavaScript. Go check it out!

Multi-Page TIFFs with C++

If you are dealing with high-speed cameras or other imaging equipment capable of producing many images in a short time you may find it handy to put many images into a single file. There are several reasons to do so:

  • Dealing with thousands of files in a single directory or spreading them over a directory hierarchy may be slow and cumbersome depending on the tools.
  • Storing many images together may communicate better that they belong together, e.g. to the same scan.
  • Handling and transmitting fewer files is often easier than juggling with many.

TIFF is a wide-spread lossless image format capable of handling many individual images in a single file. Many image viewers and image manipulation tools are able to work with multi-page TIFF files so you are quite flexible in working with such files.

But how do you produce these files from your programs? I found some solutions with different strengths and weaknesses:

Using Magick++

Magick++ is the C++ API for ImageMagick – a powerful image manipulation library. If you can hold all the images for one file in memory the code is easy and straightforward:

#include &amp;lt;string&amp;gt;
#include &amp;lt;Magick++.h&amp;gt;

class TiffWriter
{
public:
    TiffWriter(std::string filename);
    TiffWriter(const TiffWriter&amp;amp;) = delete;
    TiffWriter&amp;amp; operator=(const TiffWriter&amp;amp;) = delete;
    ~TiffWriter();

    void write(const unsigned char* buffer, int width, int height);

private:
    std::vector&amp;lt;Magick::Image&amp;gt; imageList;
    std::string filename;
};

TiffWriter::TiffWriter(std::string filename) : filename(filename) {}

// for example for a 8 bit gray image buffer
void TiffWriter::write(const unsigned char* buffer, int width, int height)
{
    Magick::Blob gray8Blob(buffer, width * height);
    Magick::Geometry size(width, height);
    Magick::Image gray8Image(gray8Blob, size, 8, "GRAY");
    imageList.push_back(gray8Image);
}

TiffWriter::~TiffWriter()
{
    Magick::writeImages(imageList.begin(), imageList.end(), filename);
}



The caveat is that you need to  hold all your images in memory before writing it to the file on disk. I did not manage to add and persist images on the fly to disk.


In our environment it was absolutely necessary to do so because of the amount of data and the I/O required to persist all image in time. So I had to implement a slightly more low-level solution using libtiff and its C API.


Using libtiff


#include &amp;lt;string&amp;gt;
#include &amp;lt;tiffio.h&amp;gt;

class TiffWriter
{
public:
    TiffWriter(std::string filename, bool multiPage);
    TiffWriter(const TiffWriter&amp;amp;) = delete;
    TiffWriter&amp;amp; operator=(const TiffWriter&amp;amp;) = delete;
    ~TiffWriter();

    void write(const unsigned char* buffer, int width, int height);

private:
    TIFF* tiff;
    bool multiPage;
    unsigned int page;
};

TiffWriter::TiffWriter(std::string filename, bool multiPage) : page(0), multiPage(multiPage)
{
    tiff = TIFFOpen(filename.c_str(), "w");
}

void TiffWriter::write(const unsigned char* buffer, int width, int height)
{
    if (multiPage) {
        /*
         * I seriously don't know if this is supposed to be supported by the format,
         * but it's the only we way can write the page number without knowing the
         * final number of pages in advance.
         */
        TIFFSetField(tiff, TIFFTAG_PAGENUMBER, page, page);
        TIFFSetField(tiff, TIFFTAG_SUBFILETYPE, FILETYPE_PAGE);
    }
    TIFFSetField(tiff, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
    TIFFSetField(tiff, TIFFTAG_IMAGEWIDTH, width);
    TIFFSetField(tiff, TIFFTAG_IMAGELENGTH, height);
    TIFFSetField(tiff, TIFFTAG_SAMPLEFORMAT, SAMPLEFORMAT_UINT);
    TIFFSetField(tiff, TIFFTAG_ROWSPERSTRIP, TIFFDefaultStripSize(tiff, (unsigned int) - 1));

    unsigned int samples_per_pixel = 1;
    unsigned int bits_per_sample = 8;
    TIFFSetField(tiff, TIFFTAG_BITSPERSAMPLE, bits_per_sample);
    TIFFSetField(tiff, TIFFTAG_SAMPLESPERPIXEL, samples_per_pixel);

    std::size_t stride = width;
    for (unsigned int y = 0; y &amp;lt; height; ++y) {
        TIFFWriteScanline(tiff, buffer + y * stride, y, 0);
    }

    TIFFWriteDirectory(tiff);
    page++;
}

TiffWriter::~TiffWriter()
{
    TIFFClose(tiff);
}



Note that line 14 is needed if you do not know the number of images to store in the file in advance!