Month: April 2022

Reading a conanfile.txt from a conanfile.py

I am currently working on a project that embeds another library into its own source tree via git submodules. This is currently convenient because the library’s development is very much tied to the host project and having them both in the same CMake project cuts down dramatically on iteration times. Yet, that library already has its own conan dependencies in a conanfile.txt. Because I did not want to duplicate the dependency information from the library, I decided to pull those into my host projects requirements programmatically using a conanfile.py.

Luckily, you can use conan’s own tools for that:

from conans.client.loader import ConanFileTextLoader

def load_library_conan(recipe_folder):
    text = Path(os.path.join(recipe_folder, "libary_folder", "conanfile.txt")).read_text()
    return ConanFileTextLoader(text)

You can then use that in your stage methods, e.g.:

    def config_options(self):
        for line in load_library_conan(self.recipe_folder).options.splitlines():
            (key, value) = line.split("=", 2)
            (library, option) = key.split(":", 2)
            setattr(self.options[library], option, value)

    def requirements(self):
        for x in load_library_conan(self.recipe_folder).requirements:
            self.requires(x)

I realize this is a niche application, but it helped me very much. It would be cool if conan could delegate into subfolders natively, but I did not find a better way to do this.

Full-text Search with PostgreSQL

If you want to add simple text search functionality to an application backed by an SQL database one of the first things that may come to your mind is the SQL LIKE operator. The LIKE operator and its case-insensitive sibling ILIKE find substrings in text data via wildcards such as %, which matches any sequence of zero or more characters:

SELECT * FROM book WHERE title ILIKE '%dog%'.

However, this approach satisfies only very basic requirements for text search, because it only matches exact substrings. That’s why application developers often use an external search engine like Elasticsearch based on the Apache Lucene library.

With a PostgreSQL database there is another option: it comes with a built-in full-text search. A full-text search analyzes text according to the language of the text, parses it into tokens and converts them into so-called lexemes. These are strings, just like tokens, but they have been normalized so that different forms of the same word, for example “pony” and “ponies”, are made alike. Additionally, stop words are eliminated, which are words that are so common that they are useless for searching, like “a” or “the”. For this purpose the search engine uses a dictionary of the target language.

In PostgreSQL, there are two main functions to perform full-text search: they are to_tsvector and to_tsquery. The ts part in the function names stands for “text search”. The to_tsvector function breaks up the input string and creates a vector of lexemes out of it, which are then used to perform full-text search using the to_tsquery function. The two functions can be combined with the @@ (match) operator, which applies a search query to a search vector:

SELECT title
  FROM book
  WHERE to_tsvector(title) @@ to_tsquery('(cat | dog) & pony')

The query syntax of ts_query supports boolean operators like | (or), & (and), ! (not) and grouping using parentheses, but also other operators like and <-> (“followed by”) and * (prefix matching).

You can specify the target language as a parameter of to_tsvector:

# SELECT to_tsvector('english', 'Thousands of ponies were grazing on the prairie.');

'graze':5 'poni':3 'prairi':8 'thousand':1

Here’s another example in German:

# SELECT to_tsvector('german', 'Wer einen Fehler begeht, und ihn nicht korrigiert, begeht einen zweiten (Konfuzius)');

'begeht':4,9 'fehl':3 'konfuzius':12 'korrigiert':8 'wer':1 'zweit':11

PostgreSQL supports dictionaries for about 80+ languages out-of-the-box.

The examples in this article are just a small glimpse of what is possible with regards to full-text search in PostgreSQL. If you want to learn more you should consult the documentation. The key takeaway is that there is another option between simple LIKE clauses and an external search engine.

Effective computer names with DNS aliases

If you have a computer in a network, it has a lot of different names and addresses. Most of them are chosen by the manufacturer, like the MAC address of the network device. Some are chosen by you, like the IP address in the local network. And some need to be chosen by you, like the computer’s name in your local DNS (domain name service).

A typical indicator for an under-managed network is the lack of sufficiently obvious computer names in it. You want to connect to the printer? 192.168.0.77 it is. You need to access the network drive? It is reachable under nas-producer-123.local. You can be sure that either of these names change as soon as anything gets modified in the network.

Not every computer in a network needs a never-changing, obvious name. If you connect a notebook for some hours, it can be addressable only by 192.168.0.151 and nobody cares. But there will be computers and similar network devices like printers that stay longer and provide services to others. These are the machines that require a proper name, and probably not only one.

Our approach is a layered one, with four layers:

  • MAC-address, chosen by the manufacturer
  • IP address, chosen by our DHCP
  • Device name, chosen by our DNS
  • Device aliases, chosen by our DNS

Of course, our DHCP and our DNS is told by our administrator what addresses and names to give out. Our IP addresses are partitioned into sections, but that is not relevant to the users.

The device name is a mapping of a name on an IP address. It is chosen by the administrator in case of a server/service machine. It will tell you about the primary service, like “printer0”, “printer1” or “nas0”. It is not a creative name and should not be remembered or used directly. If the machine has a direct user, like a workstation or a notebook, the user gets to choose the name. The only guideline is to keep it short, the rest is personal preference. This name should only be remembered by the user.

On top of the device name, each machine gets one or several additional DNS names, in the form of DNS aliases (CNAME records). These are the names we work with directly and should be remembered. Let’s see some examples:

I want to print on the laser printer: “laserprinter.local” is the correct address. It is an alias to printer0.local which is a mapping to 192.168.0.77 which resolves to a specific MAC address. If the laser printer gets replaced, every entry in this chain will probably change, except for one: the alias will point to the new printer and I don’t have to care much about it (maybe I need to update my driver).

I want to access the network drive: “nas.local” is one possibility. “networkdrive.local” is another one. Both point to “nas0” today and maybe “nas1” tomorrow. I don’t need to care which computer provides the service, because the service alias always points to the correct machine.

I want to connect to my colleague’s workstation: Because we have different naming preferences, I cannot remember that computer’s name. But I also don’t have to, because the computer has an alias: If my colleague’s name is “Joe”, the computer’s alias is “joe.local”, which resolves to his “totallywhackname.local”, which points to the IP address, etc. There is probably no more obvious DNS name than “joe.local”.

Another thing that we do is give a service its purpose as a name. This blog is run by wordpress, so we would have “wordpress.local”, but also “blog.local” which is the correct address to use if you want to access the blog. Should we eventually migrate our blog to another service, the “blog.local” address would point to it, while the “wordpress.local” address would still point to the old blog. The purpose doesn’t change, while the product that provides it might some day.

Of course, maintaining such a rich ecosystem of names and aliases is a lot of work. We don’t type our zone files directly, we use generators that supply us with the required level of comfort and clarity. This is done by one of our internal tools (if you remember the Sunzu blog post, you now know 2 out of our 53 tools). In short, we maintain a table in our wiki, listing all IP addresses and their DNS aliases and linking to the computer’s detail wiki page. From there, the tool scrapes the computer’s name and MAC address and generates configuration files for both the DHCP and DNS services. We can define our whole network in the wiki and have the tool generate the actual settings for us.

That way, the extra effort for the DNS aliases is negligible, while the positive effects are noticeable. Most network modifications can be done without much reconfiguration of dependent services or machines. And it all starts with alias names for your computers.

Applying the KonMari method to your IT supplies room

Our company is rather small, with less than ten people working in one big room on two floors (yes, the room is divisioned vertically, not horizontally). There are a few additional rooms, like a bathroom or a kitchen, but everything else has to find a place in our working space.

There are two exceptions to this rule:

  • A small room holds all cleaning utilities
  • A bigger room holds all things IT, like our servers and our IT supplies

None of these rooms “spark joy”, as Marie Kondo would describe them. You open the door, search around while ignoring the mess, grab the thing you came for and close the door again. When it is time to put the thing back, you more or less place it where you’ve found it. The state of these rooms is slow deterioration, because it can only get worse, but not better.

The situation became unfortunate for the IT room, because it contained far more things than storage space. Cables piled up on shelves, harddisks lingered on tables at specific locations that probably indicated something. A huge collection of CDs and DVDs waited in boxes for a second installation – most of our computers don’t even have a drive for them anymore. Every drawer contained some kind of main theme (manuals, adapters, cables), but a lot of surprises, too. The time it took to find something only went up and most of the time, it was cheaper to just buy the device (again) than search for it. And if you don’t use it anymore? Put it in the IT room.

A few years back, the KonMari method of cleaning up and organizing things was promoted by Marie Kondo. It is intended for your wardrobe and kitchen, but the guidelines can also be applied to your toolshed – and your IT room:

  • Not keeping a thing is the default
  • Concentrate on only keeping useful things (things that you use regularly or that make you happy)
  • If you keep a thing, it needs a dedicated place
  • Dedicate places by “category” and don’t deviate from your categorization
  • Provide a container for each category
  • Try to stack upright in horizontal direction, not vertically

The last guideline was really eye-opening for me: Every time I dedicated a box for things, like software CDs, the stacks grew upwards. This means that “lower layers” aren’t in direct access anymore and tend to be forgotten. If you dig to the ground of the box, you find copies of obscure software like “Windows 2000” or “Nero burning rom” that you’ve not thought about in ten years or even longer.

At the bottom of our cables box, we found a dozen cables for the parallel port, an interface that was forgotten the minute USB came around in 1996. The company was founded in 2000 and we never owned a device that used this port. We also found disks for the zip 100 drive, which might have used it – we don’t remember.

These things spark nostalgia (something else than joy), but serve no practical purpose anymore. And even if somebody came around with a zip disk, we wouldn’t remember that we have the cables at the bottom of our box.

If you try to stack your things upright, everything is visible and in fast access. There is no bottom layer anymore. Applied to CDs, this means that every CD case’s spine is readable. Every CD that you want to keep needs to be in a labled case. The infamous mainboard driver CD in a paper box with drivers from 2002 for a mainboard you scrapped in 2009 has no place in this collection.

The fitting categorization of things is the most important part of the process, in my opinion. Let me explain it by a paradigm shift that made all the difference for me:

In the early days our categories were like manual, CD, cable, screw, etc. Everytime a new computer was bought, the accompanying utilities box (often the mainboard carton) got looted for these categories – manuals to the manuals, CDs to the CDs. It was easy to find the place where the CDs were stored, but hard to find the right CD.

Now, we provide a small carton for each computer and put everything related to it in this carton. It is labeled with the computer’s number and stored like a book on the shelf. If you search anything for this computer – a CD, a screw, whatever – it is in this carton. If we get rid of the computer, the carton follows suit.

We now categorize by device and not by item type. This means that the collection of 10,000 screws that were collected over the years can be discarded. They simply aren’t needed anymore. They never sparked joy.

Another topic are the cables. While most cables can be associated with a computer or a specific device, there are lots of cables that are “unbound”. Instead of lumping them all together (and forming the aforementioned layers of parallel, serial and USB1 cables), we sort them by main connector and dedicate a box for this connection type. If you search a DisplayPort cable, you grab the DisplayPort box. If you require a VGA cable – well, we’ve thrown this specific box out last year. Look in the “exotic” box.

Each box is visible and clearly labeled. Inside each box are only things that you would expect. This means that there is a lot of boxed air. But it also means that you have to think about what to store and what not – simply because the number of boxes is limited.

And this is where “sparking joy” comes into play. The IT room is not an archive for all things digital. It is also not a graveyard for discard electronics. If you can’t see yourself using the part in the future and having joy using it, don’t keep it.

We have a box labeled “random loot” that defies this filter. It contains things that we can’t categorize, don’t have an immediate use case for, but hesitate to throw away. Every household has a similar thing with “that drawer”. Our plan is to add a year label to the box and just throw it away unopened if it is older than X years.

We need to evolve the categories of the room to keep it useful. An example are USB cables that are all stored in one cable box. With USB-C on the rise, the need to separate into different USB “layers” became apparent. We will soon have at least two USB cable boxes. And perhaps, one day in the future, we might throw the non-USB-C box away.

The IT room was transformed from a frustrating mess to a living and evolving storage space that solves your concern in an efficient way. The typical use cases of the room are adressed right away, with a structure that is maintainable without too much effort.

The inspiration and guidelines of Marie Kondo and the thoughts about proper categorization helped us to have an IT room that actually sparks joy.

Forced Acronyms are not that S.M.A.R.T.

A while back, I noticed that quite a lot of people are following that trend to unify a bunch of talking points to a more or less memorizable acronym. Sometimes, this is a great mnemonic device to make the essence of a thing clear in seconds – but for some reason, there are few stories acknowledged in which such attempts actually fail.

However, one of the most prominent acronyms in project management is the idea of S.M.A.R.T. goals. That easily dissolves into S for Specific, M for Measureable, and… hm… T is… something about Time, and then there are A and R, and they very clearly… well well. let’s consult wikipedia… span up a multidimensional vector space out of {Achievable, Attainable, Assignable, Agreed, Action-oriented, Ambitious, Aligned with corporate goals, Realistic, Resourced, Reasonable, Results-based}.

Now this is the point where it’s hard to follow. These are somehow too much possibilities, with no clear assignment. There are probably lots of people out there with their very specific memorization and their very specific interpretation of these letters; and it might very well be true that this forced acronym holds some value. In their specific case.

But why shouldn’t we be honest about it? If you have such a situation, you are not communicating clearly anymore. You have gone beyond that point. There is not a clear, concise meaning anymore.

These are the points where you would be honest to leave your brilliant acronym behind. If you ever sit in a seminar where someone wants to teach you some “easily memorizable acronym” with lots of degrees of freedom, open to interpretation and obviously changing over time, just – complain. Of course, everyone is entitled to using their own memory hook (“Eselsbrücke”) in order to remember whatever his or her goal is. That is not my point.

My Issue is with “official” acronyms that are not clear and constant. We as software developers have a responsibility to treat such inconsistencies as very dangerous and more harmful than helpful. With this post, I want to bring the idea out there that one should rather more often complain about a bad acronym than just think “weeeeell, but I really like how it sounds and I don’t care that it’s somewhat tainted.”

Or am I completely bullheaded in that regard? What is your opinion?

PS: If you are German and remember the beginning of 2021, a similar laziness happened there when our government tried to make their Covid rules clear and well-known. Note that this remark does have nothing to do with politics. Anyway: they invented this acronym of “AHA” (which, in German, is also that sound of having a light bulb appear over your head.) Not that bad of an idea. However, one of that “A”s originally meant “you just need a non-medical mask (Alltagsmaske) everywhere” – until some day, it was changed to “you need a medical face mask in everyday life (im Alltag)”. They just thought it clever to keep the acronym, but change one letter to mean its near opposite.

This is dangerous. Grossly negilent. Just for the sake of liking your old acronym too much, you needlessly fails to communicate clearly. Which is, for a government as much as for a software developer, usually your job.

Naming things 😉