Author: schneide

On the usefulness of volatile memory

Observing ants is a wonderful source of insights. Here are some thoughts about memory and information storage gathered from ants.

I have a strange hobby: Raising ants as pets. Not literally pets, because I don’t give them names and they probably don’t know I even exist, but as tended animals in a controlled and restricted environment, the formicarium. This doesn’t sound too strange if you know three additional things about me: I was fascinated by ants since kindergarten, I play dwarf fortress for fun and I’m interested in computers.

My main colony is a lasius niger nest with about 1500 individuals. Lasius niger or black garden ant is a very common and easy to raise ant that is small enough to not require too much space but big enough to be observed and tracked.

I could tell you hours and hours of ant facts, but let’s concentrate on the topic of this blog post: An ant colony can be perceived as one big organism. Each individual ant has a clear role in the hierarchy:

  • The queen ant is the sole egg layer. In an established colony, she won’t do anything else. She will be fed, cleaned and protected by her workers. She was born a queen and cannot be replaced. If she dies, the whole colony will slowly fade away because nobody produces new ants anymore. Worker ants don’t know if they still have a queen and don’t care anyway. If a queen dies, she will be fed to the remaining larvas as soon as her scent disappears.
  • The male ants are born, fed and kicked out of the nest, preferably when the young queens fly out to find a mate. They won’t do anything else and have a lifespan of days.
  • The worker ants are all sterile females and do all the work. Literally all the work. They are born workers and spend every moment of their lives contributing to the hive or just sitting around scrounging food. Other workers are too busy to judge them.

But keep in mind that ants only have short-range communication means (scent, antennae drumming and ground vibrations), so no single ant has complete overview about the situation. It wouldn’t have the brains to process that information anyway. Ants have limited capability to remember things but no long-time memory. It isn’t necessary for the single worker ant to store any information. The information is stored in the hive – literally.

If you abstract some details away, you can also perceive and describe an ant colony as a computer or at least as a massive parallel problem solver. The algorithms are ingrained in the worker ants and are executed ruthlessly. The problems are food, water, enemies, cleaning and nurture. If the environment (the problem space) is suitable for the algorithms, the colony will thrive. Else the colony will ultimately fail. An ant colony at the side of a busy road will lose many workers in sudden “enemy” attacks, while a colony in the middle of a meadow might find less insects killed by cars. Not a single ant in each colony is aware of these relations. But they have found a way to share information: They mark their position (and therefore their way) by special scent fluid. They use their scents to label the environment for other ants.

If you ever encountered an avid user of printed sticker labels, this is what ants do, too: They put a sticker label on everything they come in contact with. If you seem to be food, they label you as food and rush back to the hive, laying out a “food in this direction” lane. If you seem dead and inedible, they mark you as garbage and them or some other ant will pick you up and follow the garbage line (finding the garbage line often requires them to return near their hive so it seems they mistook the garbage for food). The garbage area is marked with its own scent, preferably at a cliff. My ants love to throw things down a cliff! If you didn’t get the relationship to dwarf fortress yet, it should be clear by now. There are countless more examples, but you get the idea. The environment is not only an area, it is the long-term memory of the colony. The colony’s “memory brain” is scent on dirt, stones and sticks. The ant colony has outsourced its complete knowledge about the surrounding into the surrounding itself.

The ants have invented something I would call an “inverse cloud”. The cloud in IT is a concept of data storage that exists mostly independent of physical location and provides access to that data from virtually anywhere. The ants’ inverse cloud is a concept of data storage that is tightly coupled to a physical location and provides access to that data only if you are in the immediate vincinity. If you remove a physical data storage part in the IT cloud, it gets replaced by other parts that contain mirrored copies of the data. The cloud never forgets. If you remove a physical data storage part in the ants’ inverse cloud, it is forgotten immediately. Ants accept their environment as it is right now and never look back.

Now think about what happens when it rains and all the scents are washed away.

After every rain, the ant colony enters a “new level”, a fresh environment to be discovered and labeled. They probably never grow old to rediscover the same cliff again and again. This is what happens to a computer when the power is lost. It loses its working memory. But keep in mind that the colony retains some memories: the hive is underground and often rain-proof by overlaying stones or plants. So the colony remembers that it is currently in the hive, because it always smells like hive. The colony remembers the queen chamber because it always smells like queen. The colony remembers the brood chambers because they always smell like teen spirit (SCNR).

In my formicarium, it never rains. The ants get enough water by drinking troughs, but the marked lanes are never erased. This leads to all sorts of silly situations that the ants don’t even recognize as such. For example, a strong “food here” scent lane exists long after the food is gone. So a lot of enthusiastic workers run around searching in the target area. And remember that the hive smell never fades? My ants have assimilated area after area as “hive” after enough ants have marked it. So now they react excessively to disturbances because they think they are defending the hive – and therefore the queen! – but are ant miles away from the colony. Even better, a lot of young ants that usually never leave the hive until they are older wander out into the open (“it’s still the hive, just less dark”) and panic as soon as they encounter something that shouldn’t be in a hive. The panic spreads by, you’ve guessed it already, alarm scent and soon hundreds of battle-ready ants are running around frantically without any one of them knowing why.

So, to speak in computer terms, the main memory never gets erased and the caches never flushed. A little error can spread like a wildfire (like the panic example above) and cause disadvantages like energy consumption without any real gain (no enemy to defend against) or a lot of delicate ants sitting around in plain sight of their predators. The whole system is fragile and erratic and probably wouldn’t survive in the wild. A good measure of rain would remove the odd memories and probably ease the ants because their hive, the area that needs to be defended at all costs, would shrink again.

I’ve raised neurotic ants in need of a cold shower.

How does this give us any insight into modern computing? Well, my train of thoughts is this: If modern systems are unable to forget, because memory is cheap and permanent, we might be prone to design software that acts neurotic and hyped up. The ability to forget, to really don’t remember at all, might be crucial in designing resilient parallel systems. There is the cost of losing valueable information, but the benefit of losing all results of errors seems to match it. So volatile memory might be a nuisance for us programmers, but it also provides a “blank slate” every time the system starts and is the reason for the most important question in IT: “Have you tried turning it off and on again?”

Systems that rely on “place oriented programming”  seem to have the need of regular reset phases where the working memory is cleared and the system goes into the next cycle fresh and rested. We might even call it sleep. And in case you wonder: The sleep of ants is an ongoing topic for research.

Disclaimer: I know that not all ants are as dumb as lasius niger. Some ants even teach each other facts about their environment. The wikipedia article mentions some wonderful examples. I had ant colonies with more complex ants and they were wonderful. But right now, as I’m typing this, there is a lasius niger worker that heaves a wasp husk part to the top corner of the formicarium, throws it down (did I mention they love throwing things?) and runs back down to heave it up again, probably because the corner is somehow marked as a garbage dump zone. It has repeated this process at least half a dozen times now. This is how a biological infinite loop looks like. Some ants even parallelize such a loop to exhaustion:

The Four Steps of Complex Tasks (Part I)

Trying to succeed in a complex task without solid experience is a challenging endeavour. A simple framework with four steps can help you with it.

In software development, we often face complex tasks that need to be addressed with nearly certain success rates. “Failure is not an option” is an easy sentence to say, but hard to implement. As experienced software developers, we’ve learnt our share from failures or near-misses and have probably derived an implicit problem-solving framework for our typical tasks. But what if the tasks are not so typical? What if we need to do complex work where we lack experience – and know it?

Adopting a problem-solving framework

In order to keep this blog entry as practical as possible, lets tell the theory with the help of an example that will be foreign to most software developers: You need to participate in a call for proposals for a big software project. You are responsible for the complete process from downloading and reading all documents for the call for proposals, making sense of the requirements, making a plan and an estimate, writing everything up and delivering your proposal in time and form at the right address. You cannot rely on the help of more experienced colleagues. You will probably feel overwhelmed and a bit helpless.

Now is the time to look for proven problem-solving frameworks that will at least help you coordinate your work and remind you about necessary steps that might go by the board otherwise. There are probably numerous frameworks out there, but I’ve found one to be easily applicable and reliable in the outcome. It’s the four steps of warfare. “Warfare” is a heavy word that isn’t in the dictionary of most people. It really means “to wage war” on somebody or something. But it doesn’t imply death, injury or even a “loser”. If you can wrap your head around the following sentence, we can easily forget its ancestry and follow the steps without any bloodshed: We are about to wage war on the call for proposals.

Being on a mission

We are now on a mission! Every mission starts with the most important piece of information: The mission statement. In this statement, the primary goals of the mission need to be declared as clear and unambiguous as possible. In our example, the mission statement might read like “I want to deliver a complete, believable and feasible proposal for the software project while adhering to all formalities”. It is worthwhile to write the mission statement down and keep it visible. This is your goal, this is why you wage war. Anything you do during the mission that doesn’t help you further your case in regard to the mission statement can be called a distraction (or in agile speak: “waste”).

Analyzing our example’s mission statement, we can conclude that we have two fields of work to cover: the content of the proposal and the formalities of the call. Both need to be covered successfully in order to achieve mission success. It is not enough to have a formidable proposal but send it in too late or without certain required characteristics. And it doesn’t do good to send something awful in that fulfills all technical requirements of the proposal but stands no chance because of its content. You’ve just discovered that your war will be fought on two fronts (content and formalities) and neglecting either one of them will not result in victory. That is an important discovery for the moment you employ your “troops” – time, attention, effort and material in our case.

The four steps

But slow down! We are far away from employing our troops! We first need to learn about the four steps of warfare. With our mission statement at hand, we have to make a plan for these steps:

  • Reconnaissance
  • Maneuver
  • Offensive
  • Defensive

Nearly every military mission can be broken down into these four steps in that order. We cover the first step in this blog post and the following three in the second part next week:

Step one: Reconnaissance

hunter looking through binocularsEvery warfare mission has important things at stake. There would be no mission otherwise, because each mission is expensive. With something big at stake, you want to reduce risk as much as possible. The military way to reduce risk is to gather knowledge about the enemy, to “increase intelligence”. Our “enemy” is the call for proposals with its deadline and rules and requirements. It is an easy enemy in a way, because all information is already made available in clear form and will not change over time. Or is it? Our first move should be to gather and revise all documents that tell us something about the call for proposals. Are we in possession of all available information? Nothing will break our neck more effectively than forgetting to read all documents. The sentence “I didn’t know that!” might sound like a good explanation why the mission failed, but it only indicates that you already failed in the reconnaissance step. Everything you do after a botched reconnaissance has a high risk of being ineffective, useless or downright counter-productive. You need to be sure to survey the complete terrain (read all the documents). A general that says “we don’t need to look behind this mountain, there will be nothing behind it” is in danger of failure as sure as you are if you say “that’s probably all information there is, what else should one want from me?”. Don’t underestimate your enemy! Just because you can download and read everything in regard to your call of proposals doesn’t mean it can be found at one place or will be conveniently available. Double-check that you’ve followed each hint about additional rules (like in general terms and conditions). Read all documents carefully!

When you’ve gathered a complete map of the terrain (have all rules and requirements in readable form), it’s time to mark the problematic features. Make a list of all formalities you need to adhere to. This list is already part of your battle plan. You will later need to employ resources to check off each point of this list. Mark all requirements that you find problematic or surprising. Those are like the strongholds on a battlefield. They need extra attention from you later on. Take your list of simple tasks and tack it on your mission board. If you didn’t have one yet, you’ll have it now. The mission board will look exactly like in the movies when you complete your first step. On top goes the mission statement, with your task list in whatever fashion suits you beneath it (some prefer the ordered list, some might draw a graph, whatever works for you). The mission board is for you and you only, nobody else needs to understand it. You’ve just created your headquarter! Now you have all ingredients to assemble your troops on the battlefield: you know the terrain, where the enemy resides, where you are and what your mission is.

Now is the time to get moving – in part two of this blog entry series next week. Stay tuned!

We are changing our locale

This is a bilingual posting:

We decided to continue this blog in english language, though we are no way of native speakers. Old postings may be translated in the future, if worth the effort.

The Softwareschneiderei remains a german-speaking company.

—-

Dies ist ein zweisprachiger Eintrag:

Wir haben uns entschieden, diesen Blog auf englisch weiterzuführen, obwohl wir keine englischen Muttersprachler sind. Alte Einträge könnten zukünftig übersetzt werden, wenn sie den Aufwand wert sind.

Die Softwareschneiderei bleibt eine deutschsprachige Firma.