# What is it with Software Development and all the clues to manage things?

As someone who started programming a long time ago (roughly 20 years, now that I think about it), but only in recent years entered the world of real software development, the mastery of day-to-day-challenges happens to consist of two main topics: First, inour rapidly evolving field we never run out of new technologies to learn, and then, there’s a certain engineering aspect underlying, how to do things in a certain manner, with lots of input every year.

So after I recently shared some of such ideas with my friends — I indeed still have a few ones of those), I wondered: How is it, that in the modern software development world, most of the information about managing things actually comes from the field itself, and rather feeding back its ideas of project management, quality, etc. into the non-software-subspaces of the world? (Ideas like the Agile movement, Software Craftmanship, the calls of doing things Lean and Clean, nowadays prospered so much that you see their application or modification in several other industries. Like advertisement, just as an example.)

I see a certain kind of brain food in this question. What tells software development apart from other current fields, so that there is a broad discussion and considerable input at its base level? After all, if you plan on becoming someone who builds houses, makes cars, or manages cities, you wouldn‘t engage in such a vivid culture of „how“ to do things, rather focussing on the „what“.

Of course, I might be mistaken in this view. But, by asking: what actually tells software development apart from these other fields of producing something, I see a certain kind of brain food, helpful for approaching every day tasks and valuing better tips over worse ones.

So, what can that be?

1. Quite peculiar is the low entry threshold in being able to call yourself a “programmer”. With the lots of resources you get at relatively little cost (assumption: you have a computer with a working internet connection), you have a lot of channels by which you can learn the „what“ of software development first, and saving the „how“ for later. If you plan on building a house, there‘s not a bazillion of books, tutorials, and videos, after all.

2. Similarly, there‘s the rather low cost of failure when drafting a quick hobby project. Not always will a piece of code that you write in your free time tell yourself „hey man, you ever thought about some better kind of architecture?“ – which is, why bad habits can stick and even feel right. If you choose the „wrong“ mindset, you don‘t always lose heaps of money, and neither do you, if you switch your strategy once in a while, you also don‘t automatically. (you probably will, though, if you are too careless in this process).

3. Furthermore, there‘s the dynamic extension of how your project is going to be used („Scope Creep“). One would build a skyscraper in a different way than a bungalow (I‘m not an expert, though), but with software, it often feels like adding a simple feature here, extending the scope there, unless you hit a point where all its interdependencies are in a complex state of conflict…

4. Then, it‘s a matter of transparency: If you sit in a badly designed car, it becomes rather obvious when it always exhausts clouds of black smoke. Or your house always smells like scents of fresh toilet. Of course, a well designed piece of software will come with a great user experience, but as you can see in many commercial products, there also is quite some presence of low-than-average-but-still-somehow-doing-what-it-should software. Probably users are more tolerant with software than with cars?

5. Also, as in most technical fields, it is not the case that „pure consultants“ are widely received in a positive light. For most nerds, you don‘t get a lot of credibility if you talk about best practices without having got your hands dirty over a longer period of time. Ergo, it needs some experienced software programmers in order to advise less experienced software programmers… but surely, it‘s questionable whether this is a good thing.

6. After all, the requirements for someone who develops a project might be very different in each field. From my academical past in computational Physics, I know that there is quite some demand for „quick & dirty“ solutions. Need to add some Dark Matter in your model here? Well, plug this formula in and check the results. Not every user has the budget or liberty in creating a solid structure of your program. If you want to have a new laboratory building, of course, you very well want it it do be designed as good as it can get.

All in all, these observations somehow boil down to the question, whether software development is to be seen more like a set of various engineering skills, rather like a handcraft, an art, or a complex program of study. It is the question, whether the “crack” in this field is the one who does complex arithmetics in its head, or the one who just gets what the customer wants. I like thinking about such peculiar modes of thought, as they help me in understanding what kinds of things I should learn next.

Or is there something completely else to it?

# Compiling Agda 2.6.2 on Fedora 32

Agda is a dependently typed functional programming language that I like very much. Its latest versions have some special features which are not supported by any more well known languages, for example higher inductive types. Since I want to use all the special features of Agda, I regularly compile the latest version. This is a procedure which comes with a few surprises more often than not, so this post is about saving you the time it took me to figure out what to do.

I like to compile Agda using the Haskell tool Stack, which can be installed with

`curl -sSL https://get.haskellstack.org/ | sh`

The sources of Agda have to be checked out with all submodules – otherwise there will be some weird “cabal” (another Haskell tool, more basic than Stack) errors which I was not able to understand. This can be done with:

`git clone --recurse-submodules https://github.com/agda/agda.git`

Now if you go to the Agda folder

`cd agda`

There should be files called “stack-8.8.4.yaml” and similar. Those files can can tell Stack how to build Agda. The number is the version of the Haskell compiler which is used to build Agda. I usually use the latest, you do not have to figure out which (if any) is installed on your system, since Stack will just download the appropriate compiler for you.

However, just running stack failed for my on Fedora 32 due to some linking problem in the end. It turned out, that “libtinfo.so” was not found by the linker. “libtinfo.so.6” is available on Fedora 32, so adding a link to it fixed the problem:

`sudo ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so`

Now, you can tell Stack to get all necessary things and compile and install Agda with:

`stack install --stack-yaml stack-8.8.4.yaml`

This also installs a binary into “~/.local/bin” which is in PATH on Fedora 32 by default, so you should be able to call agda from the command line. Also, you can use “agda-mode” to configure emacs for agda.

# Unintentionally hidden application state

What do libraries like React and Dear Imgui or paradigms like Data-oriented design and Data-driven programming have in common?
One aspect is that they all rely on explictly modeling an application’s state as data.

## What do I mean by hidden state?

Now you will probably think: of course, I do that too. What other way is there, really? Let me give you an example, from the widely used Qt library:

```QMessageBox box;
msgBox.setText("explicitly modeled data!");
msgBox.exec();
```

So this appears to be harmless, but is actually one of the most common cases were state is modeled implicitly. But we’re explicitly setting data there, you say. That is not the problem. The `exec()` call is the problem. It is blocking until the the message box is closed again, thereby implicitly modeling the “This message box is shown” state by this thread’s instruction pointer and position on the call stack. The instruction pointer is now tied to this state: while it does not move out of that function, the message box is still shown.

This is usually not a big problem, but it demonstrates nicely how state can be hidden unintentionally in an application. And even a small thing like this already prevents you from doing certain things, for example easily serializing your complete UI state.

This cannot really be avoided while keeping some kind of conceptual separation between code and data. It can sure be minimized. But is it bad? Well, it is certainly good to know when you’re using implicit state like that. And there’s one critiria for when it should most certainly minimized as much as possible: highly interactive programs, like user interfaces, games, machine control systems and AI agents.
These kinds of programs usually have some spooky and sponteanous interactions between different, seemlingly unrelated objects, and weird transitions between states. And using the call-stack and instruction pointer to model those states makes them particularly unsuitable to being interacted with.

## Alternatives?

So what can you use instead? The tried and tested alternative is always a state-machine. There are also related alternatives like behavior trees, which are actually quite similar to a “call stack in data” but much more flexible. Hybrid solutions that move only bits of the code into the realm of data are promises/futures and coroutines. Both effectively allow a “linear” function to be decoupled from its call-stack, and be treated more like data. And if their current popularity is any indicator, that is enough for many applications.

What do you think? Should hidden state like this always be avoided?

# Changing the keyboard navigation behaviour of form inputs

The default behaviour in HTML forms is that you can move the focus from one input element to the next via the tab key and submit the form via the enter key. This is also how dialogs work on most operating systems when using the native UI components. This behaviour is consistent across all browsers, and changing it messes with the user’s expectations and reduces accessibility. So I would normally advise against changing this behaviour without good reasons.

However, one of our customers wanted a different behaviour for an application developed by us. This application replaced an older application where the enter key did not submit the form, but moved the focus to the next input element. The ‘muscle memory’ effect made users accidentally submit the form by hitting the enter key, causing frustration. Since this application is not a public web site, but merely a web technology based intranet application with a small and specialized user base, changing the default behaviour is acceptable if the users want it.

So here’s how to do it. The following JavaScript function `focusNextInputOnEnter` takes a form element as a parameter and changes the focus behaviour on the input elements within this form.

```function focusNextInputOnEnter(form) {
var inputs = form.querySelectorAll('input, select, textarea');
for (var i = 0; i < inputs.length; i++) {
var input = inputs[i];
return function(event) {
if (!isEnter(event.which)) {
return;
}
var nextIndex = index + 1;
while (nextIndex < inputs.length) {
var nextInput = inputs[nextIndex];
if (nextInput.disabled) {
nextIndex++;
continue;
}
nextInput.focus();
break;
}
};
})(i));
}

function isEnter(keyCode) {
return keyCode === 13;
}
}
```

It works by handling the `keypress` events on the input elements and checking the key code for the enter key (code 13). It has an additional check so that disabled input elements are skipped.

To apply this change in behaviour to a form we have to call the function when the DOM content is loaded:

```<form id="demo-form">
<input type="text">
<input type="text" disabled="disabled">
<input type="checkbox">
<select>
<option>A</option>
<option>B</option>
</select>
<textarea></textarea>
<input type="text">
<input type="text">
</form>

<script>
focusNextInputOnEnter(document.getElementById('demo-form'));
});
</script>
```

I want to reiterate my warning that you should definitely not do this for public web sites, and elsewhere only if you know that this is what your users want.

# Using (elastic)search with .NET Core

Many modern applications require powerful search mechanisms to become useful and make their users more productive. That is in large part due to the amount of data available to work with. Thankfully there are already powerful tools to index your data and make it searchable.

One of the most well known state-of-the-art solutions is ElasticSearch and it has an API to be used from .NET called NEST. While the documentation is ok I want to give a quick rundown on how to add searching capabilities to your .NET Core application. Some ideas are borrowed from the great post “Using Elasticsearch with ASP.NET Core and Docker“.

## Getting ElasticSearch running

The easiest way to get up and running with elasicsearch is to use their docker images and just run the container on your development machine. I like to a docker compose file like the following to get elasticsearch and its tooling application kibana up and running fast:

```version: '3.8'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
container_name: elastic
environment:
- node.name=elastic
- cluster.initial_master_nodes=elastic
ports:
- "9200:9200"
- "9300:9300"
volumes:
- type: bind
source: ./esdata
target: /usr/share/elasticsearch/data
networks:
- esnetwork
kibana:
image: docker.elastic.co/kibana/kibana:7.8.0
ports:
- "5601:5601"
networks:
- esnetwork
depends_on:
- elasticsearch
volumes:
esdata:
networks:
esnetwork:
driver: bridge
```

After you run it with docker compose you can talk to the search service on http://localhost:9200/ and the kibana management GUI on http://localhost:5601/. On the Kibana UI especially the Dev Tools and its console are interesting for experimenting with search queries.

## Access ElasticSearch from your .NET Core app

I find it quite elegant to write a extension method for the IServiceCollection to configure the ElasticClient and register it as a Singleton to the dependency injection framework of .NET Core like so

```    public static class ElasticSearchExtension
{
this IServiceCollection services, IConfiguration configuration)
{
var url = configuration["elasticsearch:url"];
var settings = new ConnectionSettings(new Uri(url))
.DefaultMappingFor<SearchableDevice>(deviceMapping => deviceMapping
.IndexName("devices")
.IdProperty(dev => dev.Id)
)
;
var client = new ElasticClient(settings);
var response = client.Indices.Create("devices", creator => creator
.Map<SearchableDevice>(device => device
.AutoMap()
)
);
// maybe check response to be safe...
}
}
```

The configuration block looks like following

```  "ElasticSearch": {
"url": "http://localhost:9200/"
},
```

and allows for future extension.

Our search service has to be registered in `Startup `of course:

```public void ConfigureServices(IServiceCollection services)
{
...
}
```

## Indexing our objects

In the above example we built a SearchableDevice class whose public properties are going to be indexed by ElasticSearch. The API allows for a much more fined grained control about what and how to index but we want to keep things simple without having to worry about excluding Properties and so on. If you have set it up that way indexing a `SearchableDevice `is merely one simple call:

```// SearchClient is the injected IElasticClient
// mySearchableDevice is an instance of SearchableDevice
SearchClient.IndexDocument(mySearchableDevice);
```

## Searching for objects

When developing a search query I like to try it in the Kibana Dev Tools and then transform it to a NEST call. A simple query to look in all device properties if they start with “needle” looks like this:

```{
"query": {
"multi_match": {
"type": "phrase_prefix",
"fields": ["*"],
"query": "needle"
}
}
}
```

The nice thing about the Kibana Dev Tools console is that it can display and complete the possible values for fields like “type” in your multi_match query.

That search query can then be translated to a NEST call in a straightforward way and looks this way:

```var response = SearchClient.Search<SearchableDevice>(sd => sd
.Query(q => q
.MultiMatch(query => query
.Type(TextQueryType.PhrasePrefix)
.Fields("*")
.Query("needle")
)
)
);
```

The search response contains the hits with their source objects and some metadata like the score and result count.

Bear in mind that ElasticSearch only returns the first/best 10 matches by default, so specifying the result size often might be closer to what you want.

## Wrapping it up

Getting started with ElasticSearch in .NET Core does not require too much boiler plate an setup work if you use tools like docker and the NEST library. Making it usable and tuning the indexing and querying may require a lot of work to achieve the best results. On the other hand smaller applications can start-off with a simple search setup like shown above and simply evolve it when need be.

# The Bird in the Jungle

When I was a young boy and attending school, there were art classes that I kind of liked. You either were told cool stories about lunatic painters or painted cool pictures by yourself. I still remember one particular assignment that sticks out to me: draw a bird!

In case you ask what this blog entry is all about: It is about project management and expectation management. It will stray a little bit into process territory. But mostly, it’s about a picture of a bird in a jungle.

When the teacher told us the task for the next weeks, my imagination ran wild. Draw a bird! I envisioned a stork-like bird with long legs and a long beak, fishing by a river in a tropical setting. The river would flow through a full-blown jungle with lots of plants and insects and things to discover. The bird would be in the center of the picture, drawing your attention to its eyes as it hovered over a fish in the water, ready for the strike that would provide its supper. But if your eyes follow the water, you would see that the bird is just a piece in a complex ecosystem with lots of untold stories. My pen would tell some of those stories!

We had rougly two hours of art class, so I had to move fast. Because the bird was the essential piece of my picture, I had to draw it last, on top of the background jungle and aligned with all the secondary motives. In the first hour, the jungle grew on my sheet of paper, following the outlined river. If I grew weary of drawing plant leaves, I added a bug or a little snake. Somewhere, a happy little jungle squirrel peaked through the green (Ok, that’s not true. I didn’t know about Bob Ross back then and my squirrel was probably not happy).

The thing was, the art class was suddenly over. My jungle wasn’t completed yet. Worse, the entire river, fish and bird were still missing. The teacher asked me if I misunderstood the assignment. What a joke! Of course I understand, my bird will be majestic – once it is finished next week.

I peeked at the pictures of my classmates. Most of them had some bird-like outlines, a beak or a foot. Nobody had finished their drawing. As far as I was concerned, I was well within expectations. Many classmates took their picture home to work on it in the afternoon. I waited until next week, I was certain about my plan.

Next week came and a sudden realization followed: Most of my classmates had indeed worked on their bird at home. Their drawings were energetic, strong and defined. They looked like black-and-white copies of bird photographies and nothing like the awkward lines from last week. They had traced an image through their sheet and called it a day. Two classmates had even traced the same image and produced essentially identical birds. Those cheaters. Not me!

When half the class had already turned in the assignment, the pressure began to rise for the strugglers. My plans were in jeopardy and I skipped several trees and mountains in the outskirts of the picture to save time. This also meant I had to fade out the picture at the edges to make it seem deliberately.

This art class went over so fast, I still hadn’t drawn my bird when the bell rang. The teacher wanted to collect our pictures, but I had to bargain for a week of work at home. With that much time at my disposal, I could implement my plans. I drew six evenings and half the weekend. My jungle was never right. My river wasn’t as dynamic as envisioned. But most of all, my bird was nowhere near majestic. It was a stork-like bird, but way too small. It didn’t hover over the fish, I had to make it dive into the water to even be near its prey. In the end, I had spend around 20 hours on a two-hour assignment that produced a drawing of a jungle that happened to have a hungry, desperate bird in the middle. Somewhere in the outskirts, a squirrel laughed.

The teacher gave me a mediocre grade out of pity. It was clear I didn’t copy from anywhere. But it was also clear as the water of the river in my jungle that I didn’t follow the instructions at all.

Looking back now, I can smile about the naive boy that set out to tackle a task that nobody asked for while getting lost and receiving an average grade for an enormous effort. You can probably already see that the return on investment was awful.

The interesting thing for me to see clearly now is that the project was doomed from the start. Mostly because I didn’t work on the teacher’s assignment, I worked on my own assignment. And my own assignment was much bigger than the original one, but the time budget was not. If you don’t see it as clearly, just count the words in the teacher’s assignment (“draw a bird” – 3 words) and just the nouns in my plan. Calculate one hour for every noun and you can see how this would never have worked. So this is the first take-away: Always make sure that you work on your customer’s project and not your own. If the customer didn’t use a specific noun in the mission statement, why did we include it in our project plan? In my story, the teacher never even mentioned a “background”. Just a bird. Draw it and be done. Go home and draw the jungle in your spare time.

The next thing that sticks out now for me is that I delayed the essential work (the bird) as much as possible. This is exactly the wrong approach if you want to act at least a little bit agile. Cover the essential things first and fill in the details later. In my today’s work, we call it the “risk first approach”. The most critical part comes first. It might not be pretty, it might not be complete, but it already works. Applying this to my story: If I had drawn the bird first (big and majestic in the middle), I could have called it quits anytime afterwards. Nobody asked for more and my teacher wouldn’t grade the background – only the bird.

The third thing is about expectation management: The teacher expected us to try, fail and cheat at home. Those clearly traced pictures matched the assignment better than my work. I just refused to fail and start over. I didn’t set out to match the expectations of my teacher, I wanted to fulfill my own expectations that were not grounded in reality. I even ignored the grading scheme of my teacher (my customer, but I couldn’t see it that way at that time) and got upset when all my effort just yielded a mediocre grade.

Today, I see all these things and can handle them successfully. As a young boy, those were foreign concepts for me. Between the two points in time, I acquired concepts, terms and words to reflect on the process. But I’m still learning. Perhaps you see something in the story that I haven’t seen yet? Tell me in a comment below!

Nevertheless, I often think about my bird in the jungle as a reminder to keep on track.

# Shooting Troubles With Toys

As software grows, one of the typical challenges is keeping track of the quirks and subtleties of all the languages, third-party libraries, frameworks, IDEs / toolchains or whatnot you, at places, need to maneuver in order not to construct everything yourself. That‘s often just a matter of familiarization and after you stumbled across a particular type of problem – once or a few times – it gets assimilated as a trivial thing. It sometimes gets so far as keeping certain warnings or harmless exceptions in your software – technical debt. (Alas, your customer doesn‘t usually care for the perfect product, or, let‘s say, wait that long and pay for the perfect product..).

Now, once in a while all the third-party implementations you rely on interact in a odd manner. These are the cases where you get a „Cannot update a component while rendering a different component“ in a React/Recoil application, a NoClassDefFoundError in a Java / Grails application, a general SegFault in a C / C++ program, or your database does weird stuff. U name it.

So even when you encounter a problem that your then, time is a critical thing. So what you do? Google it. Find fellow victims on Stack Overflow, GitHub, etc. – but this only goes so far, depending on how common your problem ist.

Now you should always have a version control system at hand, of course. This has the huge advantage of being able to simplify your problem. Just enter a new branch that no one cares about, and you can completely get rid of all the confusing mess that is your reliance on third party content. Of course, this is a possibility one can always know about. Point here being, do it as a habit. Learn it as a habit. If it‘s only in „deactivating this probably useless flag“, „hardcoding this localhost into this URL in order to make progress quickly“ – you do not want to risk carrying this into production code. Just know that keeping experimental branches open for a longer time is a bad habit, either. So think of them e.g. as „in two hours, I will either merge or delete this branch, there‘s no way about it“. There‘s nothing in there that you wouldn‘t be able to reproduce if required.

And sometimes, it‘s more effective to work from the other end. Instead of going from „very close to where we want to be“, start from a place completely unpolluted by your technical debt. Start a toy project. Use exactly the dependencies that you have in your real project, and try to set up your error scenario. In our case, this method helped us in understanding a completely meaningless NoClassDefFoundError, because suddenly – with exactly the same JDK and Grails packages that we had in the real code – IntelliJ IDE just felt more like telling us verbosely what the actual problem is. Which you can then see without all the clutter.

Even more, this procedure does help your with your own Rubber Ducking – after all, you want to describe to yourself a scenario, where „Actually I don‘t get it, I am just doing this and that and…“, well, are you? Or is there more to it that your eyes don‘t see? Just find out.

Of course, this is just the precursor to a more test driven approach. Toy projects aren‘t really anything else, they are just isolated environments in which you completely see what is going on, with an essential setup and a clear expectation. These are tests. Now if you already wrote them, why not think about including them in your projects as tests? Especially if you‘re kind of new to test driven development, you can make this habit of toy boxing a guide on the road to a more test driven way of thinking.

Or maybe, just don‘t make errors. If you ever have the option – just choose that [I guess then you have time to fix mine, too? :)]

# Leibniz would have known how to override equals

Equality is a subtle and thorny business, in programming as well as in pure mathematics, physics and philosphy. Probably every software developer got annoyed somtime by unexpected behaviour of some ‘equals’ method or corresponding operators and assertions. There are lot’s of questions that depend on context and answering them for some particular context might cost some pain and time – here is a list of examples:

• What about objects that come with database-ids? Should they be equal for the objects to be equal?
• Are dates with time zones equal if they represent the same instant but have a different time zone?
• What about numbers represented by functions that compute digits up to a given precision?

## Leibniz’ Law

This post is about applying an idea of Leibniz I like, to the problem of finding good answers to the questions above. It is called “Leibniz’ law” and can be phrased as a definition or characterization of equality:

Two objects are equal, if and only if, they agree in all properties.

If you are not familiar with the phrase “if and only if”, that’s from mathematics and it is a shorthand for saying, that two things are true:

• If two objects are equal, then they agree in all properties.
• If two objects agree in all properties, then they are equal.

Lebniz’ law is sometimes stated using mathematical symbols, like “$\forall$“, but this would be besides the point of this post – what those properties are will not be defined in a formal mathematical way. If I am in doubt about equality while programming, I am concerned about properties relevant to the problem I want to solve. For example, in almost all circumstances I can imagine, for a list, a relevant property would be its length, but not the place in the computers memory where it is stored.

But what are relevant properties in general? For me, such a property is the result of running some piece of meaningful code. And what meaningful code is, depends on your judgement how the object in question should be used. So in total, this boils down to the following:

Two instances of a type are equal, if and only if, they yield the same results in any meaningful piece of code.

Has this gotten us anywhere? My answer is yes, since the question about equality was reduced to a question about use cases of a type, which might be a starting point of defining a new type anyway.

## Turtles all the way down

Please take a moment to note what a sneaky beast equality can be: Above I explained equality by using equality – right where I said “same results”. It is really hard to make statements about anything at all without using some notion of equality in some way. Even in programming, where you can freely define when two objects are equal, you can very well forget that you are using a system, namely your programming language, which usually already comes with an intricate notion of equality defined on the syntax you are using to define your notion of equality…

On a more practical note, that means that messed up notions of equality usually propagate if you define new kinds of objects from known ones.

## Relation to Liskov’s Principle

With our above definition, we are very close to an informal interpretation of Liskov’s Substitution Principle, which we can rephrase as:

In all meaningful code for a type, an instance of a subtype has to behave the same way.

For comparison, the message of this post stated in the same tongue:

In all meaningful code for a type, two equal instance of the type should behave the same way.

# Mastering programming like a martial artist

Gichin Funakoshi, who is sometimes called the “father of modern karate”, issued a list of twenty guiding principles for his students, called the “Shōtōkan nijū kun”. While these principles as a whole are directed at karate practitioners, many of them are very useful for other disciplines as well.
In my understanding, lifelong learning is a fundamental pillar of both karate and programming, and many of those principles focus on “learning” as a more fundamental action. I’d like to focus on a particular one now:

Formal stances are for beginners; later, one stands naturally.

While, at first, this seems to focus on stances, the more important concept is progression, and how it relates to formalities.

## Shuhari

It is a variation of the concept of Shuhari, the three stages of mastery in martial arts. I think they map rather beautifully to mastery in programming too.

The first stage, shu, is about learning traditions and movements, and how to apply them strictly and faithfully. For programming, this is learning to write your first programs with strict rules, like coding conventions, programming patterns and all the processes needed to release your programs to the world. There is no room for innovation, this stage is about absorbing what knowledge and practices already exist. While learning these rules may seem like a burden, the restrictions are also a gift. Because it is always clear what is right and what is wrong, and decisions are easy.

The second stage, ha, is about breaking away from these rules and traditions. Coding conventions, programming-patterns etc. are still followed. But more and more, exceptions are allowed, and encouraged, when they serve a greater purpose. A hack might no longer seem so bad, when you consider how much time it saved. Technical dept is no longer just avoided, but instead managed. Decisions are a little harder here, but there’s always the established conventions to fall back to.

In the final stage, ri, is about transcendence. Rules lose their inherent meaning to purpose. Coding conventions, best-practices, and patterns can still be observed, but they are seen for what they are: merely tools to achieve a goal. As thus, all conventions are subject to scrutiny here. They can be ignored, changed or even abandoned completely if necessary. This is the stages for true innovation, but also for mastery. To make decisions on this level, a lot of practice and knowledge and a bit of wisdom are certainly required.

## How to use this for teaching

When I am teaching programming, I try to find out what stage my student is in, and adapt my style appropriately (although I am not always successful in this).

Are they beginners? Then it is better to teach rigid concepts. Do not leave room for options, do not try to explore alternatives or trade-offs. Instead, take away some of the complexity and propose concrete solutions. Setup rigid guidelines, how to code, how to use the IDEs, how to use tools, how to communicate. Explain exactly how they are to fulfill all their tasks. Taking decisions away will make things a lot easier for them.

Students in the second, or even in the final stage, are much more receptive to these freedoms. While students on the second stage will still need guidance in the form of rules and conventions, those in the final stage will naturally adapt or reject them. It is much more useful to talk about goals and purpose with advanced students.

# Querying gaps between date ranges in Oracle SQL

Let’s say we have a database table with date ranges, each range designated by a RANGE_START and a RANGE_END column:

```CREATE TABLE date_ranges (
range_start DATE,
range_end   DATE
);
```
```RANGE_START	RANGE_END
-----------	---------
05/02/2020	01/04/2020
02/04/2020	15/04/2020
16/04/2020	01/05/2020
01/06/2020	20/06/2020
21/06/2020	01/07/2020
02/07/2020	31/07/2020
05/08/2020	30/08/2020
```

We are now interested in finding the gaps between these date ranges. If we look at this example data set we can see that there are two gaps:

```RANGE_START	RANGE_END
05/02/2020	01/04/2020
02/04/2020	15/04/2020
16/04/2020	01/05/2020
-- gap --
01/06/2020	20/06/2020
21/06/2020	01/07/2020
02/07/2020	31/07/2020
-- gap --
05/08/2020	30/08/2020
```

What would be the SQL query to find these automatically? With standard SQL this would be a difficult task. However, there are some special functions in Oracle SQL called analytic functions that greatly help with this task. Analytic functions compute an aggregate value based on a group of rows. They differ from aggregate functions in that they return multiple rows for each group. In this case we will use the analytic functions MAX and LEAD:

```SELECT * FROM (
SELECT
MAX(range_end)
OVER(ORDER BY range_start) + 1 gap_start,
OVER(ORDER BY range_start) - 1 gap_end
FROM date_ranges
) WHERE gap_start <= gap_end;
```

The result of this query are the date range gaps we are interested in:

```GAP_START	GAP_END
---------	-------
02/05/2020	31/05/2020
01/08/2020	04/08/2020
```

Note that the MAX function in the query is the analytic MAX function, not the aggregate MAX function, indicated by the OVER keyword with an analytic clause. It operates on a sliding window. The LEAD analytic function allows you to access the following row from the current row without using a self-join.