Tag: command line

Quotes Are Not Part of the Argument

Recently, we upgraded a project from an older Java version to a newer one. Many of the changes were routine: update dependencies, replace deprecated APIs, fix a few compiler errors and run the test suite.

One of these changes concerned the invocation of an external Windows program.

The application used the deprecated overload:

Runtime.getRuntime().exec(command);

The command was assembled as one long string. The external program accepted command-line parameters of the following form:

/a="value of A" /b="value of B" /c="value of C"

Because the parameter values could contain spaces, we enclosed them in double quotes. We even had unit tests that explicitly verified the quoting. It was an important detail, or so we thought.

As part of the upgrade, we switched to the recommended overload that accepts the executable and its arguments separately:

Runtime.getRuntime().exec(commandArray);

The migration seemed straightforward. Instead of joining the executable and all parameters into one command string, we put them into a string array:

String[] command = {
    "program",
    "/a=\"value of A\"",
    "/b=\"value of B\"",
    "/c=\"value of C\""
};
Runtime.getRuntime().exec(command);

The unit tests were still green. The quotes were still present. Everything looked correct.

Then the application was deployed.

Invalid switch

In production, the external program stopped accepting our invocation. Its only diagnostic message was:

Invalid switch

This was not particularly helpful.

We inspected the parameters. All switches were present. Their spelling was correct. Their order was correct. The values were correct. The quote characters were exactly where our tests expected them to be.

Even more confusingly, the command worked perfectly when entered manually in a Windows command prompt:

program /a="value of A" /b="value of B" /c="value of C"

The executable clearly supported these parameters. The shell command clearly worked. And our Java code appeared to produce the same command.

But it did not.

Asking the receiving program

After spending some time comparing strings and staring at quote characters, we decided to stop reasoning about what the external program ought to receive. Instead, we wrote a small program that showed us what it actually received:

void main(String[] args) {
    for (var i = 0; i < args.length; i++) {
        System.out.println("args[" + i + "]: '" + args[i] + "'");
    }
}

We packaged it as a JAR and invoked it from cmd.exe using the same parameter structure:

java -jar exec-experiment.jar /a="value of A" /b="value of B" /c="value of C"

The output was:

args[0]: '/a=value of A'
args[1]: '/b=value of B'
args[2]: '/c=value of C'

The double quotes were gone.

This was the missing piece.

The quotes in a shell command are not necessarily characters intended for the receiving program. They are instructions to the command-line parser. They tell it that a sequence containing spaces belongs to one argument.

The value

/a="value of A"

does not mean that the program receives an argument containing two quote characters. It means that the program receives one argument rather than three:

/a=value of A

The quote characters control parsing. They are not part of the resulting argument.

Quotes can appear in surprising places

To verify this interpretation, we performed a slightly more unusual experiment:

java -jar exec-experiment.jar /"a="va"lue of A" /b="value of B" /c="value of C"

This command is certainly not how anybody would normally write the parameters. Nevertheless, its output was unchanged:

args[0]: '/a=value of A'
args[1]: '/b=value of B'
args[2]: '/c=value of C'

The quotes can surround different portions of a token. Their purpose is to influence how the command line is divided into arguments. Once parsing is complete, they disappear.

This distinction is easy to overlook because a command line is usually presented as a string. It looks as though this string is passed to the program. In reality, there are two different representations involved:

program /a="value of A"

is a textual command line that still needs to be parsed.

By contrast,

new String[] {
    "program",
    "/a=value of A"
}

already describes the result of that parsing: an executable followed by one complete argument.

We had moved the parsing boundary

With Runtime.exec(String[]), every array element already represents one argument. Spaces inside an element do not split it into additional arguments.

By retaining the quotes, we had changed their meaning. They were no longer syntax interpreted by a shell-like parser. They had become literal characters inside the argument:

/a="value of A"

That was not what the external program expected. It expected:

/a=value of A

The error message “Invalid switch” was therefore accurate, but not very illuminating. The switch looked correct in our logs because we were looking at its command-line representation rather than at the argument format expected by the program.

The fix was simple:

String[] command = {
    "program",
    "/a=value of A",
    "/b=value of B",
    "/c=value of C"
};
Runtime.getRuntime().exec(command);

Or, preferably, using ProcessBuilder:

Process process = new ProcessBuilder(
    "program",
    "/a=value of A",
    "/b=value of B",
    "/c=value of C"
).start();

After removing the quote characters, the external program worked again.

Ironically, the quotes that our old implementation and its tests had treated as essential were exactly what broke the new implementation.

The takeaway

A command line and an argument array are not interchangeable representations.

When constructing a command line, quoting may be required to preserve spaces during parsing. When constructing an argument array, parsing has already happened conceptually. Each element is one argument, spaces included.

Do not ask:

How would I type this command in a shell?

Ask:

What exact strings should the receiving program find in its argument array?

The answer to the second question is what belongs in Runtime.exec(String[]) or ProcessBuilder.

Sometimes an API migration changes more than a method signature. It moves a boundary – in this case, the boundary between formatting a command line and supplying already separated arguments.

And when that boundary moves, yesterday’s carefully tested solution can become today’s bug.

Learning about Class Literals after twenty years of Java

The story about a customer request that led to my discovery of a cool feature of Java: Class Literals (you’ve probably already heard about them).

I’ve programmed in Java nearly every day for twenty years now. At the beginning of my computer science studies, I was introduced to Java 1.0.x and have since accompanied every version of Java. Our professor made us buy the Java Language Specification on paper (it was quite a large book even back then) and I occassionally read it like you would read an encyclopedia – wading through a lot of already known facts just to discover something surprising and interesting, no matter how small.

With the end of my studies came the end of random research in old books – new books had to be read and understood. It was no longer efficient enough to randomly spend time with something, everything needed to have a clear goal, an outcome that improved my current position. This made me very efficient and quite effective, but I only uncover surprising facts and finds now if work forces me to go there.

An odd customer request

Recently, my work required me to re-visit an old acquaintance in the Java API that I’ve never grew fond of: The Runtime.exec() method. One of my customer had an recurring hardware problem that could only be solved by rebooting the machine. My software could already detect the symptoms of the problem and notify the operator, but the next logical step was to enable the software to perform the reboot on its own. The customer was very aware of the risks for such a functionality – I consider it a “sabotage feature”, but asked for it anyway. Because the software is written in Java, the reboot should be written in Java, too. And because the target machines are exclusively running on Windows, it was a viable option to implement the feature for that specific platform. Which brings me to Runtime.exec().

A simple solution for the reboot functionality in Java on Windows looks like this:


Runtime.exec("shutdown /r");

With this solution, the user is informed of the imminent reboot and has some time to make a decision. In my case, the reboot needed to be performed as fast as possible to minimize the loss of sensor data. So the reboot command needs to be expanded by a parameter:


Runtime.exec("shutdown /r /t 0");

And this is when the command stops working and politely tells you that you messed up the command line by printing the usage information. Which, of course, you can only see if you drain the output stream of the Process instance that performs the command in the background:


final Process process = Runtime.exec("shutdown /r /t 0");
try (final Scanner output = new Scanner(process.getInputStream())) {
    while (output.hasNextLine()) {
        System.out.println(output.nextLine());
    }
}

The output draining is good practice anyway, because the Process will just stop once the buffer is filled up. Which you will never see in development, but definitely in production – in the middle of the night on a weekend when you are on vacaction.

Modern thinking

In Java 5 and improved in Java 7, the Runtime.exec() method got less attractive by the introduction of the ProcessBuilder, a class that improves the experience of creating a correct command line and a lot of other things. So let’s switch to the ProcessBuilder:


final ProcessBuilder builder = new ProcessBuilder(
        "shutdown",
        "/r",
        "/t 0");
final Process process = builder.start();

Didn’t change a thing. The shutdown command still informs us that we don’t have our command line under control. And that’s true: The whole API is notorious of not telling me what is really going on in the background. The ProcessBuilder could be nice and offer a method that returns a String as it is issued to the operating system, but all we got is the ProcessBuilder.command() method that gives us the same command line parts we gave it. The mystery begins with our call of ProcessBuilder.start(), because it delegates to a class called ProcessImpl, and more specific to the static method ProcessImpl.start().

In this method, Java calls the private constructor of ProcessImpl, that performs a lot of black magic on our command line parts and ultimately disappears in a native method called create() with the actual command line (called cmdstr) as the first parameter. That’s the information I was looking for! In newer Java versions (starting with Java 7), the cmdstr is built in a private static method of ProcessImpl: ProcessImpl.createCommandLine(). If I could write a test program that calls this method directly, I would be able to see the actual command line by myself.

Disclaimer: I’m not an advocate of light-hearted use of the reflection API of Java. But for one-off programs, it’s a very powerful tool that gets the job done.

So let’s write the code to retrieve our actual command line directly from the ProcessImpl.createCommandLine() method:


public static void main(final String[] args) throws Exception {
    final String[] cmd = {
            "shutdown.exe",
            "/r",
            "/t 0",
    };
    final String executablePath = new File(cmd[0]).getPath();

    final Class<?> impl = ClassLoader.getSystemClassLoader().loadClass("java.lang.ProcessImpl");
    final Method myMethod = impl.getDeclaredMethod(
            "createCommandLine",
            new Class[] {
                    ????, // <-- Damn, I don't have any clue what should go here.
                    String.class,
                    String[].class
            });
    myMethod.setAccessible(true);

    final Object result = myMethod.invoke(
            null,
            2,
            executablePath,
            cmd);
    System.out.println(result);
}

The discovery

You probably noticed the “????” entry in the code above. That’s the discovery part I want to tell you about. This is when I met Class Literals in the Java Language Specification in chapter 15.8.2 (go and look it up!). The signature of the createCommandLine method is:


private static String createCommandLine(
        int verificationType,
        final String executablePath,
        final String cmd[])

Note: I didn’t remove the final keyword of verificationType, it isn’t there in the original code for unknown reasons.
When I wrote the reflection code above, it occurred to me that I had never attempted to lookup a method that contains a primitive parameter – the int in this case. I didn’t think much about it and went with Integer.class, but that didn’t work. And then, my discovery started:


final Method myMethod = impl.getDeclaredMethod(
        "createCommandLine",
        new Class[] {
                int.class, // <-- Look what I can do!
                String.class,
                String[].class
        });

As stated in the Java Language Specification, every primitive type of Java conceptionally “has” a public static field named “class” that contains the Class object for this primitive. We can even type void.class and gain access to the Class object of void. This is clearly written in the language specification and required knowledge for every earnest usage of Java’s reflection capabilities, but I somehow evaded it for twenty years.

I love when moments like this happen. I always feel dumb and enlightened at the same time and assume that everybody around me knew this fact for years, it is just me that didn’t get the memo.

The solution

Oh, and before I forget it, the solution to the reboot command not working is the odd way in which Java adds quote characters to the command line. The output above is:


shutdown /r "/t 0"

The extra quotes around /t 0 make the shutdown command reject all parameters and print the usage text instead. A working, if not necessarily intuitive solution is to separate the /t parameter and its value in order to never have spaces in the parameters – this is what provokes Java to try to help you by quoting the whole parameter (and is considered a feature rather than a bug):


final String[] cmd = {
        "shutdown",
        "/r",
        "/t",
        "0",
};

This results in the command line I wanted from the start:


shutdown /r /t 0

And reboots the computer instantaneous. Try it!

Your story?

What’s your “damn, I must’ve missed the memo” moment in the programming language you know best?