Islands in the streams

This just in!

Update: So, I thought my problems where with the error 209, ACTION_UNKNOWN seen below in a few logs. Not so. Tonight I just realized that I'm playing around with streams that don't support stuff like available() (natively using IDOS->GetFilePosition() and such), i.e. any streaming data with no real end, like STDIN. Like, duh... Anyhow. I've solved this by calling IDOS->IsInteractive() on the file handle prior calling any un-supported functions on it.

Now, this minor "progress" hasn't solved everything, it just renders much of yesterdays ramblings regarding processes interchanging streams invalid. Much is however still valid. I'll try to remove the false parts hastily.

Continuing my Mauve testing, I'm currently occupied in trying to get one process to read data from another process' streams. In the Java world it's really easy to just send an object to any other thread and do stuff with it. Not so much in the AmigaOS 4 world. It's much the same problem I had with the sockets, where each process executes in its own context, i.e. data stream pointers are actually just pointers to internal structures valid only in the current process context.

Problems described in the context of trying to tell something about how Java works

Now, I try to write these blogs with the mixed intent of giving a powerpointy overview in combination with gory tech-details that might spark the attention of Amiga and Java coders, and at the same time getting order to my own thoughts. I sometimes get stuck in complex explanations and an overall eager to combine words in long sentences without seemingly coming to any conclusion. Sometimes I just keep it short. The text today probably requires some degree of technical knowledge. Hopefully, I will have some more real-world usage stuff to talk about in later blogs.

Starting with how Mauve executes tests

The Mauve test suite works by spawning new processes for each test to perform. When you start a Mauve test run, Mauve starts another instance of JamVM, using a simple Runtime.getRuntime().exec("jamvm RunnerProcess test") call, which in Amiga native code is realized using IExec->SystemTags(). While the test JamVM instance is running, there are a few Mauve controller threads keeping track of the test suite processes, aborting any failed or long-running tests. These controller threads talks to the tests using simple "console redirection", i.e. one programs console output decides what will happen next; the controller threads are only loops checking for data on System.in (i.e. STDIN or IDOS->Input()) and performs various tasks depending on input.

A simplified Java code example can be seen below:

01    
02    public static void main(String[] args) {
03        // Exec the other jamvm process and 
04        // set up in/out communications with it.
05        Process process = Runtime.getRuntime().
06                         exec("jamvm RunnerProcess test");
07        // Our controller thread keeping track of
08        // System.in actions
09        InputPiperThread pipe = 
10                 new InputPiperThread(
11                       System.in,
12                       process.getOutputStream());
13        // Start our InputPiperThread
14        pipe.start();
15        
16        // Do other stuff here...
17        ...
18    }
19    private static class InputPiperThread extends Thread {
20        // The controller starts by executing run 
21        public void run() {            
22            do {
23                // Check what's available on 
24                // parent's System.in
25                if (in.available() < 0) {
26                    ch = in.read();
27                    ...
28                } else {
29                    Thread.sleep(200);
30                }
31            } while (ch != -1);
32        }
33        InputStream in;
34        OutputStream out;
35        InputPiperThread(InputStream i, OutputStream o) {
36            in = i;out = o;int ch = 0;
37        }
38        
39    }
40}

The problem -- so far

The problem I had was that call to in.available() on line 25 wasn't interrupted correctly -- it just kept on going on forever. I've simplified the code above, so you can't see that the main Mauve process will eventually try to kill the InputPipeThread, and this should lead to an exception halting the thread, ending the entire process. What I noticed, was that I could not read the data at all -- there was no data to be found, my input stream (in) is actually not even a valid input stream.

Investigation begins

Before I understood what was wrong I had to see how System.in eventually is mapped to IDOS->Input(). This info gives you a good look at what actually goes around in a Java Virtual Machine and its different parts. I have a nice screenshot of it:

1. In the center of the picture you see the first step in java.lang.System where the actual reference to STDIN is (Java coders will certainly recognize System.in)
2. You can see it is being statically initialized to VMSystem.makeStandardInputStream(), with VMSystem being the GNU Classpath specific VM native layer.
3. There we wrap a FileDescriptor.in in a few buffered streams.
4. FileDescription.in is a statically initialized and wrapped version of FileChannelImpl.in, which in turn is a statically initialized to the value of VMFileChannel.getStdIn().
5. VMFileChannel.getStdIn() is finally where we get to the actual core of the poodle: stdin_fd().
6. The stdin_fd() function is native, so in the last step we can see the utterly simple C counterpart (Amiga C coders will recognize this part): the actual call to IDOS->Input().

So, in summary, System.in is the file handle returned by IDOS->Input() wrapped in various Java fluff.

My first thought was that something was awry with the file handle. Looking through logs however showed it wasn't so. Then, remembering my socket problems I realized that I probably can't send one Amiga process's STDIN file handle to any other Amiga process with the same ease I can send data from one Java thread to the other.

Going back to the Mauve threads and processes:

In the picture above we see the first process (marked 1) spawning a new jamvm process (number 2 in the picture). We can also see the Mauve controller threads, 3 and 4, as started by process 1. What to notice here is that to the Amiga, all four boxes represent a unique Amiga DOS Process. However, in the Java virtual machine, process 1, 3 and 4 are regarded as equals all executing in one virtual machine able to share data without much fuss. Process 2 could from the Java viewpoint of process 1 actually be any Amiga native process, like SketchBlock, OWB or whatnot, but in this particular context happens to be another jamvm instance.

Where's the Grim Reaper?

Now, anyone who's been coding on AmigaOS 4 knows that the Grim Reaper will manifest itself if any process tries to mess with privately allocated data by another process. This is also true for sockets and data stream pointers (i.e. our file handle to STDIN as returned by IDOS->Input()), but without the Grim Reaper showing up, since the pointers are only identifiers easily detected by the API as non-valid data. Looking at the logs for one of my experiments clearly show this:

Level PID Message TRACE 260 Returning stdin: 19248530 DEBUG 260 Available for file 19248530 Input: 19248530 Output:19248630 DEBUG 271 Available for file 19248530 Input: 18271650 Output:18271550 ERROR 271 IoErro reading examine data: 209 (ACTION_UNKNOWN)

What you can see is the main jamvm process with ProcessID (PID) 260, returning a file handle to System.in, STDIN (19248530), to be handed over to the controller thread (PID 271). The next line is again process 260 successfully checking for available data on STDIN. And, on the third line you see the controller thread 271 trying the exact same thing, but on the fourth line evidently failing. (The numbers after Input and Output on the second and third line are what IDOS->Input/Output, returns in each process, just to be sure we have the pointer we want.)

And, the solution?

This perhaps doesn't come as a surprise to knowledgeable Amiga programmers (I'm also not surprised), but it does take a while to come to this conclusion given the number of things that can be wrong.

I haven't yet come to a solution, but i think it can involve using a PIPE:. Looking again at the Java example above, around line 12 we actually have a PIPE: in use under the bonnet:

05        Process process = Runtime.getRuntime().
06                         exec("jamvm RunnerProcess test");
07        // Our controller thread keeping track of
08        // System.in actions
09        InputPiperThread pipe = 
10                 new InputPiperThread(
11                       System.in,
12                       process.getOutputStream());

When getting the output and input stream from a Java spawned Amiga process I am already using Amiga pipes, and this works without problems. Now I only need to open a new PIPE: and then pipe the real IDOS->Input() to it. But I don't yet know how to do it.