måndag 20 maj 2013

A telnet-client in Java on AmigaOS 4.1

I'm happy to announce the first working usable application for Java on Amiga: a telnet-client.

What can be seen is the class "org.jamiga2.jta.JTAConsole" telnetting into my Mac. It's really slow at the moment, but that's due to the extensive logging, and currently my Mac's login credentials are hardcoded in the Java file. And when quitting, one of the Java processes crashes due to some memory I might be freeing too many times. But besides that: it's alive!

Now what?

What's left to do, is everything else. Fix the crash, try out real SSH, fix the client from proof-of-concept status into a real SSH client, and lots of other stuff. But it's really nice to see something usable actually running.

måndag 25 mars 2013

Islands in the streams

This just in!

Update: So, I thought my problems where with the error 209, ACTION_UNKNOWN seen below in a few logs. Not so. Tonight I just realized that I'm playing around with streams that don't support stuff like available() (natively using IDOS->GetFilePosition() and such), i.e. any streaming data with no real end, like STDIN. Like, duh... Anyhow. I've solved this by calling IDOS->IsInteractive() on the file handle prior calling any un-supported functions on it.

Now, this minor "progress" hasn't solved everything, it just renders much of yesterdays ramblings regarding processes interchanging streams invalid. Much is however still valid. I'll try to remove the false parts hastily.

Continuing my Mauve testing, I'm currently occupied in trying to get one process to read data from another process' streams. In the Java world it's really easy to just send an object to any other thread and do stuff with it. Not so much in the AmigaOS 4 world. It's much the same problem I had with the sockets, where each process executes in its own context, i.e. data stream pointers are actually just pointers to internal structures valid only in the current process context.

Problems described in the context of trying to tell something about how Java works

Now, I try to write these blogs with the mixed intent of giving a powerpointy overview in combination with gory tech-details that might spark the attention of Amiga and Java coders, and at the same time getting order to my own thoughts. I sometimes get stuck in complex explanations and an overall eager to combine words in long sentences without seemingly coming to any conclusion. Sometimes I just keep it short. The text today probably requires some degree of technical knowledge. Hopefully, I will have some more real-world usage stuff to talk about in later blogs.

Starting with how Mauve executes tests

The Mauve test suite works by spawning new processes for each test to perform. When you start a Mauve test run, Mauve starts another instance of JamVM, using a simple Runtime.getRuntime().exec("jamvm RunnerProcess test") call, which in Amiga native code is realized using IExec->SystemTags(). While the test JamVM instance is running, there are a few Mauve controller threads keeping track of the test suite processes, aborting any failed or long-running tests. These controller threads talks to the tests using simple "console redirection", i.e. one programs console output decides what will happen next; the controller threads are only loops checking for data on System.in (i.e. STDIN or IDOS->Input()) and performs various tasks depending on input.

A simplified Java code example can be seen below:

01    
02    public static void main(String[] args) {
03        // Exec the other jamvm process and 
04        // set up in/out communications with it.
05        Process process = Runtime.getRuntime().
06                         exec("jamvm RunnerProcess test");
07        // Our controller thread keeping track of
08        // System.in actions
09        InputPiperThread pipe = 
10                 new InputPiperThread(
11                       System.in,
12                       process.getOutputStream());
13        // Start our InputPiperThread
14        pipe.start();
15        
16        // Do other stuff here...
17        ...
18    }
19    private static class InputPiperThread extends Thread {
20        // The controller starts by executing run 
21        public void run() {            
22            do {
23                // Check what's available on 
24                // parent's System.in
25                if (in.available() < 0) {
26                    ch = in.read();
27                    ...
28                } else {
29                    Thread.sleep(200);
30                }
31            } while (ch != -1);
32        }
33        InputStream in;
34        OutputStream out;
35        InputPiperThread(InputStream i, OutputStream o) {
36            in = i;out = o;int ch = 0;
37        }
38        
39    }
40}

The problem -- so far

The problem I had was that call to in.available() on line 25 wasn't interrupted correctly -- it just kept on going on forever. I've simplified the code above, so you can't see that the main Mauve process will eventually try to kill the InputPipeThread, and this should lead to an exception halting the thread, ending the entire process. What I noticed, was that I could not read the data at all -- there was no data to be found, my input stream (in) is actually not even a valid input stream.

Investigation begins

Before I understood what was wrong I had to see how System.in eventually is mapped to IDOS->Input(). This info gives you a good look at what actually goes around in a Java Virtual Machine and its different parts. I have a nice screenshot of it:

  1. In the center of the picture you see the first step in java.lang.System where the actual reference to STDIN is (Java coders will certainly recognize System.in)
  2. You can see it is being statically initialized to VMSystem.makeStandardInputStream(), with VMSystem being the GNU Classpath specific VM native layer.
  3. There we wrap a FileDescriptor.in in a few buffered streams.
  4. FileDescription.in is a statically initialized and wrapped version of FileChannelImpl.in, which in turn is a statically initialized to the value of VMFileChannel.getStdIn().
  5. VMFileChannel.getStdIn() is finally where we get to the actual core of the poodle: stdin_fd().
  6. The stdin_fd() function is native, so in the last step we can see the utterly simple C counterpart (Amiga C coders will recognize this part): the actual call to IDOS->Input().
So, in summary, System.in is the file handle returned by IDOS->Input() wrapped in various Java fluff.

My first thought was that something was awry with the file handle. Looking through logs however showed it wasn't so. Then, remembering my socket problems I realized that I probably can't send one Amiga process's STDIN file handle to any other Amiga process with the same ease I can send data from one Java thread to the other.

Going back to the Mauve threads and processes:

In the picture above we see the first process (marked 1) spawning a new jamvm process (number 2 in the picture). We can also see the Mauve controller threads, 3 and 4, as started by process 1. What to notice here is that to the Amiga, all four boxes represent a unique Amiga DOS Process. However, in the Java virtual machine, process 1, 3 and 4 are regarded as equals all executing in one virtual machine able to share data without much fuss. Process 2 could from the Java viewpoint of process 1 actually be any Amiga native process, like SketchBlock, OWB or whatnot, but in this particular context happens to be another jamvm instance.

Where's the Grim Reaper?

Now, anyone who's been coding on AmigaOS 4 knows that the Grim Reaper will manifest itself if any process tries to mess with privately allocated data by another process. This is also true for sockets and data stream pointers (i.e. our file handle to STDIN as returned by IDOS->Input()), but without the Grim Reaper showing up, since the pointers are only identifiers easily detected by the API as non-valid data. Looking at the logs for one of my experiments clearly show this:

Level PID Message TRACE 260 Returning stdin: 19248530 DEBUG 260 Available for file 19248530 Input: 19248530 Output:19248630 DEBUG 271 Available for file 19248530 Input: 18271650 Output:18271550 ERROR 271 IoErro reading examine data: 209 (ACTION_UNKNOWN)

What you can see is the main jamvm process with ProcessID (PID) 260, returning a file handle to System.in, STDIN (19248530), to be handed over to the controller thread (PID 271). The next line is again process 260 successfully checking for available data on STDIN. And, on the third line you see the controller thread 271 trying the exact same thing, but on the fourth line evidently failing. (The numbers after Input and Output on the second and third line are what IDOS->Input/Output, returns in each process, just to be sure we have the pointer we want.)

And, the solution?

This perhaps doesn't come as a surprise to knowledgeable Amiga programmers (I'm also not surprised), but it does take a while to come to this conclusion given the number of things that can be wrong.

I haven't yet come to a solution, but i think it can involve using a PIPE:. Looking again at the Java example above, around line 12 we actually have a PIPE: in use under the bonnet:

05        Process process = Runtime.getRuntime().
06                         exec("jamvm RunnerProcess test");
07        // Our controller thread keeping track of
08        // System.in actions
09        InputPiperThread pipe = 
10                 new InputPiperThread(
11                       System.in,
12                       process.getOutputStream());
When getting the output and input stream from a Java spawned Amiga process I am already using Amiga pipes, and this works without problems. Now I only need to open a new PIPE: and then pipe the real IDOS->Input() to it. But I don't yet know how to do it.

tisdag 29 januari 2013

The colour Mauve

As briefly mentioned in my previous entry, trying anything more fancy than sending a little data over the network, causes JamVM to wildly crash. I've yet not found the reason for this, it only happens for one particular example (that I've come across) and I can't find any obvious cause for it. It happens right after JamVM has actually ended, and the crash is according to Grim Reaper caused by the process of name "", which milliseconds before was named "jamvm.exe". I'm guessing it has something to do with the C-library trying to close/free something that I've already freed, or something like that. One upside of this problem is that I finally got around to sorting out my issues regarding which C-library I'm using.

JamVM currently uses clib2

So, there, I've said it. Its in black and white. I'm using clib2, and I'm sticking to it. I find that I constantly ask myself about design decisions I've made, and wondering both when, and why I came up with a certain solution. I will not use newlib, for the simple reason that I just wrote that "JamVM currently uses clib2". The JNI libraries (java-net, java-io, etc) should preferably use no C-library, or clib2 when needed. Being libraries, we're not using the C-library startup code. Some libraries however dependes on the reference GNU Classpath implementation, which uses c-library functions. So, in some cases it is more convenient to use a C-library.

I'm not entirely one-hundred percent sure that you can mix different c-libraries in your shared libraries, but I really do not see why not. As long as you don't try to make them talk to each other by interchanging implementation specific handles like sockets or such. But I see no reason why one library calling clib2's printf should interfere with another library calling newlib's printf. Steven Solie has a few good pointers on AmigaOS 4 programming in general, and specifically for networking. He also mentions that clib2 is open-source, and therefore a good choice in terms of debugging options. Which I believe it will prove to be, if I don't stumble upon my problem by accident.

What about that colour?

My previously mentioned doodling with mauve (the test suite for JVM's) continues. Instead of hopelessly banging my head against the keyboard, I thought it wise to continue my mauve runnings. With that I might be able to pinpoint the cause of crash more accurately, instead of wading through hierarchies of Java classes. And that is also one step closer to finishing the bounty.

Note to self

Like I said, my memory fails me. So I thought I'd write down how I run mauve. And it isn't that complicated. I have a mauve installation on my DH1, and it's only a matter of putting it in the classpath to be able to run the "Harness" class. However, putting stuff in the classpath should be as easy as setting an environment variable. But clib2's getenv implementation only has a buffer size of 256 bytes, which means that my scattered long windling classpath doesn't fit. Instead of solving the problem, I hardcoded the classpath. Fugly. But it works. Anyhow, with that in place, starting mauve is as easy as:

jamvm.exe -vm=jamvm.exe Harness java.lang.String

The "java.lang.String" reference is the test to actually run. Not entering this will run all tests. Entering something like java.* will run anything in the "java" package.
Edit: Just remembered that you need to say which VM you want to test. If none is specified, it uses "java", which in my particular case will execute the original JAmiga VM, and not JamVM.

Finding bugs as we speak!

Trying to run mauve, and simultaneously writing this blog entry, I'm actually bumping over a bug of "getenv" in java.lang.VMSystem not being linked correctly. How coincidental. Which concludes my rambling for the night. Off to get that bug!

onsdag 16 januari 2013

Jamiga net fishing

I've had some more progress with the networking. Instead of boring you with text about it, I thought I'd show you a clip of the progress and give an insight in how a day coding JAmiga might look.

Enjoy:

Well, its almost like that.

Anyhow, what the clip tries to illustrate is the Amiga running a Java program opening a server socket on port 7777, with my Mac telnetting to that port, and then exchanging a few strings. You can see a more academic documentary version below:

The current running network solution will most likely only work in single threaded environments, since using bsdsockets.library in multiple processes requires some extra care. My previous described "one-amiga-process-for-each-socket approach is currently disabled. There were other issues with that solution. I haven't yet decided how to do it, but starting an extra process for each socket is really not a satisfying solution -- it can be sufficient to get past the obstacle, but in the long run its really not a good way to do it.

What's next?

Next, I'll focus on getting one of the Milestone 1 possible applications to work, namely the Telnet/SSH client. At the moment, the whole thing crashes wildly, but I'm assuming its a simple mistake I've made.

onsdag 12 december 2012

Status report

With the bounty in place, my aim was to update the blog more often -- preferably when I had some sort of update. However, my time has been lacking the last few weeks, and I haven't made as much progress as I wanted. But the bounty has surely made me more motivated -- your donations are much appreciated. In order to get some well needed progress, I've earmarked two "mellandagar" (i.e. the days between christmas and new year) to fully focus on Jamiga. I'm hoping nothing will interfere with these plans.

Controlling the signal

So, why this blog entry? Well, its because I've actually made some progress! I've managed to sort out the signal handling. Now I cannot only run multi-threaded Java programs, I can also break them forcefully by pressing Ctrl-C. It might not seem like a big whoop, but it is really, really nice to be able to halt the Java process, and being able to do so in a stable manner. I have described the signaling previously, but I'll take it in more detail now.

Signal ping-pong

JamVM is aimed to run on a pthread capable OS. I've managed to wrap all pthread functions needed (as in needed by JamVM, and specifically tied to JamVM -- it is not a general purpose pthread implementation). There are at least four threads needed for one single instance of JamVM:

  • The main process
  • Finalizer thread
  • Reference handler
  • Signal handler which should catch any signals, i.e. Ctrl-C and such
  • ... plus any number of possible Java threads
The way it's intended, in a pthread environment, the signal handler catches all signals sent to the JamVM process. Upon receiving Ctrl-C the signal handler executes the Java class "System.exit()", and the exits the program by using the C function exit(). There is however a problem with this approach in the Amiga environment. A child process cannot run exit() to end the main process, so AmigaOS decides to only do a RemTask for any child processes calling exit(), which will leave all the other processes running. I managed to circumvent this exit()-calling by some signal sending back and forth; the signal handler actually signals the main process to shut down the virtual machine and the call exit().

In a real pthread environment, all the four threads are run inside one actual OS process, so the OS only sees one process and signals sent can be redirected to either thread. On AmigaOS the four threads are are actually four different processes, so anyone of them can be signaled independently (from like Ranger), and the main process is the most likely target. For instance, when running a Java console application and pressing Ctrl-C, the main process gets signaled, and not the Signal handler. However, this is not a big problem. I merely redirect the signal to the Signal handler, by setting up an exception handler (with SetExcept()) for each process that catches all signals and sends it to the Signal handler. The Signal handler calls Java method "System.exit()", and signals any running Java threads to stop what they're doing, and then signals the main process, which in turn signals any running JVM process (the Finalizer, Reference handler and Signal handler) to stop what they're doing. And then we're done.

What's next?

When the signaling bits are fully done (I must re-factor and cleanup the mess its currently in), I will resurrect my networking stuff. I ended up in this signaling mess trying to end a network process that hung for some reason, and I was fed up with rebooting my Amiga just to release the TCP-ports. You can see a small screenshot above of my attempts (and partial success) with running two Java instances sending data from client to server. The reason I cannot break this using Ctrl-C is that the networking process eats the Ctrl-C. And I guess that's what I'll be doing on my two "mellandagar".


The selected top Shell process with the three child processes is the Java network client (you can see the command "jamvm.exe ServerClientExample", which runs the ServerClientExample class). The next Shell process with the three familiar child processes is the other JamVM instance, running the Java server. This also has two children, the Jamiga socket processes, which are the processes handling the networking, as I described about a year ago.

lördag 10 november 2012

Coffee and bounty

Coffee and a bounty. What could be better?

We've just set up a bounty on AmigaBounty.net for Java on AmigaOS 4.1. The plan forward consist of three milestones. Milestone 1 covers the basic functionality of a working Java implementation, which also offers some novel applications to AmigaOS. Milestone 2 and 3, will offer even more in terms of user gain, with GUI and graphics capabilities, and applet support.

The reason why this bounty is set up at this point, is that I now feel confident that the goal can be reached. By taking the JamVM route, the number of unimplemented cans of worms the original Jamiga VM had, has greatly been reduced.

Milestone 1 possible applications

I thought I'd collect a small collection of tools that will run on Jamiga2 when milestone 1 is done (more info on what milestone 1 is will follow). These will consist of either currently available console applications, or existing Java open-source frameworks packaged by me (or someone else) using minimal coding.

Telnet/SSH client

A very simple Telent/SSH client, based on the JTA framework. It actually has a GUI client, but with minor effort a very simple console based telnet client has been made working. See below for a screenshot of it running on my amigaized JamVM source and GNU Classpath 0.99 (with GUI support). The system is Ubuntu Linux in a VMWare on my MacBook Pro, so its Intel, not PPC.

What you see is JamVM spitting out stuff like "[Opened native library...]", "libopen returning true". The text "Connected!" and the following lines are actually the Java program connecting to the telnet daemon on MacOS. I entered "uptime" which was sent to MacOS, who so kindly responded with the uptime. I wrote "quit", which the Java program takes as a signal I want to quit, and then JamVM spits out "exit vm", and exits.

Again, nota bene: this runs on Linux and Intel right now, which you probably can deduct from it not looking like Amiga. But just to be clear, so I don't fool anyone.

Console based Twitter client

By using Twitter4J a simple console application allowing for status updates and listing should be feasible. Together with some clever ARexx scripting, integration with f.i. Ringhio should not be impossible.