Coffee and bounty

Coffee and a bounty. What could be better?

We've just set up a bounty on AmigaBounty.net for Java on AmigaOS 4.1. The plan forward consist of three milestones. Milestone 1 covers the basic functionality of a working Java implementation, which also offers some novel applications to AmigaOS. Milestone 2 and 3, will offer even more in terms of user gain, with GUI and graphics capabilities, and applet support.

The reason why this bounty is set up at this point, is that I now feel confident that the goal can be reached. By taking the JamVM route, the number of unimplemented cans of worms the original Jamiga VM had, has greatly been reduced.

Milestone 1 possible applications

I thought I'd collect a small collection of tools that will run on Jamiga2 when milestone 1 is done (more info on what milestone 1 is will follow). These will consist of either currently available console applications, or existing Java open-source frameworks packaged by me (or someone else) using minimal coding.

Telnet/SSH client

A very simple Telent/SSH client, based on the JTA framework. It actually has a GUI client, but with minor effort a very simple console based telnet client has been made working. See below for a screenshot of it running on my amigaized JamVM source and GNU Classpath 0.99 (with GUI support). The system is Ubuntu Linux in a VMWare on my MacBook Pro, so its Intel, not PPC.

What you see is JamVM spitting out stuff like "[Opened native library...]", "libopen returning true". The text "Connected!" and the following lines are actually the Java program connecting to the telnet daemon on MacOS. I entered "uptime" which was sent to MacOS, who so kindly responded with the uptime. I wrote "quit", which the Java program takes as a signal I want to quit, and then JamVM spits out "exit vm", and exits.

Again, nota bene: this runs on Linux and Intel right now, which you probably can deduct from it not looking like Amiga. But just to be clear, so I don't fool anyone.

Console based Twitter client

By using Twitter4J a simple console application allowing for status updates and listing should be feasible. Together with some clever ARexx scripting, integration with f.i. Ringhio should not be impossible.

JamVM running, albeit with a limp

I've been battling with pthreads and signals. I couldn't seem to get Amiga processes and signaling system to function correctly using a simple pthread mapping. I should've figured this, since there is no simple pluginable replacement. There is the AmigaOS 4 pthread implementation, but it doesn't seem to take signals into consideration.

But, now, I've finally managed to get JamVM running and ending in non-crashable manner. I've basically implemented a lot of the pthread and signal functionality, with much regard taken toward adapting it to Amiga-JamVM interaction, rather than a all-purpose pthread re-implementation. I will commit these changes, and also post a blog with more details on my particular solutions sometime (I dare not add "in the near future", since that would probably be lying).

Threading dark alleys

What i now have running is JamVM, with its finalizer, signal handling and garbage collector processes. These JamVM's "utility threads" are Amiga processes, but I'm thinking of changing them to tasks instead, since they really don't do any DOS stuff, and can't be easily interrupted. Tasks can be removed with a simple RemoveTask(), whereas DOS processes has other stuff attached to it that needs closing. I am actually cheating on this point already, since I call abort() in the child processes, which newlib complains about (and actually decides to do a RemoveTask() on the processes -- I'm assuming this is a big memory leak). None the less, the overall functionality is there, practically without alterations to the original JamVM code base! I've tried to only add sources to the architecture and OS specific drawers. Only a few modifications has been made to JamVM, and I believe I'll be able to catch most of the special cases in the Amiga code.

Work continues with the classpath libraries

With JamVM finally running and producing stack traces, I can continue my work in adapting JAmiga's GNU classpath library implementation. I've found that some native reflection methods and fields aren't loaded correctly. I think/hope this is only a matter of better naming in the JAmiga libraries.

How does the native libraries work, anyway?

The native classpath libraries are implemented as ordinary Amiga libraries. This differs from how JamVM handle's dynamic libraries on other platforms. As I've previously mentioned, JamVM uses either *nix shared objects (.so) or Windows DLL. Both these can export actual function names, which Amiga libraries can't, at least not in the same way. Shortly put, when writing libraries, the actual function name can be exported, and looked up by whatever code wants to use them. A Java method that is executed in native code (i.e. written in C or similar), is looked up by a mangled name.

For instance, the isDirectory(String) method in class java.io.VMFile is in the DLL called Java_java_io_VMFile_isDirectory__Ljava_lang_String_2. This is handled dynamically by the VM, and it can try with various mangled name variants for the same method before deciding it doesn't exist. In a Windows DLL or .so object, it is enough to have a function called "Java_java_io_VMFile_isDirectory__Ljava_lang_String_2" and prefixed with an JNI_EXPORT define. For the JAmiga libraries, we must however create a mapping that defines the class and method as simple strings, and then make a manual lookup. This was the way the JAmiga libraries was implemented already. However, the JAmiga mangling scheme doesn't always adhere to the actual Java mangling standard. Sometimes the functions are mangled with the prefix "Jamiga_" instead of "Java_", and the signatures isn't always entirely correct. So, that is the next step: make sure all native methods are correclty mangled.

JamVM in the making

I have actually received a few donations! I'd like to start by saying thank you, you know who you are!

After a much needed vacation on the French Riviera, I still have a few days off work, and I can put a wee bit more effort into JAmiga/JamVM.

As previously stated I'm focused on JamVM. I've just switched from version 1.5.3 to the latest Git-hub version which have all the OpenJDK stuff and new shiny features. I had to make the switch since, a) its the wise thing to do; why bother with old stuff, and b) I stumbled upon the same JNI function that wasn't implemented in the JAmiga VM (*sigh*).

Things we miss in AmigaOS 4 land which makes porting harder

Automake/autoconf etcetera

Currently I'm moving my 1.5.3 changes to the new JamVM, and trying to sort out the makefiles. Which is tedious, boring and I suck at it. JamVM uses Makefile.am which should be automake'd with tools we don't have for AmigaOS (at least I haven't found them in the usual channels). So I'm doing it by hand. Which isn't that hard. Just boring and time consuming.

This is basically done now -- I'm not entirely happy having the makefiles manually tweaked, but it was the most pragmatic solution for the time being.

Signal stuff

• sigaction()
This function

allows the calling process to examine and/or specify the action to be associated with a specific signal

according to the docs.

This is used in JamVM's initializeSignals() to setup the suspendHandler() function being called when SIGUSR1 is signaled to the thread. I have changed this to instead use IExec->SetExcept(), to call the same function upon receiving SIGUSR1.

• pthread_sigmask()
From the man-page:

The pthread_sigmask() function shall examine or change (or both) the calling thread's signal mask

This does the same thing as the function sigprocmask(), but only for one thread. The latter we actually have in clib2's signal.h. We probably don't have it in pthread implementation, since it seems to be using some sort of signaling which AmigaOS don't support by standard.

But what does it do exactly? It is used in JamVM in the functions enableSuspend() and disableSuspend() (and their variations). For enableSuspend the SIGUSR1 signal is unblocked (SIG_UNBLOCK), and for disableSuspend it is blocked (SIG_BLOCK) -- i.e. when suspend is enabled, the thread will be able to receive the SIGUSR1 signal, and when disabled not. This has impact on the sigaction stuff just mentioned, so my theory is that I will be able to again call IExec->SetExcept() and disable/enable the SIGUSR1 exception catching in enableSuspend and disableSuspend, respectively.

• sigsuspend()
The docs says that

sigsuspend() replaces the current signal mask with the set of signals pointed to by sigmask and then suspends the thread until delivery of a signal whose action is either to execute a signal-catching function or to terminate the process

This is used in the JamVM function suspendLoop(), which is called from enableSuspend() and suspendHandler(). The suspend loop suspends execution until any signal, except SIGUSR1 and SIGTERM (these are removed from the mask using sigdelset prior the call) is sent to the thread, while the JamVM thread flag "suspended" is true (JamVM:s internal structure for its threads). Basically the thread is suspended until signalled not to be.

I think this could be interchanged with a Wait() on the appropriate signals. If a SIGTERM is sent to the thread, I'm guessing the thread should be terminated... but on the other hand, the signal mask given to sigsuspend has SIGTERM explicitly unset. I'll have to experiment with this, and investigate further.

PThreads

We (AmigaOS 4 users) do actually have the shared library thread.library which implements most of the pthread functionality, however not the pthread_sigmask (as described in the Signal section above), and these functions:

• pthread_kill()
AmigaOS can't "kill" a process. We could ask it (by sending Ctrl-C or something), but that's not the same thing. So for now this a dummy function that does nothing implemented by me.

sched.h

We don't have the sched.h include, and thus not these JamVM needed functions:

• sched_yield()
According to docs, this

forces the running thread to relinquish the processor until it again becomes the head of its thread list.

I believe that this can in some way be interchanged with SetTaskPri(FindTask(NULL), 0), which "changes priority of a task [and] a reschedule is performed", where the important part being the rescheduling. I have heard that we really can't trigger an explicit rescheduling in AmigaOS (with the pre-emptive multitasking, I assume), so this is the closest we'll get.

Next actions

I will be implementing and testing my theories above. In a future blog I will report more thoroughly on my chosen methods to implement the missing functionality found. Hopefully this can be of use for other porters in Amiga land.

Again, I wish to thank for the donations. If anyone else wan't to donate, the button is at your right. I have been thinking of starting a bounty, but I currently don't know what to expect from my efforts. But I want you all to know that I won't give up until I can run Eclipse and other java stuff on my Amiga, perhaps not mainly on my AmigaONE XE, but more likely (time wise) on my X1000 (which I hope will come before the end of the year).

Looking at JamVM again

In 2009 I had some attempts at JamVM. Now I have actually managed to get JamVM running on AmigaOS 4. You can read more on what was the turning point at amigans.net. So currently I'm going JamVM, all in!

What does this then mean, exactly?

Well, for starters, from my point of view: in february 2010 I posted a blog on my JAmiga efforts. This was in 2010. Now, more than two years later, I have still not managed to get a 0.0.7 version out. Although work has been progressing, it has been slow. I have had the aim to add network support, but it has been a sisyphean task. Whenever I have thought; "just these few things need to be done, then I will have this feature working", there have always been new stuff lurking round the corner.

The things lurking have been mostly involved with stuff not fully implemented in the actual JAmiga virtual machine, i.e. binary executing Java classes by translating the bytecode into native processor instructions. There have also been issues with some JNI calls not fully implemented; like in my previous post regarding shuffling data to and from byte arrays. The way I have been solving it this far have been just trying to get rid of the problem quickly, by either copying stuff from other projects or returning something invented, adding up to quite some hefty technical debt.

Basically, I've been reinventing the wheel, when what I want to concentrate on is the connection between the JVM and the Amiga -- the classpath library. Basically, what I begun implementing in terms of network support. By taking the JamVM route, I have a fully invented wheel, and can concentrate on the Amiga native stuff, like network, GUI/Swing, audio, creating a really tight Amiga integration with ARexx and stuff.

JamVM features

To round of, a few motivating facts about JamVM:

• PowerPC support, quoting Robert Lougher on JamVM:s webpage: "for many years my main platform, so this is well tested".
• Small codebase -- I've got a good overview of all the files. I've looked at Oracle's HotSpot VM, and that is a hairier monster.
• Widely used, JamVM is the primary JVM for ARM Ubuntu
• Thoroughly tested
• Supports both GNU Classpath and OpenJDK

The plan

I've just checked in the latest JamVM branch into the JAmiga SVN, and this will be altered with Amiga modifications. The plan is to first have it compile for AmigaOS 4, using pthreads, and added modifications to load JAmiga's classpath libraries. In its current form JamVM could theoretically use OS 4's .so-object support, but since JAmiga's classpath library is implemented with ordinary Amiga libraries, and this is the way we ultimately want it, I choose to prioritize like this.

Byte off more than one can buffer

This is more of a mental note to self. But it gives a small insight to what I'm battling with.

I've just traced in which way VMNetChannel allocates its ByteBuffer (ByteBuffers are used for pretty much anything handling data, not only network stuff, so its a pretty crucial thing to have).
Starting in VMNetChannel, we find a ByteBuffer.allocate(int capacity). It goes something like this:

1. ByteBuffer.allocate(capacity) calls
2. DirectByteBufferImpl.allocate(capactiy) calls
3. DirectByteBufferImpl.ReadWrite(capacity) ReadWrite is an internal class which calls:
4. ReadWrite constructor calling its super constructor:
5. DirectByteBufferImpl(cap) yep, ReadWrite extends DirectByteBufferImpl
6. DirectByteBufferImpl constructor calls its super (back to ByteBuffer) with a few arguments:
7. ByteBuffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0) which passes on to its super, Buffer
8. Buffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0)

And there we have it!

VMDirectByteBuffer.allocate(capacity) is native call in javanio.library, and actually does something else than throwing itself back and forth.
VMDirectByteBuffer.allocate() returns a Pointer which will be stored in Buffer().

Back to net stuff

After my latest musings with JNI and JVM interfaces, I am finally back at implementing the network support. I have implemented it pretty like described in previous entry and also made the necessary classpath changes. Furthermore, the classpath implementation, java-net library, now fully depends on the various JVM_-functions exported by the main JAmiga engine, i.e. JVM_Socket, JVM_Bind and so on. This means that the BSDSocket library now can be removed from java-net library, since this is all handled by the JAmiga engine. Hopefully this will lead to a more manageable code base. To conclude, what is left to get the network support fully functioning and releasable is:

• cleanup of code
• make sure the java-net library only does what it is supposed to
• test, test, test

But, first and foremost, what I found late last night: start to implement needed java-nio functionality in the JVM interface. There are some ByteBuffer things that need sorting. These things should already be implemented, somewhere in the code. Probably some connections were lost in my JNI/JVM interfacing.