Status report

With the bounty in place, my aim was to update the blog more often -- preferably when I had some sort of update. However, my time has been lacking the last few weeks, and I haven't made as much progress as I wanted. But the bounty has surely made me more motivated -- your donations are much appreciated. In order to get some well needed progress, I've earmarked two "mellandagar" (i.e. the days between christmas and new year) to fully focus on Jamiga. I'm hoping nothing will interfere with these plans.

Controlling the signal

So, why this blog entry? Well, its because I've actually made some progress! I've managed to sort out the signal handling. Now I cannot only run multi-threaded Java programs, I can also break them forcefully by pressing Ctrl-C. It might not seem like a big whoop, but it is really, really nice to be able to halt the Java process, and being able to do so in a stable manner. I have described the signaling previously, but I'll take it in more detail now.

Signal ping-pong

JamVM is aimed to run on a pthread capable OS. I've managed to wrap all pthread functions needed (as in needed by JamVM, and specifically tied to JamVM -- it is not a general purpose pthread implementation). There are at least four threads needed for one single instance of JamVM:

• The main process
• Finalizer thread
• Reference handler
• Signal handler which should catch any signals, i.e. Ctrl-C and such
• ... plus any number of possible Java threads

The way it's intended, in a pthread environment, the signal handler catches all signals sent to the JamVM process. Upon receiving Ctrl-C the signal handler executes the Java class "System.exit()", and the exits the program by using the C function exit(). There is however a problem with this approach in the Amiga environment. A child process cannot run exit() to end the main process, so AmigaOS decides to only do a RemTask for any child processes calling exit(), which will leave all the other processes running. I managed to circumvent this exit()-calling by some signal sending back and forth; the signal handler actually signals the main process to shut down the virtual machine and the call exit().

In a real pthread environment, all the four threads are run inside one actual OS process, so the OS only sees one process and signals sent can be redirected to either thread. On AmigaOS the four threads are are actually four different processes, so anyone of them can be signaled independently (from like Ranger), and the main process is the most likely target. For instance, when running a Java console application and pressing Ctrl-C, the main process gets signaled, and not the Signal handler. However, this is not a big problem. I merely redirect the signal to the Signal handler, by setting up an exception handler (with SetExcept()) for each process that catches all signals and sends it to the Signal handler. The Signal handler calls Java method "System.exit()", and signals any running Java threads to stop what they're doing, and then signals the main process, which in turn signals any running JVM process (the Finalizer, Reference handler and Signal handler) to stop what they're doing. And then we're done.

What's next?

When the signaling bits are fully done (I must re-factor and cleanup the mess its currently in), I will resurrect my networking stuff. I ended up in this signaling mess trying to end a network process that hung for some reason, and I was fed up with rebooting my Amiga just to release the TCP-ports. You can see a small screenshot above of my attempts (and partial success) with running two Java instances sending data from client to server. The reason I cannot break this using Ctrl-C is that the networking process eats the Ctrl-C. And I guess that's what I'll be doing on my two "mellandagar".

The selected top Shell process with the three child processes is the Java network client (you can see the command "jamvm.exe ServerClientExample", which runs the ServerClientExample class). The next Shell process with the three familiar child processes is the other JamVM instance, running the Java server. This also has two children, the Jamiga socket processes, which are the processes handling the networking, as I described about a year ago.

Coffee and bounty

Coffee and a bounty. What could be better?

We've just set up a bounty on AmigaBounty.net for Java on AmigaOS 4.1. The plan forward consist of three milestones. Milestone 1 covers the basic functionality of a working Java implementation, which also offers some novel applications to AmigaOS. Milestone 2 and 3, will offer even more in terms of user gain, with GUI and graphics capabilities, and applet support.

The reason why this bounty is set up at this point, is that I now feel confident that the goal can be reached. By taking the JamVM route, the number of unimplemented cans of worms the original Jamiga VM had, has greatly been reduced.

Milestone 1 possible applications

I thought I'd collect a small collection of tools that will run on Jamiga2 when milestone 1 is done (more info on what milestone 1 is will follow). These will consist of either currently available console applications, or existing Java open-source frameworks packaged by me (or someone else) using minimal coding.

Telnet/SSH client

A very simple Telent/SSH client, based on the JTA framework. It actually has a GUI client, but with minor effort a very simple console based telnet client has been made working. See below for a screenshot of it running on my amigaized JamVM source and GNU Classpath 0.99 (with GUI support). The system is Ubuntu Linux in a VMWare on my MacBook Pro, so its Intel, not PPC.

What you see is JamVM spitting out stuff like "[Opened native library...]", "libopen returning true". The text "Connected!" and the following lines are actually the Java program connecting to the telnet daemon on MacOS. I entered "uptime" which was sent to MacOS, who so kindly responded with the uptime. I wrote "quit", which the Java program takes as a signal I want to quit, and then JamVM spits out "exit vm", and exits.

Again, nota bene: this runs on Linux and Intel right now, which you probably can deduct from it not looking like Amiga. But just to be clear, so I don't fool anyone.

Console based Twitter client

By using Twitter4J a simple console application allowing for status updates and listing should be feasible. Together with some clever ARexx scripting, integration with f.i. Ringhio should not be impossible.

JamVM running, albeit with a limp

I've been battling with pthreads and signals. I couldn't seem to get Amiga processes and signaling system to function correctly using a simple pthread mapping. I should've figured this, since there is no simple pluginable replacement. There is the AmigaOS 4 pthread implementation, but it doesn't seem to take signals into consideration.

But, now, I've finally managed to get JamVM running and ending in non-crashable manner. I've basically implemented a lot of the pthread and signal functionality, with much regard taken toward adapting it to Amiga-JamVM interaction, rather than a all-purpose pthread re-implementation. I will commit these changes, and also post a blog with more details on my particular solutions sometime (I dare not add "in the near future", since that would probably be lying).

Threading dark alleys

What i now have running is JamVM, with its finalizer, signal handling and garbage collector processes. These JamVM's "utility threads" are Amiga processes, but I'm thinking of changing them to tasks instead, since they really don't do any DOS stuff, and can't be easily interrupted. Tasks can be removed with a simple RemoveTask(), whereas DOS processes has other stuff attached to it that needs closing. I am actually cheating on this point already, since I call abort() in the child processes, which newlib complains about (and actually decides to do a RemoveTask() on the processes -- I'm assuming this is a big memory leak). None the less, the overall functionality is there, practically without alterations to the original JamVM code base! I've tried to only add sources to the architecture and OS specific drawers. Only a few modifications has been made to JamVM, and I believe I'll be able to catch most of the special cases in the Amiga code.

Work continues with the classpath libraries

With JamVM finally running and producing stack traces, I can continue my work in adapting JAmiga's GNU classpath library implementation. I've found that some native reflection methods and fields aren't loaded correctly. I think/hope this is only a matter of better naming in the JAmiga libraries.

How does the native libraries work, anyway?

The native classpath libraries are implemented as ordinary Amiga libraries. This differs from how JamVM handle's dynamic libraries on other platforms. As I've previously mentioned, JamVM uses either *nix shared objects (.so) or Windows DLL. Both these can export actual function names, which Amiga libraries can't, at least not in the same way. Shortly put, when writing libraries, the actual function name can be exported, and looked up by whatever code wants to use them. A Java method that is executed in native code (i.e. written in C or similar), is looked up by a mangled name.

For instance, the isDirectory(String) method in class java.io.VMFile is in the DLL called Java_java_io_VMFile_isDirectory__Ljava_lang_String_2. This is handled dynamically by the VM, and it can try with various mangled name variants for the same method before deciding it doesn't exist. In a Windows DLL or .so object, it is enough to have a function called "Java_java_io_VMFile_isDirectory__Ljava_lang_String_2" and prefixed with an JNI_EXPORT define. For the JAmiga libraries, we must however create a mapping that defines the class and method as simple strings, and then make a manual lookup. This was the way the JAmiga libraries was implemented already. However, the JAmiga mangling scheme doesn't always adhere to the actual Java mangling standard. Sometimes the functions are mangled with the prefix "Jamiga_" instead of "Java_", and the signatures isn't always entirely correct. So, that is the next step: make sure all native methods are correclty mangled.

JamVM in the making

I have actually received a few donations! I'd like to start by saying thank you, you know who you are!

After a much needed vacation on the French Riviera, I still have a few days off work, and I can put a wee bit more effort into JAmiga/JamVM.

As previously stated I'm focused on JamVM. I've just switched from version 1.5.3 to the latest Git-hub version which have all the OpenJDK stuff and new shiny features. I had to make the switch since, a) its the wise thing to do; why bother with old stuff, and b) I stumbled upon the same JNI function that wasn't implemented in the JAmiga VM (*sigh*).

Things we miss in AmigaOS 4 land which makes porting harder

Automake/autoconf etcetera

Currently I'm moving my 1.5.3 changes to the new JamVM, and trying to sort out the makefiles. Which is tedious, boring and I suck at it. JamVM uses Makefile.am which should be automake'd with tools we don't have for AmigaOS (at least I haven't found them in the usual channels). So I'm doing it by hand. Which isn't that hard. Just boring and time consuming.

This is basically done now -- I'm not entirely happy having the makefiles manually tweaked, but it was the most pragmatic solution for the time being.

Signal stuff

• sigaction()
This function

allows the calling process to examine and/or specify the action to be associated with a specific signal

according to the docs.

This is used in JamVM's initializeSignals() to setup the suspendHandler() function being called when SIGUSR1 is signaled to the thread. I have changed this to instead use IExec->SetExcept(), to call the same function upon receiving SIGUSR1.

• pthread_sigmask()
From the man-page:

The pthread_sigmask() function shall examine or change (or both) the calling thread's signal mask

This does the same thing as the function sigprocmask(), but only for one thread. The latter we actually have in clib2's signal.h. We probably don't have it in pthread implementation, since it seems to be using some sort of signaling which AmigaOS don't support by standard.

But what does it do exactly? It is used in JamVM in the functions enableSuspend() and disableSuspend() (and their variations). For enableSuspend the SIGUSR1 signal is unblocked (SIG_UNBLOCK), and for disableSuspend it is blocked (SIG_BLOCK) -- i.e. when suspend is enabled, the thread will be able to receive the SIGUSR1 signal, and when disabled not. This has impact on the sigaction stuff just mentioned, so my theory is that I will be able to again call IExec->SetExcept() and disable/enable the SIGUSR1 exception catching in enableSuspend and disableSuspend, respectively.

• sigsuspend()
The docs says that

sigsuspend() replaces the current signal mask with the set of signals pointed to by sigmask and then suspends the thread until delivery of a signal whose action is either to execute a signal-catching function or to terminate the process

This is used in the JamVM function suspendLoop(), which is called from enableSuspend() and suspendHandler(). The suspend loop suspends execution until any signal, except SIGUSR1 and SIGTERM (these are removed from the mask using sigdelset prior the call) is sent to the thread, while the JamVM thread flag "suspended" is true (JamVM:s internal structure for its threads). Basically the thread is suspended until signalled not to be.

I think this could be interchanged with a Wait() on the appropriate signals. If a SIGTERM is sent to the thread, I'm guessing the thread should be terminated... but on the other hand, the signal mask given to sigsuspend has SIGTERM explicitly unset. I'll have to experiment with this, and investigate further.

PThreads

We (AmigaOS 4 users) do actually have the shared library thread.library which implements most of the pthread functionality, however not the pthread_sigmask (as described in the Signal section above), and these functions:

• pthread_kill()
AmigaOS can't "kill" a process. We could ask it (by sending Ctrl-C or something), but that's not the same thing. So for now this a dummy function that does nothing implemented by me.

sched.h

We don't have the sched.h include, and thus not these JamVM needed functions:

• sched_yield()
According to docs, this

forces the running thread to relinquish the processor until it again becomes the head of its thread list.

I believe that this can in some way be interchanged with SetTaskPri(FindTask(NULL), 0), which "changes priority of a task [and] a reschedule is performed", where the important part being the rescheduling. I have heard that we really can't trigger an explicit rescheduling in AmigaOS (with the pre-emptive multitasking, I assume), so this is the closest we'll get.

Next actions

I will be implementing and testing my theories above. In a future blog I will report more thoroughly on my chosen methods to implement the missing functionality found. Hopefully this can be of use for other porters in Amiga land.

Again, I wish to thank for the donations. If anyone else wan't to donate, the button is at your right. I have been thinking of starting a bounty, but I currently don't know what to expect from my efforts. But I want you all to know that I won't give up until I can run Eclipse and other java stuff on my Amiga, perhaps not mainly on my AmigaONE XE, but more likely (time wise) on my X1000 (which I hope will come before the end of the year).

Looking at JamVM again

In 2009 I had some attempts at JamVM. Now I have actually managed to get JamVM running on AmigaOS 4. You can read more on what was the turning point at amigans.net. So currently I'm going JamVM, all in!

What does this then mean, exactly?

Well, for starters, from my point of view: in february 2010 I posted a blog on my JAmiga efforts. This was in 2010. Now, more than two years later, I have still not managed to get a 0.0.7 version out. Although work has been progressing, it has been slow. I have had the aim to add network support, but it has been a sisyphean task. Whenever I have thought; "just these few things need to be done, then I will have this feature working", there have always been new stuff lurking round the corner.

The things lurking have been mostly involved with stuff not fully implemented in the actual JAmiga virtual machine, i.e. binary executing Java classes by translating the bytecode into native processor instructions. There have also been issues with some JNI calls not fully implemented; like in my previous post regarding shuffling data to and from byte arrays. The way I have been solving it this far have been just trying to get rid of the problem quickly, by either copying stuff from other projects or returning something invented, adding up to quite some hefty technical debt.

Basically, I've been reinventing the wheel, when what I want to concentrate on is the connection between the JVM and the Amiga -- the classpath library. Basically, what I begun implementing in terms of network support. By taking the JamVM route, I have a fully invented wheel, and can concentrate on the Amiga native stuff, like network, GUI/Swing, audio, creating a really tight Amiga integration with ARexx and stuff.

JamVM features

To round of, a few motivating facts about JamVM:

• PowerPC support, quoting Robert Lougher on JamVM:s webpage: "for many years my main platform, so this is well tested".
• Small codebase -- I've got a good overview of all the files. I've looked at Oracle's HotSpot VM, and that is a hairier monster.
• Widely used, JamVM is the primary JVM for ARM Ubuntu
• Thoroughly tested
• Supports both GNU Classpath and OpenJDK

The plan

I've just checked in the latest JamVM branch into the JAmiga SVN, and this will be altered with Amiga modifications. The plan is to first have it compile for AmigaOS 4, using pthreads, and added modifications to load JAmiga's classpath libraries. In its current form JamVM could theoretically use OS 4's .so-object support, but since JAmiga's classpath library is implemented with ordinary Amiga libraries, and this is the way we ultimately want it, I choose to prioritize like this.

Byte off more than one can buffer

This is more of a mental note to self. But it gives a small insight to what I'm battling with.

I've just traced in which way VMNetChannel allocates its ByteBuffer (ByteBuffers are used for pretty much anything handling data, not only network stuff, so its a pretty crucial thing to have).
Starting in VMNetChannel, we find a ByteBuffer.allocate(int capacity). It goes something like this:

1. ByteBuffer.allocate(capacity) calls
2. DirectByteBufferImpl.allocate(capactiy) calls
3. DirectByteBufferImpl.ReadWrite(capacity) ReadWrite is an internal class which calls:
4. ReadWrite constructor calling its super constructor:
5. DirectByteBufferImpl(cap) yep, ReadWrite extends DirectByteBufferImpl
6. DirectByteBufferImpl constructor calls its super (back to ByteBuffer) with a few arguments:
7. ByteBuffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0) which passes on to its super, Buffer
8. Buffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0)

And there we have it!

VMDirectByteBuffer.allocate(capacity) is native call in javanio.library, and actually does something else than throwing itself back and forth.
VMDirectByteBuffer.allocate() returns a Pointer which will be stored in Buffer().

Back to net stuff

After my latest musings with JNI and JVM interfaces, I am finally back at implementing the network support. I have implemented it pretty like described in previous entry and also made the necessary classpath changes. Furthermore, the classpath implementation, java-net library, now fully depends on the various JVM_-functions exported by the main JAmiga engine, i.e. JVM_Socket, JVM_Bind and so on. This means that the BSDSocket library now can be removed from java-net library, since this is all handled by the JAmiga engine. Hopefully this will lead to a more manageable code base. To conclude, what is left to get the network support fully functioning and releasable is:

• cleanup of code
• make sure the java-net library only does what it is supposed to
• test, test, test

But, first and foremost, what I found late last night: start to implement needed java-nio functionality in the JVM interface. There are some ByteBuffer things that need sorting. These things should already be implemented, somewhere in the code. Probably some connections were lost in my JNI/JVM interfacing.

JNI and JVM interface

Currently I am working on implementing the JVM interface from OpenJDK. This work is done in parallell with GNU Classpath. I.e. I will try to use OpenJDK's JVM interface in JAmiga's GNU Classpath implementation. Basically I have refactored the classpath specific code from the JAmiga binary into its own library, gnuclasspath.library (an upcoming library will of course be openjdk.library). This will then in turn get access to the JAmiga VM's exported JVM interface, in the same way OpenJDK would do it. In this work I have also found a need to implement more of the JNI standard, and also add support to native libraries for JNI_OnLoad and JNI_OnUnload. These two functions are run when a native library is loaded into JAmiga. The JNI_OnLoad function will, besides whatever it might do that's specific to the library, also initialize the JVM interface.

To summarize what is currently going on with JAmiga:

• Native libraries export JNI_OnLoad/OnUnload
• Implementation of JVM interface
• Implementation of more JNI functions
• Implementation of java-net stuff

The last point is actually what has motivated the other updates. OpenJDK's java-net uses the JVM interface for much of its functionality, and rather than implementing the methods in my own abstraction, I chose the existing one.

Native libraries export JNI_OnLoad/OnUnload

Since there still are native libraries that don't implement JNI_OnLoad/OnUnload, the library needs to be of at least version 2.x. I don't know a way to determine if an Amiga library only implements a few functions. Since the function pointers aren't set, I can't really look for NULL. Perhaps it could be solved with AmigaOS 4 interfaces, but that is not an option.

Implementation of JVM interface

I have started with a struct with function pointers to each JVM function (their basically functions with a name like JVM_Open, JVM_Close, JVM_ArrayCopy and so on). this struct gets copied to each native library when JNI_OnLoad is called. I thought of letting JAmiga registering a "jvm.library" that each native library could open if needed. I kind of like that idea. I might actually go that way in the future.

Implementation of more JNI functions

While moving GNU Classpath stuff, I found lots of places where both JVM interface functions should be used, and also where some new JNI 1.6 methods could be used, for instance the DirectByteBuffer functions are used by java-nio, and java-net.

Implementation of java-net stuff

This is how all the above started. I wanted java-net support. And kind of had it. But then I stumbled upon problem. And I got think that "why not do this, and that". So, java-net was kind of a can of worms. Or, rather, the entire JAmiga is a can of worms.

So, what does this all mean — when will we get Java on Amiga?

Sorry. But this actually doesn't change the actual state of JAmiga — the functionality is still the same as JAmiga 0.0.6. But all these actions taken are very important steps into taking JAmiga to OpenJDK support which is our only chance of getting Java 8 to Amiga, and with that JavaFX (enhanced Swing, i.e. GUI), cool lambda expression (geeky parallell stuff) and, and, all sorts of other stuff.

PS. I also added a Donate-button to the right. If you feel I deserve a danish while coding, any contribution is welcome. A +1 or comment is even more welcome! If enough of you +1 JAmiga I promise to update the blog more.

OpenJDK explorations

I created a VMWare image with Ubuntu 10.04.3 to use for building OpenJDK 8.
With a clean Ubuntu install, including latest updates, I followed this guide: http://hg.openjdk.java.net/jdk8/jdk8/raw-file/tip/README
1. Installed mercurial
apt-get install mercurial

2. Created dir "Projects", in which I cloned the OpenJDK Mercurial repository:
hg clone http://hg.openjdk.java.net/jdk8/jdk8 JAmigaOpenJDK
cd JamigaOpenJDK
sh ./get_source.sh

3. The readme on howto install the Bootstrap JDK tells us to install JDK 6. However, JDK 8 needs JDK 7. This turned out not to be as easy as "aptitude openjdk-7", since the Ubuntu 10 aptitude repository doesn't contain JDK 7. Google helped me find: instructions on how to do it. You apparently should download the thing from Oracle directly. So:
3.1. Download Java 7 SDK from Oracle.
3.2. Untar and move to /usr/lib/jvm
tar xf jdk-7u3-linux-i586.tar.gz
mv jdk1.7.0_3 /usr/lib/jvm
export LANG=C ALT_BOOTDIR=/usr/lib/jvm/jdk1.7.0_3

Now the sanity check should complete ok (remember to CD to JAmigaOpenJDK):
make sanity
...
Sanity check passed.


4. FreeType needs to be in version 2.3 or newer. Running freetype-config --ftversion said version 2.3.11 was installed. Must be good, right?

5. Apache Ant v 1.7.1 is also required, which already is available in my Ubuntu.

6. Next, fire away a make all and patiently wait.
make all

7. The build/linux-i586/j2re-image directory contain the stuff built.

in order to ease my development I downloaded a JDK branch: hg.openjdk.java.net/jdk8/jdk8/jdk/rev/0b052b7f3f83 using the ZIP-link at hg.openjdk.java.net, jdk8/jdk8/jdk. This contains the bsd/linux/macosx/solaris/windows and share stuff. This should be the place to put JAmiga specific stuff. My aim is to build this on the Amiga, and use j2re-image. I'm assuming this won't work in the long run; I'll probably have to change some stuff in my JAmigaOpenJDK-ubuntu install and rebuild to get everything correct.

At it, again

After my last JAmiga-hiatus, I am now trying to remember what I previously did, and what and I should do next. Looking at my own local classpath files, I am kind of confused over what I have done, which files have change, and what I haven't done. Meanwhile, the GNU Classpath has been released in version 0.99, which is somewhat a surprise to me, since I thought it was more or less abandoned in favour of OpenJDK. Anyhow, my two options for my continued effort to get JAmiga more amture is to either clean up the mess I have with the GNU Classpath, or do some testing with OpenJDK. I am really tempted to take a look at OpenJDK 8, since it really seems to be the most wise thing to do. Either way, I have already begun the work in creating a JVM-interface which should work with both GNU Classpath and OpenJDK. So at worst, I will get some knowledge in OpenJDK but resort to continuing with the GNU Classpath on JAmiga.

Currently, my Mac is at hard work installing Ubuntu, on to which I will do my OpenJDK explorations. And this time I will document every single thing I do.

Changing channel

Picking up my networking work. I created a VMNetChannel which holds all socket/net-related operations and altering VMChannel to only contain file I/O. Looking at where VMChannel is used, I can only find the entries listed in the screenshot.

Hopefully it is only a matter of:
1. Removing all net-related methods in VMChannel
2. Replacing VMChannel with VMNetChannel in net-classes
3. Recompile, ship and release! ;) Well... sort of.

Looking closer at the screenshot reveals that VMChannel is used in the VMChannelOwner interface. This interface makes sure that Channel implementations (i.e. any *ChannelImpl-class) can return a VMChannel... so I'm assuming this would also have to be replicated with a VMNetChannelOwner interface, which consequently must be implemented by any net/socket-related ChannelImpl-classes (i.e. *SocketChannelImpl). This will most likely lead to even more dependencies since each class using these implementations rely on VMChannel methods. Another solution would be to change the VMChannel class to an interface implemented by two new classes, VMFileChannel and VMNetChannel. They would throw exceptions for any un-implemented methods. That way, the classpath would still compile without to much change, but could/would(/should?) fail in runtime. One perk of the second solution would be that changes would only (looking quickly at the search result) be made in the reference-classes (the last three entries in the vm/reference/-drawer), which is good from a maintainability point-of-view. (Any VM-specific stuff should go in the vm-drawer, so any VM can use the non-VM-specific stuff [not in /vm/reference/]. If I were to make changes in other classes, these classes would have to be moved to the vm-drawer and affect more parts of the GNU Classpath.)

Don't know how much time I will have for Jamiga the next few months though... Hopefully some, but probably not much.