Looking at JamVM again

In 2009 I had some attempts at JamVM. Now I have actually managed to get JamVM running on AmigaOS 4. You can read more on what was the turning point at amigans.net. So currently I'm going JamVM, all in!

What does this then mean, exactly?

Well, for starters, from my point of view: in february 2010 I posted a blog on my JAmiga efforts. This was in 2010. Now, more than two years later, I have still not managed to get a 0.0.7 version out. Although work has been progressing, it has been slow. I have had the aim to add network support, but it has been a sisyphean task. Whenever I have thought; "just these few things need to be done, then I will have this feature working", there have always been new stuff lurking round the corner.

The things lurking have been mostly involved with stuff not fully implemented in the actual JAmiga virtual machine, i.e. binary executing Java classes by translating the bytecode into native processor instructions. There have also been issues with some JNI calls not fully implemented; like in my previous post regarding shuffling data to and from byte arrays. The way I have been solving it this far have been just trying to get rid of the problem quickly, by either copying stuff from other projects or returning something invented, adding up to quite some hefty technical debt.

Basically, I've been reinventing the wheel, when what I want to concentrate on is the connection between the JVM and the Amiga -- the classpath library. Basically, what I begun implementing in terms of network support. By taking the JamVM route, I have a fully invented wheel, and can concentrate on the Amiga native stuff, like network, GUI/Swing, audio, creating a really tight Amiga integration with ARexx and stuff.

JamVM features

To round of, a few motivating facts about JamVM:

• PowerPC support, quoting Robert Lougher on JamVM:s webpage: "for many years my main platform, so this is well tested".
• Small codebase -- I've got a good overview of all the files. I've looked at Oracle's HotSpot VM, and that is a hairier monster.
• Widely used, JamVM is the primary JVM for ARM Ubuntu
• Thoroughly tested
• Supports both GNU Classpath and OpenJDK

The plan

I've just checked in the latest JamVM branch into the JAmiga SVN, and this will be altered with Amiga modifications. The plan is to first have it compile for AmigaOS 4, using pthreads, and added modifications to load JAmiga's classpath libraries. In its current form JamVM could theoretically use OS 4's .so-object support, but since JAmiga's classpath library is implemented with ordinary Amiga libraries, and this is the way we ultimately want it, I choose to prioritize like this.

Byte off more than one can buffer

This is more of a mental note to self. But it gives a small insight to what I'm battling with.

I've just traced in which way VMNetChannel allocates its ByteBuffer (ByteBuffers are used for pretty much anything handling data, not only network stuff, so its a pretty crucial thing to have).
Starting in VMNetChannel, we find a ByteBuffer.allocate(int capacity). It goes something like this:

1. ByteBuffer.allocate(capacity) calls
2. DirectByteBufferImpl.allocate(capactiy) calls
3. DirectByteBufferImpl.ReadWrite(capacity) ReadWrite is an internal class which calls:
4. ReadWrite constructor calling its super constructor:
5. DirectByteBufferImpl(cap) yep, ReadWrite extends DirectByteBufferImpl
6. DirectByteBufferImpl constructor calls its super (back to ByteBuffer) with a few arguments:
7. ByteBuffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0) which passes on to its super, Buffer
8. Buffer(capacity, capacity, 0, -1, VMDirectByteBuffer.allocate(capacity), null, 0)

And there we have it!

VMDirectByteBuffer.allocate(capacity) is native call in javanio.library, and actually does something else than throwing itself back and forth.
VMDirectByteBuffer.allocate() returns a Pointer which will be stored in Buffer().

Back to net stuff

After my latest musings with JNI and JVM interfaces, I am finally back at implementing the network support. I have implemented it pretty like described in previous entry and also made the necessary classpath changes. Furthermore, the classpath implementation, java-net library, now fully depends on the various JVM_-functions exported by the main JAmiga engine, i.e. JVM_Socket, JVM_Bind and so on. This means that the BSDSocket library now can be removed from java-net library, since this is all handled by the JAmiga engine. Hopefully this will lead to a more manageable code base. To conclude, what is left to get the network support fully functioning and releasable is:

• cleanup of code
• make sure the java-net library only does what it is supposed to
• test, test, test

But, first and foremost, what I found late last night: start to implement needed java-nio functionality in the JVM interface. There are some ByteBuffer things that need sorting. These things should already be implemented, somewhere in the code. Probably some connections were lost in my JNI/JVM interfacing.

JNI and JVM interface

Currently I am working on implementing the JVM interface from OpenJDK. This work is done in parallell with GNU Classpath. I.e. I will try to use OpenJDK's JVM interface in JAmiga's GNU Classpath implementation. Basically I have refactored the classpath specific code from the JAmiga binary into its own library, gnuclasspath.library (an upcoming library will of course be openjdk.library). This will then in turn get access to the JAmiga VM's exported JVM interface, in the same way OpenJDK would do it. In this work I have also found a need to implement more of the JNI standard, and also add support to native libraries for JNI_OnLoad and JNI_OnUnload. These two functions are run when a native library is loaded into JAmiga. The JNI_OnLoad function will, besides whatever it might do that's specific to the library, also initialize the JVM interface.

To summarize what is currently going on with JAmiga:

• Native libraries export JNI_OnLoad/OnUnload
• Implementation of JVM interface
• Implementation of more JNI functions
• Implementation of java-net stuff

The last point is actually what has motivated the other updates. OpenJDK's java-net uses the JVM interface for much of its functionality, and rather than implementing the methods in my own abstraction, I chose the existing one.

Native libraries export JNI_OnLoad/OnUnload

Since there still are native libraries that don't implement JNI_OnLoad/OnUnload, the library needs to be of at least version 2.x. I don't know a way to determine if an Amiga library only implements a few functions. Since the function pointers aren't set, I can't really look for NULL. Perhaps it could be solved with AmigaOS 4 interfaces, but that is not an option.

Implementation of JVM interface

I have started with a struct with function pointers to each JVM function (their basically functions with a name like JVM_Open, JVM_Close, JVM_ArrayCopy and so on). this struct gets copied to each native library when JNI_OnLoad is called. I thought of letting JAmiga registering a "jvm.library" that each native library could open if needed. I kind of like that idea. I might actually go that way in the future.

Implementation of more JNI functions

While moving GNU Classpath stuff, I found lots of places where both JVM interface functions should be used, and also where some new JNI 1.6 methods could be used, for instance the DirectByteBuffer functions are used by java-nio, and java-net.

Implementation of java-net stuff

This is how all the above started. I wanted java-net support. And kind of had it. But then I stumbled upon problem. And I got think that "why not do this, and that". So, java-net was kind of a can of worms. Or, rather, the entire JAmiga is a can of worms.

So, what does this all mean — when will we get Java on Amiga?

Sorry. But this actually doesn't change the actual state of JAmiga — the functionality is still the same as JAmiga 0.0.6. But all these actions taken are very important steps into taking JAmiga to OpenJDK support which is our only chance of getting Java 8 to Amiga, and with that JavaFX (enhanced Swing, i.e. GUI), cool lambda expression (geeky parallell stuff) and, and, all sorts of other stuff.

PS. I also added a Donate-button to the right. If you feel I deserve a danish while coding, any contribution is welcome. A +1 or comment is even more welcome! If enough of you +1 JAmiga I promise to update the blog more.

OpenJDK explorations

I created a VMWare image with Ubuntu 10.04.3 to use for building OpenJDK 8.
With a clean Ubuntu install, including latest updates, I followed this guide: http://hg.openjdk.java.net/jdk8/jdk8/raw-file/tip/README
1. Installed mercurial
apt-get install mercurial

2. Created dir "Projects", in which I cloned the OpenJDK Mercurial repository:
hg clone http://hg.openjdk.java.net/jdk8/jdk8 JAmigaOpenJDK
cd JamigaOpenJDK
sh ./get_source.sh

3. The readme on howto install the Bootstrap JDK tells us to install JDK 6. However, JDK 8 needs JDK 7. This turned out not to be as easy as "aptitude openjdk-7", since the Ubuntu 10 aptitude repository doesn't contain JDK 7. Google helped me find: instructions on how to do it. You apparently should download the thing from Oracle directly. So:
3.1. Download Java 7 SDK from Oracle.
3.2. Untar and move to /usr/lib/jvm
tar xf jdk-7u3-linux-i586.tar.gz
mv jdk1.7.0_3 /usr/lib/jvm
export LANG=C ALT_BOOTDIR=/usr/lib/jvm/jdk1.7.0_3

Now the sanity check should complete ok (remember to CD to JAmigaOpenJDK):
make sanity
...
Sanity check passed.


4. FreeType needs to be in version 2.3 or newer. Running freetype-config --ftversion said version 2.3.11 was installed. Must be good, right?

5. Apache Ant v 1.7.1 is also required, which already is available in my Ubuntu.

6. Next, fire away a make all and patiently wait.
make all

7. The build/linux-i586/j2re-image directory contain the stuff built.

in order to ease my development I downloaded a JDK branch: hg.openjdk.java.net/jdk8/jdk8/jdk/rev/0b052b7f3f83 using the ZIP-link at hg.openjdk.java.net, jdk8/jdk8/jdk. This contains the bsd/linux/macosx/solaris/windows and share stuff. This should be the place to put JAmiga specific stuff. My aim is to build this on the Amiga, and use j2re-image. I'm assuming this won't work in the long run; I'll probably have to change some stuff in my JAmigaOpenJDK-ubuntu install and rebuild to get everything correct.

At it, again

After my last JAmiga-hiatus, I am now trying to remember what I previously did, and what and I should do next. Looking at my own local classpath files, I am kind of confused over what I have done, which files have change, and what I haven't done. Meanwhile, the GNU Classpath has been released in version 0.99, which is somewhat a surprise to me, since I thought it was more or less abandoned in favour of OpenJDK. Anyhow, my two options for my continued effort to get JAmiga more amture is to either clean up the mess I have with the GNU Classpath, or do some testing with OpenJDK. I am really tempted to take a look at OpenJDK 8, since it really seems to be the most wise thing to do. Either way, I have already begun the work in creating a JVM-interface which should work with both GNU Classpath and OpenJDK. So at worst, I will get some knowledge in OpenJDK but resort to continuing with the GNU Classpath on JAmiga.

Currently, my Mac is at hard work installing Ubuntu, on to which I will do my OpenJDK explorations. And this time I will document every single thing I do.

Changing channel

Picking up my networking work. I created a VMNetChannel which holds all socket/net-related operations and altering VMChannel to only contain file I/O. Looking at where VMChannel is used, I can only find the entries listed in the screenshot.

Hopefully it is only a matter of:
1. Removing all net-related methods in VMChannel
2. Replacing VMChannel with VMNetChannel in net-classes
3. Recompile, ship and release! ;) Well... sort of.

Looking closer at the screenshot reveals that VMChannel is used in the VMChannelOwner interface. This interface makes sure that Channel implementations (i.e. any *ChannelImpl-class) can return a VMChannel... so I'm assuming this would also have to be replicated with a VMNetChannelOwner interface, which consequently must be implemented by any net/socket-related ChannelImpl-classes (i.e. *SocketChannelImpl). This will most likely lead to even more dependencies since each class using these implementations rely on VMChannel methods. Another solution would be to change the VMChannel class to an interface implemented by two new classes, VMFileChannel and VMNetChannel. They would throw exceptions for any un-implemented methods. That way, the classpath would still compile without to much change, but could/would(/should?) fail in runtime. One perk of the second solution would be that changes would only (looking quickly at the search result) be made in the reference-classes (the last three entries in the vm/reference/-drawer), which is good from a maintainability point-of-view. (Any VM-specific stuff should go in the vm-drawer, so any VM can use the non-VM-specific stuff [not in /vm/reference/]. If I were to make changes in other classes, these classes would have to be moved to the vm-drawer and affect more parts of the GNU Classpath.)

Don't know how much time I will have for Jamiga the next few months though... Hopefully some, but probably not much.