Not Not Porting 9front to Power64

In my last blog I documented some admittedly frustrating interactions with getting this Raptor Computing Talos 2 up and running again. It ended with me not certain about if I was going to get this running again or if I was going to wind up with a very expensive paper weight. So it is with great excitement that I say that indeed the talos is working again.

The rest of the story is that with a couple more weeks of patience and back and forth I was able to have Raptor identify some components that were damanged on the board, have them identify the values for said components, and had a very good friend of mine attempt the repair. Happy to say everything went as planned and the Talos is currently sitting beside me runing Debian stable.

So compared to my previous blog I do owe some apology, Raptor did end up helping me get things fixed and I'm grateful that we were able to get here. However, I do still stand by my evaluation that the customer support is a bit... less than ideal. Talking to some other members of the talos IRC seems to confirm that this is a somewhat common occurance unfortunately, and I think this is worth mentioning to those who are interested in getting one of these machines. But enough grumbling, let's get to the fun stuff.

Start of the port

Now with a working device I had no excuse to start the porting process. I rebased my old branch on top of current 9front and set to work again. Where I had left things was a userspace (compiler, linker, assembler, libc) that was capable of building the entire system and a proof of concept for running a binary from skiboot. However to move forward I had mostly decided that we should build a kernel that is compatible with petitboot(linux kexec) in order to have others be able to boot the machine from their existing firmware.

So first up was figuring out how kexec worked, no documentation of course so straight into the code it is. What we found was that powerpc64 kexec drops you in to a random spot in physical memory, passes you your current location in a register and says "good luck". This is slightly annoying because in 9front we don't generate relocatable binaries, we pick one spot for things to be in memory and go with it, which meant that we had to do relocation here. Not too bad, a couple of assembly loops later and we had our kernel in a spot in physical memory where it expected itself to be, not too bad. There were a couple quirks along the way with this, like how kexec only uses the first segment of the ELF file, or how linux kexec limits where the sections of the ELF request itself to be put, even if none of that is honored by the kexec userspace utility, but whatever it works.

Most of our console is going to be done through the bmc console, which means we use OPAL calls in order to print and read. Not too big of a deal a couple of assembly shims and now we're printing this to the console. There is some of a mismatch between the ABI to OPAL and our own internal ABI between functions that makes writing the assembly shims a bit dangerous, but we got them working. At this point I was back to where I was before with my little POC before with skiboot.

Next up was getting us over in to C, which was pretty easy just have to be slightly careful around how to set the static base (TOC in ppc terminology) because we still don't have the MMU up, but nothing too difficult. What we did find however was that our linker and compiler were choking on 64 bit constants, so I had to spend a day or so modifying those to generate acceptable (for the time being anyway) instruction synthesis for 64 bit constant operations. This took a little whiles and lots of staring at the ISA to see if there was a better way to do things. For a worst case 64 bit immedaite, a load is 5 instructions (ouch!) but you can get away with less instructions for portions of that 64 bit number that are zerod. So what the linker does now and try to fit a constant within one of those cheaper paths, then falling back to the expensive path. A better solution may be to have the compiler generate a constant pool and do 64 bit loads here instead of using immediates however with how we bootstrap our statis base, we'll need to use that worst case at least once before we can use SB to address data in the DATA segment. The arm64 compiler on 9front does things a bit different and instead just generates constants into the instruction stream itself, and uses PC relative 64 bit loads. This would also work for boostrappnig the SB, but is a bit more complex to implement. Lots to think about...

With that mostly out of the way (at least for now), we need to focus on bringing up the MMU. Which for the time being is lots of time reading the ISA manual and checking against what other systems do. Interestingly POWER9 has two different methods of configuring the MMU, there is a more classical "Hashed Page Table" design and a newer radix tree implementation. The radix tree is a bit more typical to other systems, but only works on POWER9 and up and I am curious to try the "true powerpc way" with the hashed page tables. I haven't attempted to write the code for bringup yet but hope to get to that soon, still need to read over that section of the ISA a couple more times though to get a proper understanding.

That's all for now, maybe I'll write another blog post once I get further along.