Blabbing About my Foenix Device Experiments

I wanted to write just a quick post (EDIT: Ok, it turned out to not be so quick) to share some updates about what I've been working on these past months. I originally thought this year that I would return to some DOS projects. That may still happen later on in the year, but not right now anyway.

Instead I actually returned to the Foenix "new-retro" computers I have, which now includes the F256K which I wrote about previously and as of December, now also includes an A2560K.

I originally received my F256K in February 2024, and played around with the variant of BASIC that it ships with, SuperBASIC, for a while. I never actually produced anything noteworthy with it, but it was fun to mess with. For most of 2024 however, it sat in a box unused.

Then in late December, I got notice that my A2560K that I originally ordered back in Spring 2022 was going to ship soon. I had actually kind of forgotten about this honestly (I was looking forward to it, but it had been so long it was just out of my mind for the most part), so getting that email notification was a nice surprise. Anyway, it arrived a week or so later, and it's pretty darn cool.

The A2560K ships with FoenixMCP loaded onto it, which acts as the operating system essentially.

I had been looking forward to the A2560K, as I'm much more interested in the Motorola 68000 CPU in retro-devices than I am in the 6502/65816 CPU that the F256K has. I'll freely admit that I don't particularly like 6502 assembly. It is simple though, which is admittedly a refreshing change. However, 8 and 16-bit CPUs in general are just a little bit too "primitive" for my personal tastes. I generally draw the line for myself at whether a CPU has built-in hardware support for multiplication and division (even if such hardware support for it is slow). I just don't really get all that excited about having to write my own low-level basic math functions for such fundamental things like that. I know a lot of others in the retro-world disagree though.

So I started finally really digging into the Calypsi toolchain which supports the A2560-line of Foenix devices too. Put together some basic "hello world"-esque projects easily enough and got them running on my A2560K. However after not too long, I ended up running into weird issues. Stuff like graphical corruption or other weird issues that only happened "once in a while." You know, those kind of really fun to debug issues that everyone loves. After digging around in the Foenix Discord for anything about this (can I take this opportunity to say again how much I hate projects where all the information is locked away in Discord? ugh!) I came to realize that this was (maybe?) somewhat of a known issue.

The problem was that I was using FoenixMgr's --run-pgz option to load a .pgz file (essentially a binary executable) and run it directly on device. The way that FoenixMgr does this currently is it loads the PGZ into memory over the USB debug port and then directly modifies the CPU's reset vector to be equal to the PGZ's execution start memory address, and then it performs a CPU reset which will start execution of the PGZ due to the modified reset vector. This works but apparently it was not a great setup if you were writing code that utilized functionality provided by FoenixMCP (such as basic text/console I/O support, e.g. using stuff like printf in C). Apparently this method of starting up a PGZ would eventually cause some memory corruption. At the time I didn't really understand enough of the low-level details to know what the actual issues were, although as I've grown to learn much more about how Calypsi's C startup routines work as well as having spent a lot of time looking at FoenixMCP over the past months, I feel like now I understand far better what was happening. However, as was recently discovered (and fixed), another issue was that on the A2560 devices, the USB debug port has 32-bit alignment requirements, and FoenixMgr was not adhering to this. So this was also almost certainly accounting for at least some of the weird issues I was experiencing.

However, back in December, this was an annoying problem to hit! But since this was all open-source stuff, I figured, why not dig in myself? I cloned the FoenixMCP Git repository, installed VBCC (which is what FoenixMCP is built with, not Calypsi) and played around with some simple modifications to FoenixMCP, just to get my feet wet, and flashed them back to my A2560K. This seemed to work fine for a very short while, but usually after using my modified builds of FoenixMCP for more than a few minutes, I started noticing other "only happens once in a while"-style issues. After a whole lot of time troubleshooting and investigating, I realized that my own builds of the FoenixMCP when using completely un-modified source code (that is, immediately after a fresh git clone) were not producing identical results to what had originally been flashed to my A2560K (after Stefany had kindly provided the exact source code for the fork of FoenixMCP that ships on the A2560K40 and A2560K60's that people like me are now receiving in 2024/2025). I tried some different versions of VBCC to see if it was just some version difference, but no alternative version would ever produce identical compiled output. To be clear, what I was comparing was the intermediate assembly output produced by the compiler, where there was always some differences (excluding very minor things like register usage differences).

Frustrated, I put my A2560K back in the box. This was also almost exactly around the time that the A2560M was announced, which further made me sad. "Great", I said to myself "I spent two years waiting to receive this, and by the time I receive it, it's effectively discontinued." This disappointment was even further compounded by the realization after searching through various Discord messages that the FPGA implementation of the graphics support on the existing A2560K devices had some known issues and possibly was even missing some features outright. These FPGA issues were acknowledged by Stefany and she promised to address them, but that she needed to spend time refactoring the FPGA for all A2560 devices to make maintenance and future enhancements easier.

So ... that's fine? I guess? I waited two years already, I guess I can wait some more.

After being a bit grumpy about all of this for a few weeks, I decided to take my F256K back out of the box it had been sitting in and finally give it some serious use. This was mainly because I was intrigued by what I'd been reading about via the Foenix Discord regarding the Foenix Toolbox. This was a sort of improved version of the FoenixMCP, but built with Calypsi this time. And for F256 devices, was specifically targetting the 65816 CPU, using an alternative FPGA load that allows the 65816's native 24-bit memory addressing mode. Basically, enabling you to use a "flat" memory model instead of the more familiar mode of the 6502 that was used on devices such as the Commodore 64 where the CPU has a 64KB "window" into the total memory available, which could be paged in or out via an MMU. Using such a "flat" memory model on the F256K was infinitely more interesting to me than messing about with an MMU. I'm sorry to anyone reading this, but my retro nostalgia only goes so far! I feel like I'm more willing to tolerate MS-DOS's 640KB segmented memory model limitations just because 640KB is much more than 64KB, heh. But to be clear, I don't much like MS-DOS's segmented memory model either!

The Foenix Toolbox is pretty bare. When you boot it up, you get greeted by a simple graphical bootloader that does nothing more than scan for bootable code in various places, such as the SD card, flash, and RAM.

As I understand it, the design of the Foenix Toolbox is intended to be minimal. It aims to basically only provide a means to boot your code and other applications, as well as to provide kernel functions that user code can use and initialize the hardware for use at bootup. This is a bit different from the FoenixMCP which did those things too, but also provided a neat command prompt / CLI interface.

There were some issues I had noticed with the Foenix Toolbox, and so because I am never satisfied with anything, I once again figured, why not dig in myself?

This too, ended up being a somewhat frustrating experience. In particular I noticed that once again, my custom builds would exhibit some odd behaviours at times. Thankfully though, it was nothing like what I had noticed with FoenixMCP and VBCC. I could always produce identical 1:1 binaries from the original source. The issues that I would notice were almost always regarding bootup where sometimes it wouldn't finish booting (most often getting stuck at an empty blue screen with only a blinking text cursor). Finally hooking up a USB serial cable to the DB9 port on the F256K and monitoring the debug logging output sent via the serial connection, showed that it wasn't always getting stuck at the same places, and worse, that it seemed like making completely unrelated modifications to the code, rebuilding and booting would "fix" the problem. Ugh.

So after stewing about it for about a month, back in late February, I decided on a kind of silly course of action. I was convinced that there was some memory corruption issue or something of that nature in the Foenix Toolbox. But that at least the Calypsi toolchain seemed to be a bit more sane / approachable compared to VBCC, so I decided that I was going to do a super detailed "deep-dive" over the entire Foenix Toolbox code, using a method I've used usually on older "legacy code" (to be clear, I'm not calling Foenix Toolbox "legacy") projects at various jobs I've had over the years that I've started to call a "code transplant." The idea is you slowly take code from the original project, bit by bit, into a new project and test it at every single step of the way. I most often use this approach when I'm trying to upgrade legacy code to a new version of a major framework it uses where the new version requires a great many changes to be made, or something of that sort. My idea with using this approach to help me find and diagnose projects here with the Foenix Toolbox was that it would give me a great opportunity to go over the code line-by-line, function-by-function and along the way, gain a better understanding of where it might be doing things "incorrectly" that might be contributing to the problems I had been seeing. Maybe. Overkill? Possibly, yes.

One thing I did as I went, was I stripped out support for anything other than the F256K along the way. This was unfortunate, but it was done out of desparation. The F256K is the only F256 device I have, and thusly, the only one I can test on, and to be quite frank, with how frustrated I was feeling, it was also the only device I actually cared about supporting at this point.

Anyway, I was luckily able to fix some problems I found along the way by doing this. However, of note, none of these problems really seemed to be of the sort that I was expecting to find when I started.

Because I had grown much more accustomed to the code, I also was able to graft in a version of the command prompt interface found in the FoenixMCP.

There's a number of modifications that I've made and am still in the process of making. In particular I've made a bunch of changes to the boot source scanning and bootup process which align much more with my particular preferences. This screenshot doesn't really showcase it at all of course. I'll write more about the changes in a future post, probably when I feel comfortable sharing the code.

A quick side-bar: If the original design intent of the Foenix Toolbox was to be extremely minimal, why did I add a CLI back in? I guess it's because I disagree with the minimal design intent, somewhat. I've so far found the built-in CLI to be extremely valuable. In particular because it includes POKE and PEEK commands, as well as the ability to dump parts of memory, it doubles as a debugging tool in some cases. Because it's running in different memory/code spaces from what normal user code utilizes, you don't clobber application code by using the CLI. So upon hitting the reset button on the F256K, the previously running app is still in memory, but you're now booted up into the CLI so you can easily inspect things in memory. It's not a great debugger, but it's better than nothing. You could of course also specially craft a linker script to have a non-built-in CLI app also not clobber "normal" user code, but on such a hardware-limited device, it's frankly just easier if you include it directly in the Foenix Toolbox I think. Plus there is still plenty of spare flash space (which gets used as ROM by the F256K) for this kind of thing.

However, the biggest improvement so far, is that it's been pretty darn stable.

Well, for the most part.

Over the past couple weeks, I've started to notice some odd things happening again. This was pretty strange, as since I started this project in late February, I've been testing builds a lot every single step of the way. At no point until just the past couple weeks did I ever notice anything like before with the original Foenix Toolbox code where sometimes a build would fail to boot completely for some unknown reason, and then making a completely unrelated change (how do I know it's unrelated? By watching the USB serial debug output to see where exactly the bootup process gets stuck) such as adding a new printf("Hello, world!\n"); somewhere, and then loading that build which would then suddenly and miraculously work.

I've seen some things which are starting to make me think that the Calypsi compiler is sometimes emitting incorrect code, possibly related to the overall code size. But I'm not at all certain about this.

For example, recently I added an ftell() equivalent API call to the Foenix Toolbox's kernel / jumptable functions since such a function was curiously missing. I spent a few hours struggling with an issue where the underlying FatFs call, f_tell() would return a junk value unless I added a dummy for-loop or almost anything else that was otherwise innocuous to that particular kernel function. I thought this was perhaps related to some timing requirement by the hardware, but when I looked at what f_tell() was actually doing, it was just reading a value from a struct that was already loaded into memory! No actual hardware access was being performed. When I dumped that internal struct's memory to the USB serial port, I could see it actually had the correct value, even when a junk value was being returned to my code.

The cherry-on-top with that issue was that when I later modified the linker flags used in the project to include --rtattr exit=simplified (for unrelated reasons, it was just something I had noticed in the Calypsi manual and thought that it made sense to add this to save some code space) which reduced the total code size by a few kilobytes, the problem with f_tell() magically vanished.

Another example of such a weird issue with the Calypsi compiler's code generation, was where turning compiler optimizations off actually caused a very simple delay function to turn into an infinite loop. I feel like it's usually the other way around for this type of thing where you notice issues when turning compiler optimizations on!

I haven't yet been able to reproduce these types of issues in small, isolated projects, but I probably need to spend some more time attempting this, as I would obviously love to have a great reproduction scenario to log a bug with.

Despite some of these more recent frustrations, I actually feel like I've been having quite a bit of fun with it all, as I'm becoming more familiar with the hardware, the compiler toolchain and its quirks, as well as it just feeling so darn good to feel like I'm finally building something and making progress rather than banging my head against a wall.

The benefit of learning much more about the Calypsi toolchain is that I was able to return to my A2560K for just a single day and whip out this example project (mostly just for myself for future reference) that is a simple "hello world"-style project showing how to make a bootable and flashable project for the A2560K that doesn't rely on FoenixMCP at all. I remember trying to figure this out back in late December in response to a question by someone else on the Foenix Discord and I didn't get anywhere with it, heh.

I would like to share the code for this project as it might be of interest to some other F256K owners, but I think I want to keep it to myself for a little while longer yet until I make some further improvements as well as spend some more time trying to determine if some of these issues are dumb things I've been doing, or actual issues with Calypsi. I'm still definitely considering this all "experimental" for now!