Exploring Watcom C/C++
A couple weeks ago I picked up a complete boxed copy of Watcom 10.0a. I was also able to find an almost complete set of the printed documentation/manuals for Watcom 9.5 from a somewhat-local seller.
I do love having physical copies of books, especially reference material like this. Most people will probably think I'm weird, but I find this more convenient to refer to in many (but not all) cases. Being able to physically flip through pages at will and back and forth between several different pages as needed just feels a ton more convenient to me then doing the same with a PDF. And I've always preferred reading text like this on real paper. The obvious downsides versus an electronic copy is the lack of ability to do a Ctrl-F to find something specific and also not being able to copy+paste something from it.
So what's so special about Watcom anyway? Well, for me personally, it was the first C compiler I ever used. Not this version though. In 1996 I remember my dad bought me a book, "C DiskTutor" by L. John Ribar (published in 1992) that came with a disk containing sample code and a stripped down copy of the Watcom C 8.5 compiler. The disk actually had a few bad spots on it and I remember he called the publisher and they sent out another disk. I didn't realize it at the time, but the version of the compiler on the disk lacked the ability to compile code with any memory model other then the small model, so it wasn't suitable for really any "real world" use. Certainly good enough for introductory exercises though, which was obviously the intended purpose.
I didn't spend a whole lot of time using it though. I remember getting frustrated with fixing my own bugs as there was no included debugger with this stripped down version. At least I don't remember there being one. But also I remember trying to make sense of the poor compiler error messages was tough for me being completely new to C at the time. Finally, I also remember really hating how I had to quit out of EDIT.COM to compile and run my code. The disk included some simple text editor which seemed to be written specifically for this book. It did have the ability to run the compiler from within the editor, but I vaguely remember arriving at the conclusion that this editor was not very good and a bit buggy. EDIT.COM ended up being a less-frustrating option for writing code. Keep in mind my only other programming experience at this time was a little over a year with QBasic which had a much faster and easier edit/compile/run cycle. I didn't understand why anyone would want to use something else when it was so much easier! Of course years later I learnt that many people at that time used better editors then EDIT.COM and actually really liked writing code in their editor of choice and using external build scripts / Makefiles. Heh.
Beyond my nostalgia though, Watcom was used by all sorts of game developers who cared about code efficiency in the early-to-mid 90's, such as id Software with Doom and 3D Realms with Duke Nukem 3D amongst many others. Plus it shipped with a royalty-free version of the well known DOS extender DOS/4GW. However, while it was better at code generation then the competition (Borland, Microsoft, etc), it lacked in the end-user experience. Borland and Microsoft had better tooling in the form of IDE's (Watcom didn't ship an IDE with their compiler until 10.0) and easier to use command line tools.
Eventually other compilers caught up and Watcom converted to an open-source project when Sybase (who owned Watcom) decided to end-of-life the compiler.
So far I've been using DJGPP (which is just a DOS port of GCC) for all my DOS coding projects. I'm using an old version that corresponds roughly with what was current when I first discovered it for no other reason then that it feels "true" to the whole spirit of these DOS coding projects.
DJGPP is pretty awesome honestly. Since it's just an older version of GCC for DOS, it feels very familiar. Lots of the
same tools are present, including GDB and MAKE. And the code generation is really quite excellent. I think my only
gripes with DJGPP are that it uses the AT&T syntax for inline assembly and that I kind of have a tiny bit of a
love/hate relationship with the CWSDPMI DOS extender and DPMI server it uses. It's nothing major really, just minor
gripes relating around how it seems more "complex" to use on the surface as compared to something like DOS/4G. But I
realize that there are good reasons for it. And personally I believe that CWSDPMI is probably a bit more robust then
DOS/4GW, but I could be wrong. Feels that way sometimes though.
RHIDE is an IDE for DJGPP that seems to be largely inspired by Borland's DOS IDEs. It's also
pretty great and is very quick to get up and running. You don't even need to do any fiddling about with GCC or MAKE
directly when using RHIDE as it will handle that all for you. Awesome. Though it's not without its quirks and
limitations. The editor has some weird (to me) ideas about indentation that I've only been able to "resolve" by using a
very specific combination of options that involves mixing tabs and spaces in a way that makes me sad.
Additionally, while the GDB integration is good, it definitely feels like it's missing some extra polish. There's no
automatic locals watching (have to explicitly add anything you want to watch the value of). The watch viewer itself
is quite simplistic and watching the values of structs is ... a poor experience. Well, not until you start wanting
to watch more complex values anyway. The whole watch window really needed to be implemented as an expanding/collapsing
node tree type of viewer, but it's not sadly. And there's no memory viewer. The disassembler viewer really lacks a
source code + assembly code combined mode. It's not the end of the world, as when you're stepping through code you can
see both and the current line is highlighted in both places. But still... it makes browsing the assembly view somewhat
more annoying than it needed to be. Bleh. Oh, and there really isn't any better GDB frontend for DJGPP/DOS that I'm
aware of. This is pretty much the best option. And to be fair, for the time, it wasn't bad at all. It's just that my
nostalgia only goes so far.
So anyway, back to Watcom. The version I picked up (10.0a) is 4 years older then the version of DJGPP I've been using so far. So I didn't have any expectations about getting a "better" compiler or anything like that. But I was pretty excited to try it out because it really feels much more "true" to the spirit of using a 486 to do DOS programming as Watcom was the compiler that the majority of my favourite DOS games were written with. A silly reason, for sure, heh.
I'd heard the "new" IDE that Watcom introduced with 10.0 was kind of not good, and certainly that was my reaction to trying it out as well. It requires Windows 3.x and it does work. But it's pretty weird. The "IDE" is actually comprised of a series of separate executables that are all invoked from the main "IDE" (what is actually called the "IDE" by Watcom). It's more of a "project manager and GUI build tool" then anything else. You can see all the source files in your project and configure various compile and link options. Clicking on a source file causes a separate text editor application (and hence, a separate window) to open. Same goes with things like Windows resource files, etc. All separate tools. It's pretty weird. No other IDEs even at that time were like that. Certainly not any of the well known ones anyway. Kind of makes me curious what their reasons were for going with this design. Anyway, it's certainly not something I want to use. Additionally, I found that debugging graphical DOS programs launched from within Windows was problematic and resulted in crashes sometimes while stepping through code. So the Watcom IDE is out.
I had to admit at this point that I was really quite unfamiliar with what DOS text editors were out there. Turns out there was quite a lot. Many even had fancy features like syntax highlighting. Some also had fancy macro languages for additional configurability. Nice. Should be easy to find something suitable.
Heh.
It would take too long to write fully about this experience, but suffice it to say that while I didn't try everything on that list, I did try quite a lot. I realized that I am quite picky about the text editor I use. In weird ways anyway, probably mostly born out of habit and fear of change. Also anything based on VIM or Emacs I disregard because that's just how I roll.
I basically wanted the following features at an absolute minimum:
- Syntax highlighting.
- Multiple files open at once and able to easily switch between them.
- External tools launching (to launch batch scripts / makefiles to run/compile/debug).
- Ability to run in 80x50 text mode.
Not too much to ask for I think. Multiple editors had support for all of these in fact. The first one I decided to look
at was SETEDIT because it's what RHIDE was made from (or is that the other way
around?). I quickly discovered that launching DOS/4G programs from within CWSDPMI programs can be problematic. The DOS
version of SETEDIT is built with DJGPP which means it uses the CWSDPMI DOS extender. Programs that I was going to be
building with Watcom would make use of DOS/4GW. Additionally some of the Watcom tools, such as the DOS debugger WD,
also use DOS/4GW. I was unable to launch the Watcom WD debugger from within SETEDIT without causing crashes. While
some test programs I built which also used DOS/4GW could be launched fine, I didn't like the idea that down the road
there might be problems. Ugh.
As it turns out, a lot of the better text editors were also built with DJGPP. All of these that I tried which supported
launching external tools/programs also exhibited the same crashing problems when launching some DOS/4GW programs like
WD. Ergh.
I do want to specifically say that I really liked FTE. Just a great, simple, minimalistic, easy to use editor which for some reason really resonated with me. However, it is built with DJGPP so it was sadly a no-go. That being said, I have a "conversion" project on my backlog list now, heh. If I decide to use Watcom over the long term, I intend to fully investigate whether it's possible to convert FTE to compile with Watcom so as to hopefully remove the crashing issues with DOS/4G programs.
Another editor I wanted to point out is something called "Power View IDE" which is actually a full DOS IDE for Watcom C/C++ that some guy wrote! I was ecstatic when I found this linked on some random forum post, seemingly as an afterthought by the poster. I don't think there is any existing project website for it, but it can be downloaded here. I couldn't believe that there was so little information out there about this and basically no one ever mentioned it anywhere.
This editor I think could have been exactly what I was looking for. Except it wasn't. It had some show-stopping bugs which made me oh so very sad. For one, when scrolling through text using the arrow keys on the keyboard (as I sometimes do), the editor would periodically insert random numbers ... !? What the heck? Maybe a bug in the keyboard handler? I took a look at the code and nothing stood out to me, so not sure what to make of that one. But yeah, an editor that randomly inserts random characters into your code is instantly disregarded. Another issue was that it didn't seem to correctly handle restoring the video mode when returning from executed programs. This became problematic when running the editor in 80x50 text modes which was a requirement of mine. Oh well. I guess the buggy nature of it is why no one ever really seemed to talk about it from what I could see.
To make this discussion of text editors not go on for many more paragraphs I'll cut to the chase and say that I eventually settled on Aurora. It was actually my last resort option, heh. While it has quite a lot of weird quirks that were initially off-putting for me, it is an amazingly customizable editor due to it's weird macro language "AML." Everything in the editor itself is largely implemented in AML, from how text editor behaviours work, to the menus, to the keyboard shortcuts, to file open/close dialogs, etc, etc. The DOS executable is basically just a runtime environment for AML and the Aurora text editor just comes with a pre-built set of compiled scripts that give you a text editor. Neat. It's reasonably well documented and the macro language has support for quite a lot. It appears to be a 16-bit real-mode DOS executable so no problems executing DOS/4GW programs from within the editor.
However, it had no real built in support for some nice "quality of life" things such as capturing build errors and displaying them nicely in the editor. "Well, with all the things AML supports, how hard can it be to implement this myself in exactly the way I want it?" This line of thinking kind of spiraled out of control quickly. Suffice it to say, there were way more quirks with AML then I had expected and the debugging facilities within Aurora were barebones at best. I think the hardest part was actually managing the lifecycle of a plugin that is meant to be called again and again from a menu item. It wasn't straightforward and was complicated by the fact that the description of how AML's "objects" work made me think more of traditional OOP style programming ... and it kind of isn't like that in a lot of ways. The extensive documentation unfortunately doesn't really tell you anything about actually building plugins for the editor specifically. Nothing in the way of concrete examples with explanations for how to do common things or why it's done this way, etc. Instead, it tends to just focus on the language basics and some API stuff... nothing to tie it all together. Oh well. I should be happy that it was as well documented as it was really. After suffering through some bizarre and hard to debug editor crashes from poor lifecycle management, I got something working!
At this point, I had put off "fixing" all the subjectively weird behaviours and keybindings that Aurora has out of the box (even when selecting a keybinding scheme during installation that was supposedly based on an editor I was mostly familiar with). Finally got that all mostly sorted out and at this point was thinking to myself "wow, this REALLY all had better have been worth it!" Meaning of course, that I really hoped I wouldn't grow to hate Watcom, heh.
The "WMAKE" menu I added contains options for Build, Rebuild, Clean, Run, Debug and Show Errors. Build errors will show up automatically after any action has finished, but it's a quick key-bound menu item to easily bring back up the error window if it gets hidden. These WMAKE actions basically require a makefile that contains targets for each menu item. Which is totally fine, as that was basically how I envisioned the makefile structure I was going to use anyway.
target_config = debug
target_name = dgl_test
object_files = &
blit.obj &
clipping.obj &
dgl.obj &
draw.obj &
gfx.obj &
internal.obj &
keyboard.obj &
mathext.obj &
mouse.obj &
pcx.obj &
util.obj &
test.obj
cc_flags_debug = /d2 /zp4 /5r /fp3
cc_flags_release = /d1+ /zp4 /5r /fp3 /oneatx
cc_flags = /mf $(cc_flags_$(target_config))
link_flags_debug = debug all
link_flags_release = debug all
link_flags = $(link_flags_$(target_config))
: : : : .NOCHECK
:
.NOCHECK
: .NOCHECK
: Basically relying on WMAKE to launch what I need once builds complete.
If that one other weirdo like me who wants to use Watcom C/C++ with the Aurora Text Editor comes by this site and reads this, here's my WMAKE.AML plugin. The two main things that are currently missing are "click on error to goto sourcefile/line" support and some kind of better window management (like, e.g. automatically resizing existing editor windows to have the errors window always visible on screen). The former will likely be added soon as it should be simple to add.
So anyway ... surely I would not have to all of this effort without having some idea of whether it was going to be worth it at the end right? Of course. Heh.
I'm quite a big fan of the Watcom debugger WD (previously called WVIDEO... not sure where the "video" part of that
name comes from honestly).
It's kind of got a bit of a Visual Studio C++ debugger thing going on that I like (and this was even from before Visual Studio C++'s debugger got good I think). I think the one thing I dislike about it so far is that abruptly terminating a running program can leave your system in a weird state. For example, it doesn't seem to always restore overridden interrupt handlers. So not letting your running code run to the point where it removes it's custom keyboard interrupt handler can leave you unable to do anything once the debugger exits and returns to your text editor. Oops. DJGPP and GDB (and maybe CWSDPMI?) seemed to do a much better job of cleaning things up in these cases so your system didn't crash very often (though it wasn't perfect).
As for Watcom in general, I like that DOS/4GW seems to get out of your way a little bit more than CWSDPMI does. For
example, the first 1MB of memory is directly accessible out of the box, so I can write into 0xA0000 immediately, or
read directly from some memory in the BIOS to e.g. read the RTC tick count, or the BIOS fonts, etc. Just as DOS should
be! Additionally, I can just call _dos_setvect and _dos_getvect directly as I would in a 16-bit realmode program
and it will basically "just work" when I want to implement my own interrupt handler. Of course, it's not quite that
simple in all cases, like when you want to use some bit of shared DOS memory and communicate with some process or piece
of hardware that is running in 16-bit mode. But for the simple cases, it is simple. Don't need to have a bunch of
_go32_dpmi_blahblah calls around. This will come back to bite me in the ass though, as DJGPP provides a relatively
nice API for interfacing with CWSDPMI when you (inevitably) need to for the more complicated scenarios. DOS/4GW doesn't
provide you with as nice an API... you just call the standard DPMI server interrupt. However, this is mostly documented
in my stack of Watcom manuals, so that's a nice plus! Even some examples such as how to implement bimodal interrupt
handlers.
On the topic of code generation though, the version of GCC from DJGPP that I was using is definitely better(as expected, it is 4 years newer).
We can take a look at the following very simplistic example which shows how something simple like function inlining is
handled between these two compilers. In this example, I was using GCC flags -O2 -m486 -ffast-math and WCC386 flags
/zp4 /5r /fp3 /oneatx both of which are recommended flags for optimum performance (could do -O3 with GCC of course
though).
This example is something I noticed while running some simplistic benchmarks to see how much performance I would lose with Watcom. This snippet isn't even what I was benchmarking... it's just some setup code to generate a random 16x16 sprite for the actual benchmark that appeared later in the code. However I noticed it wasn't inlining a call the way I was expecting when looking at the assembly output.
// in a header file
static int
static byte*
static void
// elsewhere, in main()
bmp = ;
for
As a quick aside, before anyone looks at the above code and says anything like "Wow, that would be slow anyway! You
really don't want to be using anything like an explicit pset type of function call anywhere! Look at that slow
multiplication for god's sake!" I know. For any code that's running in an inner loop in a performance-sensitive
scenario, I would always use, e.g. the above surface_pointer function to get a direct pointer to the memory before
the loop and then use that pointer in the inner loop itself. And if it was really important to save every last CPU
cycle, I would probably write out the surface_offset calculation directly without the multiplication anyway. These
are all just convenience functions to be used as appropriate. The above for loop code is just an example!
GCC inlines the call to surface_pset_f as I would expect:
21:test.c **** bmp = surface_create(16, 16);
185 007e 6A10 pushl $16
186 0080 6A10 pushl $16
187 0082 E879FFFF call _surface_create
187 FF
188 0087 8945E8 movl %eax,-24(%ebp)
22:test.c **** for (x = 0; x < 16; ++x) {
190 008a 31F6 xorl %esi,%esi
191 008c 83C410 addl $16,%esp
192 008f 90 .align 2,0x90
193 L238:
23:test.c **** for (y = 0; y < 16; ++y) {
195 0090 31DB xorl %ebx,%ebx
196 0092 89F6 .align 2,0x90
197 L242:
24:test.c **** surface_pset_f(bmp, x, y, rand()%255);
218 0094 E867FFFF call _rand
218 FF
219 0099 89C2 movl %eax,%edx
220 009b BFFF0000 movl $255,%edi
220 00
221 00a0 99 cltd
222 00a1 F7FF idivl %edi
223 00a3 8B45E8 movl -24(%ebp),%eax
224 00a6 8B38 movl (%eax),%edi
225 00a8 0FAFFB imull %ebx,%edi
226 00ab 01F7 addl %esi,%edi
227 00ad 037808 addl 8(%eax),%edi
228 00b0 8817 movb %dl,(%edi)
230 00b2 43 incl %ebx
231 00b3 83FB0F cmpl $15,%ebx
232 00b6 7EDC jle L242
234 00b8 46 incl %esi
235 00b9 83FE0F cmpl $15,%esi
236 00bc 7ED2 jle L238
25:test.c **** }
26:test.c **** }
However, I noticed wcc386 did not. Not only that, no amount of tweaking of compiler settings and/or #pragmas would
get it to. It only partly inlines the call to surface_pset_f:
bmp = surface_create(16, 16);
142a ba 10 00 00 00 mov EDX,00000010H
142f 89 44 24 04 mov +4H[ESP],EAX
1433 89 d0 mov EAX,EDX
for (x = 0; x < 16; ++x) {
1435 31 f6 xor ESI,ESI
1437 e8 00 00 00 00 call surface_create_
143c 89 c7 mov EDI,EAX
for (y = 0; y < 16; ++y) {
143e 31 c9 L27 xor ECX,ECX
surface_pset_f(bmp, x, y, rand()%255);
1440 e8 00 00 00 00 L28 call rand_
1445 89 c2 mov EDX,EAX
1447 bb ff 00 00 00 mov EBX,000000ffH
144c c1 fa 1f sar EDX,1fH
144f f7 fb idiv EBX
1451 89 14 24 mov [ESP],EDX
1454 89 cb mov EBX,ECX
1456 89 f8 mov EAX,EDI
1458 89 f2 mov EDX,ESI
145a e8 00 00 00 00 call surface_pointer_
145f 8a 14 24 mov DL,[ESP]
}
1462 41 inc ECX
1463 88 10 mov [EAX],DL
1465 83 f9 10 cmp ECX,00000010H
1468 7c d6 jl L28
}
146a 46 inc ESI
146b 83 fe 10 cmp ESI,00000010H
146e 7c ce jl L27
Elsewhere in that same module, surface_pointer_ can be found:
0070 surface_pointer_:
0070 0f af 18 imul EBX,[EAX]
0073 8b 40 08 mov EAX,+8H[EAX]
0076 01 da add EDX,EBX
0078 01 d0 add EAX,EDX
007a c3 ret
007b 90 nop
So, Watcom partially inlined the call to surface_pset_f but left a call to surface_pointer as a fairly lightweight
function call. Note how Watcom uses registers to pass parameters to functions pretty much everywhere it can by
default, thusly the surface_pointer_ function has none of the typical stack management boilerplate we might expect.
From what I've read, this method of passing function parameters is one of the reasons it kept ahead of the pack
regarding performance compared to other compilers at the time.
Whilst playing with compiler flags and other things, I decided to see how wpp386, the Watcom C++ compiler, would
handle this:
bmp = surface_create(16, 16);
0063 ba 10 00 00 00 mov EDX,00000010H
0068 89 44 24 08 mov +8H[ESP],EAX
006c 89 d0 mov EAX,EDX
for (x = 0; x < 16; ++x) {
006e 31 f6 xor ESI,ESI
0070 e8 00 00 00 00 call __16hkttSURFACE near * near surface_create( int, int )
0075 89 c1 mov ECX,EAX
0077 eb 06 jmp L3
0079 46 L2 inc ESI
007a 83 fe 10 cmp ESI,00000010H
007d 7d 2a jge L5
for (y = 0; y < 16; ++y) {
007f 31 db L3 xor EBX,EBX
surface_pset_f(bmp, x, y, rand()%255);
0081 e8 00 00 00 00 L4 call rand_
0086 89 c2 mov EDX,EAX
0088 c1 fa 1f sar EDX,1fH
008b f7 fd idiv EBP
008d 89 14 24 mov [ESP],EDX
0090 8b 11 mov EDX,[ECX]
0092 0f af d3 imul EDX,EBX
0095 8b 41 08 mov EAX,+8H[ECX]
0098 01 f2 add EDX,ESI
009a 01 c2 add EDX,EAX
009c 8a 04 24 mov AL,[ESP]
}
}
009f 43 inc EBX
00a0 88 02 mov [EDX],AL
00a2 83 fb 10 cmp EBX,00000010H
00a5 7d d2 jge L2
00a7 eb d8 jmp L4
Interesting! wpp386 fully inlines surface_pset_f. The exact same compiler flags were used in both cases.
After discovering this difference, I decided to compare wpp386 vs wcc386 with some other pieces of code and saw that
there are definitely some other code efficiency differences. I've not investigated fully yet, but it does seem like
wpp386 generates slower code in some cases too. I'll probably do a follow up post once I've dug into it a little bit
more. One of the things I want to play with is re-working the code for surface_pset_f, surface_pointer and
surface_offset to see if it's just a problem caused by the code optimizer "giving up" at some point because of how
the code is written. It definitely was a thing that you'd have to at times help the compiler out by simplifying or
otherwise breaking up complex expressions into separate chunks so it could do a better job.
Once nice thing about using wpp386 for even 100% C code is that it at least gives you more warnings and the support
for toggling off warnings that you don't care about (like "W202" with wcc386 ... ugh) actually works. Turning up
the warning level with wcc386 is kind of useless for this reason. I'm assuming this is just a bug and was probably
fixed with 10.5 or 11.0.
Inline assembly is similar in some ways to GCC, but no need to deal with the AT&T syntax which I dislike. The way you do
it with Watcom 10.0 is unfortunately more complicated then using asm as you might with other C compilers of the time
(and modern compilers too), but it has it's benefits too:
void ;
Then you could simply call REP_MOVSD in your C code in the exact same way you would call any other C function. Inline
assembly written in this manner would be inserted into your code at compile time in the same way as an C function that
is being inlined by the compiler would be.
Compare to how you'd do it with DJGPP/GCC:
Works the same way when calling from your C code (REP_MOVSL can be called as any other function, and is inlined,
etc.). Also the GCC __asm__ is more flexible like what you saw with other compilers and the asm keyword, so you
could just use it wherever in your code with ease (so you don't have to use it in a macro as I'm doing here).
If I'm being honest, both methods are super ugly to me. I will never be able to commit either form to memory and so,
every time I need to write inline assembly I'll need to be referring to some documentation to copy it out. Watcom 11.0
apparently added support for the asm keyword, but apparently even with that, Watcom recommended the #pragma method
because of its ability to indicate to the compiler what registers are used to pass values in and what registers were
modified by the inline assembly code, something that the compiler couldn't do with the asm keyword I guess.