Reading and Writing

I made inroads into writing out a game file and got stuck on some iterators. The original game dumps the data in structs as one binary blob, I'm not sure of what data is in there, but I know I wound up off by 192 bytes. There was a substructure I wasn't quite finding and not matching up with my saved template. I put a demarcation where the data started going off the rails and moved to the routine that loads the game from disk instead. I'm borrowing data files from the original game and trying to load that save from 1991. So far it's progressing, but the fleet structure has about 4 sub-structures  and one sub-sub-struct member that I'm not iterating correctly though. When the game loads the data proper I should be placed into a spreadsheet-like fleet overview. From there the game proper should start.

When I have the game proper loaded, this should make it easier for me to see what loads to where and then reverse the process back on the disk.

Goals: Once I get the game proper to work, I'll be compiling the stand-alone race builder and will be sending that off to the artist. Then I'll be looking into getting a "configure" button for my game settings, and adding in bits and bobs in my growing todo list.

New Marshal in Town

Counting bytes saved

So I'm in the process of getting the main game running. When creating a new game, all the relevant game variables are saved into the disk and then your commander is "locked in" to that scenario. The game data is kind of a culmination of all the past structs I've worked with all rolled into one big blob. Right now I'm hand serializing all the data and comparing it to a game file I created in the original game. This requires lots of byte counting and making sure each variable is saved in the right order. I want to chat about marshaling for a bit.

It seems lots of people use marshaling and serializing interchangeably. It would appear that  serializing is the conversion of data blocks into a stream of data that can be saved. This is what I'm doing right now...by hand. Marshaling seems to be the active "teleportation" of data from one place to other via a serialized method. When I was a database programmer once upon a time, I just had to pass whatever data object to a serialization method and it was automatically marshaled away. I don't have such a luxury in C. I also have to contend with the old 2 byte/4 byte/big-endian/little-endian integer issues.

So the big question is why am I doing this?

Well, one of the game's touchstones is that it interconnects with other games. "Rules of Engagement" links with "Breach 2". The interprocess communication (IPC) between the games was done via file transfer. Right now the game is binary-compatible with the original. This means, that I should be able to link the new version of ROE to the old version of Breach 2 and have them talk back and forth. (Also, if things go well, port Breach 2 next).

 Also, in today's networked world, it would also be cool to trade captains, ships, races, missions, and systems online. Each are stand alone and don't require dependencies. (You don't need the G'shook Empire race on your HDD to share a mission with them in it.)

The game file serialization is the second to last filesystem thing I have to convert. The last is the auto-play demo files. It's partially implemented now, but is broken.

Trello

I'm working the weekend today. I set up a public Trello page to keep track of my goals. I've used this is the past with my old employer. It's pretty sparse as I didn't feel like putting a few months of resolved issues into it. Also I've just put in my short term goals and what I'm currently working on in the moment. When I run across something that need to pop in I will. I'll be adding things into it as I see but It's a good start.

Link to Trillo Page

In other news I have to battle the two byte/four byte integers again. More manually serializing data. It's par for the course when updating the filesystem functions and I've gotten quite used to it. 

Through the Jaws of Defeat

So it seems that Allegro does not allow you to update a display from a thread unless the thread made the display. It's due to TLS (security) reasons. Now I understand that we live in a world where memory is protected, and I don't know much about the mechanations of thread management, but that just seemed silly..

That also means that the thread can not alter the graphics on it's own.

I was about to create an experimental fork in Git and refactor the whole game. The plan was to put in a proper input -> update -> render loop. Using a thread appears to be a dead end. Makes sense, the original code base was not designed for one. I should of refactored this mess anyway, and actually set it as a goal at the beginning of phase 2. However, it was working (kind of) and I thought I could cheat with a thread (That I kept commented out for two months. Should of been a warning sign right there)

The "normal" thing to do is to set up an event system that can get polled by the main game. It will simply look in the queue and see if 55 milliseconds has passed and then run a function. Time and time again I try to explain why ROE does not have a main loop, and it's not possible to implement an event queue because there is no central place to poll the queue from.The game is made up of "screens" and each have their own loops, but some loops don't render anything and some don't take user input. It was only when I looked up overlays on Wikipedia did it show an example of this type of linking

                       +--------------+
                       | Root Segment |
                       | MOD1, MOD2   |
                       +--------------+
                               |
                    +----------+----------+
                    |                     |
             +-------------+       +-------------+
             |  Overlay A  |       |  Overlay B  |
             |  MOD3       |       |  MOD7       |
             +-------------+       +-------------+
                    |
           +--------+--------+
           |                 |
    +-------------+   +-------------+
    | Overlay AA  |   | Overlay AB  |
    | MOD4, MOD5  |   | MOD6        |
    +-------------+   +-------------+

This was why I was having a hard time explaining it. The program flow is hierarchical and control does not stay in the root segment. Each part of the game is loaded when it's needed from the HDD and then purged when another overlay is needed. I had long since combined everything into one big executable, but the program flow was the same. The root segment hangs out in memory, and that where my graphics lib hangs out, but nothing really loops there. This is also why, when I implemented the graphics lib with the Allegro wrapper, everything downstream started to "just work" for the most part. The 1Ch thread was independent of this tree and why I thought a thread would be a good idea.

GrafX2 - A common kinship

 So whilst whining about on the internet about my plight, I had another dev get in touch with me. When I mentioned that the game was setup with .OVL overly linking  and I didn't have a "main loop", This guy knew all about this. He one one of the developers that ported GrafX2 from DOS to Windows and used the same exact architecture. He showed me where I could shim in an event handler.

 It took a new function and 4 entry points.... I felt so stupid.

I was about ready to rip apart this whole program and now I have a functioning "inner loop" I guess. Now my next goal is to work on the game proper. I already have to abstract the file system in that part of the code to load/save/create and existing game. I also have to make the Race Builder stand alone so the artist can  make some cool alien bits .

Newp!

Thread fixing is going well... NOT!

It taunts me. I figured out how to implement my Mutex only to find out that's not what the problem is. In the Nermal lib, there are 42 times that the target bitmap can be changed. It's not that one blit function that is taking over the other. The problem is that while in the blit function, al_change_target_bitmap() is being ran elsewhere. That's in issue I need to battle. Documentation tells me that there are thread considerations for setting the target. Goodie! I will get to the bottom of this. I'm on a mission now. Sadly, this also means I'm kinda spamming my problems to other programmers for help. So if any of you are reading this.. sorry. On the flip side, I'm going to emerge from the other side a thread expert I think. I mean, there are only so many ways I can nuke an executable and not wind up with a crater, right?

The good news is the bug is replicateable and all I have to do is jam a fork into the keyboard to duplicate it. I still have a few ideas to try tonight so I'm not finished quite yet. I also have resources left untouched. Wish me luck!

Augh!

The bummer with doing under-the-hood things is that I can't really produce a cool screenshot of anything. However today I called an early defeat. Every attempt to keep the 1Ch thread from flipping tables was unsuccessful. I tried to use a few globals to pause the thread or control it but each would fail in unpredictable ways. It's mostly because each side is running blindly and autonomously from each other.

After watching access errors and deadlocks for about three hours I discovered I was just coding on accident to get a result and it wasn't getting anywhere. I sat and watched twitch streams for the rest of the day. I did it knowing full well that this was even less productive, but I wasn't glaring at broken code anymore. Frustration can be devilish sometimes.

I hope those reading don't mind, but I'm going to rubber duck this a bit.

The issue:
When BitBlit() is called from the thread, and something is being updated on a screen buffer, they will collide.

Why:
When using Allegro drawing commands, you must set a target on which the operation is preformed. While BitBlit is being ran in the thread, it is stealing the target away from the main thread, causing miswrites. At the same time, the target can be stolen by the main game while the thread has control.

How to fix:
The thread needs to signal the main code that it wants control of the target. The main code needs to signal to the thread that it currently has control and that it must wait.
If the main code wants to take the target, it will check if the thread requested it. It will then pause and signal the thread to go ahead, wait for the thread to flag a finish signal, and then continue.

Problem:
The thread can request, but if there is nothing being drawn by the main code, the request will go unanswered. I have no way to tell the thread it's OK to take the target unless the main code is about to take the target itself, or has finished with the target and can pause relinquish control

Try again:
Main code: "I'm going to need the target"
Main code: *check signals, is there a lock?*
                    yes  ---> pause and hawk the lock signal. When lock is clear set lock and go
                    when done clear lock

Thread:  "I need the target"
Thread: *check signals, is there a lock?*
               yes ---->pause and hawk the lock signal. When lock is clear, set lock and go
               when done clear lock.


I think that works? I don't think was was resetting the lock when the other half cleared it and before I took the target. It's 11:13 right now so I'll be looking at this with fresh eyes tomorrow.

Thread the Needle

So today was the day. I decided to get the 1Ch thread working once and for all, only to have is spectacularly blow up in my face. Turns out, threads are hard, and when you introduce them to code that was never designed for them, it becomes a gigantic mess.

 Here's a little synopsis to get people up to speed.

In the original game there was a function that was attached to interrupt 28. In hexadecimal this is 1C, which is why I call it the 1Ch thread. How the interrupt originally worked was every 55 milliseconds game control was taken away and the interrupt would update various variables and things on the screen. Then it would give control back to the main game again. ROE is funny because it doesn't really have a "main loop" like you would assume. The game is screen based and each screen has it's own input loop. When the input is captured the state of the screen is changed.  The "master" loop is actually the interrupt that runs independently of the screens.

Because interrupts are not available and I couldn't use a program loop to poll events, I decided to use a thread that just woke up every 55 milliseconds, did it's thing and went to sleep. The original interrupt was controlled  by globals that directed what parts would run. I used the same variables to control my thread.

Everything seemed to work fine. My first hurdle was to gracefully kill the running thread at shutdown. Normally just returning from the function killed the thread, but I set mine up to run forever in a while() loop. Because the thread also contained allegro drawing functions, when I shut down allegro on exit, the thread would crash. Using a no control global I was able to get the game to exit gracefully. Look OK so far.

The thread is corrupting my blits :(

But it's the thread's drawing functions that are wrecking havoc on my game. In the original game, when the interrupt took control, everything stopped. My thread, however, can run at the same time as the game, and actually execute the same function that is currently being run. This is causing my drawing buffers to be snatched away by the thread as things are being drawn and making garbage get drawn everywhere. (That or crashing the game entirely)


I'm going to need to implement a mutex, which is a thing I learned today, so that I can politely wait until the drawing surface is clear so I can write to it with my thread. How fun.

I knew that was going to be hard. I was warned about using a thread too. I didn't exactly think it would cause everything to blow up in my face though.

The Color of the Skin...

Well, I fought though the funk I've been in for the last week or so  and got a goal done. The enemy skin colors are now working. If you have played the original game you will notice that the colors don't seem as vibrant as they used to be. This is because the underlying luma is grey and things are a little washed out. This isn't a problem as the aliens are going to be redone anyway. This means that Gorn, Robbie the Robot, The Borg, The Chicago Bulls, Tholeans, Ninja Turtles, and others will be will be gone. 
When adding functions to the game, such as high resolution and alpha blending, I'm trying to keep Allegro code away from the actual program loop. What I have been doing is actually changing the Nermal functions and extending them a bit. For example I added a new type of blit called B_TINT what will copy over bitmaps with whatever color is set in a global tint variable. I couldn't add an argument to the BitBlit() function least it break all the blits in the game. (I counted , there are 891 calls to that function alone.) I just added a mode and I'm taking the extra data from a global. It's terrible and hacky, but it works. 

My next goal tomorrow is the master 1Ch thread I keep saying I will do over and over again. When that is complete... I will run the game proper for the first time from end to end and it all should just work. I know there are still filesystem functions in the game proper I need to replace, but the builders, for the most part are functional. They are not "done" by any stretch of the imagination. The graphic update in Phase 3 is going to pass over those. I also need to add an options button and implement UI themes.

But, it would be nice to play a mission in the actual game first.

Coloring

Step one of getting the alien pigmentation to work properly is to set up the graphics to allow for the kind of shading I need to do. In the original game, each pixel was captured, the color was changed, and then put into the buffer. If the color was black or white, the colors were not altered. I'm giving the game a layer approach. The first layer is the black and white luma, then a color tint is applied over top, and then the static color layer is on top of that. This mean things like metal arms don't get colored when the skin does.

I spent today separating out the color layers. Keep in mind these are not the aliens I'm planning to use in the final game, but I do need to get the proof of concept working. You can see here the idea I'm going to have when it's done. There are parts that are colored and other parts that stay the same color. That's important.

Laid out, you can see why I need some of the aliens re-made.

I'll probably be spending tomorrow getting this implemented using the colorbar code I have already written. Once that is all done, I will get a build of the game off to to the artist to go wild with new character designs.

Touchy Touchy Touchy

Man, when you have you mojo broken, it's tough to get to back into the swing of things. I had to take a few days to help out family and I super did not want to open Visual Studio when I got back for some reason. I think it's because the alien pigmentation thing is going to require more than three steps to implement. I have to redraw the alien body parts and set up the alpha shader and then the color overlay. It's a thing that needs do be done because then I can send a copy of the alien builder to my artist so he can make cool alien parts as I move on to the timer thread.

It's not like I don't know what I have to do next. but my motivation is critically low and I have been having a penchant to watch Subnautica let's plays.

But there is a little progress. One of the reasons why I'm porting this game to something a little more cross-platform is because I want to take advantage of touch interfaces. It's kind of what the game was designed for, just about 15 years too early.

Well... Touch works :)

The original game only understood relative mouse movements. Here every button is a discrete position on the screen. I have a snaking feeling resolution is going to bite me in the read later, but as a proof of concept I love the fact I'm on the right track.

Family things

Small update.. I'm helping some family with a trip to Florida today (Monday) and tomorrow (Tuesday). Regular updates continue in the middle of the week.

In this time I'll try and come up with an alien pigmentation solution.

Transparencies work... Sorta

I had to battle with alpha channels today. The program put up a fight, but I was able to get the transparent parts to cooperate. It's not done, as the game will try and overwrite aa transparent object with another one and blend them together. I still have to do pigmentation, but now that the alpha channel is working that should be the last big hurdle for the builders.

I was fighting with the alpha channel all day because my method of clearing buffers with a transparent  background dropped my framerate to the point where it was taking three minutes
to render a screen with an alpha channel in it. It was resolved, but It's still cosmetically.... iffy so I have more work ahead of me while I fix up the pigmentation portion.

Today was a slog. Bugs suck.  

Great taste with only 10% of the warnings.

So I'm now down to 131 warnings. I had some personal stuff to do today so I didn't get a good crack at the code. I had a filesystem issue that came up where not all the files were being read in the default data folder. Turns out it was a simple off-by-one error. I gave a crack at working the alpha channels today and the transparencies are not working the way I thought they would. Turns out the game takes my transparent sprites and copies them to a black bitmap for holding before they are transferred to the target buffer. This destroys the alpha channel so I need to do some research into how I can retain that.

When I need to experiment, I never do it in live code. I actually have a whole other project in Visual Studio dedicated to trying out ideas or figuring out what's wrong. The colorbar/tinting experiments were conducted there along with some of my findfirst() and findnext() code. Now it looks like I'll be playing with alpha channels to get things copied right.

I've also been seeing some routines that are XORing graphics with itself to clear the screen. That's cool when you have direct access to the framebuffer, but I'm using a virtual display for the resolution boost and that is SUPER expensive. As I work the code more, I'll probably be seeing more Allegro encroach into the game proper. The game logic itself is solid and does not need to be touched at all. This is just in the name of cosmetic touch-ups.

Friday should be more productive. 

Refactoring

While getting the code up to C99 standard,  IntelliSense as become much more useful. I can now see all the functions is the solution explorer and flip back to function declarations with ease. I decided while I was doing this, I should get rid of some of those 600 some-odd warnings that have been hanging around. This required me to start writing prototypes so that those undefined declarations can get cleared out. I imported the obsolete proto.h file and started to update all the prototypes in it.

I'm now down to 305 warnings. When I first got the code to compile I was sitting just shy of 1000 so I've knocked it down to a third of that. I'll try and shoot for zero warnings but some of these are going to stick around as lots of leftovers are architectural. The game proper still has lots of legacy stuff in it, and I have another whole module to convert to Allegro's filesystem. The good news is IntelliSense is making some of the more glaring errors much more obvious.

I didn't work on the builders as the refactoring took priority. It wasn't a lost day as these are all things that need to get done.

Making Intellisense

Alien paperdolls are not going together correctly.

All of the old stdio.h code has been kicked out of the filesystem module. I can now read and write mission files. My next task is to get the transparency to work in the builders. that game had a mode where it would "copy except for black" however I didn't support that mode and instead I'm going to use alpha channels. As you can see my alien races are being rendered with unsightly black corners. Hopefully, just adding a an alpha to those parts will make them look better. Also, I will need to make them white so my tint code will work on them properly. Where there is a B_CUT blit that does the "everything but black" graphic paste, I get a warning generated. I'll do the graphic transparency fix and change it a B_COPY so the warning is removed. I will need to do that to all the builders to get things working.

Another thing I'm doing is cleaning up the code so that the functions show up in Intellisense. It's not hard, but going though each .c file and rewriting the declarations is a tedious process. However, it makes things much more searchable Originally, I couldn't see any functions in my solution explorer and now they are slowly coming back so I can zero in on what function I need to fix/alter.  I have Intellisense finished for all the builders, but haven't started with the game code yet. I want to get the builders done so I can actually play the game to see what kind of mess I have.

Before the game can run though, the 1Ch needs fixing. I feel that's going to to a few days as I have to make it able to flow with the proper global control variables. There is a bit of legacy code that is randomly hiding my mouse too. The original game hid the mouse pointer before screen draws so it woudn't get caught in cut/paste functions. This is no longer needed. The mouse will only be hidden when the game is running on a touch device.

Yea, That's going to be a fun project too :)

Saving Missions

Just a quick update. I'm still working on the Mission file save routine. It's pretty big as I have to parse out all the data and put it all in the right order. I also have to be mindful of "blank" data that was usually reserved for linked list addresses that I have to skip over.  All in all I have to manually organize and save anywhere between 300-500 pieces of data.

It's taking a while.

While I was at it I also worked on the loading and saving of "Game lists". The game supports more than one fleet commander. (in other words Multiple players), and you can assign you commander to a mission. While that commander is in that game, they are unavailable for any other missions. This means that when I have the mission data save function working it will be one step closer to playing the actual game...

But, I need to fix the enemy and captain builders first. The paperdolls don't work quite right yet and my color routines need to go in to color the enemy races properly.

Nothing big and surprising yet. Progress is being made.

I'm on a Mission

Mission builder with the original game running in DOSBox

Another Saturday of full-time work. I have the mission builder able to load mission files. I can't help but to side-eye it a little. I had to manually load all the data members and skip over unneeded data. I literally had the  mission file open in a hex editor and the original game running in DOSBox so I can see what data was where. On Monday I'll be writing the mission save function. There is a function to import a Breach 2 captain, but I have a tiny problem with that as the path only goes up to 15 characters and windows using 255. Also I realized that mission files are stand alone and can be shared. I will leverage this later.

The other big thing I realized is that this is a Star Trek game. I mean, I know it has some Star Trek like elements, but when looking deeper at the way everything is put together, it's Next Gen through and though.

On a completely unrelated note. I was thinking of implementing interface themes in the game. I already have the ability to load the "old skool" graphics, and have been using them as a place holder for now. The game will obviously have a new skin. It would be nice to give the users the ability to create their own galactic empire and have the interface themed to match it.

By default, of course, I'll be using an updated FWS theme of the original game.

Bob the Builder

The original developer's save file from 1992

I can now load and save the commanders, the ships, the captains and their enemies. The Mission builder is the next big thing on the list. This is kind of the coup de grâce of the builders because it requires all the other builders to be working. (You need to load Captains, Enemies, Ships, and Solar Systems to make a mission).

If finding more and more of those "detached" functions that aren't showing up in Intellisense. I'm still on the filesystem module, but after that I'll probably going to fix up all the other files in numerical order.

Captain builder

I may experiment with the enemy builder tomorrow. That's a part of the game that requires my new tint routines and transparencies to put the paperdolls together. I really amped to get into the game proper so I can see just how badly it's broken. When I first started this project, I was so timid about leaving the graphic layer. Now I seriously want to tear things apart.

On the business side, I need to drop a chunk of change to Uncle Sam in taxes this year. The income for funding this project was given to me in November, so it's on my income statement. Truth be told, I've been neglecting the "business" side of things and need to get that straightened out too.

Ship builder.

I already did the "Disturbed " joke when I was sick last time.

I was always more of a fan of the Richard Cheese version of "Down with the Sickness" anyway.

I cut and pasted a few lines of code, changed a read function to a write one, then I fell asleep at my desk. I went to bed and now I writing this after a day long nap and a powerful headache on my Chromebook. Looking forward to my sleeping schedule getting all botched up again.  And it was going so well too...

As it's the  beginning of the month and I super don't feel like dropping code at the moment, maybe I can take some quiet time to see where I'm at.

My current project is to continue to abstract the hardware away form the code. As I do this the game is slowly transforming into something playable. Things are "just working" which is making me happy. When I run the game, I have a terminal window open that spills out all my debug information. During phase one,the graphics and sound libraries were largely dummied out in order to get the program barely functional. "TODO:" implementation alerts would cascade on my screen. Now that things are getting fleshed out, the screen just lists warnings to keep track of. The big warning I get is that the 1Ch thread is disabled. In the original code, this was an interrupt and I documented how I changed it to an Allegro thread earlier in this blog. What's cool is the game does detect if this is disabled and places a useful "STANDBY" on the game screen itself where a "Winky" is supposed to be. (You can see this is previous screenshots). The 1Ch thread is going the be the last "big thing" I'm going to implement. First, because it kind of relies on everything else working, and Second, when I botch the thread up, it tends to cause Visual Studio's debugger to hang.

The other warning I get is when the game is expecting a graphic with a transparent alpha channel. This is a new functionality I have almost implemented, however, I'm still using the old graphics so I don't have any transparent assets just yet. (My artist is working on those :))

I also have to implement the "color tint" function I was experimenting with a few posts ago. That's going to bu used to "color" the alien races. That code will show up when I work on the Enemy Builder. (After the Captain Builder and Ship Builder are complete).

After the builders are done, it will be time to attack the 1Ch thread once and for all. After that... The game should just "work". If everything goes will the game will officially be in Beta and then it will be onto Phase Three.

Phase three will be graphic polishing and code optimization. Right now, the game eats 30% of my CPU when it runs, and I have an i7 with 32 GB of ram. Keep in mind, the *emulated* version in DosBox runs at 0.2%. Much of this is because I am doing lot's of per-pixel things on "virtual screen" in RAM and not using video memory effectively... or at all in many cases. I'm also using Allegro debug DLLs that are huge and spew log files all over the place.

This is also when I'll start the marketing. That will include website, landing page, social networking, and that whole business. Actually, the earlier I can start this the better. However, I would like to have a functional product before I start making promises.

That's my post for today.