Advanced questions on saved game hex editing

[Dmitri Fatkin]

Good day, everyone.  At the closing week concluding the last year, I began working on design of a project which is currently inwardly known as Dune: A Harder Path, and represents itself about 3 times harder version of the original Cryo Interactive's campaign, with redesigned armies, appearances of their leaders, resources, locations and many other things which resulted in about 24 kilobytes changelog and 2 starting saved game files compatible with both 2.1 and 3.7 versions.

 

However, along that 4 weeks road, I've started experiencing certain difficulties in terms of what I felt could have been improved by a change, and yet, discovering how exactly such modification should be done would have easily resulted in multiple days of searching for the exact data offsets, thus drawing me away from the outlined development plan, the main philosophy of which consists of having a stable, playable and well-balanced Alpha version which contains thoroughly-maintained changelog and lacks merely sufficiently-summarized readme files.

 

Below I'm submitting the inquiries in regards to particular areas of advanced editing of saved game files, hoping that some of you possess more information on the topics than I am, thus being able to offer own help in order to slightly speed-up the extensive campaign-making process. I'll be expanding that list of questions on occasion when further-ones arise.

 

1) How specifically and to what extent the F7 repeat byte compacts game data? Is it possible to extend/alter its utilization to the degree where putting additional data within certain offset frames will not cause collision of data retrieval from the loaded save file by the game, thus allowing to station more data such as identities of troops past 67?

 

2) Based off the first question, do you think it's technically-possible to extend the number of IDs past what the game expects it to be & loads at day 1, apart troops reallocation? (That's what I've done as of now). Are troop IDs past 67 inherently limited by how the program recognizes them and are there any workarounds for making such a troop be movable from sietch to a sietch & respond to clicking queries properly?

 

3) Since there are 4 principal ways of launching the conquest against Harkonnen fortresses in the game (through Bledan, Tsympo, Haga, or directly Arrakeen area), and one of them consists of assembling and training the forces below Arrakeen region, slightly down from Tuono-Tabr, I'm interested in how exactly the swapping of Smuggler NPCs between Arrakeen-Pyons and Tuono-Pyons can be achieved, resulting the one at the latter location to never haggle over the price, or perhaps, there's a way to set that up separately for the existing Tuono-Pyons NPC, without the necessity to switch the traders between locations? The data associated with the Smugglers in the place from where character allocation & dialogues display information is taken doesn't seem to have anything related to every Smuggler in particular, providing a generic specification of "0C00023080006F93".

 

4) Is there a chance someone could elaborate on the detailed description of character data which follows the definition of Troops, the one Hugslab has already briefly touched upon in his reply there? Would be nice to know which bytes control a character's willingness to follow the player and at which offsets are the ones responsible for designation of accompanying NPCs.

 

So far, my own brief description of corresponding data consists of this:

The first byte is sprite identificator of the character.
The third byte is location of the room within the place.
The fourth byte is type of the place the character stays in (that byte is specified in field C of sietch data).
The 6th byte is used to determine whether a character is active (it's hidden when it's set to FF). Also serves as a pointer to the exact place the character is at.
The 7th byte refers to the NPC the player will have a dialogue with upon clicking the sprite, it's also linked with the last byte (either 92 or 93).

 

5) As we know, somewhere there's a byte which determines Game Stage (or, how Rémi Herbulot called it - Phase) variable, directly influencing player's discovering capabilities or the course of events that are about to happen after it has reached certain value. Where exactly is that byte defined? And, no-less interestingly, is Harkonnens' intent to attack is directly linked with it, or it's treated as a separate variable which gets turned on upon reaching of military Game Stage? I'm asking since internal development console seems to allow to activate the option named "HARKO ATTACK".

 

6) Finally, of a much lower priority, and yet, no-less interest of being aware about for an extra occasion lies the question regarding specification of offsets and the description of bytes responsible for depiction of Controlled Areas on the strategical planetary map, provided through "See Results". I'll be surprised if someone extensively researched that, however, nothing prevents us from thinking & investigating the subject a bit further, right?

 

Thanks for answering any of this, and, if some of you will be willing to assist or exchange ideas further, I can be reached either as cyber_smoke in Skype or Smoke Nightvogue#1044 on Discord. Alternatively, you can simply dispatch a private message here to discuss anything related to the topic or the world of Dune novels in general.

Edited August 15, 2020 by Dmitri Fatkin
Found the answer to 3rd question a couple of weeks ago

[Nyerguds]

I found specs on the savegames, and... it seems the 7F thing is a simple flag-based RLE compression.

https://sites.google.com/site/duneeditor/savegame-editing

http://www.shikadi.net/moddingwiki/RLE_Compression

That's not really something you want to handle with hex editing though. It's much better to have a tool to uncompress the data, and recompress it afterwards.

 

[Lord Winner]

I kept 49 successive copies of DUNE21S3.SAV from the game run where I could launch the final attack on the morning of day 17 ( https://forum.dune2k.com/topic/20497-dune-cheats/?tab=comments#comment-395491 ). These might help pinpointing the game stage / phase variable, without having to playing through the entire game again. I can somehow send you a set of savegames for quick runs (17 to 20 in-game days) if you wish

 

EDIT: In order to research 5), I built a couple C++ programs, a RLE decompressor with hard-coded F7 byte and a program to detect bytes whose values have strictly increased between the N given files. Nothing fancy, and in fact, while posting this, I'm thinking that this decompressor is incorrect: I didn't special-case the F7 02 00 bytes at the beginning. Also, the "chars[1] < chars[argc - 1]" condition is superfluous, it's a leftover from an earlier state of the code.

#include <cstdio>
#include <cstring>

int main(int argc, char ** argv) {
    FILE * fin;
    FILE * fout;

    if (argc < 3) {
        fprintf(stderr, "Usage: %s <input file, - for stdin> <output file, - for stdout>\n", argv[0]);
        return 1;
    }

    if (!strcmp(argv[1], "-")) {
        fin = stdin;
    }
    else {
        fin = fopen(argv[1], "rb");
    }

    if (nullptr != fin) {
        if (!strcmp(argv[2], "-")) {
            fout = stdout;
        }
        else {
            fout = fopen(argv[2], "w+b");
        }

        if (nullptr != fout) {
            while (!feof(fin)) {
                int c1 = fgetc(fin);
                if (c1 != EOF) {
                    if (c1 != 0xF7) {
                        fputc(c1, fout);
                    }
                    else {
                        int len = fgetc(fin);
                        if (len != EOF) {
                            int data = fgetc(fin);
                            if (data != EOF) {
                                for (int i = 0; i < len; i++) {
                                    fputc(data, fout);
                                }
                            }
                        }
                    }
                }
            }
            fclose(fout);
        }
        fclose(fin);
    }

    return 0;
}

#include <cstdio>
#include <cstring>
#define __STDC_FORMAT_MACROS
#include <cinttypes>

int main(int argc, char ** argv) {

    if (argc < 3) {
        fprintf(stderr, "Usage: %s <input file 1> <input file 2> ... <input file n>\n", argv[0]);
        return 1;
    }
    printf("%d\n", argc);

    FILE * fptr[argc + 1];
    int chars[argc + 1];

    bool error = false;
    for (int i = 1; i < argc; i++) {
        fptr[i] = fopen(argv[i], "rb");
        if (nullptr == fptr[i]) {
            error = true;
        }
    }
    if (!error) {
        uint32_t offset = 0;
        while (!feof(fptr[1]) && !error) {
            for (int i = 1; i < argc; i++) {
                chars[i] = fgetc(fptr[i]);
                if (EOF == chars[i]) {
                    error = true;
                }
            }
            if (!error) {
                bool candidate = true;
                for (int i = 1; i < argc - 1; i++) {
                    if (chars[i] >= chars[i + 1]) {
                        candidate = false;
                    }
                }
                if (candidate && chars[1] < chars[argc - 1]) {
                    fprintf(stdout, "%04" PRIX32 " ", offset);
                    for (int i = 1; i < argc; i++) {
                        printf("%02X ", chars[i]);
                    }
                    printf("%d ", candidate);
                    fputc('\n', stdout);
                }
                offset++;
            }
        }
        for (int i = 1; i < argc; i++) {
            if (nullptr != fptr[i]) {
                fclose(fptr[i]);
            }
        }
    }

    return 0;
}

I chose a subset of the savegame files from the aforementioned 17-day run of the game, decompressed them with the first program (producing 22056-byte files), then fed them to the second program thusly:

./findincreasedbytes DUNE21S3_day2_afterfindingtuonoclam_decompressed.SAV DUNE21S3_day2_afterfindingharah_decompressed.SAV DUNE21S3_day3_afterfirstvisionandviewingmessage_decompressed.SAV DUNE21S3_day5_afterfindingstilgarandrecruitingtroops_decompressed.SAV DUNE21S3_day8_beforegoingtochani_2_decompressed.SAV DUNE21S3_day9_aftermeetingchani_decompressed.SAV DUNE21S3_day10_attackingtsympos_decompressed.SAV DUNE21S3_day17_5charactersinarrakeentuek_decompressed.SAV

The output was pretty small:

9
43DF 07 09 0C 14 15 16 1B 28 1 
43E1 08 0C 0D 16 19 1D 22 3B 1 
43E4 10 14 15 2C 44 48 55 68 1 
4CA4 0D 16 24 3D 4C 74 97 FB 1 
4CB0 0E 10 12 18 1C 1E 24 36 1 

For Dune "21", the variable at offset 43E4 (with the above decompressor, probably at a slight offset from that with a fixed decompressor) looks like a great candidate for the game stage / phase variable, since multiple values of that byte match the first byte of the section F of multiple sietches:

• Tuono-Clam can be found when the game stage is >= 0x10;
• Harah is in Tuono-Timin, which can be found at stage 0x14, and Habbanya-Tabr, Habbanya-Timin and Habbanya-Tuek have also 0x14;
• the Ergsun region and Sihaya-Clam have 0x28 or 0x2C;
• Chani is in Oxtyn-Tabr which has 0x44;
• after meeting Chani and she falls in love with Paul, sietches such as Oxtyn-Tuek and Habbanya-Harg become accessible, they have 0x48;
• 0x55 is the value of Sihaya-Tuek, which means I started attacking the Tsympo fortresses remotely (as described in the other topic, I had the character drink the water of life in Tuono-Tabr on day 9), before meeting Liet Kynes;
• the last save is from the step before I could launch the final attack, long after meeting Liet Kynes, which raises the variable to 0x5C.

Then, I checked a DUNE21S3_day11_attackingtsympos2_decompressed.SAV savegame between these two last savegames passed to my program: the byte at offset 43E4 is 0x5C. Looks like I met Liet Kynes in the meantime.

In my fast runs, the Harkonnens never get a chance to attack Fremen, I don't leave any mining troops close to the Arrakeen region and I have at most one army troop from the Carthag - Tuono - Habbanya region. The game stage / phase variable is not the only input to the algorithm deciding whether the Harkonnens should attack, that's all I can guess

A couple final notes:

• the byte at offset 4CA4 might be the charisma byte, given that it's the only value among the four candidates which goes over 100 at the end of the game. But maybe the set of savegame files I fed into the program contains two consecutive states where the charisma didn't increase, causing it to be filtered. In any case, easy to check by changing that byte and reloading the save; if it's not the byte at offset 4CA4, diffing fewer savegames with more change between savegames would help. EDIT2: nope, that byte isn't the charisma, and in fact, I can't find any byte containing the values displayed in the results screen.
• I should definitely special-case F7 02 00 in my decompressor, and then, I'd be curious to try using a different byte for the RLE compression (not that it makes much sense in practice). Maybe that's the reason for that odd F7 02 00 sequence at the beginning of the savegame: storing the RLE byte inside the savegame.

Edited February 12, 2020 by Lord Winner
Adding the programs, the output of the second program, and commentary.

[Lord Winner]

About 6), a wild guess: in the first 12 KB of decompressed savegame files (offsets ~6 to ~0x3080), the patterns are interesting (mostly 0, 2, A and 8 hex digits, but occasional F or E), and the proportion of set bits raises over time, AFAICS, it's higher in a day 27 pre-final-attack savegame than in a day 17 pre-final-attack save games. I don't currently have access to the computer which has savegames of higher in-game times and wider controlled areas.

I hope I kept the day 200+ savegame, where I had Fremen troops build wind-traps into all sietches which didn't have any, converted all troops to ecology, and reached over 70% controlled area without cheating (nearly, or slightly more than, 80% after resorting to manual water refilling in the savegame file ).

EDIT the next day: a bit of testing by direct savegame editing (in compressed form) proved this wild guess to be correct. The first F7 BD 00 in DUNE21S0.SAV, at file offsets 6-8, is a perfect playground for experimenting with the control status of a fairly large area by changing a single byte. I played with a single hex digit, the other half of the byte behaves the same way. The effect applies to the northern area of the planet, north of Tsympo-Pyort, and it is as follows:

00 (0b0000): nothing
01 (0b0001): light Atreides control, light vegetation (only green)
02 (0b0010): light Atreides control, no vegetation
03 (0b0011): light Harkonnen control, no vegetation
04 (0b0100): light Atreides control, light vegetation
05 (0b0101): heavier Atreides control, heavy vegetation (green and brown)
06 (0b0110): heavier Atreides control, light vegetation
07 (0b0111): mixed Atreides / Harkonnen control, light vegetation
08 (0b1000): light Atreides control, no vegetation
09 (0b1001): heavier Atreides control, light vegetation
0a (0b1010): heavier Atreides control, no vegetation
0b (0b1011): mixed Atreides / Harkonnen control, no vegetation
0c (0b1100): light Harkonnen control, no vegetation
0d (0b1101): mixed Atreides / Harkonnen control, light vegetation
0e (0b1110): mixed Atreides / Harkonnen control, no vegetation
0f (0b1111): heavier Harkonnen control, no vegetation

Therefore, pairs of bits have the following meaning:

00: no area control or vegetation;
01: Atreides control, vegetation;
10: Atreides control, no vegetation;
11: Harkonnen control.

 

Edited February 10, 2020 by Lord Winner