Dune II editor with 1.07 support

[Nyerguds]

How about these projectiles then?

[MrFlibble]

Heh, I've always wondered why in the original scenarios, unit IDs are never below 22 (sometimes they start around 27); sandworms have IDs above 22 as well :) Most third-party scenario editors change the IDs so that they form a normal numerical list starting with 0 or 1 though.

[MrFlibble]

Noticed something related to the "Required PAK file" thingy. If a custom made campaign (like Stefan's; or just series of modified levels) in the form of raw INI files into your Dune II folder, the game will load the next mission from the folder if you win. However, if you lose and are prompted to choose another of the remaining regions, the game will load the scenario file from SCENARIO.PAK, ignoring its counterpart in the folder (to get the custom mission working in this situation, you need to save the mission, restart the game, load the mission and restart it). This behavior only happens if SCENARIO.PAK is tagged as "Required PAK file". If it is not, scenarios are loaded from the directory after losing a mission as well.

From this it can be speculated that the "Required PAK file" has something to do with telling the game where to look for the files first: in the game directory or in a specific PAK archive. I also wonder why MENTAT#.ENG files edited by SSTEditor freeze the game only if inside a "required" PAK archive. Clearly there is something in the edited files' structure that causes incorrect game behavior, but why does it only happen for PAK files that are handled differently from the others by the game? ???

[Nyerguds]

Heh... It just means that just like in C&C, they got different functions for opening files; one that only looks in the pak files, and another that looks in the game folder first and then in the pak files. The problem here is not these functions, but when the programmers decided to use which one.

Judging from that reload system, they probably planned to make the "required PAK file" setting disallow reading of a pak's contents from the game folder, but looking at the reality, it's clear they either made poorly-implemented exceptions on it (for the scenarios) or just plain forgot about it and stopped implementing the rule in the rest of the code they wrote. This isn't too far-fetched, seeing as there is a version of the game which doesn't have the Required PAK thing set on any of its files.

It's even possible that they used this as beta feature; disabling the Requirement on the files to test stuff from the game folder, and only later packing it all together and changing the file settings.

But yeah, I'm not surprised that half-implemented stuff like that can cause problems.

On a side note, are the MENTAT#.ENG originally in a Required PAK file?

[MrFlibble]

Judging from that reload system, they probably planned to make the "required PAK file" setting disallow reading of a pak's contents from the game folder, but looking at the reality, it's clear they either made poorly-implemented exceptions on it (for the scenarios) or just plain forgot about it and stopped implementing the rule in the rest of the code they wrote.

On a side note, are the MENTAT#.ENG originally in a Required PAK file?

Yes, ENGLISH.PAK is a required one (whereas GERMAN.PAK and FRENCH.PAK are not). BTW, if I'm not mistaken, the US version will read DUNE.ENG from the folder first, but all other files from ENGLISH.PAK will be read from the PAK unless they are removed from it, in which case the game will use their counterparts in the folder (if present).

[Nyerguds]

I see.. bizarre. I wonder what makes it freeze then, if the modified strings file DOES work outside of the required pak...

[Nyerguds]

Kinda off topic, but still fun...

Is it known where this is handled?

Harkonnen WOR (from level 2, upgrades in 4, instead of 5 and 6)

Ordos Siege Tank (from level 7 instead of 6)

I just found ALL of this data in C&C95... it's all together in the function that determines if the player is allowed to get a certain item as construction option or not. Really cool. This allows me to unlock the helicopter pad and APC for Nod :)

[TrueBrain]

I've noticed an odd thing concerning the "File type" variable. As I mentioned before, some of the PAK files are marked as "Required PAK file" (as we called it), namely:

DUNE.PAK

ENLGISH.PAK

INTRO.PAK

MENTAT.PAK

SCENARIO.PAK

SOUND.PAK

This is true for all versions except the EU version (which also seems to be the latest one, judging by file dates and the presence of features that are missing from other releases), where none of the PAK files are tagged as "required". What is interesting is that by default, the EU version does not display the "Taking advantage of extended memory. Please wait..." message during initialization (given that digitized sounds and the use of extended memory have been enabled), unlike any other version (some even have this message translated into German and French). However, when I changed the above-mentioned files' status to "required" in the EU version, it started showing the extended memory message when appropriate.

I have no idea what to make out of this, but it seems possible that these files have something to do with memory management at game startup or something along those lines...

Noticed something related to the "Required PAK file" thingy. If a custom made campaign (like Stefan's; or just series of modified levels) in the form of raw INI files into your Dune II folder, the game will load the next mission from the folder if you win. However, if you lose and are prompted to choose another of the remaining regions, the game will load the scenario file from SCENARIO.PAK, ignoring its counterpart in the folder (to get the custom mission working in this situation, you need to save the mission, restart the game, load the mission and restart it). This behavior only happens if SCENARIO.PAK is tagged as "Required PAK file". If it is not, scenarios are loaded from the directory after losing a mission as well.

From this it can be speculated that the "Required PAK file" has something to do with telling the game where to look for the files first: in the game directory or in a specific PAK archive. I also wonder why MENTAT#.ENG files edited by SSTEditor freeze the game only if inside a "required" PAK archive. Clearly there is something in the edited files' structure that causes incorrect game behavior, but why does it only happen for PAK files that are handled differently from the others by the game? ???

To come back on this, this feature is not "Required PAK file", it is "Load to XMS/EMS" flag. More detail:

When you set this flag on a file (or PAK, both are valid), it will load the file in question into the XMS/EMS memory when opening it for the first time. All next requests for and actions on this file come from memory. This removes a lot of overhead of opening the file on the disk. For some reason, this flag is always off in the EU version (which gave me a nice suprise :P).

This also explains your scenario problem: as soon as the PAK file with scenarios is loaded, all the scenarios are put in memory from the PAK file. So as long as you keep on loading scenarios outside the PAK file, you are fine. As soon as you load one from inside the PAK file, all next loads will be from inside the PAK file.

You can freely alter a scenario outside a PAK at runtime. Restarting the level reloads all data (even the length of the file). Altering a scenario file inside a PAK file on runtime will result in breaking, assuming you have the above mentioned flag off. Also, enabling the flag on a scenario file itself, will make it use that content for the rest of the run of the application.

I hope the above all makes a bit sense to you :)

Now my only remaining question is: why did they disable the 'Load to XMS/EMS' in the EU version?

[Nyerguds]

Oh wow, this is interesting information :)

As for the EU version... the question is if it's disabled in the exe itself, or only in the files table.

[TrueBrain]

Oh wow, this is interesting information :)

As for the EU version... the question is if it's disabled in the exe itself, or only in the files table.

As the files table is in the exe, I would say the exe. But when enabled, it simply works, so all code is intact and there.

[Nyerguds]

Meh, technicalities. I meant if the XMS code was in the exe :P

[MrFlibble]

Thanks for clearing this issue, TrueBrain :)

Now my only remaining question is: why did they disable the 'Load to XMS/EMS' in the EU version?

IIRC the three-language release in question was compiled in the UK, and certain code modifications were done compared to the original US version; I guess they might have turned it off for some reason and forgot to turn it back on :)

I still wonder why the game will crash if a mentat database file that was modified by SSTEditor is placed inside a "Load to XMS/EMS" PAK file, but will work fine if such files are read from the directory or from an unmarked PAK file.

[TrueBrain]

Meh, technicalities. I meant if the XMS code was in the exe :P

That I tried to say: yes :) Else voices can never work. (if you have no XMS/EMS, voices are never loaded. It doesn't fit in the normal memory). So all code is intact, just disabled for PAK usage.

Thanks for clearing this issue, TrueBrain :)

IIRC the three-language release in question was compiled in the UK, and certain code modifications were done compared to the original US version; I guess they might have turned it off for some reason and forgot to turn it back on :)

I still wonder why the game will crash if a mentat database file that was modified by SSTEditor is placed inside a "Load to XMS/EMS" PAK file, but will work fine if such files are read from the directory or from an unmarked PAK file.

That is indeed odd. What is the name of such file? (haven't touched any Mentat code yet ;))

[MrFlibble]

That is indeed odd. What is the name of such file? (haven't touched any Mentat code yet ;))

MENTATA.ENG, MENTATH.ENG, MENTATO.ENG (GER/FRE), inside ENGLISH.PAK. They are unique in that all other sting table files just contain text strings (compressed or not), these also contain a sort of hierarchy of strings that defines what text goes to what element of the database. As far as I know, SSTEditor is the only tool capable of handling those. When the modified files are put back into ENGLISH.PAK, the game will freeze if you go to the Mentat menu and select any topic from the database (I suppose it might be that the game can't find the necessary WSA file, but no error message is shown). Sometimes (very rarely) the modified file will work, but normally it won't. If you turn the "Load into EMS/XMS" option off for ENGLISH.PAK, or put the modified files into another PAK file without this tag, they will work fine.

Oh, and maybe you would be able to shed some light on the Unknown 006 (bit 6) option in the units section? It's turned on in v1.07 for units, and turned off for everything in v1.0. I wonder what it could be for. Maybe other unknowns could be also explained as well? :) (Just asking ;D)

[TrueBrain]

MENTATA.ENG, MENTATH.ENG, MENTATO.ENG (GER/FRE), inside ENGLISH.PAK. They are unique in that all other sting table files just contain text strings (compressed or not), these also contain a sort of hierarchy of strings that defines what text goes to what element of the database. As far as I know, SSTEditor is the only tool capable of handling those. When the modified files are put back into ENGLISH.PAK, the game will freeze if you go to the Mentat menu and select any topic from the database (I suppose it might be that the game can't find the necessary WSA file, but no error message is shown). Sometimes (very rarely) the modified file will work, but normally it won't. If you turn the "Load into EMS/XMS" option off for ENGLISH.PAK, or put the modified files into another PAK file without this tag, they will work fine.

If you put the files in another PAK, they should not be loaded. Dune2 is very picky where a file should be, and if it is not where it expects it to be, it doesn't load it.

Either way, I see no logic reason why the fact it is loaded in the EMS/XMS could cause such problems. The code looks clean and bug-free, as far as I can tell. Maybe it makes some assumption on its location .. dunno. We will see when we get to the Mentat stuff, but I am afraid it will remain a mystery ;) For sure it will work in OpenDUNE btw.

Oh, and maybe you would be able to shed some light on the Unknown 006 (bit 6) option in the units section? It's turned on in v1.07 for units, and turned off for everything in v1.0. I wonder what it could be for. Maybe other unknowns could be also explained as well? :) (Just asking ;D)

What is Unknown 006 (bit 6)? (I never used the editor, and most likely never will ;) It will only bias me while converting. When I would have known you guys thought the above mentioned flag did what you think it would do, it would have taken me much longer to understand the code :P).

Check:

http://svn.opendune.org/trunk/src/unit.h?mode=view#filecontent

Under 'UnitInfo'. This should match the data you talk about. Or at least I hope that the mapping from the disk to this struct is one-on-one. Either way, that file documents what we think the variables do (and for those without '??', we are pretty sure). If you have any specific questions, let me know :)

[MrFlibble]

If you put the files in another PAK, they should not be loaded. Dune2 is very picky where a file should be, and if it is not where it expects it to be, it doesn't load it.

Of course I meant that if you put it into another PAK files and add it to the respective file list - that's how I added extra content in Super Dune II: Classic Edition and Dune 2 eXtended.

What is Unknown 006 (bit 6)?

Just a bit flag; we do not know what it does :)

Check:

http://svn.opendune.org/trunk/src/unit.h?mode=view#filecontent

Under 'UnitInfo'. This should match the data you talk about. Or at least I hope that the mapping from the disk to this struct is one-on-one. Either way, that file documents what we think the variables do (and for those without '??', we are pretty sure). If you have any specific questions, let me know :)

Thanks, that should be informative :)

[TrueBrain]

(..)

Just a bit flag; we do not know what it does :)

(..)

Yes, but what does it mean? Does Unknown 006 point to the 6th variable in the struct or the 6th byte? I am not familiar with what syntax is used :)

[MrFlibble]

I was referring to the way it is defined in the Editor. I have not idea how to put it any other way :)

[Nyerguds]

From my editor's code:

    // unitOptsList
         (
(typeid: 02; text: 'Short name ID          '; helpid: 02), // size: 2
(typeid: 03; text: 'Name code reference    '; helpid: 03), // size: 4
(typeid: 02; text: 'Long name ID           '; helpid: 02), // size: 2
(typeid: 03; text: 'WSA file code reference'; helpid: 03), // size: 4
(typeid: 12; text: 'Air unit shadow        '; helpid: 00), // size: 1
(typeid: 13; text: 'Unknown 005 (bit 2)    '; helpid: -1), 
(typeid: 14; text: 'Unknown 005 (bit 3)    '; helpid: -1), 
(typeid: 15; text: 'Unknown 005 (bit 4)    '; helpid: -1), 
(typeid: 16; text: 'Unknown 005 (bit 5)    '; helpid: -1), 
(typeid: 17; text: 'Worm camouflage        '; helpid: 00), 
(typeid: 18; text: 'Turret Rotates         '; helpid: 00), 
(typeid: 19; text: 'Unknown 005 (bit 8)    '; helpid: -1), 
(typeid: 12; text: 'Pick up for repairs    '; helpid: 00), // size: 1
(typeid: 13; text: 'Unknown 006 (bit 2)    '; helpid: -1),
(typeid: 14; text: 'Unknown 006 (bit 3)    '; helpid: -1),
(typeid: 15; text: 'Force all script cycles'; helpid: 19), // 'ForceScriptCycles' - relates to TacticalPos
(typeid: 16; text: 'Can target air units   '; helpid: 00),
(typeid: 17; text: 'Unknown 006 (bit 6)    '; helpid: -1), // for all units: 0 in demo/1.00, 1 in 1.07 versions
(typeid: 18; text: 'Unknown 006 (bit 7)    '; helpid: -1),
(typeid: 19; text: 'Unknown 006 (bit 8)    '; helpid: -1),
...

(sizes taken from the getDataLen function in the code, by TypeID. All condensed bit flags words are split into 2 separate bytes)

So the value MrFlibble is talking about is the second bit of the byte at address [unit]+0Dh, which means the second byte of the condensed bit flags word variable you got as variable_0C in UnitInfo.

Looks like we got more info on that than he does, MrFlibble...

By the way, TrueBrain... I noticed that the the next one after these 2 bytes is marked with "??" on that page... it's the chance an infantry unit spawns when the unit is destroyed.

You might get some more useful information from my editor code... we put quite some work into the research of these variables.

[TrueBrain]

I was referring to the way it is defined in the Editor. I have not idea how to put it any other way :)

And I was asking how it was defined ;) But Nyerguds answered nicely ;) I will see tomorrow if I can find what that bit does :)

From my editor's code:

(..)

(sizes taken from the getDataLen function in the code, by TypeID. All condensed bit flags words are split into 2 separate bytes)

So the value MrFlibble is talking about is the second bit of the byte at address [unit]+0Dh, which means the second byte of the condensed bit flags word variable you got as variable_0C in UnitInfo.

Tnx!

(..)

By the way, TrueBrain... I noticed that the the next one after these 2 bytes is marked with "??" on that page... it's the chance an infantry unit spawns when the unit is destroyed.

You might get some more information from my editor code... we put quite some work into the research of these variables.

Please read my pre-last post :) I explain why we don't, and won't :) Independent research is always better. If we are done, we match up and see where we differ :)

[Nyerguds]

Well a lot of this was also researched by Segra by disassembling. I mean come on, you don't even have the prerequisites identified there... he even found the functions that handle those. There's a lot of other obvious stuff like the unit commands.

btw, your "spriteID" var is kinda wrong. That's not the spriteID of the unit, but of the icon it uses. The unit's sprite ID is at index 44h

This is my editor's data, converted to indexes, with var sizes and descriptions. Take it or leave it, your choice. :P

00 length: 02  // Short name ID
02 length: 04  // Name code reference
06 length: 02  // Long name ID
08 length: 04  // WSA file code reference
0C length: 01  // Byte of bit flags: Air unit shadow / Unknown / Unknown / Unknown / Unknown / Worm camouflage / Turret Rotates / Unknown
0D length: 01  // Byte of bit flags: Pick up for repairs / Unknown / Unknown / Force all script cycles / Can target air units / Unknown / Unknown / Unknown
0E length: 02  // Infantry spawn chance
10 length: 02  // Hit points
12 length: 02  // Sight   
14 length: 02  // Sidebar icon
16 length: 02  // Cost    
18 length: 02  // Build Time
1A length: 02  // Tech level
1C length: 02  // Prerequisites part 1
1E length: 02  // Prerequisites part 2
20 length: 01  // Build icon order
21 length: 01  // Upgrades needed
22 length: 02  // Sidebar command #1
24 length: 02  // Sidebar command #2
26 length: 02  // Sidebar command #3
28 length: 02  // Sidebar command #4
2A length: 02  // [CHOAM status - unused]
2C length: 01  // Unknown 022
2D length: 02  // Unknown 023
2F length: 02  // Threat level for AI
31 length: 01  // Owner   
32 length: 02  // Unit array range min
34 length: 02  // Unit array range max
36 length: 01  // Byte of bit flags: Unknown / Explode on target / Explode when dying / Sonic Immunity / Bumpy Movement / Tracked crushing / Has user control / Stay on map
37 length: 01  // Byte of bit flags: Unknown / Unknown / Fires Twice / Sand impact (if weapon) / Unknown Unknown / Unknown / Not a weapon type
38 length: 02  // Unknown 030
3A length: 02  // Unknown 031 // related to AI building more or less of this unit?
3C length: 02  // Movement type
3E length: 02  // Move animation speed
40 length: 02  // Speed   
42 length: 02  // Turning speed
44 length: 02  // Unit graphics
46 length: 02  // Turret graphics
48 length: 02  // Default AI command
4A length: 02  // Unit display mode (frame usage)
4C length: 02  // Death animation
4E length: 02  // Weapon rate of fire
50 length: 02  // Weapon range
52 length: 02  // Weapon damage
54 length: 02  // reference to special animation script (see segras unit scripts explanation) e.g. worm eat.
56 length: 02  // Weapon type
58 length: 02  // Weapon sound

[TrueBrain]

Well a lot of this was also researched by Segra by disassembling. I mean come on, you don't even have the prerequisites identified there... he even found the functions that handle those. There's a lot of other obvious stuff like the unit commands.

btw, your "spriteID" var is kinda wrong. That's not the spriteID of the unit, but of the icon it uses. The unit's sprite ID is at index 44h

This is my editor's data, converted to indexes, with var sizes and descriptions. Take it or leave it, your choice. :P

(..)

As I said above: I will get back to you in a few months. For now, my approach is much better, for the reasons above :) Tnx anyway.

[Nyerguds]

Ah, right. But for things we ARE 100% sure of, your way just takes longer.

Ah well, it's your hobby, have fun with it; you can always come back here and look at the list after you found enough stuff :)

But please tell us the stuff you found that we don't know yet, then ;)

I'm not trying to do things "the right way", I'm just trying to make the editor as complete and correct as possible ;D

[MrFlibble]

Ah, right. But for things we ARE 100% sure of, your way just takes longer.

I think the difference is in that the editor's purpose is to modify the existing EXE to achieve a desired effect, while OpenDUNE team's research has the goal of actually understanding all the processes behind game behavior, and all the possible interrelations between different functions. In this respect, getting a known result from the editor's database actually means skipping some important steps in general research of Dune II code.

[Nyerguds]

So? I guarantee you he'll find the entire scope of the worm ripple effect a LOT faster if he can trace it from the bit that actually CAUSES it. Which, imo, IS a valid origin point to start a trace from. You can always trace in 2 directions, and see where it links back into known code.

It's just, the way he does it, he'll never get the remake finished ::)