Editing .SHP and .ENG files

[MrFlibble]

Do you know of any programs that would allow to edit (not just view) and create new Dune 2 .SHP images and .ENG string tables? AFAIK, XCC Mixer views most of Dune 2 SHPs and some ENGs (unfortunately, not mentat*.eng), but SHPs created with XCC Mixer don't work, as they are for C&C games, not Dune 2 Any help will be appreciated.

[ant222]

While there are no replies,

let me show my incompetence and ask this question:

Is the text compressed/encrpted in the .eng files? If it's not then you might try a hex-editor. However direct editing may broke the file. Are the format's specs open?

[ant222]

Hey, try this one!

It's reported to work with .eng files from Dune-II: Battle for Arrakis.

P.S.: If you haven't found it yourself already...

[MrFlibble]

Hi ant222, I've tried Siberian_GRemlin's editor, but alas, it only supports some of Dune 2's .eng files (not mentat*.eng and text*.eng - at least, for version 1.00). I just forgot to mention this editor in my first post.

The ENG files are, of course, encrypted, and looks like there's no tool available yet. However, the readme for the unofficial Dune 2 Russian localization mentions that one Wladimir Mutel' has been able to successfully decrypt the Dune 2 string tables, and even wrote a program to edit the ENG files. Maybe I'll contact this person and ask him if this program, or some documentation on the Dune 2 ENG format at least, still exists after the decade-or-so when the program was created.

[ant222]

They mentioned Battle for Arrakis. Can it have some differences from The Building of a Dynasty? Anyway you might try tocontact them...

[MrFlibble]

I've looked a bit deeper into things, and here's what I've come up with:

Siberian_GRemlin's editor can edit text*.eng after all, but the initial text is garbled. For example, the phrase

Welcome. I am your Mentat, and you may call me Ammon.

will look like

WЋЪш. IЃэyіБMЉ

[ant222]

Hmmm. Looks like they encode letter sequences by these &#NNNN. For exampple, the word Ammon is "Amъn" which yields that #1098 encodes "mo".

They probably measure some statistics and, basing thereon, decide which letter sequences are profitable to encode (met with higher frequency and having longer length) and which should be left as is (lower frequency, shorter length).

Maybe they didn't go that far and use a precalculated encoding table.

Anyway, there's not enough text in the game to make any compression useful. A surplus 150K is nothing compared to the 3,8 MBytes of the total volume...

[ant222]

Moreover, it seems fixed. And every &#NNNN encodes two letters.

Welcome -> WЋЪш = W+el+co+me.

Note, that &#1096 encodes "me". Then we find "me" encoded the same way in "call me...".

Some alphabetical order definitely is present:

1033 - en (in Mentat)

1035 - el (in Welcome)

So, we can guass 1034 should stand for "em" because m is between l and n in the alphabet.

1096 - me (Welcome)

1097 - ma (in "you may")

1066 - co (Welcome)

1067 - ca ("may call")

That's it!

The only problem I see is with the way they encoded "Mentat":

M + 1033(=en) + -t + 1027.

But 1027 is "am" (from "I am"). And in the above it should stand for "at".

But that's not a dead end. This "-t" is not just "t". Probably, "-" denotes that in the next encoded pair the last symbol is replaced by the character after "-".

[MrFlibble]

Hey, cool. Maybe you can decode it? I have the entire Ordos briefings and Mentat descriptions which I have manually put down from the game for D2TM - maybe this will help you? ::) I have also established correspondence between briefing text and string numbers in texto.eng for your convenience.

And BTW, I have contacted Siberian_GRemlin and sent him the same materials. He says he'll look into it when his exams are over.

Hmmm. Looks like they encode letter sequences by these &#NNNN. For exampple, the word Ammon is "Amъn" which yields that #1098 encodes "mo".

The &#NNNN is actually the number of the symbol that FED2k/IE/server assigns to it. SSTEditor represents these as symbols taken from a ususal Windows character table (see screenshot).

[ant222]

So, you need the texts for Atreides and Harkonnen automatically decoded and converted to plain text files just like those which you attached to the post?

Or do you need a program for decoding strigtables? What should it take on input? A string table file, or a normal text file with compressed text? The latter is a bit easier to do.

Well, I could try it when I have time (just like Siberian_GRemlin, I have examinations time). From you I'll want the compressed texts in plain .txt files, not in string tables so I can work with them easiy. I am not going to write yet another stringtable editor ;)

And for what purpose do you want the orinal texts?

[MrFlibble]

And for what purpose do you want the orinal texts?

I thought that it'd be just nice to have the ENG files decoded.

I'll try to figure out things myself in the meantime, and maybe edit mentat*.eng files manually.

[MrFlibble]

I have found out that a tool called WestPak2, which, by the way, I have downloaded from Siberian_GRemlin's site, successfully reads compressed text in text*.eng files, although seemingly doesn't allow to edit them. Mentat*.eng files are not supported (yet) though :(

[Ultraq]

Do you know of any programs that would allow to edit (not just view) and create new Dune 2 .SHP images and .ENG string tables? AFAIK, XCC Mixer views most of Dune 2 SHPs and some ENGs (unfortunately, not mentat*.eng), but SHPs created with XCC Mixer don't work, as they are for C&C games, not Dune 2 :( Any help will be appreciated.

Have you happened to find any?  I'm also kinda interested in the Dune 2 SHP file.

I'm currently working on a FreeCNC-like program, and a request came in from a modder of the original C&C for a Dune 2 SHP file format creation utility (the C&C mouse cursors are based on this format, and he wishes to modify the cursors).  So far I've managed to read/load the Dune 2 SHP file into the game engine I've developed.  The next step would obviously be saving a Dune 2 SHP file.

I'm well on the way to doing that (maybe a few weeks until completion, you can thank some recently released games for that ;D), but I'm still looking around for any additional resources to help me out.  So far I've got the source code to XCC Utilities, FreeCNC, as well as this semi-helpful document on the Dune 2 SHP file.  With this topic being about a month old now, I'm wondering if you've managed to find anything which could be helpful?

Thanks.

[MrFlibble]

Well, the only course of action available for me is to create a C&C SHP with the desired palette using XCC Mixer and then, referring to information from the XCC source code, try to manually edit the file to "convert" it into a Dune 2 SHP/WSA (looks like Dune 2 WSAs are even closer to C&C SHPs than Dune 2 SHPs). So far, I have been able to "resize" frames in menshp*.shp files to 1x1 pixel so as to remove obsolete facial animations. I'm planning to try a manual C&C SHP -> Dune 2 WSA convertion, but that won't be soon, as I've got a lot of other things to do.

[Ultraq]

Carrying-on from my last post, I've written a basic C&C SHP File -> Dune 2 SHP file converter.  Details can be found in this forum thread.  It seems to work for C&C's mouse cursors, so I'm not 100% sure if it will work with Dune 2.

[MrFlibble]

I will certainly check ASAP 8)

[minniat]

Here is a table with letter pairs needed to decrunch .ENG text files.

if character is >0x7F then it's a letter pair

(4 bits for first letter, 3 bits for second letter).

char pred[16] =

{

  ' ','e','t','a','i','n','o','s','r','l','h','c','d','u','p','m'

};

char succ[16][8] =

{

/* ' ' */  't','a','s','i','o',' ','w','b',

/* 'e' */  ' ','r','n','s','d','a','l','m',

/* 't' */  'h',' ','i','e','o','r','a','s',

/* 'a' */  'n','r','t','l','c',' ','s','y',

/* 'i' */  'n','s','t','c','l','o','e','r',

/* 'n' */  ' ','d','t','g','e','s','i','o',

/* 'o' */  'n','r',' ','u','f','m','s','w',

/* 's' */  ' ','t','e','p','.','i','c','a',

/* 'r' */  'e',' ','o','i','a','d','u','r',

/* 'l' */  ' ','l','a','e','i','y','o','d',

/* 'h' */  'e','i','a',' ','o','t','r','u',

/* 'c' */  'e','t','o','a','k','h','l','r',

/* 'd' */  ' ','e','i','u',',','.','o','a',

/* 'u' */  'n','s','r','c','t','l','a','i',

/* 'p' */  'l','e','o','i','r','a','t','p',

/* 'm' */  'e','a','o','i','p',' ','b','m'

};

If You don't understand feel free to write.

[MrFlibble]

The upcoming version of Siberian String Table Editor will support compressed text and text hierarchy in Dune 2 text*.eng and mentat*.eng files.

[drnovice]

Anyone knows a Dune 2 .shp and .cps files editor??

[MrFlibble]

I will certainly check ASAP 8)

Well that was not exactly ASAP, but I finally tried out the program. First, the good news: it compiles valid Dune 2 SHP files that are recognized by XCC Mixer. Cool huh? Now onto the less cheerful fact: AFAIK it doesn't (yet) allow to create Dune 2 SHPs that contain images with different dimensions. While most, if not all, actual Dune 2 SHP files are of this latter type. So no new Mentat animations yet guys...

I also tried creating a blank 1x1 pixel menshp*.shp file to replace the one I did by hand. Unfortunately, it screws up animation and the whole Mentat screen :( My guess is that menshp*.shp is supposed to use a different compression type (there are about 3 I think) than that offered by Ultraq's program...

[MrFlibble]

Some good news: the SHPs created by Ultraq's utility are compatible with Dune 2 1.07. Looks like different versions of Dune 2 use slightly different file formats (this is also true for WSA). Also, the problem with different frame sizes within a SHP can be solved by adding extra space filled with transparent color so compensate for frame size differences.

[MrFlibble]

For anyone interested, the latest version of the Siberian String Table Editor has integrated full support for Westwood *.ENG files used in Dune 2, allowing to read and edit both compressed and uncompressed text files, including MENTAT?.ENG files that have specific structure. Click here to download.

[drnovice]

MENTAT?.ENG files are a bit different about other .ENG text files.
Opening with SSTEditor you can't add new entries on categories, but we know that MENTATH.ENG file has 2 extra entries respect MENTATA.ENG & MENTATO.ENG files.
Sardaukar entry on Vehicles (Units) category and Frigatte entry on Specials category. Curious to see that they are completely in German but ingame they are readed in English, I must investigate about this oddity.
So this means that the game "can read" more entries ingame on Mentat Menu.
 
Here the schema-type of these files (example by MENTATA.ENG):
 
FORM  00 00 04 2d  MENTNAME  00 00 03 5d

11 00 00 03 7a  00Briefing           00 00
0f 00 00 03 8b  01Advice             00 01
0f 00 00 03 96  01Orders             00 01
0f 00 00 03 a1  10Houses             00 00
17 00 00 03 b2  11House Atreides     00 01
18 00 00 04 cc  11House Harkonnen    00 01
14 00 00 05 ba  11House Ordos        00 01
13 00 00 06 d4  20Structures         00 00
11 00 00 06 e5  21Barracks           00 02
16 00 00 07 cf  21Concrete Slab      00 01
1a 00 00 08 5a  21Construction Yard  00 01
16 00 00 08 d6  21Heavy Factory      00 04
1a 00 00 09 b1  21High-Tech Factory  00 05
0b 00 00 0a 2d  21IX                 00 07
16 00 00 0a dd  21Light Factory      00 03
10 00 00 0b 9a  21Outpost            00 02
0f 00 00 0c 60  21Palace             00 08
11 00 00 0d 5b  21Refinery           00 01
18 00 00 0d fc  21Repair Facility    00 05
16 00 00 0e 85  21Rocket Turret      00 06
14 00 00 0f 41  21Spice Silos        00 01
11 00 00 0f f8  21Starport           00 06
0f 00 00 10 bf  21Turret             00 05
0d 00 00 11 45  21Wall               00 04
12 00 00 11 cc  21Wind Trap          00 01
0c 00 00 12 ef  21Wor                00 03
11 00 00 13 ef  30Vehicles           00 00
11 00 00 14 00  31Carryall           00 01
14 00 00 14 d6  31Combat Tank        00 04
12 00 00 15 92  31Harvester          00 01
17 00 00 16 a1  31Heavy Troopers     00 02
17 00 00 17 23  31Light Infantry     00 01
0c 00 00 17 d5  31MCV                00 04
15 00 00 18 f0  31Ordos Raider       00 01
14 00 00 19 e3  31Ornithopter        00 07
0d 00 00 1a ad  31Quad               00 02
14 00 00 1b bf  31Rocket Tank        00 05
13 00 00 1c c3  31Siege Tank         00 06
0e 00 00 1d 89  31Trike              00 01
11 00 00 1e b3  40Specials           00 00
13 00 00 1e c4  41Death Hand         00 08
13 00 00 1f d0  41Devastator         00 08
11 00 00 20 d6  41Deviator           00 07
0f 00 00 21 b9  41Fremen             00 08
11 00 00 22 9d  41Saboteur           00 08
12 00 00 23 eb  41Sand Worm          00 01
13 00 00 24 cc  41Sonic Tank         00 07

00  DESC  00 00 22 59

               No file?NO Desc   00
        3  2a  No desc   00
        0  2a  No desc   00
               No file?NO Desc   00
FARTR.WSA  2a  Caladan:  0c  Our ancestral home is the lush planet of [...]

[...]

00  INFO  00 00 00 0c 00 00 00 2f 00 00 01 4e 00 00 00 d1
 

• FORM section: orange bytes indicate total length of file. Just count the offset of last row in eof and go back of 7 byte. The correct offset address is the next byte of yellow byte.
• Entry section: The first FIVE bytes regards:
  1    => length (hex value) of complete string, including this byte and the name string, until last 2 bytes
  2-5 => room for number (hex value) of bytes to jump to read the DESC (descriptions) section
  then there is name string of entry with refering number of category (e.g. 41 means entry of Specials category).
  the last TWO bytes after the name string indicate the mission number that entry becomes available ingame (note that macro entries like Houses, Structures, Specials should have always 0)
• DESC section: that every description is refered to relative entry (where it's specified how many bytes jumps to start to read), it has .WSA image and description text (0c byte splits title description and mentat text description) .
• INFO section: where you must specify number (hex value) of entries (yellow byte).

 
Other bytes I don't know them yet, probably they regard link to other files.