Format80 decompression

[Ultraq]

Re: Dune 2 SHP file format -

In conjunction with the document mentioned above, I added my findings about the format, and wrote something about it in the source code for my Dune 2 SHP file converter:

/* 
 * ======================
 * Dune 2 SHP file format
 * ======================
 * 
 * The Dune 2 SHP file is a multiple image filetype, where each image can have
 * it's own set of dimensions.  The file is structured as follows:
 * 
 * File header:
 *   NumImages (2 bytes)    - the number of images in the file
 *   Offsets[NumImages + 1] - offset to the image header for an image.  The last
 *     (2 or 4 bytes each)    offset points to the end of the file.  The offsets
 *                            don't take into account the NumImages bytes at the
 *                            beginning, so add 2 bytes to the offset value to
 *                            get the actual position of an image header in the
 *                            file
 * 
 * The size of the offsets can be either 2 or 4 bytes.  There is no simple way
 * to determine which it will be, but checking the file's 4th byte to see if
 * it's 0, seems to be a commonly accepted practice amongst existing Dune 2 SHP
 * file readers:
 * 
 * eg: A 2-byte offset file: 01 00 06 00 EC 00 45 0A ...
 *     A 4-byte offset file: 01 00 08 00 00 00 EC 00 ...
 *                                       ^^
 * The marked byte will be 0 in 4-byte offset files, non 0 in 2-byte offset
 * files.
 * Lastly, like C&C SHP files, there is an extra offset, pointing to the end of
 * the file (or what would have been the position of another image header/data
 * pair).
 * 
 * Following the file header, are a series of image header & image data pairs.
 * The image header is structured as follows:
 * 
 * Image header:
 *   Flags (2 bytes)    - flags to identify the type of data following the
 *                        Datasize field, and/or the compression schemes used
 *   Slices (1 byte)    - number of Format2 slices used to encode the image
 *                        data.  Often this is the same as the height of the
 *                        image
 *   Width (2 bytes)    - width of the image
 *   Height (1 byte)    - height of the image
 *   Filesize (2 bytes) - size of the image data in the file
 *   Datasize (2 bytes) - size of the image data when Format2 encoded/compressed
 * 
 * Regarding the flags, there seem to be 4 values:
 *  - 00000000 (0) = Decompress with Format80, then Format2
 *  - 00000001 (1) = (see 3)
 *  - 00000010 (2) = Decompress with Format2
 *  - 00000011 (3) = A small 16-byte lookup table follows, and the image data
 *                   should be decompressed with Format80 then Format2.
 *  - 00000101 (5) = The first byte gives the size of the lookup table that
 *                   follows, and the image data should be decompressed with
 *                   Format80 then Format2.
 * 
 * And after this image header is the image data.
 */

[DK]

I've noticed that the DOS WWPak extractor tool is in fault. It doesnt extract the files to their correct sizes - some are cut off in length. Seems to be a buffer over-run failure that leaves the buffer truncated. HFINALB.WSA is some 100Kb+ in size. There might be other differences depending on the PAK extractor used.

Well, I think I've made a correct extractor that work with all Westwood .pak files (do not know WWPak...). So, the file .wsa I've extracted is correct.

Once I've decoded the WSA structures and image formats I'll voice my findings.

yes, let me know.

There seems to be a number of different versions - Nyerguds has release a copy of my HitSquad version of Dune 2 that seems to be the last version released.

I know exist different "version" of Dune2, maybe that's the problem, but the file is the same in all the distributions.

But I'll check this one.

Regarding the .MAP file found in DUNE.PAK. Do you know what the first part of the file contains ? Seems to references to random tiles and some are out of range but there might be some header information I'm unaware of in the MAP file. Last part of the file seems to be references to construct the 1x1, 2x2, 3x2, 3x3 structures and their "frames" (flag animations, landing pad blinks, radar turning). I havnt researched that particular file that closely yet - there might be something I've missed.

aaahhhh, long time passed, unfortunately I've saved only the decoding part of ICN, I still haven't see the MAP specs. And I cannot find it anymore :( - all I have is a todo "Reconstruct Dune2 ICN using .MAP" but cannot find .MAP specs, sorry.

If is still active, you can try to contact Christian Morgner, maybe he still have or remember the specs. I've found only a picture of the decoding result Christian send to me. I'll attach for reference, maybe can be useful.

Figured out that the TBL file is a palette cross-reference for grey-scaling used in the Starport ordering screen.

interesting!

In conjunction with the document mentioned above, I added my findings about the format, and wrote something about it in the source code for my Dune 2 SHP file converter:

Great work!

Here's my addition/comments to this strange SHP "variable" :) format:

I've found some strange offset in SHP. Some Dune2 shp doesn't have a terminator and the offset is 2byte too long.

So, I've made this modification that seems to work:

if (offset == 6) offset = 4;

Lastly, like C&C SHP files, there is an extra offset, pointing to the end of the file (or what would have been the position of another image header/data pair).

That not always pointing to exact end of the file, sometime is FileSize - 2 (see DUNE.PAK -> MENSHPH.SHP). I do not know why.

* Image header:

*  Flags (2 bytes)    - flags to identify the type of data following the

*                        Datasize field, and/or the compression schemes used

*  Slices (1 byte)    - number of Format2 slices used to encode the image

*                        data.  Often this is the same as the height of the

*                        image

....

Usually, absolute offset point 2 bytes before this header. And those 2 bytes indicate last image end or final data from previous image. That's not important for Dune2 but for other SHP version can be useful to loop animation.

[Nyerguds]

Ehh... why did you put that on a map? :S

[DK]

Ehh... why did you put that on a map? :S

I don't know, it's Christian work, maybe... with the map as background, it's more inspiring :D

[ArrakisResearchCo]

The ICON.MAP file seems to contain some 743 Word entries (double bytes). Correctly encoded they each contain (for the most part) a tile number from 0 to 398. Starting at index 238 is the "House Palace" followed by the rest of the structures. Here's how to read each structure - they are stored in this order :

Index - size - frames - name

228  -  3x3  -  4  -  House Palace (36 entries total)

274  -  2x2  -  6  -  Light Factory (24 entries total)

298  -  3x2  -  8  -  Heavy Factory (48 entries total)

346  -  3x2  -  8  -  Hi-Tech Factory (48 entries total)

394  -  2x2  -  4  -  IX Research (16 entries total)

410  -  2x2  -  4  -  WOR Trooper (16 entries total)

426  -  2x2  -  4  -  Construction Yard (16 entries total)

442  -  2x2  -  4  -  Barracks (16 entries total)

458  -  2x2  -  4  -  Windtrap (16 entries total - animation is a palette rotation - palette index 223 is used)

474  -  3x3  -  10  -  Starport (90 entries total)

564  -  3x2  -  10  -  Refinery (60 entries total)

624  -  3x2  -  10  -  Repair (60 entries total)

684  -  1x1  -  10  -  Turret (10 entries total)

694  -  1x1  -  10  -  Rocket Turret (10 entries total)

704  -  2x2  -  4  -  Spice Silo (16 entries total)

720  -  2x2  -  6  -  Radar (24 entries total)

Structures are encoded top-left to bottom-right - each frame is just an offset of size.

[Nyerguds]

Hm... so the walls and concrete aren't mentioned in it?

[ArrakisResearchCo]

Well ... when I'm browsing the tiles directly from the ICON.MAP files - here's what I get :

Index 1 to 2: track marks (tile 27 and 28).

Index 3 to 7: misc. wall tiles (34, 40, 46, 52, 60)

Index 8: tile 135 - a sand/rock tile.

Index 9 & 10: tiles 151 and 154 - dune tiles.

Index 11 & 12: tiles 235 and 237 - bottom left & bottom right parts of the House Palace.

Index 13: tile 273 - top-middle of the Hi-Tech factory; middle part of the open-dome animation.

Index 14: tile 297 - part of the construction yard being built animation.

Index 15: tile 345 - bottom part of the landing pad for the Refinery - idle status.

Index 16: tile 393 - Outpost - radar pointing west.

Index 17 to 27: out of range values.

Index 28: tile 0 - cracked concrete slab.

Index 29 to 34: tiles 1, 2, 3, 4, 5, 6 - rock craters with "transparent" background (palette index 0 is transparent)

Index 35 to 40: tiles 7, 8, 9, 10, 11, 12 - sand craters width "transparent" background (palette index 0 is transparent)

Index 41 & 42: crashed Thopter/Carryall (tile: 13 & 14)

Index 43: sand crater (tile 15).

Index 44 & 45: another crashed Thopter/Carryall (tile 16 & 17)

Index 46: sand crater (tile 18)

Index 47 to 50: corpses (tiles: 19, 20, 21, 22) - first "set" is one soldier followed by a sand shadow - second set is two soldiers followed by a sand shadow.

Index 51 & 52: blood (tiles 23 & 24).

Index 53 to 60: track marks with "transparent" background (palette index 0 is transparent) - all directions (tiles: 25 to 32).

Index 61: gravel with "transparent" background (palette index 0 is transparent).

Index 62 to 135: Wall tiles - first set of tiles (index 62-73) are full wall tiles - the rest are damaged wall tiles and destroyed wall parts.

Index 136 to 151: "Fog-of-War" tiles.

Index 152 to 154: Following the FoW tiles are 3 "frames" of 1x1 concrete tiles: first (152) is a cracked tile, second (153) is a rock tile with a crater in it, third (154) is a normal concrete slab.

Index 155 to 235:

A set of sand / rock / mountain / dunes / spice tiles (index 155 to 235) in (I'm guessing) the same order as the FoW tiles.

Sorry about my typo in my last post regarding the House Palace - it's index 238 - not 228.

I havnt yet figured out what index 17 to 27 contain but I havnt really looked into these indexes to figure them out.

I wrote a "Tile Viewer" that can display all the 399 tiles from the game. I loaded the ICON.MAP file into a Word (2-byte) array and simply just set my Tile Viewer to display tiles found in these Words. Some seem to be out of range - some seem to be out of range but I'm guessing that further analysis will cast some light on a header that might contain direct references to the tile combinations I've found by just playing around with the indexes and frame offsets. I'll throw ICON.MAP into a hex viewer and analyse it further in the coming days - hope to find some sort of system in the .MAP file. I'll post a document on my site with the complete description of ICON.MAP when I've decoded the first part of the file.

[MrFlibble]

I know exist different "version" of Dune2, maybe that's the problem, but the file is the same in all the distributions.

The initial release of Dune II (1.0) uses different SHP and WSA formats than the subsequent 1.07 releases (both EU and US), but I cannot tell you the difference. The WSA files from v1.0 are not recognized by XCC Mixer.

[ArrakisResearchCo]

If I could get my hands on a copy of the different versions including the 1.0 I could take a look at them when I finish my SHP/WSA viewer. I might be able to figure out how to handle them.

[MrFlibble]

Super Dune 2 is based on the version 1.0. You can get it here.

[ArrakisResearchCo]

Great - thanks - I'll look into it as soon as I can :)

[MrFlibble]

I thought I'd bring this matter up. Is it possible to fix some of the building frames? E.g. the IX Research Center structure has some areas that get remapped while they shouldn't be (seen with all sides other than Harkonnen):

Also one of the frames for the Gun Turret has a part of it cut (I'll get a screenshot later).

[ArrakisResearchCo]

In my experiments with the tilemap for both maps and structures I've noticed the problem with the IX tiles. All the raw extracts I've made of these tiles all have the same pixel / colortable values in all of the IX tiles and they all have wrong values for certain parts of the structure. I wrote a tile viewer that started out getting the raw pixel values which showed me the problem with the IX tiles. Since these tiles are roughly the same regardless of version it's possible to remap the faulty values when extracting the tiles. I'll take a look at my sourcecode and see if I can setup a remap table for these specific pixels in these specific tiles.

[Nyerguds]

Wait, remap isn't just determined by colour indexes?

[ArrakisResearchCo]

Ok - have a look at this screenshot :

http://www.dune2.dk/tileremap1.jpg

The screenshot is made up of four sections - should be pretty easy figuring out what is what. The top one of the two tilemap images is rendered by my program - the bottom one was loaded from a bitmap I downloaded somewhere. The zoomed copy of one of the IX tiles shows the pixel colors and their respective indexes. The program reads the tiles from ICON.ICN.

As you can see I have the same "problem" as the author of the person who made the original bitmap shown at the bottom. When reading the ICON.ICN file each tile has all the flags mapped as Fremen colors - these are then replaced by each house color.

I commented out the replace part for the IX tiles to allow viewing without my remapping of the IX tiles. What I did was this (pseudo-code) :

If (TileIndex >= 286) And (TileIndex <= 290) Then

 Replace ColorIndex 200 with ColorIndex 152

 Replace ColorIndex 201 with ColorIndex 153

 Replace ColorIndex 202 with ColorIndex 154

End If

The end result is all IX tiles shown with correct colors. This program currently only supports the ICON.ICN files with size 54231 bytes - sorry - can't remember which version this was for but I'm planning a rewrite of the tile extractor / viewer that will support all the versions I currently have.

I'm not entirely sure but the wrong colors shown on the IX tiles might be a math error in the decompression somewhere however if it's only a math error more tiles should be faulty but they are not.

Someone described the FORM format for me a while back (sorry - my bad memory can't tell who it was) and I will change my tile viewer to load the ICON.ICN file using the FORM format instead of hardwired reads from the specific file used when I wrote the original program.

Hope this helps a bit :)

[edit: may 4th '10] Changed the URL to the correct new site

[MrFlibble]

OK, as I've understood, it is possible to fix the screwed IX remap, which is good :)

This program currently only supports the ICON.ICN files with size 54231 bytes - sorry - can't remember which version this was

That's the file from Dune II v1.0, which has those extra frames for Heavy factory, High-Tech factory and Repair facility which were removed, for reasons unknown, from v1.07.

[ArrakisResearchCo]

I've since checked the file sizes vs. version codes and can only confirm MrFlibble's findings :)

You have understood it correctly - it is possible by very simple programming to fix the House of IX pixels. I'll be moving all my material from www.junkyard.dk to www.dune2.dk very soon and once everything is moved I'll go through all my notes and start reposting all the information I've gathered sofar about the different file formats and structurs - and I'll include some notes about the IX tiles and how / what to do to fix the "wrong" pixels.

(This past summer has been a real life killer but I'll start posting stuff again on a regular basis - sorry if I've missed questions / comments / notes here and there)

[Nyerguds]

it's really odd that ingame it remaps correctly for Harkonnen but does it wrong for the other sides though...

[ArrakisResearchCo]

Yeah - it's very odd but then again it could be a bug in the game that Westwood never got around to fixing. It's great that we'r able to fix it now for our own purposes tho :)

[MrFlibble]

Hm, it looks like the truncated frame of the Gun Turret was fixed in v1.07 (judging by this image). It's the second frame of this structure, the one with the gun barrel pointing in the north-eastern direction. The left side of the turret base in that frame is missing (you can see this on your tile remap image, albeit in JPEGged quality).

[ArrakisResearchCo]

I've loaded the v1.0 ICON.ICN into my tileviewer again and can confirm MrFlibble's comment. Seems that some of the shadow pixels and bright pixels are missing or miscalculated in v1.0. I havnt had the time to mess with my tile viewer - I still need to fix it so it can read both versions but they seem fixed in v1.07.

[MrFlibble]

That's good news. A fixed version of ICON.ICN could be very handy for my Dune II patch.

BTW, if the v1.07 ICON.ICN has some other fixes not present in v1.0, maybe it would be wiser to add the missing animation frames to it, rather than fix more stuff in v1.0?

Another question to ArrakisResearchCo, are you planning to make an .ICN converter tool that would support widely known graphical formats like BMP or PNG?

[ArrakisResearchCo]

I never really thought about creating a two-way tool for the graphics in Dune 2 but given your "drive" in your project MrFlibble I'll give it a shot. Most of my tools currently support export to 8-bit bitmaps (*.bmp) and they usually apply IBM.PAL as default palette but a palette changer should also be easy to put in.

I'll have to take another look at the ICON.MAP file for each of the two versions. This file contains the actual tile references for all the structure animations - there might be differences there too.

Most of my experiment tools are written in VB6 so they may not be the fastest tools around but they get the job done (note that some of them may not work under Vista). Sourcecodes will be available in the future.

My inet uplink (aswell as my site) will be down the next couple of days so I'll re-check my notes concerning the ICON.* files and see if I can come up with a compressor to reverse my decompressors work while I'm offline :)

[MrFlibble]

I never really thought about creating a two-way tool for the graphics in Dune 2 but given your "drive" in your project MrFlibble I'll give it a shot.

Thanks a lot! I really appreciate your efforts concerning Dune 2, and it makes me happy that there are still people who wish both to understand and to improve this game, even despite its age (as a side note, I do hope that in future, there will be much less people who'd dismiss a game simply because "it's old" or "it's not 3D").

[ArrakisResearchCo]

Well - most new games all have the same flaws: gigantic budgets, stunning graphics but no real game play.  Just because it's 3D and has stunning graphics doesnt mean it's a great game. It's true that Westwoods later games in the C&C series has a lot of improvements to the user interface but they are still basically the same game in different clothes. Dune 2 is perfectly ballanced where the C&C series is more "power before game play".

I still go back to the good old days and play Dune 2 when all the new games either crash or become boring. Kills the fun a bit tho when the last scenario has been played ;)

Introversion (www.introversion.co.uk) has release a few games that keep the old 2D form for the most part - great games because they keep focus in the right place: game play before stunning graphics.