Format80 decompression

[401Kill]

Hi,

I'm trying to understand the Format80 compression method used in Dune II, C&C etc.

I have been working on converting Vladan Bato's Format80 code sample (http://www.geocities.com/SiliconValley/8682/ccfiles4.txt) to VB6, as the syntax is very similar.

The images produced (see attached for an example) are nearly correct, however something isn't quite working properly. Any ideas?

Thanks,

Ben

 

' Custom functions -' GetBit (byte, bit number)' GetBits (byte, start bit number, end bit number)' String index  1 2 3 4 5 6 7 8' Bit           7 6 5 4 3 2 1 0Function Format80toRAW(Format80_Image_Data As String) As String   Dim Source() As String, Dest() As String, OutBuffer As String    ' Data arrays and working buffer   Dim SP As Long, DP As Long    ' File pointers (0 based)      Dim Com As Byte, Colour As Byte   Dim B7 As Integer, B6 As Integer   Dim Count As Long, Pos As Long, i As Long   Source = Split(ToHexStr(Format80_Image_Data), " ")   ReDim Preserve Source(UBound(Source) - 1) ' remove blank entry from end of array      ' zero pointers   SP = 0   DP = 0          Do Until False       Com = CByte("&H" & Source(SP)) ' source data in hex, convert to byte       SP = SP + 1       B7 = getBit(Com, 7)       Select Case B7       Case 0    ' Copy command (2)           ' Bits 4-6 + 3           Count = BinaryToLong(getBits(Com, 4, 6))           Count = Count + 3           ' Bits 0-3 + next byte           Pos = BinaryToLong(getBits(Com, 0, 3))           Pos = Pos + CLng("&H" & Source(SP))           Pos = (DP - Pos)    ' Start pos = current pos - calculated value           SP = SP + 1           For i = Pos To Pos + Count - 1               ReDim Preserve Dest(DP)               Dest(DP) = Dest(i)               DP = DP + 1               DoEvents           Next       Case 1           ' check bit 6 of Com           B6 = getBit(Com, 6)           Select Case B6           Case 0    ' Copy As-Is command (1)               Count = Com And &H3F   ' mask 2 topmost bits               If Count = 0 Then Exit Do    ' EOF marker               For i = 1 To Count                   ReDim Preserve Dest(DP)                   Dest(DP) = Chr(CInt("&H" & Source(SP)))                   DP = DP + 1                   SP = SP + 1                   DoEvents               Next           Case 1    ' Large Copy, Very Large Copy, and Fill commands               Count = Com And CLng(&H3F)                              If Count < &H3E Then    ' Large Copy (3)                   Count = Count + 3 ' Count = bits 0-5 of com +3                                      ' Next WORD = position from start of image                   Pos = CLng("&H" & Source(SP + 1) & Source(SP))                   SP = SP + 2                                      For i = Pos To Pos + Count - 1                       ReDim Preserve Dest(DP)                       Dest(DP) = Dest(i)                       DP = DP + 1                       DoEvents                   Next               ElseIf Count = &H3F Then    'Very Large Copy (5)                   Count = CLng("&H" & Source(SP + 1) & Source(SP))                   Pos = CLng("&H" & Source(SP + 3) & Source(SP + 2))                   SP = SP + 4                   For i = Pos To Pos + Count - 1                       ReDim Preserve Dest(DP)                       Dest(DP) = Dest(i)                       DP = DP + 1                       DoEvents                   Next               ElseIf Count = &H3E Then    ' Fill (4)                   Count = CLng("&H" & Source(SP + 1) & Source(SP))                   Colour = CInt("&H" & Source(SP + 2))                   SP = SP + 3                   For i = 1 To Count                       ReDim Preserve Dest(DP)                       Dest(DP) = Chr(Colour)                       DP = DP + 1                       DoEvents                   Next               End If           End Select       End Select       DoEvents   Loop   OutBuffer = Join(Dest, vbNullString)   Format80toRAW = OutBufferEnd Function

[MrFlibble]

Sadly I myself probably can't help you, as my programming skills are minimal, yet I think either Olaf van der Spek, the creator of XCC Utilities, or Ultraq (the forum about his Red Horizon tools is here) may help you. Also, visit the XCC Forum and the Project Perfect Mod forums and website, where you'll probably find people who'll help.

[Nyerguds]

The XCC code is all public on sourceforge anyway...

[401Kill]

Thanks guys.

Will check those links... I did find the XCC source, but I'm not competent enough in C to make much of it.

I think its going to be one of those horribly annoying, trivial bugs...

[ArrakisResearchCo]

First off - you'll have to multiply the RGB values of the palette by 4 since they are 64 based and not 256 based - VB6 / Win uses 256 setting based RGB values. I'm unsure which image you are attempting to decompress so I can't tell if it's correct or not - I can just see that the palette values are wrong. AFAIK the C&C palettes are also 64 based.

Once I finished my *.FNT extractor today I started working on SHP files. They are supposed to use Format80 compression and I'm currently in the process of designing a VB6 Format80 decompression function. I might be able to help you later on - once I've cracked the nutttttts so to speak :)

Arrakis Research Company's new site is at www.dune2.dk

[edit: may 4th '10] Changed the URL to the correct new site

[401Kill]

First off - you'll have to multiply the RGB values of the palette by 4 since they are 64 based and not 256 based - VB6 / Win uses 256 setting based RGB values. I'm unsure which image you are attempting to decompress so I can't tell if it's correct or not - I can just see that the palette values are wrong. AFAIK the C&C palettes are also 64 based.

Once I finished my *.FNT extractor today I started working on SHP files. They are supposed to use Format80 compression and I'm currently in the process of designing a VB6 Format80 decompression function. I might be able to help you later on - once I've cracked the nutttttts so to speak :)

Arrakis Research Company's new site is at www.junkyard.dk

The image I uploaded was 'setup.cps' from the original C&C, however all CPS images I decompress with my function have much the same effect.

I have implemented RGB multiplication as suggested, this has improved the look significantly. Thanks for that info.

Attached are a couple of shots from the new function. Many of the SHP images appear to work correctly, which leads me to believe that it is one of the larger CPS "commands" which has a bug.

[ArrakisResearchCo]

Well - sure looks like you'r heading in the right direction with your code. Since VB6 doesnt have SHR and SHL built in we have to create similar functions ourselves. Bit shifting has to accurate for the result to be correct. Before Window$ came along I did most of my programming in Borland's Turbo Pascal 6 for DOS. Vladan Bato has done a lot of research into these files so I hope I can successfully convert his Pascal code into a working VB6 application. Since VB6 doesnt have SHR and SHL built in I had to create these functions myself. If I succeed I'd be happy to share the source with you and anyone else who wants it :)

[401Kill]

Well - sure looks like you'r heading in the right direction with your code.

Thats good to know  ;D

Since VB6 doesnt have SHR and SHL built in we have to create similar functions ourselves. Bit shifting has to accurate for the result to be correct.

I may have to implement these... I've just been using mid() and a function to convert numbers to binary...

Before Window$ came along I did most of my programming in Borland's Turbo Pascal 6 for DOS. Vladan Bato has done a lot of research into these files so I hope I can successfully convert his Pascal code into a working VB6 application. Since VB6 doesnt have SHR and SHL built in I had to create these functions myself. If I succeed I'd be happy to share the source with you and anyone else who wants it :)

We share a mission then... good luck and thanks for the offer!

[ArrakisResearchCo]

Keeping track of pointers and their relative positions is mind bogling ...

Status of Format80 decompression function: first attempt successful - but still have a few bugs to clear out.

I've used "IBM.PAL" (found in DUNE.PAK) and "CREDIT1.SHP" (found in FINALE.PAK) from the Dune 2 files. Seems there are a few differences between multi image SHP's and single ones - I'm going bug-hunting :P

[Edit]: Seems I was a bit too quick. Turns out I still a few bugs to clear out. Update will follow.

[ArrakisResearchCo]

Bugs seem to be squished for now.

I've finally solved the problems I had with the Format80 decompression module for VB6. Please note that this module will only decompress Format80 images. In the Dune 2 files you'll find 15 CREDIT??.SHP files - they only contain 1 image each - all of them are 182 x 110 pixels. My module will decompress all 15 shapes. It will not decompress any other .SHP files found in Dune 2 - they appear to be Format40 compressed. I havnt built a Format40 decompressor yet.

Get the Format80 decompression module here :

http://www.dune2.dk/_archives/vbsources/Format80Module.zip  (3 Kb)

A note and download link will also be posted at my site at www.dune2.dk

Enjoy :)

[edit: may 4th '10] Changed the URL to the correct new site

[Nyerguds]

About your site... there's a Dune II font viewer there. Any chance it can be turned into a font editor? :P

Looking at the interface, it already has pretty much everything you'd need for it.

[ArrakisResearchCo]

When I've finished finished my Format40 decompressor I'll take a look at a font editor.

[Nyerguds]

cool... if you do make that please add C&C font support ;D

[ArrakisResearchCo]

I'll give it a try when the time comes :)

I might need palettes / font files for trials and comparison.

[Nyerguds]

heh, no problem there. I don't think there's a single C&C1-related file I don't have :P

[ArrakisResearchCo]

Great - I'll drop you a note when I'm ready to begin trials. I'm currently focusing on getting a usable site up - I might change domain in the near future - gain better bandwidth :)

[Ultraq]

Seems I might be a bit late in helping with Format80, but I thought I'd throw this in here in-case somebody finds it useful.

MrFlibble mentioned my name somewhere in this post as I've been writing Dune 2 conversion utilities for him, and in the process have ended-up writing an encoder/decoder for the Format2/40/80 compression formats used by Dune 2 and C&C.  The thread for these tools is here: http://cnc-tech.planetcnc.gamespy.com/index.php?showtopic=3699

The source files are available with the download, but the thing is it's written in Java.  If you can read Java then you're probably most interested in the encoding/decoding part which is all done in 1 utility class in my program.

I've uploaded a copy of that class with the Format2/40/80 encoding/decoding methods here: http://ultraq.orconhosting.net.nz/CNCCodec.java

[ArrakisResearchCo]

Better late than never ;)

Thanks for sharing the info Ultraq - I'll take a look at it.

[401Kill]

Been away for a while, just come back to the project and I've just got format80 decompression working perfectly - at least for all the .cps files. Turned out to be my SHL implementation was dodgy... thanks for mentioning them, it made me actually research what they were supposed to be doing. :P

My success at decoding SHP files varies... some images are perfect and some are incomprehensible. My routine currently ignores format40 shp entries (mostly - I think it might be trying a few and giving these weird results).

I haven't worked out logic for dealing with the malformed credit##.shp files (the ones which don't indicate end of header properly) yet, does your code deal with these or just ignore the headers?

I'm going to overhaul my shp handling routine now, and then on to Format40 decompression next... joy  ;D

Just got hold of a new web server, so I'll maybe get a project page going soon...

[ArrakisResearchCo]

Both CPS and SHP files have headers though they differ slightly depending which file type it is. Headers include a reference or control byte that dictates / describes which format the image is compressed in. Vlato Bato's documentation on C&C files describes the headers aswell as the three different compression formats where Format80 is one of them. Most of the headers in the UNITS*.SHP files seem to point to Format40 compression if compared to the CREDIT*.SHP files. The WSA files seem to include all three types of compression. The headers are very important in each of the files so check up on the documentation for the headers of the SHP / CPS / WSA files and check the pointers that inform on compression/format types in each of the three header types.

All three file types contain Format80 / Format40 / Format20 compressed images - headers describe which format each image is compressed in. If you check the header in the CREDIT*.SHP files you'll see that they are all of CompressionType=1 - which is Format80. When I looked at the UNIT*.SHP files the CompressionType was = 0 for most of the images I checked. When I looked at one of the images from the UNIT*.SHP files it's first RLE byte was 00h which seems to indicate a Format40 compression - so it seems that CompressionType=0 in SHP files indicates Format40 compression and CompressionType=1 in SHP files seems to indicate Format80 compression.

Headers in files are always important. They instruct your program how to read / use the rest of the file.

I've been busy at work lately but my plan is to implement a Format40 decompressor too.

[DK]

Greetings,

I'm not new in this forum... but I've posted many time ago, so I forgot the nick... (or, at time, registration isn't necessary...)

I've made a lot of research about Dune2 formats, and I think I've decoded almost all.

They are a first version of old Westwood formats, before C&C, so the docs of Bato aren't always correct.

The CPS, format80 and format40 are the same, but for SHP and WSA there are some differences.

And some SHP is different from others, too!

You can check this document for the SHP, it isn't 100% correct, but it's a start ;)

If you need some advice, let me know; also, the Java tools posted above are very good (if you understand Java).

[ArrakisResearchCo]

Thanks for the hint DK. I've already got a copy of that document and working out my handling of files from that and a bunch of others I've found. Our friend my. Bato has left a lot of hints and fairly good descriptions of principles of handling the data for binary hackers to follow - most programmers from the good old DOS days should be able to decipher his hints.

There's a Wiki describing file formats for a number of EoB and LoL file formats virtually identical to those of Dune 2 and C&C. Most of the info only requires a small amount of tweaking to become correct. A good hex-viewer and the collected documents describing the file formats and headers gets you pretty far. Just remember the old way of doing things: bit-shifting, AND/OR handling, nibbles (4-bits), file-offset - all kept inside the 64Kb data segment allowed back then.

I still have a few things to work out regarding SHP / WSA compression formats but I'm getting close to a final solution. When I have finished the last decompression functions I'll start writing my own documentation for the formats. I'll propably use VB6 syntax to describe things in code since most programmers are able to read and understand this "simple" language.

[DK]

Thanks for the hint DK. I've already got a copy of that document and working out my handling of files from that and a bunch of others I've found. Our friend my. Bato has left a lot of hints and fairly good descriptions of principles of handling the data for binary hackers to follow - most programmers from the good old DOS days should be able to decipher his hints.

yep, you're right, however I spent lot of time on some format :)

Ah, and with help of Christian Morgner (on this forum...) we've decoded also ICN and MAP files.

There's a Wiki describing file formats for a number of EoB and LoL file formats virtually identical to those of Dune 2 and C&C. Most of the info only requires a small amount of tweaking to become correct. A good hex-viewer and the collected documents describing the file formats and headers gets you pretty far. Just remember the old way of doing things: bit-shifting, AND/OR handling, nibbles (4-bits), file-offset - all kept inside the 64Kb data segment allowed back then.

eh eh eh, are you referring to EoB wiki? I'm one of the contributors :)

And the wallset format of EoB is crazy!

I still have a few things to work out regarding SHP / WSA compression formats but I'm getting close to a final solution. When I have finished the last decompression functions I'll start writing my own documentation for the formats. I'll propably use VB6 syntax to describe things in code since most programmers are able to read and understand this "simple" language.

Should be interesting and useful making also a wiki for Dune2 formats. Let me know if you're interested.

Ah, I've still some WSA do not decode correctly (like final.pak -> HFinalC.wsa), let me know if you can figure out where is the problem; I think it must be merged with previous wsa to play correctly and not alone!

[ArrakisResearchCo]

Cool - I didnt know about your contributions to the EoB wiki :)

I've noticed that the DOS WWPak extractor tool is in fault. It doesnt extract the files to their correct sizes - some are cut off in length. Seems to be a buffer over-run failure that leaves the buffer truncated. HFINALB.WSA is some 100Kb+ in size. There might be other differences depending on the PAK extractor used. Once I've decoded the WSA structures and image formats I'll voice my findings. There seems to be a number of different versions - Nyerguds has release a copy of my HitSquad version of Dune 2 that seems to be the last version released. The version I got from abandonia has a large portion of seemingly random letters in one of the scenario.ini files - there might be similar "errors" in other files in the different versions. There's a link to the HitSquad version in the last post in the thread "Dune II editor with 1.07 support".

Regarding the .MAP file found in DUNE.PAK. Do you know what the first part of the file contains ? Seems to references to random tiles and some are out of range but there might be some header information I'm unaware of in the MAP file. Last part of the file seems to be references to construct the 1x1, 2x2, 3x2, 3x3 structures and their "frames" (flag animations, landing pad blinks, radar turning). I havnt researched that particular file that closely yet - there might be something I've missed.

Figured out that the TBL file is a palette cross-reference for grey-scaling used in the Starport ordering screen.

[Nyerguds]

My editor can change the foundation of structures... it totally messesup the graphics. not only that but it also shows that the entire building has multiple frames, and not just the one tile that actually has the flag on it.