401Kill Posted February 25, 2008 Share Posted February 25, 2008 Hi,I'm trying to understand the Format80 compression method used in Dune II, C&C etc.I have been working on converting Vladan Bato's Format80 code sample (http://www.geocities.com/SiliconValley/8682/ccfiles4.txt) to VB6, as the syntax is very similar.The images produced (see attached for an example) are nearly correct, however something isn't quite working properly. Any ideas?Thanks,Ben ' Custom functions -' GetBit (byte, bit number)' GetBits (byte, start bit number, end bit number)' String index 1 2 3 4 5 6 7 8' Bit 7 6 5 4 3 2 1 0Function Format80toRAW(Format80_Image_Data As String) As String Dim Source() As String, Dest() As String, OutBuffer As String ' Data arrays and working buffer Dim SP As Long, DP As Long ' File pointers (0 based) Dim Com As Byte, Colour As Byte Dim B7 As Integer, B6 As Integer Dim Count As Long, Pos As Long, i As Long Source = Split(ToHexStr(Format80_Image_Data), " ") ReDim Preserve Source(UBound(Source) - 1) ' remove blank entry from end of array ' zero pointers SP = 0 DP = 0 Do Until False Com = CByte("&H" & Source(SP)) ' source data in hex, convert to byte SP = SP + 1 B7 = getBit(Com, 7) Select Case B7 Case 0 ' Copy command (2) ' Bits 4-6 + 3 Count = BinaryToLong(getBits(Com, 4, 6)) Count = Count + 3 ' Bits 0-3 + next byte Pos = BinaryToLong(getBits(Com, 0, 3)) Pos = Pos + CLng("&H" & Source(SP)) Pos = (DP - Pos) ' Start pos = current pos - calculated value SP = SP + 1 For i = Pos To Pos + Count - 1 ReDim Preserve Dest(DP) Dest(DP) = Dest(i) DP = DP + 1 DoEvents Next Case 1 ' check bit 6 of Com B6 = getBit(Com, 6) Select Case B6 Case 0 ' Copy As-Is command (1) Count = Com And &H3F ' mask 2 topmost bits If Count = 0 Then Exit Do ' EOF marker For i = 1 To Count ReDim Preserve Dest(DP) Dest(DP) = Chr(CInt("&H" & Source(SP))) DP = DP + 1 SP = SP + 1 DoEvents Next Case 1 ' Large Copy, Very Large Copy, and Fill commands Count = Com And CLng(&H3F) If Count < &H3E Then ' Large Copy (3) Count = Count + 3 ' Count = bits 0-5 of com +3 ' Next WORD = position from start of image Pos = CLng("&H" & Source(SP + 1) & Source(SP)) SP = SP + 2 For i = Pos To Pos + Count - 1 ReDim Preserve Dest(DP) Dest(DP) = Dest(i) DP = DP + 1 DoEvents Next ElseIf Count = &H3F Then 'Very Large Copy (5) Count = CLng("&H" & Source(SP + 1) & Source(SP)) Pos = CLng("&H" & Source(SP + 3) & Source(SP + 2)) SP = SP + 4 For i = Pos To Pos + Count - 1 ReDim Preserve Dest(DP) Dest(DP) = Dest(i) DP = DP + 1 DoEvents Next ElseIf Count = &H3E Then ' Fill (4) Count = CLng("&H" & Source(SP + 1) & Source(SP)) Colour = CInt("&H" & Source(SP + 2)) SP = SP + 3 For i = 1 To Count ReDim Preserve Dest(DP) Dest(DP) = Chr(Colour) DP = DP + 1 DoEvents Next End If End Select End Select DoEvents Loop OutBuffer = Join(Dest, vbNullString) Format80toRAW = OutBufferEnd Function Quote Link to comment Share on other sites More sharing options...
MrFlibble Posted February 25, 2008 Share Posted February 25, 2008 Sadly I myself probably can't help you, as my programming skills are minimal, yet I think either Olaf van der Spek, the creator of XCC Utilities, or Ultraq (the forum about his Red Horizon tools is here) may help you. Also, visit the XCC Forum and the Project Perfect Mod forums and website, where you'll probably find people who'll help. Quote Link to comment Share on other sites More sharing options...
Nyerguds Posted February 25, 2008 Share Posted February 25, 2008 The XCC code is all public on sourceforge anyway... Quote Link to comment Share on other sites More sharing options...
401Kill Posted March 2, 2008 Author Share Posted March 2, 2008 Thanks guys.Will check those links... I did find the XCC source, but I'm not competent enough in C to make much of it.I think its going to be one of those horribly annoying, trivial bugs... Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 29, 2008 Share Posted March 29, 2008 First off - you'll have to multiply the RGB values of the palette by 4 since they are 64 based and not 256 based - VB6 / Win uses 256 setting based RGB values. I'm unsure which image you are attempting to decompress so I can't tell if it's correct or not - I can just see that the palette values are wrong. AFAIK the C&C palettes are also 64 based.Once I finished my *.FNT extractor today I started working on SHP files. They are supposed to use Format80 compression and I'm currently in the process of designing a VB6 Format80 decompression function. I might be able to help you later on - once I've cracked the nutttttts so to speak :)Arrakis Research Company's new site is at www.dune2.dk[edit: may 4th '10] Changed the URL to the correct new site Quote Link to comment Share on other sites More sharing options...
401Kill Posted March 29, 2008 Author Share Posted March 29, 2008 First off - you'll have to multiply the RGB values of the palette by 4 since they are 64 based and not 256 based - VB6 / Win uses 256 setting based RGB values. I'm unsure which image you are attempting to decompress so I can't tell if it's correct or not - I can just see that the palette values are wrong. AFAIK the C&C palettes are also 64 based.Once I finished my *.FNT extractor today I started working on SHP files. They are supposed to use Format80 compression and I'm currently in the process of designing a VB6 Format80 decompression function. I might be able to help you later on - once I've cracked the nutttttts so to speak :)Arrakis Research Company's new site is at www.junkyard.dkThe image I uploaded was 'setup.cps' from the original C&C, however all CPS images I decompress with my function have much the same effect.I have implemented RGB multiplication as suggested, this has improved the look significantly. Thanks for that info.Attached are a couple of shots from the new function. Many of the SHP images appear to work correctly, which leads me to believe that it is one of the larger CPS "commands" which has a bug. Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 29, 2008 Share Posted March 29, 2008 Well - sure looks like you'r heading in the right direction with your code. Since VB6 doesnt have SHR and SHL built in we have to create similar functions ourselves. Bit shifting has to accurate for the result to be correct. Before Window$ came along I did most of my programming in Borland's Turbo Pascal 6 for DOS. Vladan Bato has done a lot of research into these files so I hope I can successfully convert his Pascal code into a working VB6 application. Since VB6 doesnt have SHR and SHL built in I had to create these functions myself. If I succeed I'd be happy to share the source with you and anyone else who wants it :) Quote Link to comment Share on other sites More sharing options...
401Kill Posted March 29, 2008 Author Share Posted March 29, 2008 Well - sure looks like you'r heading in the right direction with your code.Thats good to know ;D Since VB6 doesnt have SHR and SHL built in we have to create similar functions ourselves. Bit shifting has to accurate for the result to be correct.I may have to implement these... I've just been using mid() and a function to convert numbers to binary...Before Window$ came along I did most of my programming in Borland's Turbo Pascal 6 for DOS. Vladan Bato has done a lot of research into these files so I hope I can successfully convert his Pascal code into a working VB6 application. Since VB6 doesnt have SHR and SHL built in I had to create these functions myself. If I succeed I'd be happy to share the source with you and anyone else who wants it :)We share a mission then... good luck and thanks for the offer! Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 29, 2008 Share Posted March 29, 2008 Keeping track of pointers and their relative positions is mind bogling ...Status of Format80 decompression function: first attempt successful - but still have a few bugs to clear out.I've used "IBM.PAL" (found in DUNE.PAK) and "CREDIT1.SHP" (found in FINALE.PAK) from the Dune 2 files. Seems there are a few differences between multi image SHP's and single ones - I'm going bug-hunting :P[Edit]: Seems I was a bit too quick. Turns out I still a few bugs to clear out. Update will follow. Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 30, 2008 Share Posted March 30, 2008 Bugs seem to be squished for now.I've finally solved the problems I had with the Format80 decompression module for VB6. Please note that this module will only decompress Format80 images. In the Dune 2 files you'll find 15 CREDIT??.SHP files - they only contain 1 image each - all of them are 182 x 110 pixels. My module will decompress all 15 shapes. It will not decompress any other .SHP files found in Dune 2 - they appear to be Format40 compressed. I havnt built a Format40 decompressor yet.Get the Format80 decompression module here :http://www.dune2.dk/_archives/vbsources/Format80Module.zip (3 Kb)A note and download link will also be posted at my site at www.dune2.dkEnjoy :)[edit: may 4th '10] Changed the URL to the correct new site Quote Link to comment Share on other sites More sharing options...
Nyerguds Posted March 30, 2008 Share Posted March 30, 2008 About your site... there's a Dune II font viewer there. Any chance it can be turned into a font editor? :PLooking at the interface, it already has pretty much everything you'd need for it. Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 30, 2008 Share Posted March 30, 2008 When I've finished finished my Format40 decompressor I'll take a look at a font editor. Quote Link to comment Share on other sites More sharing options...
Nyerguds Posted March 30, 2008 Share Posted March 30, 2008 cool... if you do make that please add C&C font support ;D Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 30, 2008 Share Posted March 30, 2008 I'll give it a try when the time comes :)I might need palettes / font files for trials and comparison. Quote Link to comment Share on other sites More sharing options...
Nyerguds Posted March 30, 2008 Share Posted March 30, 2008 heh, no problem there. I don't think there's a single C&C1-related file I don't have :P Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 31, 2008 Share Posted March 31, 2008 Great - I'll drop you a note when I'm ready to begin trials. I'm currently focusing on getting a usable site up - I might change domain in the near future - gain better bandwidth :) Quote Link to comment Share on other sites More sharing options...
Ultraq Posted March 31, 2008 Share Posted March 31, 2008 Seems I might be a bit late in helping with Format80, but I thought I'd throw this in here in-case somebody finds it useful.MrFlibble mentioned my name somewhere in this post as I've been writing Dune 2 conversion utilities for him, and in the process have ended-up writing an encoder/decoder for the Format2/40/80 compression formats used by Dune 2 and C&C. The thread for these tools is here: http://cnc-tech.planetcnc.gamespy.com/index.php?showtopic=3699The source files are available with the download, but the thing is it's written in Java. If you can read Java then you're probably most interested in the encoding/decoding part which is all done in 1 utility class in my program.I've uploaded a copy of that class with the Format2/40/80 encoding/decoding methods here: http://ultraq.orconhosting.net.nz/CNCCodec.java Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted March 31, 2008 Share Posted March 31, 2008 Better late than never ;)Thanks for sharing the info Ultraq - I'll take a look at it. Quote Link to comment Share on other sites More sharing options...
401Kill Posted April 6, 2008 Author Share Posted April 6, 2008 Been away for a while, just come back to the project and I've just got format80 decompression working perfectly - at least for all the .cps files. Turned out to be my SHL implementation was dodgy... thanks for mentioning them, it made me actually research what they were supposed to be doing. :P My success at decoding SHP files varies... some images are perfect and some are incomprehensible. My routine currently ignores format40 shp entries (mostly - I think it might be trying a few and giving these weird results).I haven't worked out logic for dealing with the malformed credit##.shp files (the ones which don't indicate end of header properly) yet, does your code deal with these or just ignore the headers?I'm going to overhaul my shp handling routine now, and then on to Format40 decompression next... joy ;DJust got hold of a new web server, so I'll maybe get a project page going soon... Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted April 6, 2008 Share Posted April 6, 2008 Both CPS and SHP files have headers though they differ slightly depending which file type it is. Headers include a reference or control byte that dictates / describes which format the image is compressed in. Vlato Bato's documentation on C&C files describes the headers aswell as the three different compression formats where Format80 is one of them. Most of the headers in the UNITS*.SHP files seem to point to Format40 compression if compared to the CREDIT*.SHP files. The WSA files seem to include all three types of compression. The headers are very important in each of the files so check up on the documentation for the headers of the SHP / CPS / WSA files and check the pointers that inform on compression/format types in each of the three header types.All three file types contain Format80 / Format40 / Format20 compressed images - headers describe which format each image is compressed in. If you check the header in the CREDIT*.SHP files you'll see that they are all of CompressionType=1 - which is Format80. When I looked at the UNIT*.SHP files the CompressionType was = 0 for most of the images I checked. When I looked at one of the images from the UNIT*.SHP files it's first RLE byte was 00h which seems to indicate a Format40 compression - so it seems that CompressionType=0 in SHP files indicates Format40 compression and CompressionType=1 in SHP files seems to indicate Format80 compression.Headers in files are always important. They instruct your program how to read / use the rest of the file.I've been busy at work lately but my plan is to implement a Format40 decompressor too. Quote Link to comment Share on other sites More sharing options...
DK Posted April 7, 2008 Share Posted April 7, 2008 Greetings,I'm not new in this forum... but I've posted many time ago, so I forgot the nick... (or, at time, registration isn't necessary...)I've made a lot of research about Dune2 formats, and I think I've decoded almost all.They are a first version of old Westwood formats, before C&C, so the docs of Bato aren't always correct.The CPS, format80 and format40 are the same, but for SHP and WSA there are some differences.And some SHP is different from others, too!You can check this document for the SHP, it isn't 100% correct, but it's a start ;)If you need some advice, let me know; also, the Java tools posted above are very good (if you understand Java). Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted April 7, 2008 Share Posted April 7, 2008 Thanks for the hint DK. I've already got a copy of that document and working out my handling of files from that and a bunch of others I've found. Our friend my. Bato has left a lot of hints and fairly good descriptions of principles of handling the data for binary hackers to follow - most programmers from the good old DOS days should be able to decipher his hints.There's a Wiki describing file formats for a number of EoB and LoL file formats virtually identical to those of Dune 2 and C&C. Most of the info only requires a small amount of tweaking to become correct. A good hex-viewer and the collected documents describing the file formats and headers gets you pretty far. Just remember the old way of doing things: bit-shifting, AND/OR handling, nibbles (4-bits), file-offset - all kept inside the 64Kb data segment allowed back then.I still have a few things to work out regarding SHP / WSA compression formats but I'm getting close to a final solution. When I have finished the last decompression functions I'll start writing my own documentation for the formats. I'll propably use VB6 syntax to describe things in code since most programmers are able to read and understand this "simple" language. Quote Link to comment Share on other sites More sharing options...
DK Posted April 7, 2008 Share Posted April 7, 2008 Thanks for the hint DK. I've already got a copy of that document and working out my handling of files from that and a bunch of others I've found. Our friend my. Bato has left a lot of hints and fairly good descriptions of principles of handling the data for binary hackers to follow - most programmers from the good old DOS days should be able to decipher his hints.yep, you're right, however I spent lot of time on some format :)Ah, and with help of Christian Morgner (on this forum...) we've decoded also ICN and MAP files.There's a Wiki describing file formats for a number of EoB and LoL file formats virtually identical to those of Dune 2 and C&C. Most of the info only requires a small amount of tweaking to become correct. A good hex-viewer and the collected documents describing the file formats and headers gets you pretty far. Just remember the old way of doing things: bit-shifting, AND/OR handling, nibbles (4-bits), file-offset - all kept inside the 64Kb data segment allowed back then.eh eh eh, are you referring to EoB wiki? I'm one of the contributors :)And the wallset format of EoB is crazy!I still have a few things to work out regarding SHP / WSA compression formats but I'm getting close to a final solution. When I have finished the last decompression functions I'll start writing my own documentation for the formats. I'll propably use VB6 syntax to describe things in code since most programmers are able to read and understand this "simple" language.Should be interesting and useful making also a wiki for Dune2 formats. Let me know if you're interested.Ah, I've still some WSA do not decode correctly (like final.pak -> HFinalC.wsa), let me know if you can figure out where is the problem; I think it must be merged with previous wsa to play correctly and not alone! Quote Link to comment Share on other sites More sharing options...
ArrakisResearchCo Posted April 7, 2008 Share Posted April 7, 2008 Cool - I didnt know about your contributions to the EoB wiki :)I've noticed that the DOS WWPak extractor tool is in fault. It doesnt extract the files to their correct sizes - some are cut off in length. Seems to be a buffer over-run failure that leaves the buffer truncated. HFINALB.WSA is some 100Kb+ in size. There might be other differences depending on the PAK extractor used. Once I've decoded the WSA structures and image formats I'll voice my findings. There seems to be a number of different versions - Nyerguds has release a copy of my HitSquad version of Dune 2 that seems to be the last version released. The version I got from abandonia has a large portion of seemingly random letters in one of the scenario.ini files - there might be similar "errors" in other files in the different versions. There's a link to the HitSquad version in the last post in the thread "Dune II editor with 1.07 support".Regarding the .MAP file found in DUNE.PAK. Do you know what the first part of the file contains ? Seems to references to random tiles and some are out of range but there might be some header information I'm unaware of in the MAP file. Last part of the file seems to be references to construct the 1x1, 2x2, 3x2, 3x3 structures and their "frames" (flag animations, landing pad blinks, radar turning). I havnt researched that particular file that closely yet - there might be something I've missed.Figured out that the TBL file is a palette cross-reference for grey-scaling used in the Starport ordering screen. Quote Link to comment Share on other sites More sharing options...
Nyerguds Posted April 8, 2008 Share Posted April 8, 2008 My editor can change the foundation of structures... it totally messesup the graphics. not only that but it also shows that the entire building has multiple frames, and not just the one tile that actually has the flag on it. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.