ArrakisResearchCo

Sorry to be correcting you MrFlibble but the domain is www.dune2.dk - without the "k" :) For now I have just moved the entire site to the new domain but I will be updating the pages again soon.

Tadaah ... :D Great - thanks for the info proyvind - I'll make sure to note those findings along with the rest of the FORM format on my site :)

Well .. first 4 bytes are easy - they define the size which is 4. The next 4 bytes are the real mystery ... I went through my notes but couldnt really find anything about these 4 data bytes. My description of the [sINF][Data] is wrong yes - should be [sINF][Data] - it's this 4 byte data block : 02 02 03 04 None of the values I calculate from them seem to refer to anything else in the file. Third byte could refer to the number of following tags :- ??? ... I'll have to re-write my FORM analyser before I can check the actual values against other sizes / headers / tags within the file. Sorry, at this point I'm clueless about these 4 bytes [Data] :( EDIT: I just checked my TilesClass.cls file in the compression method : Result = Result & "SINF" Result = Result & LongToString(4) ' Data block is 4 bytes Result = Result & Chr(2) & Chr(2) & Chr(3) & Chr(4) ' As of yet unknown data - identical regardless of version I've noted in the sourcecode that these 4 bytes [Data] are unknowns ...

Well ... most of it was dropped from memory off-hand ... sorry for any missing / wrong info - I've been away from this for some 10 months or so - mistakes are bound to be made :P I'm in the process of building my new site and I'll re-check all the research and information I've gathered when I upload everything to dune2.dk along with the new site.

I'll sure give it at try :) Ok - first off - the bytes are ordered like this: first byte of the 4 is most-significant - last byte of the 4 is least-significant. All the multi-byte values are ordered this way. That said ... some FORM formated files may have size definitions which point one byte too "early" within the FORM structure. The FixFORMSize() function "repairs" this error. This function takes the original pointer, the size already read from the data stream and a copy of the buffer. It then copies 4 characters from the buffer at the pointer position plus the size that was passed. It then checks the contents of these 4 characters against the known FORM tags. If one of the known tags turns up the size is correct and then passed back as a result. If none of the tags turn up one is added to the size and the process is repeated until the size has been corrected. For the most part the loop is run only once but for those size values that are incorrect the loop runs twice. (Thanks goes to Olaf for pointing out the FORM structure to me in the first place) In the following: "Tag" = This is a 4 character text that make up the header of a section. "Pointer" / "Size" = A 4 byte value creating a longint value (as described above) "Data" = A block of data the length of the size given along with the tag. When reading from the data stream remember to move your pointer +4 from the current position after reading the 4 byte value you need ! The FORM format is made up of a tag that defines the section followed by a size value. The ICON.ICN file has the following structure : [FORM] : the first entry in the FORM structure - the header - followed by the FORM structure size. Should be the size of the file or filesize -4. Differs. [iCON] : defines that this FORM structure contains ICONs or tiles. This tag does NOT have a size ! [sINF][Data] : StructureINFormation - describes that this block contains a data structure. The first defines the size of the following structure - this is usually 4 - i.e. the next info is the second . The second defines the total number of binary bytes the [Data] block contains. Note that sometimes this second may be faulty in size - this is where the FixFORMSize() function described above is needed to correct for this error. When you've read the second remember to move your pointer 4 bytes so you end up at the beginning of the [Data] block. I never needed to figure out what the [Data] part in this section is actually for so I can't be sure - sorry. [sSET] : defines the StructureSET data block. This block contains TileCount * 128 bytes of "compressed" tile data. [RPAL] : defines the ReferencePALette data block. This block contains the palette cross reference sorted color tables. [RTBL] : defines the ReferenceTable data block. This block contains TileCount bytes of cross references to the RPAL block. I've described each of the last 3 data-blocks in more detail here : http://forum.dune2k.com/index.php?topic=20441.msg339067#msg339067 I can't explain why you get that large number - your pointer could be placed wrong, you are encoding the longint value in reverse order, the file may be corrupt. Hope this helps you a bit :)

The 8 bytes right after the "SINF" tag make up two longint values (4 bytes each). The first of these two denotes the "SINF" header size - usually it's 4 indicating the next longint size. The next longint size denotes the total size of the data block in the "SINF" tag. If you hit my site and download the sourcecode for my Tile Editor you'll find a file called "TilesClass.cls" in the ClassesTiles folder. There's a Sub called "DecodeICON" where all the bytes and tags are handled and also commented ... Part of the FORM decoder in my class : Case "SINF" ' ICON.ICN ' Get the size of the SINF block SINFSize = GetLong(Posi, Buffer) ' Move the pointer past the size block Posi = Posi + 4 ' Fix up the Size SINFSize = FixFORMSize(Posi, SINFSize, Buffer) ' Get the data block of the SINF block SINFData = Mid(Buffer, Posi, SINFSize) ' Move the pointer past the data block to the next header Posi = Posi + SINFSize As you can see the first 4 bytes are fetched as a longint value. It usually has a value of 4 indicating that you should move the pointer an additional 4 bytes for the next value however I discovered that some versions of the file has a header-offset error which is why I implemented the FixFORMSize() function to handle this error and find the correct position in case the header-offset is faulty. The next 4 byte block simply holds the size of the following data block. It should be fairly easy reading VB code - the TilesClass.cls file contains the entire code for handling the ICON.ICN file both ways: decompression and compression including color count check for each tile. http://www.dune2/d2te/ [edit: may 4th '10] Changed the URL to the correct new site

Well - I've already released the sourcecode at my temporary site : http://www.dune2.dk/d2te/ You can download the Tile Editor executable and the VB6 sourcecode from the above page :) Oh - the Tiles class contains all the code for decompressing / compressing the ICON.ICN file. The TileMap class handles the ICON.MAP file. EDIT: Uhm - 8 bytes ? are you referring to the 8 bytes at the beginning of the data block inside ICON.ICN or do you mean in any of the SHP/WSA files ? [edit: may 4th '10] Changed the URL to the correct new site

I've been out of country on a job most of this year but I'm back again. I wrote a Tile Editor that can edit ICON.ICN and ICON.MAP ... it's written in VB6 - should be fairly easy to read the code. Let me know if anyone wants a copy of the two classes that handle those two files. When I get the time to move my site I'll release all the VB6 tools I originally wrote along with the VB6 sourcecodes for each of the tools.

After I fixed my (de)compressor my House of IX didnt have any visible remapping with wrong colors. Maybe there's a structure bug or something or maybe some of the tiles have been encoded wrongly / maybe it has been patched already. I'm not sure why the wrong colors persit. Anyhow - I hope my editor has been of some use and just let you guys know: I'm still working on a few more tweaks to my editor. I'm working on adding the option to edit foundation size for the structures in the editor and to keep the same zoom view regardless of foundation size. Things are a bit slow on the project at the moment tho because of other things :)

Thanks for adding my editor MrFlibble :) I'm still working on the changes we've discussed in the Format80 thread and I'll inform you when the changes are done and the final release is out.

Nyerguds: I'll keep your request about the zoom view in mind :) dvalin: Thanks for the hint. I've already looked at your fine library but it took me a while to figure out which repository tool was needed to download the library. Bazar does the trick.

The grid will "auto-show" regardless of foundation dimensions but will fit the foundation dimensions of the specific TileMap reference (one of the 26). So if one of the TileMap references - say the Repair Facility - is changed from 3x2 to 3x3 the grid will auto-resize to the new dimensions. I do have to copy/shift the previous setup into the new dimensions and then fit the new "empty" tiles with new tile references. My plan is to drop concrete slabs into these "empty" tiles. I have to re-write the TileMap reference setup first though: to read the dimensions from the EXE. This will present a problem: if your editor has been used to change the foundation dimensions the ICON.MAP file will not match and will result in under-/overflow and seriously messed up references. Alternatively I have to add code to patch the EXE from the Tile Editor when changing the foundation size in the editor. I'm still looking into these things :)

Yup :) As I said: I've tried making the editor as simple to use as possible. The latest beta release has the graphics changed for the third tool.

After I cleaned up the original code and rewrote my own (de)compressor I left out the parts that replace colors in the tiles. The result should be that all tiles should be correctly indexed in regards to the colors used. There are minute differences beteween the versions but primarily different selections of the cyan colors used in the IX Research Center. I've tried loading ICON.ICN from all four versions and all seem to be displayed without the remapping errors we have seen on other screenshots. Changing the house colors in the palette will definately screw things up in the game. At least if I look at the changes I've tried making with my editor - like when I changed the concrete slab to use some of the Harkonnen colors instead. Result was that all concrete slabs were colored like the house they belonged to and sometimes when a unit died on one of my slabs it changed color to that enemy units house. Looked pretty cool :) If you want to draw something on a tile that has to be the house color you should use the harkonnen colors to draw it. Notice that my editor shows all house specific parts in the harkonnen colors. The game replaces these pixels with the respective house colors when needed.

I've setup a page describing the editor a bit ... http://www.dune2.dk/d2te/ You can get the beta of the editor here : http://www.dune2.dk/D2TE(Beta).exe Grab a copy of your DUNE.PAK file and load the copy into the editor using File->Open. The first link hopefully describes the tools enough - if not please let me know :) [edit: may 4th '10] Changed the URL to the correct new site

Well .. I'm having a hard time trying to answer your question Nyerguds :) As you know - the basics of the editor is to allow visual editing of the images that make up the tiles. In case the person drawing a new tile or changing an existing tile ends up having too many colors in the tile that person would have to manually correct the contents of the image to stay inside the 16 color limit. I could build in a hardwired restriction that prevents the user from adding the 17th color but if the user trieds to change a number of shades or colors - say change the overall color of the Starport Facility to something else - it would be nice if the user could temporarily be allowed to have too many colors in the tile as long as the end result keeps the 16 color limit. Auto-correcting for the limit would prevent the user from making the adjustments to the image contents as needed. It would be up to the user to correct any tiles that have too many colors. The three tools above the palette: pen, replacer and picker - work outside the palette. The third tool, the Picker, only works on the zoomed image (the one in the middle of the window) and can be used to pick up a color from the zoomed image. This will select the picked up color in the palette. What happens inside the palette is specific to the palette itself. The top part of the palette displays the two colors selected: the border is the color locked to the right mouse button while the color in the middle is the color locked to the left mouse button. The pen allows you to draw with these two colors, while the replacer only replaces a pixel if one of the colors match according to which button is pressed. I think it's very hard trying to describe how the editor works and how the tools can and should be used ... I'll try setting up a page that describes how everything works in the editor ..

A little experimenting later ... I've added a "huge" visual warning to tiles containing too many colors. I've attached a sample screenshot of the editor. The first tile displayed has 17 colors. Notice that the tile has been painted red in the TileView area (top-left of the window) and I've added a red/orange square thingie over each zoomed pixel of the tile. The palette shows one of the bluish colors with a small black box around it - showing that this is the color selected for the left mouse button. The view changes from standard zoomed view to this "error-view" the second the 17th color is added to the tile. Screenshot also shows that the actual pixel colors are behind the red warning squares on the tile. Once the 17th color is replaced on the tile the zoomed view returns to normal - and the tile will be repainted normally in the TileView area aswell. I'll describe the limits and restrictions in the documentation for the editor when I release it but please let me know if this is enough or if I should change something or make it different. I've decided to pre-release a beta version of the editor. NOTE! please that I wont be supporting this version in any way because the final editor is to be released in the near future. The editor requires that you load from DUNE.PAK and will only save to DUNE.PAK again. Make a backup of your DUNE.PAK before attempting to edit the tiles ! "Save to" saves to a different DUNE.PAK file if needed. "Save" will save to the same DUNE.PAK as you loaded. The "TileMap" menu has not yet been implemented so no entries in this menu work yet. You can add/remove tiles. The editor requires VB6 runtimes SP6. The magnifying glass tool is NOT a magnifier - it's a colorpicker - use it to pick a color in a tile. When selecting TileMaps you can drag-n-drop tiles from the TileView onto the zoomed TileMap frame. A green border will be shown around the tile in the frame where you can drop the new tile. Get the editor here (216Kb) : http://www.dune2.dk/D2TE(Beta).exe This is the program file - not an archive and not an install package. Please play with the editor and let me know of anything that should be changed or if it crashes for any reason. If it crashes please note what you did before it crashed and please try to make it crash the same place again. [edit: may 4th '10] Changed the URL to the correct new site

I've tried keeping the interface and workspace as simple to use as possible. The color picker allows you to pick any of the 256 colors as you see fit .. but: if you place the 17th color on a tile the entire tile will be shown in red colors in the tile-view area (top-left of the screen) thereby showing the this particular tile contains too many colors. I've put in three tools: pen, color-replacer and color-pick. The last of these tools currently has the wrong tool icon: there's a magnifying glass displayed - I'm working on putting in a color-pick icon instead (which will be in the final release). When using the third tool you can pick a color from the tile (doesnt matter if it's in single tile mode or tilemap mode). When using the third tool to pick a color from the tile, the selection will be activated in the palette / color picker thereby showing which color is currently selected. Note that all three tools have double functions: left-click to use color on the left mouse button or right-click to use color on the right button. The tools default to having palette index zero (the transparent black) on the right mouse button while the left mouse button defaults to palette index 255 which is a white color. If you use the color-pick tool to select a color from the tile you can press either of the two mouse buttons to lock this color to that button. Note that only the color used on the left mouse button will be "highlighted" in the palette. When I started designing this tool I didnt put much thought to keeping the user informed about which colors are used in each tile. As long as the editor internally keeps track of this limit and somehow warns about and restriction violations it should be enough. It doesnt really matter which 16 colors are used for each tile as long as the total number of colors used in the tile doesnt exceed 16. I've solved this by re-drawing the tile in all red colors in the TileView area thereby warning that this specific tile contains too many colors. Before attempting to save the editor runs through all tiles and checks that the total color count of each tile doesnt exceed 16. If just one tile has more than 16 colors the editor prevents you from saving and warns that X number of tiles have too many colors. I have been thinking about planting some kind of indicator on the zoomed view along with the red-coloring of faulty tiles but I havnt done anything about this yet. When I release the editor I'll put in a readme.txt file that describes the functions in the editor and what to look for.

Nyerguds: The reordering of the data blocks when "compressing" the tiles into the ICON.ICN file is based on each pixel entry in each tile. When ever a tile has to be compressed again the code will read all 256 pixel value entries in the tile and count up the number of colors found in the tile. Each color reference will then be stored in a seperate array that will be reordered lowest color to highest color. The actual compression is made up of zero to 15 indexes in this new array. I'll supply the source code for the Tile Editor when it's finished - should be fairly easy to read the VB6 code used. As long as you respect the 16 color limit per tile this new array can be created and re-used when compressing the tiles. It's all in the code and I've commented a large portion of the class that does all the work so it should be fairly easy to read and understand :) MrFlibble: I have not yet posted a download for the Tile Editor. I'd like to finish the last few things in the editor before releasing it but I'll post info when I've released it. And not to worry: I'll release the editor and the source code for it before I finish writing the new site for dune2.dk :)

Thanks Nyerguds :) I'll see what I can do about the foundation dimensions. We would have to use your editor to change the foundation dimensions in the .EXE though. I chose to hardwire the IBM.PAL in the editor so I would only have to load the two ICON.* files since they are the ones changed by the editor. Changing the palettes in the game shouldnt be a problem. For my web design purposes I write a palette editor a few years back that allows easy palette building. See the attached screenshot of "Blend My Colors". I could add this interface to the Tile Editor and load all the palettes from the game into this palette editor. How would that be ? I'm not entirely sure what you have in mind about "automatic sorting when replacing a color" - could you explain please ? I've already added my 8-bit bitmap class to the Tile Editor for export purposes and I'm also thinking about adding "grey-scale" export options using the GRAYRMAP.TBL swap table from the game but if palettes are changed this swap table should also be changed.

First test runs of the changed made by the Tile Editor has resulted in : (Testing on v1.07-HS) Green "Fog-of-war" .. although Dune2 internally seems to ignore the completely black Fog-of-war tile and simply just paints everything black where it's supposed to be. Border of the Fog-of-war is green as it's supposed to be. Windtraps where the structure itself is colored like the Starport Facility. Sandish / skin colored. Full animation returned for the Repair Facility. In v1.07-* the middle sequences of open/close animation are missing. Full animation returned for the Hi-Tech Factory. In v1.07-* the middle sequences of open/close animation are missing. Concrete slabs now have house color. The tiny red lights on the Palace and Spice Refinery have been changed to yellow instead of red. The Tile Editor currently prevents saving if one or more tiles have too many colors. Limit is 16 colors pr. tile. Tile references in the TileMap (ICON.MAP) can be changed by simple drag-n-drop. Looks funny when some of the animation frames have been changed :) The pixel editor and TileMap editor are functional now. The last thing I need to add is the ability to add/remove tiles and add/remove frames in the TileMap. I'm not entirely sure about this but I'm guessing that those entries in the TileMap list that contain landscape parts, wall parts, turrets and so on should be kept locked in regards to the number of frames they have. So I'll leave those locked for now.

Tile Editor update: I've resolved the UI design issues I had and I'm back on track. Currently D2TE (Dune 2 Tile Editor) is able to load ICON.ICN / ICON.MAP, display all tiles, display all tile references found in ICON.MAP (each structures seperate frames - tiles used for each frame are highlighted) and you can drag-n-drop tiles to a selected frame (thereby editing which tiles are displayed in the frame). The contents of IBM.PAL is hardwired into D2TE and palette selection is also up and running. I still need to implement the pixel editor (pen / color-replacer / color-picking). Color count restriction is already built but not yet implemented since the pixel editor isnt running yet. A load/save dialog also needs to be constructed. Currently I have a few hardwired load menus for test purposes while I'm working on the editor. These will be removed and replaced with proper dialogs. Attached is a screenshot of the editor in 800x600.

I've had a lot of things on my hands since Christmas along with a windows driver crash when I put in my new ATI card so I havnt had that much time for my projects. I'm having a few design issues with the tile editor - I'm hoping that my calendar will allow me to get back on track during the next week. At least for the tile editor.

If you should stumble over the tile/color relations please let us know. If you should stumble over any information about spice value I'd really appreciate getting a few hints in that direction. With your permission Nyerguds I'd like to post a page containing your your findings and information found in your source code for your editor. I'd also like to wish everyone a merry Christmas :)

I've been pretty busy today so I havnt really had the time to go on but before I left this morning I started working on the pixel editor. When I have an editor it will be easy to change the tiles. "radar colour" ? You mean the minimap at the bottom right part of the screen ? I have not looked into that part because I have not yet stumbled across anything that compares pixel values to tile numbers. My guess is that DUNE2.EXE internally somewhere handles tile-to-color conversion to create the minimap.