ICN & MAP format specifications?

[dvalin]

Heyo!

I'm in the process of cleaning up the code in libeastwood for reading icon.icn & icon.map (which I didn't originally write myself..) and I notice that there are fields in the header that's not processed nor documented.

I'm having a hard time finding much other documentation or code covering the format, so would anyone else know about any resources?

It's not really *that* important for me to find out as I have no problems reading the format, I'm mostly curious and also wondering if it contains any additional information that could be of any use.

[MrFlibble]

Have you checked out this: http://www.junkyard.dk/icon_map.asp? Also, here's some info: http://forum.dune2k.com/index.php?topic=20441.msg339718#msg339718

BTW, it's icon.icn and icon.map, not dune.icn/dune.map.

[dvalin]

Have you checked out this: http://www.junkyard.dk/icon_map.asp? Also, here's some info: http://forum.dune2k.com/index.php?topic=20441.msg339718#msg339718

BTW, it's icon.icn and icon.map, not dune.icn/dune.map.

Seems cool!

Reading through it now. :)

Yupp, fixed the filenames in my previous post now. :)

[MrFlibble]

Glad to help :)

[dvalin]

Seems cool!

Reading through it now. :)

Yupp, fixed the filenames in my previous post now. :)

D'oh, seems to only cover the MAP format, that one is a simple enough format that I got all bits and pieces in place for already. :)

[MrFlibble]

Well, then how ArrakisResearchCo would be able to make an utility to successfully edit both those formats?

Just browse that site and the thread here, I think you'll find what you're looking for.

[dvalin]

D'oh, seems to only cover the MAP format, that one is a simple enough format that I got all bits and pieces in place for already. :)

Double d'oh, I spoke too early, I noticed some description on the ICN format later in the forum thread. :)

thx:)

[ArrakisResearchCo]

I've been out of country on a job most of this year but I'm back again.

I wrote a Tile Editor that can edit ICON.ICN and ICON.MAP ... it's written in VB6 - should be fairly easy to read the code.

Let me know if anyone wants a copy of the two classes that handle those two files. When I get the time to move my site I'll release all the VB6 tools I originally wrote along with the VB6 sourcecodes for each of the tools.

[dvalin]

I might be interested, dunno if I really have any need for it though, my code reads the formats just fine

and implementing writing to the format from that should probably be fairly easy I presume..

Only thing I'm wondering about, and that kinda was what I made this post for is what the 8 bytes

starting at offset 18 is for, reading through the Format80 thread in here, it seems that I'm not

the only one wondering either..

[ArrakisResearchCo]

Well - I've already released the sourcecode at my temporary site :

http://www.dune2.dk/d2te/

You can download the Tile Editor executable and the VB6 sourcecode from the above page :)

Oh - the Tiles class contains all the code for decompressing / compressing the ICON.ICN file. The TileMap class handles the ICON.MAP file.

EDIT:

Uhm - 8 bytes ? are you referring to the 8 bytes at the beginning of the data block inside ICON.ICN or do you mean in any of the SHP/WSA files ?

[edit: may 4th '10] Changed the URL to the correct new site

[dvalin]

no, 8 bytes at offset 18 of ICON.ICN:

00 00 00 04  02 02 03 04

just after "ICONSINF" and before "SSET"

[dvalin]

here's the possible values as I see it btw. (where I assume that the first 4 bytes is the size of the field as big endian, being 4):

little endian: 514(0x202) 1027(0x403) 67305986(0x4030202)

514+1027= 1541(0x605)

big endian: 514(0x202) 772(0x304) 33686276(0x2020304)

514+772 = 1286(0x506)

first two values are 16 bit values (first as decimal, then same value as hexadecimal) of the next four bytes, while the third is a 32 bit value of all the four bytes..

(values are most likely big endian..)

Anyone able to make anything out of this? :)

[ArrakisResearchCo]

The 8 bytes right after the "SINF" tag make up two longint values (4 bytes each). The first of these two denotes the "SINF" header size - usually it's 4 indicating the next longint size. The next longint size denotes the total size of the data block in the "SINF" tag.

If you hit my site and download the sourcecode for my Tile Editor you'll find a file called "TilesClass.cls" in the ClassesTiles folder. There's a Sub called "DecodeICON" where all the bytes and tags are handled and also commented ...

Part of the FORM decoder in my class :

Case "SINF" ' ICON.ICN

 ' Get the size of the SINF block

 SINFSize = GetLong(Posi, Buffer)

 ' Move the pointer past the size block

 Posi = Posi + 4

 ' Fix up the Size

 SINFSize = FixFORMSize(Posi, SINFSize, Buffer)

 ' Get the data block of the SINF block

 SINFData = Mid(Buffer, Posi, SINFSize)

 ' Move the pointer past the data block to the next header

 Posi = Posi + SINFSize

As you can see the first 4 bytes are fetched as a longint value. It usually has a value of 4 indicating that you should move the pointer an additional 4 bytes for the next value however I discovered that some versions of the file has a header-offset error which is why I implemented the FixFORMSize() function to handle this error and find the correct position in case the header-offset is faulty. The next 4 byte block simply holds the size of the following data block.

It should be fairly easy reading VB code - the TilesClass.cls file contains the entire code for handling the ICON.ICN file both ways: decompression and compression including color count check for each tile.

http://www.dune2/d2te/

[edit: may 4th '10] Changed the URL to the correct new site

[dvalin]

The 8 bytes right after the "SINF" tag make up two longint values (4 bytes each). The first of these two denotes the "SINF" header size - usually it's 4 indicating the next longint size. The next longint size denotes the total size of the data block in the "SINF" tag.

If you hit my site and download the sourcecode for my Tile Editor you'll find a file called "TilesClass.cls" in the ClassesTiles folder. There's a Sub called "DecodeICON" where all the bytes and tags are handled and also commented ...

Part of the FORM decoder in my class :

Case "SINF" ' ICON.ICN

  ' Get the size of the SINF block

  SINFSize = GetLong(Posi, Buffer)

  ' Move the pointer past the size block

  Posi = Posi + 4

  ' Fix up the Size

  SINFSize = FixFORMSize(Posi, SINFSize, Buffer)

  ' Get the data block of the SINF block

  SINFData = Mid(Buffer, Posi, SINFSize)

  ' Move the pointer past the data block to the next header

  Posi = Posi + SINFSize

As you can see the first 4 bytes are fetched as a longint value. It usually has a value of 4 indicating that you should move the pointer an additional 4 bytes for the next value however I discovered that some versions of the file has a header-offset error which is why I implemented the FixFORMSize() function to handle this error and find the correct position in case the header-offset is faulty. The next 4 byte block simply holds the size of the following data block.

It should be fairly easy reading VB code - the TilesClass.cls file contains the entire code for handling the ICON.ICN file both ways: decompression and compression including color count check for each tile.

http://www.junkyard.dk/d2te/

Hm, I think I might be stupid.. :P

To be honest, I find VB code truly horrible to read.. :(

Could you explain what the SINF section is for? and how FixFORMSize() actually works?

Reading it as a 32 bit int, long endian, the value I get is 33686276, which cannot be right..?

[ArrakisResearchCo]

I'll sure give it at try :)

Ok - first off - the bytes are ordered like this: first byte of the 4 is most-significant - last byte of the 4 is least-significant. All the multi-byte values are ordered this way.

That said ... some FORM formated files may have size definitions which point one byte too "early" within the FORM structure. The FixFORMSize() function "repairs" this error. This function takes the original pointer, the size already read from the data stream and a copy of the buffer. It then copies 4 characters from the buffer at the pointer position plus the size that was passed. It then checks the contents of these 4 characters against the known FORM tags. If one of the known tags turns up the size is correct and then passed back as a result. If none of the tags turn up one is added to the size and the process is repeated until the size has been corrected. For the most part the loop is run only once but for those size values that are incorrect the loop runs twice.

(Thanks goes to Olaf for pointing out the FORM structure to me in the first place)

In the following:

"Tag" = This is a 4 character text that make up the header of a section.

"Pointer" / "Size" = A 4 byte value creating a longint value (as described above)

"Data" = A block of data the length of the size given along with the tag.

When reading from the data stream remember to move your pointer +4 from the current position after reading the 4 byte value you need !

The FORM format is made up of a tag that defines the section followed by a size value. The ICON.ICN file has the following structure :

[FORM] : the first entry in the FORM structure - the header - followed by the FORM structure size. Should be the size of the file or filesize -4. Differs.

[iCON] : defines that this FORM structure contains ICONs or tiles. This tag does NOT have a size !

[sINF][Data] : StructureINFormation - describes that this block contains a data structure. The first defines the size of the following structure - this is usually 4 - i.e. the next info is the second . The second defines the total number of binary bytes the [Data] block contains. Note that sometimes this second may be faulty in size - this is where the FixFORMSize() function described above is needed to correct for this error. When you've read the second remember to move your pointer 4 bytes so you end up at the beginning of the [Data] block. I never needed to figure out what the [Data] part in this section is actually for so I can't be sure - sorry.

[sSET] : defines the StructureSET data block. This block contains TileCount * 128 bytes of "compressed" tile data.

[RPAL] : defines the ReferencePALette data block. This block contains the palette cross reference sorted color tables.

[RTBL] : defines the ReferenceTable data block. This block contains TileCount bytes of cross references to the RPAL block.

I've described each of the last 3 data-blocks in more detail here :

http://forum.dune2k.com/index.php?topic=20441.msg339067#msg339067

I can't explain why you get that large number - your pointer could be placed wrong, you are encoding the longint value in reverse order, the file may be corrupt. Hope this helps you a bit :)

[dvalin]

[sINF][Data] : StructureINFormation - describes that this block contains a data structure. The first defines the size of the following structure - this is usually 4 - i.e. the next info is the second . The second defines the total number of binary bytes the [Data] block contains. Note that sometimes this second may be faulty in size - this is where the FixFORMSize() function described above is needed to correct for this error. When you've read the second remember to move your pointer 4 bytes so you end up at the beginning of the [Data] block. I never needed to figure out what the [Data] part in this section is actually for so I can't be sure - sorry.

This is the only one I'm wondering about, the other ones I know and understand..

But what you're writing above cannot be correct...

after [sINF] there's only 8 bytes, the first four ([sisze] bytes prolly' defines the length of the next four (what you refer to as ), but after this comes [sSET], NOT the [Data] section you're referring to..

I think you must be remembering incorrectly and what you really mean is [sINF][Data], especially since not knowing what the four bytes [Data] is for, this is from where the valus I posted above is from btw..

So I guess the conclusion must be that we're both lost on what this is for, but sure, works just fine without, I'm just curious and a bit OCD when it comes to being "complete". ;)

I'm gonna dig my way through some of the IFF specifications, perhaps I'll find some pointers of some sorts.. :)

[ArrakisResearchCo]

Well ... most of it was dropped from memory off-hand ... sorry for any missing / wrong info - I've been away from this for some 10 months or so - mistakes are bound to be made :P

I'm in the process of building my new site and I'll re-check all the research and information I've gathered when I upload everything to dune2.dk along with the new site.

[ArrakisResearchCo]

Well .. first 4 bytes are easy - they define the size which is 4. The next 4 bytes are the real mystery ...

I went through my notes but couldnt really find anything about these 4 data bytes. My description of the [sINF][Data] is wrong yes - should be [sINF][Data] - it's this 4 byte data block :

02 02 03 04

None of the values I calculate from them seem to refer to anything else in the file. Third byte could refer to the number of following tags :- ??? ... I'll have to re-write my FORM analyser before I can check the actual values against other sizes / headers / tags within the file. Sorry, at this point I'm clueless about these 4 bytes [Data]  :(

EDIT:

I just checked my TilesClass.cls file in the compression method :

  Result = Result & "SINF"

  Result = Result & LongToString(4) ' Data block is 4 bytes

  Result = Result & Chr(2) & Chr(2) & Chr(3) & Chr(4) ' As of yet unknown data - identical regardless of version

I've noted in the sourcecode that these 4 bytes [Data] are unknowns ...

[dvalin]

here's the possible values as I see it btw. (where I assume that the first 4 bytes is the size of the field as big endian, being 4):

little endian: 514(0x202) 1027(0x403) 67305986(0x4030202)

514+1027= 1541(0x605)

big endian: 514(0x202) 772(0x304) 33686276(0x2020304)

514+772 = 1286(0x506)

first two values are 16 bit values (first as decimal, then same value as hexadecimal) of the next four bytes, while the third is a 32 bit value of all the four bytes..

(values are most likely big endian..)

Anyone able to make anything out of this? :)

These are the possible values of the four bytes and various combinations etc. that I could come up with as possible values to be used, but not able to figure out any places these might make sense..

[dvalin]

with some help from the OpenDune guys, we figured what SINF was for :O)

02 02 03 04

first two bytes is size, shifted with third byte, you'll get:

2<<3 = 16

Fourth byte is bits per pixel (thx for your previous in-depth analyzis of the format, it helped us a lot in figuring out what this was for)

so..

here's from my code:

    _width = _stream.get(); // 2

    _height = _stream.get(); // 2

    shift = _stream.get(); // 3

    _width <<= shift; // 16

    _height <<= shift; // 16

    _bpp = _stream.get(); // 4

    _tileSize = ((_width*_height)<<shift)>>_bpp; // ((16*16)<<3)>>4 = 128

tada, mystery solved! :)

[ArrakisResearchCo]

Tadaah ...  :D

Great - thanks for the info proyvind - I'll make sure to note those findings along with the rest of the FORM format on my site :)

[Jongware]

... some FORM formated files may have size definitions which point one byte too "early" within the FORM structure. The FixFORMSize() function "repairs" this error. This function takes the original pointer, the size already read from the data stream and a copy of the buffer. It then copies 4 characters from the buffer at the pointer position plus the size that was passed. It then checks the contents of these 4 characters against the known FORM tags. If one of the known tags turns up the size is correct and then passed back as a result. If none of the tags turn up one is added to the size and the process is repeated until the size has been corrected.

Prolly falling under "better late than never": a tag simply always starts at the next even offset. If the size of the previous block was odd, a zero byte is inserted. (Thinking: recently had to find this out myself again -- where are the file specs when you think you don't need them anymore?)

[dvalin]

speaking about specifications, here's the specifications for the general IFF format used.. (although this thread is mostly about the non-standard types, not part of the standard)

http://bazaar.launchpad.net/~doonlunacy/libeastwood/trunk/annotate/head%3A/doc/iff.txt