Jump to content

The XBF format


Spectral Paladin

Recommended Posts

December 15th marked 15 years since posting the topic “Emperor expansion pack” in Westwood's forums, the project which became known as the pioneering Kwisatz Haderach mod. I was very young then and not very mature and fled before it was all completed. One thing that got to me was having to watch helpless as others tried to crack the XBF format, the key to the models and so to a true expansion pack.

This left me with the wish to return to the task numerous times, as my own computing skills increased. I have not really uncovered any more information than what was known back in 2002-03, which was already quite a lot actually. However, I believe that the viewer/exporter which was the product of those efforts has vast room for improvement. What follows is an outline of goals that in my opinion will conclusively resolve the XBF challenge - even though this will come 10+ years too late.

Primary Goal A: Obj importer/exporter

The obj format is virtually universal among 3D software - pretty much any of them can read it and many can export it. Thus converting to/from it opens up many possibilities to work with the models' geometry. The only caveat is that animations are not supported.

What I propose and have worked on is a set of Python scripts that serve as command-line tools to import and export to this format. It is quick to develop in Python and the code can be easily adjusted. Specifically the goal can further be broken down into:

• From XBF to obj - geometry only. This is complete (see screenshot below). I am happy to share the script after some cleaning up of the code.
• From obj to XBF - the mirror image of the above. It is sufficient to add an equivalent write() for each read()
• Texture support

Primary Goal B: Collada importer/exporter

Collada is a format created for the express purpose of sharing files across incompatible 3D software. The advantage over obj is support for animation. Thus, all Python scripts above should graduate to converting from/to Collada.

Secondary goals/other possibilities

Rather than convert to obj/collada, many 3D software tools allow the creation of plugins to support a format. In the case of Blender for example, these are written in Python and so it should be relatively simple to borrow code from the scripts above. Nonetheless, this is not a priority since pretty much any tool can work with obj and collada.

It is also possible to write a viewer/editor specifically for XBF. The sole reason to do this is a better chance to fill out the last gaps in understanding the format. Otherwise, it is a waste of effort to reinvent the wheel.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


On the off chance someone has shared a similar fixation with XBF throughout the years, it would be fantastic if you would like to post about your progress so that efforts can be combined. I am very open to sharing my own code on a public repository, assuming there is interest. Though it would hardly be a surprise if everyone has long moved on.

Edited by Spectral Paladin
Link to comment
Share on other sites

Why are you going to try this. When the people out there already have the solution by using the engine in C&C3?

There are dune2, dune2000 and EbfD mods out there in C&C3. By the sound of it. It is easier to do just that. And you have better models/graphics as a bonus too.

Link to comment
Share on other sites

  • 3 months later...
On 12/25/2016 at 9:44 PM, Spectral Paladin said:

• From XBF to obj - geometry only. This is complete (see screenshot below). I am happy to share the script after some cleaning up of the code.

Can you please share this script?

Link to comment
Share on other sites

  • 2 weeks later...

Hello - yes, sure. It looks like I was distracted by other things during the holidays.

Below is the latest version of it, I believe. It takes one command line argument, the name of an xbf file, and produces an obj file with the same name (only extension differs), with the name of the objects, their vertices and faces. All vertices are exported with their coordinates transformed; hence the use of the Numpy library.

#!/usr/bin/env python3
import struct
import numpy as np
import argparse
import os

def readInt(file):
	return struct.unpack("<i", file.read(4))[0]

def readUInt(file):
	return struct.unpack("<I", file.read(4))[0]

def readInt16(file):
    return struct.unpack("<h", file.read(2))[0]

def readUInt16(file):
    return struct.unpack("<H", file.read(2))[0]

def readMatrix(file):
	return struct.unpack("<16d", file.read(8*16))

def readByte(file):
    return struct.unpack("<c", file.read(1))[0]

class Vertex:
    def __init__(self):
        self.vertices = None

    def readFrom(self, file):
        self.vertices = struct.unpack("<6f", xbfFile.read(4 * 6))

class Face:
    def __init__(self):
        self.longs = []
        self.floats = []

    def readFrom(self, file):
        self.longs = struct.unpack("<5i", xbfFile.read(4 * 5))
        self.floats = struct.unpack("<6f", xbfFile.read(4 * 6))

VertexTotal = 0 #for writing out to obj

class Object:
    def __init__(self):
        self.children = []
        self.vertices = []
        self.faces = []

    def readFrom(self, file):
        vertexCount = readInt(file)
        flags = readInt(file)
        hasPrelight = bool(flags & 1)
        hasFaceData = bool(flags & 2)
        hasVertexAnimation = bool(flags & 4)
        hasKeyAnimation = bool(flags & 8)
        faceCount = readInt(file)
        childCount = readInt(file)
        self.transform = readMatrix(file)
        nameLength = readInt(file)
        self.name = file.read(nameLength)

        for i in range(childCount):
            child = Object()
            child.readFrom(file)
            self.children.append(child)

        for i in range(vertexCount):
            vertex = Vertex()
            vertex.readFrom(file)
            self.vertices.append(vertex)

        for i in range(faceCount):
            face = Face()
            face.readFrom(file)
            self.faces.append(face)

        if hasPrelight:
            rgb = [readInt(file) for i in range(vertexCount)]

        if hasFaceData:
            faceData = [readInt(file) for i in range(faceCount)]

        if hasVertexAnimation:
            frameCount = readInt(file)
            count = readInt(file)
            actual = readInt(file)
            keyList = [readUInt(file) for i in range(actual)]
            if count < 0: #compressed
                scale = readUInt(file)
                compCount = readUInt(file)
                compressedData = [readUInt16(file) for i in range(compCount*4)]
                if (scale & 0x80000000): #interpolated
                    interpolationData = [readUInt(file) for i in range(frameCount)]

        if hasKeyAnimation:
            frameCount = readInt(file)
            keynimationflags = readInt(file)
            actual = readInt(file)
            for i in range(frameCount+1):
                readInt16(file)
            for i in range(actual):
                struct.unpack("<12f", file.read(4 * 12))

    def writeToObj(self, file, transform=None, resetTotal=True):
        global VertexTotal
        if resetTotal:
            VertexTotal = 0

        np_transform = np.array(self.transform)
        np_transform.shape=(4,4)

        if transform is not None:
            np_transform = np_transform.dot(transform)

        name = self.name.decode()
        file.write("o "+name+"\n")

        for vertex in self.vertices:
            np_point = np.array([vertex.vertices[0], vertex.vertices[1], vertex.vertices[2], 1])
            np_result = np_point.dot(np_transform)
            file.write("v "+str(np_result[0])+" "+str(np_result[1])+" "+str(np_result[2])+" "+str(np_result[3])+"\n")
        file.write("\n")
        for face in self.faces:
            file.write("f "+str(face.longs[0]+1+VertexTotal)+" "+str(face.longs[1]+1+VertexTotal)+" "+str(face.longs[2]+1+VertexTotal)+"\n")
        file.write("\n")
        VertexTotal += len(self.vertices)

        for child in self.children:
            child.writeToObj(file, np_transform, False)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("file")
    args = parser.parse_args()

    with open(args.file, "rb") as xbfFile:
        version = readInt(xbfFile)
        appDataSize = readInt(xbfFile)
        xbfFile.read(appDataSize)
        textureNameDataSize = readInt(xbfFile)
        xbfFile.read(textureNameDataSize)

        xbfObject = Object()
        xbfObject.readFrom(xbfFile)

        xbfname = os.path.splitext(args.file)[0]
        with open(xbfname+".obj", "w") as f:
            xbfObject.writeToObj(f)

Some of the types read may not be strictly correct (i.e. int vs uint). I will have to re-immerse myself in the project to ensure I'm not forgetting something already worked out.

What prompts your interest? Have you modded Emperor or other Westwood games before? Is there something you are working on at the moment?

Link to comment
Share on other sites

Thanx a lot! I will give it a try on holidays i think, i want to create some kind of remake of Dune on Android platform. I'm using Unity engine as it has a lot of documentation online. And no, i haven't modded Emperor or other Westwood games before.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...