Spectral Paladin

emperor
The XBF format

10 posts in this topic

December 15th marked 15 years since posting the topic “Emperor expansion pack” in Westwood's forums, the project which became known as the pioneering Kwisatz Haderach mod. I was very young then and not very mature and fled before it was all completed. One thing that got to me was having to watch helpless as others tried to crack the XBF format, the key to the models and so to a true expansion pack.

This left me with the wish to return to the task numerous times, as my own computing skills increased. I have not really uncovered any more information than what was known back in 2002-03, which was already quite a lot actually. However, I believe that the viewer/exporter which was the product of those efforts has vast room for improvement. What follows is an outline of goals that in my opinion will conclusively resolve the XBF challenge - even though this will come 10+ years too late.

Primary Goal A: Obj importer/exporter

The obj format is virtually universal among 3D software - pretty much any of them can read it and many can export it. Thus converting to/from it opens up many possibilities to work with the models' geometry. The only caveat is that animations are not supported.

What I propose and have worked on is a set of Python scripts that serve as command-line tools to import and export to this format. It is quick to develop in Python and the code can be easily adjusted. Specifically the goal can further be broken down into:

• From XBF to obj - geometry only. This is complete (see screenshot below). I am happy to share the script after some cleaning up of the code.
• From obj to XBF - the mirror image of the above. It is sufficient to add an equivalent write() for each read()
• Texture support

Primary Goal B: Collada importer/exporter

Collada is a format created for the express purpose of sharing files across incompatible 3D software. The advantage over obj is support for animation. Thus, all Python scripts above should graduate to converting from/to Collada.

Secondary goals/other possibilities

Rather than convert to obj/collada, many 3D software tools allow the creation of plugins to support a format. In the case of Blender for example, these are written in Python and so it should be relatively simple to borrow code from the scripts above. Nonetheless, this is not a priority since pretty much any tool can work with obj and collada.

It is also possible to write a viewer/editor specifically for XBF. The sole reason to do this is a better chance to fill out the last gaps in understanding the format. Otherwise, it is a waste of effort to reinvent the wheel.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


On the off chance someone has shared a similar fixation with XBF throughout the years, it would be fantastic if you would like to post about your progress so that efforts can be combined. I am very open to sharing my own code on a public repository, assuming there is interest. Though it would hardly be a surprise if everyone has long moved on.

Edited by Spectral Paladin
0

Share this post


Link to post
Share on other sites

A dust scout in Blender:

dustscout_blender.png

Note the object list in the top right - this is the object tree from the xbf file (but flattened).

0

Share this post


Link to post
Share on other sites

Why are you going to try this. When the people out there already have the solution by using the engine in C&C3?

There are dune2, dune2000 and EbfD mods out there in C&C3. By the sound of it. It is easier to do just that. And you have better models/graphics as a bonus too.

0

Share this post


Link to post
Share on other sites

The engine in C&C3 (SAGE) has nothing to do with the E:BFD engine. I have seen around a misconception that the latter was a precursor to former, but it is incorrect.

Edited by Spectral Paladin
0

Share this post


Link to post
Share on other sites

I see their misunderstanding. But each game actually has their own engine. Completely build from scratch. Very unlikely compared to how Blizzard used the WC3 engine for SC2, but upgraded it.

0

Share this post


Link to post
Share on other sites

It's strange how, whenever Emperor is mentioned, everyone's reaction just seems to be "nope, go play something else"... Why? It'd be neat to mod Emperor, no?

0

Share this post


Link to post
Share on other sites
On 12/25/2016 at 9:44 PM, Spectral Paladin said:

• From XBF to obj - geometry only. This is complete (see screenshot below). I am happy to share the script after some cleaning up of the code.

Can you please share this script?

0

Share this post


Link to post
Share on other sites

Hello - yes, sure. It looks like I was distracted by other things during the holidays.

Below is the latest version of it, I believe. It takes one command line argument, the name of an xbf file, and produces an obj file with the same name (only extension differs), with the name of the objects, their vertices and faces. All vertices are exported with their coordinates transformed; hence the use of the Numpy library.

#!/usr/bin/env python3
import struct
import numpy as np
import argparse
import os

def readInt(file):
	return struct.unpack("<i", file.read(4))[0]

def readUInt(file):
	return struct.unpack("<I", file.read(4))[0]

def readInt16(file):
    return struct.unpack("<h", file.read(2))[0]

def readUInt16(file):
    return struct.unpack("<H", file.read(2))[0]

def readMatrix(file):
	return struct.unpack("<16d", file.read(8*16))

def readByte(file):
    return struct.unpack("<c", file.read(1))[0]

class Vertex:
    def __init__(self):
        self.vertices = None

    def readFrom(self, file):
        self.vertices = struct.unpack("<6f", xbfFile.read(4 * 6))

class Face:
    def __init__(self):
        self.longs = []
        self.floats = []

    def readFrom(self, file):
        self.longs = struct.unpack("<5i", xbfFile.read(4 * 5))
        self.floats = struct.unpack("<6f", xbfFile.read(4 * 6))

VertexTotal = 0 #for writing out to obj

class Object:
    def __init__(self):
        self.children = []
        self.vertices = []
        self.faces = []

    def readFrom(self, file):
        vertexCount = readInt(file)
        flags = readInt(file)
        hasPrelight = bool(flags & 1)
        hasFaceData = bool(flags & 2)
        hasVertexAnimation = bool(flags & 4)
        hasKeyAnimation = bool(flags & 8)
        faceCount = readInt(file)
        childCount = readInt(file)
        self.transform = readMatrix(file)
        nameLength = readInt(file)
        self.name = file.read(nameLength)

        for i in range(childCount):
            child = Object()
            child.readFrom(file)
            self.children.append(child)

        for i in range(vertexCount):
            vertex = Vertex()
            vertex.readFrom(file)
            self.vertices.append(vertex)

        for i in range(faceCount):
            face = Face()
            face.readFrom(file)
            self.faces.append(face)

        if hasPrelight:
            rgb = [readInt(file) for i in range(vertexCount)]

        if hasFaceData:
            faceData = [readInt(file) for i in range(faceCount)]

        if hasVertexAnimation:
            frameCount = readInt(file)
            count = readInt(file)
            actual = readInt(file)
            keyList = [readUInt(file) for i in range(actual)]
            if count < 0: #compressed
                scale = readUInt(file)
                compCount = readUInt(file)
                compressedData = [readUInt16(file) for i in range(compCount*4)]
                if (scale & 0x80000000): #interpolated
                    interpolationData = [readUInt(file) for i in range(frameCount)]

        if hasKeyAnimation:
            frameCount = readInt(file)
            keynimationflags = readInt(file)
            actual = readInt(file)
            for i in range(frameCount+1):
                readInt16(file)
            for i in range(actual):
                struct.unpack("<12f", file.read(4 * 12))

    def writeToObj(self, file, transform=None, resetTotal=True):
        global VertexTotal
        if resetTotal:
            VertexTotal = 0

        np_transform = np.array(self.transform)
        np_transform.shape=(4,4)

        if transform is not None:
            np_transform = np_transform.dot(transform)

        name = self.name.decode()
        file.write("o "+name+"\n")

        for vertex in self.vertices:
            np_point = np.array([vertex.vertices[0], vertex.vertices[1], vertex.vertices[2], 1])
            np_result = np_point.dot(np_transform)
            file.write("v "+str(np_result[0])+" "+str(np_result[1])+" "+str(np_result[2])+" "+str(np_result[3])+"\n")
        file.write("\n")
        for face in self.faces:
            file.write("f "+str(face.longs[0]+1+VertexTotal)+" "+str(face.longs[1]+1+VertexTotal)+" "+str(face.longs[2]+1+VertexTotal)+"\n")
        file.write("\n")
        VertexTotal += len(self.vertices)

        for child in self.children:
            child.writeToObj(file, np_transform, False)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("file")
    args = parser.parse_args()

    with open(args.file, "rb") as xbfFile:
        version = readInt(xbfFile)
        appDataSize = readInt(xbfFile)
        xbfFile.read(appDataSize)
        textureNameDataSize = readInt(xbfFile)
        xbfFile.read(textureNameDataSize)

        xbfObject = Object()
        xbfObject.readFrom(xbfFile)

        xbfname = os.path.splitext(args.file)[0]
        with open(xbfname+".obj", "w") as f:
            xbfObject.writeToObj(f)

Some of the types read may not be strictly correct (i.e. int vs uint). I will have to re-immerse myself in the project to ensure I'm not forgetting something already worked out.

What prompts your interest? Have you modded Emperor or other Westwood games before? Is there something you are working on at the moment?

0

Share this post


Link to post
Share on other sites

Thanx a lot! I will give it a try on holidays i think, i want to create some kind of remake of Dune on Android platform. I'm using Unity engine as it has a lot of documentation online. And no, i haven't modded Emperor or other Westwood games before.

0

Share this post


Link to post
Share on other sites

Yup, Unity is solid. It's always good to see something dune-related, particularly to do with Emperor. Don't hesitate to post news/updates around these forums :)

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now