OpenDune

[Nyerguds]

That'd be the Show Posts Of This Person function. I'd suggest to start with the oldest posts :P

[Xaroth]

It's been a while since the last update of thing..

As a semi-developer (read: I can read C, but writing is still a bit tricky), it's hard to go technical for those who want the juicy bits, but I'm sure TrueBrain can answer most (if not all) questions...

We(read: TrueBrain and glx) made some nice progress, I've been able to run the windows build on xp64 up until the point where the windows stack limits us (for now).. this basically means that you can watch the intro movie up until "and the evil harkonnen"...

[Nyerguds]

Seriously though, you must be doing something wrong to get such gigantic stacks...

[Xaroth]

Seriously though, you must be doing something wrong to get such gigantic stacks...

I think this is mostly due to the way we decompile things, I won't go into detail as TrueBrain is the person with knowledge about this (I just nod and wave)

However, I do know that this will be solved as we go, already a lot of the points where we used to get stack overflows are fixed (take the intro movie for one, as the windows build at first didn't even run that), but there are a lot more to fix, and I'm sure that once we fix those, we'll find more to fix.

Then again, this is what we bargained for, as we should end up with a near-perfect clone of Dune II, from which we can work.

[TrueBrain]

Seriously though, you must be doing something wrong to get such gigantic stacks...

Why wrong? It is a logic result of making C code from assembly. Assembly uses a lot of jumps. If you would convert that to C, you would end up with tons of gotos. This means in the worst case you would have 1 function of 2000 (!) lines. This is not really acceptable to make real C out of that. Therefore, every jump is in its own function. In result, it eats stack like a maniac. Linux GCC tail-call optimization nicely resolves all those, and makes it a jump again. Mingw32 GCC tail-call optimization is not 'smart' or 'advance' enough to do so. In result, it still eats stack, just a little bit less as a maniac ;) But at some point in time it does run out.

As said, it is just a stepping stone, and those places will be fixed sooner or later.

So are we doing something wrong? Not so much. Well, besides being completely insane to do this in the first place of course ..

[Xaroth]

besides being completely insane to do this in the first place of course ..

I would more call it a temporary misplacement of sanity.

[Nyerguds]

Therefore, every jump is in its own function.

Woah, what? Tons of jumps in asm can easily be recognized as being part of IF, CASE, WHILE and other structures like that... you don't have any tools to identify stuff like that?

[Xaroth]

Woah, what? Tons of jumps in asm can easily be recognizes as being part of IF, CASE, WHILE and other structures like that... you don't have any tools to identify stuff like that?

*points at TrueBrain*

he's my tool to identify stuff like that  ;)

[MarkJ]

Woah, what? Tons of jumps in asm can easily be recognized as being part of IF, CASE, WHILE and other structures like that... you don't have any tools to identify stuff like that?

I am guessing that this is something they will work on later, once they have the current version running.

But nyerguds is right. I am guessing at least 95% of those jumps should be easily converted into high-level constructs. It doesn't look like the source was optimized too much flow-wise when it was compiled, so most of the structure is still fairly straight forward.

If you want an example to see just how the compiler converted stuff, I suggest you grab a copy of the Wolfenstein 3D source code. That comes with a compiled version of the source which was compiled with the same compiler as Dune 2. So you have the compiled binary, a nice map file which tells you where everything is, and the source code itself.

[TrueBrain]

If you want an example to see just how the compiler converted stuff, I suggest you grab a copy of the Wolfenstein 3D source code. That comes with a compiled version of the source which was compiled with the same compiler as Dune 2. So you have the compiled binary, a nice map file which tells you where everything is, and the source code itself.

Good idea, tnx :)

About the rest: local jumps are easy. Even without resolving you see that with your own eyes. They are never the problem. Any compiler optimizes those fine too. The problem is that the main function is a chain of long jumps, which at the end jumps back to the first (jumps, not calls). It is safe to say that as soon as you leave the segment, you should stop auto-creating C structures out of it ;) So no tool can solve the stack overflows we are having, and resolving the local jumps is currently not worth the effort (any human sees the solution anyway). It is a balance between effort and gain ;)

About the compiler: it does a bit of optimization, but not too fancy things. It mostly tries to use registers for local variables. It does little to no loop-unwinding, and it does little to no logic reverting (reverting the if and else case). This leads to funny code from time to time. What is much worse, is that the compiler makes mistakes. An example:

As soon as the code wants to access local data from the cs:, it pushes the cs value in the ds. It still uses cs to access the memory (and ds is just never used). Example:

push ds

push cs

pop ds

mov ax [cs:1AE]

pop ds

If you for example check in the input handler (29E8 segment, for OpenDUNE code), you see that happening all the time. This is not an issue, just lame. There is also global data at 353F:7000+ (again, for OpenDUNE code). This is mostly loaded by mov ax, 353Fh, and mov ds, ax. Still fine. At 353F:700E is a mask which states which input is allowed. At one point, the code looks like this:

push ds

push cs

pop ds

mov ax, [ds:700E]

pop ds

This is wrong. 'ds' is now pointing to 'cs', and it reads some random memory (29E8:700E). Lucky enough that piece of memory is frozen (const-data), and contains a very acceptable value for the variable (0x1000). Nevertheless, it is wrong. And it is, as far as I can see, done by the compiler. Well .. it made me laugh, and I wanted to share that laugh ;) I wonder how more 'bugs' are introduced via this way.

[Nyerguds]

About the compiler: it does a bit of optimization, but not too fancy things. It mostly tries to use registers for local variables.

Um, that's normal... you always have to load stack data into registers to manipulate it. Otherwise the manipulation commands are all horribly large.

push ds

push cs

pop ds

mov ax, [ds:700E]

pop ds

Isn't that just deliberately copying ds to cs? Still, even if that's the case it's indeed a pretty bizarre way of doing it. Without seeing the code around it, I really can't tell if this is deliberate or not.

[Xaroth]

if it's of any help, I had a bookmark from when that piece of code was being talked about on IRC:

http://svn.opendune.org/filedetails.php?repname=opendune&path=%2Ftrunk%2Fdecompiled%2Fcs__29E8.c

it's the third function in that file, f__29E8_0281_0012_4D00

I have a feeling that 'bizarre' ways of things being done are just the tip of the iceberg, optimized doesn't mean it has to be readable anymore, so trying to read the optimized versions will give us a lot of headache.. but I'm sure it won't stop us for long :)

[Nyerguds]

Actually, mov commands between registers are usually just 1 or 2 bytes, and pushes and pops are normally one byte each, so I don't think this is optimization.

[Xaroth]

Actually, mov commands between registers are usually just 1 or 2 bytes, and pushes and pops are normally one byte each, so I don't think this is optimization.

Who knows, it's been a long while since this game was compiled so you can't really judge the sanity of the compilers.. or the coders for that matter :)

[Nyerguds]

Why are you so hell-bent on making all of this boil down to sanity issues? ;D

[Xaroth]

Small update again, for those who might be interested.

TrueBrain managed to make the decompiler a lot smarter, resolving some jumps into their parent functions, grouping some totally-obvious stuff together as they should, not only making the conversion bit faster/smarter, but also increasing the stability for windows by a lot...

We also decided to start our release plans, for those interested, they can be found on the forum, for those only interested in new toys to play with.. the first release will be Nov 1 (though this doesn't mean it'll be stable at that point  :P )

for now, it's a new start of a new working week, so I hope I can update those interested soon

[MrFlibble]

Nice going :) So the resulting product will be a one-to-one engine recreation of Dune II?

[Xaroth]

Nice going :) So the resulting product will be a one-to-one engine recreation of Dune II?

Our main goal for now is a complete recreation.. once we reached something we are satisfied with, we will discuss options in term of making this great game even better... ofcourse we've had our ideas, but I doubt any will be implemented before we're content with the code base.

[Nyerguds]

well, you can always use the research done here (like Segra's IDA database) to identify functions in your source code.

[MrFlibble]

Our main goal for now is a complete recreation.. once we reached something we are satisfied with, we will discuss options in term of making this great game even better... ofcourse we've had our ideas, but I doubt any will be implemented before we're content with the code base.

Actually an accurate recreation without additions or enhancements will be of more value, because most Dune II "clones" add something that wasn't there in the original, or do not recreate certain aspects of game play properly.

[Xaroth]

Actually an accurate recreation without additions or enhancements will be of more value, because most Dune II "clones" add something that wasn't there in the original, or do not recreate certain aspects of game play properly.

At some point the conversion to C will be complete, at that point in time we should (if no bugs have been created) have a perfect 'clone' of Dune II, which will be released and tagged so that people can enjoy the Dune II experience on any platform.

Our goals, however, will go much further than that :)

[Nyerguds]

And then, you can make the unit and structure settings ini-based :D

You know, for Dune II it's actually pretty possible. I think we've identified almost all settings in there.

[Xaroth]

And then, you can make the unit and structure settings ini-based :D

You know, for Dune II it's actually pretty possible. I think we've identified almost all settings in there.

Personally I was more thinking of throwing both that, and the AI together in a scripted language like LUA .. but as i said before, this won't happen until we have a complete replica of the original :)

[Nyerguds]

hehe. People actually asked me to do similar things for C&C1. While this is technically possible with the hacking/assembler programming knowledge I got atm, it's a total mess to really accomplish that though.

[MrFlibble]

The funny thing is that the developers initially wanted to use a RULES.INI-like file in Dune II called PROFILE.INI (you can get it here), but then they dropped this idea for some reason, only to return to it in Red Alert.