OpenDune

[Xaroth]

Hello everybody,

Seeing we just launched our new site, together with the whole shebang, TrueBrain and I have decided it is time for us to introduce the project to the world wide Dune community..

Below a copy of the announcement, as I'm quite the lazy git today and really can't be arsed rewriting TrueBrain's words to say the same thing.. it might be a bit too technical for some people, but I'll add some explanation below.

----------

In the past few months we created an application which can decompile 16bit real mode applications to C. As a result we can produce C code from the Dune2 binary. Together with a DOS emulation layer, we can create a working Dune2. This will be our initial version.

From this version we will slowly grow to a cross-platform, real C application. Of course it will take a lot of time to get there, but we have that time.

Currently, we have a big problem with the overlays used in Dune2, which doesn't allow us to create a static version of Dune2 (we do have a working version when we attach a JIT). Till this problem is solved, we won't release any source. We of course hope to solve this problem soon, but it is pretty complex.

-----------


The OpenDUNE project will be an Open Source project, with some ideas and concepts similar to that of OpenTTD, as we both originate from that community.

As stated in the announcement, TrueBrain is still working on the last few issues we are facing in getting the initial version on the road, this might take days, or weeks, I don't know.. but I'll be sure to let people know (most likely on this same thread) when this is working... until then we can only hope TrueBrain's brain doesn't melt down.

As mentioned, we take aim to differentiate our project from the numerous attempts at making remakes, by starting from the original game, and improving from there. Also we hope to attract people from all over the world who share our passion of Dune, to help us improve and make one of the best games, better.

Of course I'm missing a lot of details that people might want to know, at this point we are trying to focus on getting the initial versions done so we can assess the situation, and work from there.


If you're interested feel free to drop by on www.opendune.org ( forums at forum.opendune.org ) , or on irc.oftc.net at channel #openDune .

[MarkJ]

Wouldn't decompiling then repackaging the generated source code open up a whole heap of legal issues in regards to the game? I am pretty sure the license agreements for most games specifically state that you are forbidden from reverse-engineering/decompiling the binaries. It's an ancient game, and to an extent, most of the work on these boards is based on the reverse engineering work of the community, but it seems different from decompiling the whole game, then releasing source.

As for the overlays, I don't understand why that's even an issue. If you have generated the source code, why can't you just modify it and recompile it for a large or huge memory model instead of the resource limited DOS one. I would have thought that the overlay management code would have been fairly invisible (added by the compiler), or done through library calls.

[Nyerguds]

legal issues with whom? No one is sure who even has the copyrights of Dune II nowadays, and nobody seems to care either. This game is as abandoned as it could possibly be. It's pretty much the textbook definition of abandonware: "Sure, everyone downloads it 'illegally', but seeing as no one is even around to sue anyone, who cares? just let us enjoy this great old game."

Also, um, decompiled code is usually a mess. I prefer sticking to assembler, personally.

[Xaroth]

Wouldn't decompiling then repackaging the generated source code open up a whole heap of legal issues in regards to the game? I am pretty sure the license agreements for most games specifically state that you are forbidden from reverse-engineering/decompiling the binaries. It's an ancient game, and to an extent, most of the work on these boards is based on the reverse engineering work of the community, but it seems different from decompiling the whole game, then releasing source.

As for the overlays, I don't understand why that's even an issue. If you have generated the source code, why can't you just modify it and recompile it for a large or huge memory model instead of the resource limited DOS one. I would have thought that the overlay management code would have been fairly invisible (added by the compiler), or done through library calls.

Re: Legal question, all I can say at this point is that we've contacted the involved parties about our concept, and none at this point have objected to it

As for the overlay, I'll let that for TrueBrain to answer, as I (still) don't have the required knowledge to say anything useful about it beyond the generic nodding and smiling..

[TrueBrain]

As for the overlays, I don't understand why that's even an issue. If you have generated the source code, why can't you just modify it and recompile it for a large or huge memory model instead of the resource limited DOS one. I would have thought that the overlay management code would have been fairly invisible (added by the compiler), or done through library calls.

Don't make any mistakes, decompiling is not as simple as disassembling. So all I can say: it is a big issue If you want to know how big, I suggest you step by, and I can explain you in great detail (would take a lot of space to explain that here). If you then have any good suggestions for it, it would be more than welcome

To give a bit more detail: decompiling can't be done like disassembling by looking at the executable. It needs to be done runtime to make anything sane from it (for various of reason). As results, overlays kill the fun.

Also, um, decompiled code is usually a mess. I prefer sticking to assembler, personally.

Sadly enough, assembly doesn't take you away from the DOS code Also, my decompiled code currently looks VERY much like assembly

[MarkJ]

Fair enough about the legal issues. It's just that some developers seem kind of strict when people start poking around with their games, or making remakes, etc. It would be a shame to see the work and effort you have gone to halted by a 'cease and desist' order or somesuch. Yay for the age of litigation .

As for decompilation, I know what you are getting at. I am currently experimenting with decompilers (perhaps low-level to high-level translator is a better term) for c64 games

My point was that most of the overlay segments in the Dune 2 executable seem to be just collections of functions like most other segments. So if you can decompile those, then why would it matter how the overlay manager works? As long as you can resolve function calls from non-overlayed segments to those in overlayed ones, then there wouldn't seem to be any real need to understand how exactly the overlay system does it's stuff.

I suppose it's kind of like the C libraries. I am guessing that you didn't decompile those, and rather just identified where the functions were, and replaced calls to them accordingly. In regards to your generated source, it isn't really important how the C libraries were implemented in Dune 2, just that a call to the printf function actually does what printf is meant to do.

I (and it seems everyone here) have no idea how overlays work, so my assumptions might be complete bollocks, but yeah, I would certainly be interested in seeing where the complications lie .

[Nyerguds]

To give a bit more detail: decompiling can't be done like disassembling by looking at the executable. It needs to be done runtime to make anything sane from it (for various of reason). As results, overlays kill the fun.

I've seen people converting functions from asm to C to get source code though.

[TrueBrain]

I've seen people converting functions from asm to C to get source code though.

One function is simple. A set of function is simple. Code that self-modifies itself on-the-fly is not simple (Dune2 does that, btw, all overlays call trigger a rewrite of the call itself, to give 1 example).

(..)
As for decompilation, I know what you are getting at. I am currently experimenting with decompilers (perhaps low-level to high-level translator is a better term) for c64 games

I agree, a better term, lets go with that

My point was that most of the overlay segments in the Dune 2 executable seem to be just collections of functions like most other segments. So if you can decompile those, (..)

You can't just randomly decompile things. You need to have them in memory first (for various of reasons, self-modifying not being the last). As overlays are not always loaded at the same position (if I move my mouse sooner one time than the other, overlays are loaded in different order), it becomes very hard to know if a piece of code is already known, or not (different 'cs' value). To make matters worse: there are functions which are nearly identical between overlays, but have just 1 bit different. I guess this just gives a peek into the problems I am facing If you are interested in more talk about this, I suggest we move this talk to http://forum.opendune.org, as it is not really related to Dune2, more related to my decompiler.

(..)

I (and it seems everyone here) have no idea how overlays work, so my assumptions might be complete bollocks, but yeah, I would certainly be interested in seeing where the complications lie .

Overlays are the most undocumented features of DOS 16bit world. You got to love it

Btw, my JIT runs Dune2 fine, but it looks in the memory and decompiles the code there to C, which is then compiled to native machine code, and executed. Works very well And doesn't have any problem with self-modifying code, overlays, ... but is useless on long term, as it means you need to modify DOS code to change the behaviour

[Nyerguds]

One function is simple. A set of function is simple. Code that self-modifies itself on-the-fly is not simple (Dune2 does that, btw, all overlays call trigger a rewrite of the call itself, to give 1 example).

..ew

[MarkJ]

Registered on your forum, and posted questions there

[MrFlibble]

Honestly, I didn't have the patience to read through all the technical stuff above, but I like it that you're aiming at recreating Dune 2 as close as possible in terms of its internal workings.

[Xaroth]

From time to time I try to follow what TrueBrain is attempting, to then realise (again) that I should have started with C rather than C# , were I to have any chance at understanding it 

I'm also pleasantly surprised at the interest we've got in a single day, we had the concept on trying to remain low profile until we were ready, and I think we skipped a few weeks in that schedule, but the interest we gained so far definitely makes up for that

[Nyerguds]

yeah, MrFlibble pretty much revived the Dune II community here

[MrFlibble]

Always my pleasure

[TrueBrain]

For those who did follow the technical blabla (:p), I just wanted to let you guys know I solved most of the overlay stuff, and I hope to release a teaser within the next week or so. Not sure if it will be Windows compatible, as I don't have a Windows machine, but I will do my best

[Nyerguds]

Then exactly what OS would that demo be made for?

[TrueBrain]

Then exactly what OS would that demo be made for?

I run Linux, so it works on Linux. Windows is a bit tricky, as it doesn't have setitimer. I fixed that with a nice thread, but it is sub-optimal. Also, Windows runs out of stack much sooner than Linux does. And currently the code is very stack-intensive (running 500-deep is not an exception ).

The Linux binaries and the source is released on our forum, if you are interested. It is not much yet (the Westwood logo showing up, mostly), but it is a start

[Nyerguds]

Don't you define the memory for the application in advance anyway? Stack operations shouldn't matter if you reserve enough mem.

[TrueBrain]

Don't you define the memory for the application in advance anyway? Stack operations shouldn't matter if you reserve enough mem.

Hehe My decompiled version is for now rather inefficient with recursive loops, and fills the stack. This means if you have a loop which runs, say, 65535 times, it will require a stack-depth of 65535. Linux seemly allocates a bigger stack (by default) for an application than Windows does, so the latter runs sooner out of stack. It happens. It is a minor inefficiency of my decompiler, which I will fix sooner or later (as it is just a matter of unrolling loops). But for now it means OpenDUNE doesn't run on Windows.

Even on modern computers there is a limit to the amount of stack you can use. Valgrind by default allocates 88MB of stack I believe, and it happens that valgrind complains it was out of stack. It might be possible to manually tweak those values, but I don't think that is a correct approach to the problem. Windows people just have to show a bit more patient for now

[Nyerguds]

And to think I kicked off my Linux because I never booted it up because I could only play my games in Windows...

[Olaf van der Spek]

Even on modern computers there is a limit to the amount of stack you can use.

Yeah, infinite memory has yet to be invented.
Manually specifying a larger stack should be no problem on Windows though.

[TrueBrain]

Finally solved the stack overflow problem in general. Gcc can optimize that perfectly (tail-call optimization for the win!). Also we finally managed to get the Windows version to work. Which was a pita, as Windows doesn't have a serialized setitimer variant. But, after battling SDL with threads, we finally managed to get it to work. As you can understand, that makes me very happy

If you are interested, we released our second teaser today. It works on both Linux and Windows (the spot for OSX Guru is still open ). Head over to our forums for more details. All I want to say here, is that the intro works nicely

As last, do you guys know any website/thread with good documentation about inner workings of Dune2 (it is a hard job identifying functions, so any help in this area would be much appreciated). I tried searching threads, but information is very scattered.

[Nyerguds]

I'd say, do a search for all of Segra's posts in this forum... I think it's the most in-depth you'll get anywhere.

[MarkJ]

This. Segra is pretty much the central intelligence of all things to do with Dune 2's internals.

[TrueBrain]

That I noticed, but there are only snippets out there. I was hoping there was some central place I missed

[Nyerguds]

That'd be the Show Posts Of This Person function. I'd suggest to start with the oldest posts

[Xaroth]

It's been a while since the last update of thing..

As a semi-developer (read: I can read C, but writing is still a bit tricky), it's hard to go technical for those who want the juicy bits, but I'm sure TrueBrain can answer most (if not all) questions...

We(read: TrueBrain and glx) made some nice progress, I've been able to run the windows build on xp64 up until the point where the windows stack limits us (for now).. this basically means that you can watch the intro movie up until "and the evil harkonnen"...

[Nyerguds]

Seriously though, you must be doing something wrong to get such gigantic stacks...

[Xaroth]

Seriously though, you must be doing something wrong to get such gigantic stacks...

I think this is mostly due to the way we decompile things, I won't go into detail as TrueBrain is the person with knowledge about this (I just nod and wave)

However, I do know that this will be solved as we go, already a lot of the points where we used to get stack overflows are fixed (take the intro movie for one, as the windows build at first didn't even run that), but there are a lot more to fix, and I'm sure that once we fix those, we'll find more to fix.

Then again, this is what we bargained for, as we should end up with a near-perfect clone of Dune II, from which we can work.

[TrueBrain]

Seriously though, you must be doing something wrong to get such gigantic stacks...

Why wrong? It is a logic result of making C code from assembly. Assembly uses a lot of jumps. If you would convert that to C, you would end up with tons of gotos. This means in the worst case you would have 1 function of 2000 (!) lines. This is not really acceptable to make real C out of that. Therefore, every jump is in its own function. In result, it eats stack like a maniac. Linux GCC tail-call optimization nicely resolves all those, and makes it a jump again. Mingw32 GCC tail-call optimization is not 'smart' or 'advance' enough to do so. In result, it still eats stack, just a little bit less as a maniac But at some point in time it does run out.

As said, it is just a stepping stone, and those places will be fixed sooner or later.

So are we doing something wrong? Not so much. Well, besides being completely insane to do this in the first place of course ..