Now Working on a New Engine

Coordinator
Aug 1, 2010 at 11:35 PM

So...

heh...

I'm working on a new engine. It leverages SunBurn rendering and BEPU Physics while removing every single pain point in Ox. The editor is real time (physics and all), serialization happens on objects directly (no more silly 'Token' concept), and there won't likely be a schism between the Gui objects and normal 3D game objects. There are several additional features that I'll not cover, but they aid tremendously with practical development issues that Ox didn't even address. While Ox had a really organized architecture, it just grew too fat (>30,000 lines of code) to easily change. Plus, it's public API made it too hard to make changes without breaking everyone's code. Also, while Ox had reasonably extensible rendering, using SunBurn makes it possible for the rendering to keep improving, which has completely stalled with Ox. I suppose my biggest mistake was not fully embracing C#'s unique features to simply the code and usage - it's very hard to extricate oneself from the C++ paradigm once one gets so entrenched...

There are two problems -

1) The engine is being built for a client and I don't know if he'll allow it to be open sourced when the project ends, and

2) It is in no way compatible with Ox.

So, I am likely to start ramping down on Ox development (actually, this has already been happening). If my client doesn't want the code to be open sourced, I'll just have to look at rewriting it yet again using the improved development idioms. But as far as Ox goes, it's gone about as far as I can take it :/

Just wanted to give everyone an honest heads-up!

Editor
Aug 2, 2010 at 9:39 PM

PIRATE IT! (we wont tell)

Coordinator
Aug 2, 2010 at 10:10 PM
Heh... I'm afraid I have a hard enough time finding contract work as it is :) And I got this whole 'be honest to your clients' hang up :) He's a cool guy, so I don't think there will be any big issues with open sourcing it after the project is done and the engine is mature enough to be useful. cheers!
Aug 5, 2010 at 7:19 AM

Thanks for letting us know ;) I only found Ox a few weeks ago, but from what I've seen so far, it is one of the better open source XNA-based efforts out there, so it's a shame to see active development stop. Good luck with the new project!

Actually I'd be interested to hear your views of Ox's shortcomings, especially how you think it uses C# in suboptimal ways.

The token system I can see isn't really required, but serialisation direct on objects also has it's shortcomings. Something like a factory pattern to house the parsing code would definitely make it cleaner, and an adjusted object hierarchy in terms of how the manager systems work might also be good.

Coordinator
Aug 5, 2010 at 9:09 PM
Edited Aug 6, 2010 at 5:20 AM
Hi Kerome! You're welcome. I think one of the most ridiculous things that happen in open source is when a project lead simply disappears and leaves all the users in the dark. So far, with the new engine, I've found that the right thing to do isn't having scaffolding tokens like Ox, but to have a custom-built serializer object that allow each object to participate in its own serialization. In roughly 200 lines of code, I've gotten the same serialization functionality provided by .NET's default xml serializer plus the Ox's token objects. So, the real problem was expecting one-size-fits-all .NET xml serializer to be adequate. As for adding property events to facilitate undo and redo in the editor, I will try to use C#'s upcoming dynamic method resolution functionality to implement this in just a few lines of code. The big problem with Ox is high-level - it's a matter of all the pieces coming together to create an engine that is too fat for me to personally go further with. For example, I wanted to integrate SunBurn, but it turned out that it was easier to create a new engine by scavenging parts from Ox. When it's easier to write an engine from scratch than to use an existing one, the existing one is likely a failure. Had I known years ago what I know now (and had SunBurn been available), Ox would be no more than 8 - 10,000 lines of code. At that size, fundamental changes are still feasible for a single maintainer or any of the end-users. Another issue was the use of stupid .XML configuration files to communicate what objects can be edited in the editors. The proper solution is .NET attributes and reflection. Finally, there are practical development functionality requirements that Ox does not provide and is too fat to easily add (I'll not cover these but they are many). And the editor being split into multiple programs was just retarded. The new editor will be a single, project-oriented 'suite' with context-dependent editing modes (a little more 3DS Max-like), not to mention real-time, enabling you to run your game within it. When building Ox, I was only concentrating on functionality and not too worried about the size of the code base (though I did try to follow DRY). I've come to find that the true measure of a design's success is the size of the code base it leaves behind. A large code base is a design failure. A medium size code base is fundamentally flawed. A small code base is passable. An inexplicably small code base is true success. And it matters for practical reasons. A small code base can be adapted to your future needs. A large code base is applicable only against yesterday's. (Of course, small and large here are relative, but you can feel them with full force when making changes)
Aug 6, 2010 at 7:57 AM
Edited Aug 6, 2010 at 8:10 AM

That's an interesting view. Personally, my preference is a little more towards architectural consistency within an engine rather than going hell-for-leather for minimum code size, but that's born of a number of years organising large projects, where it is more important that if someone diverges outside their usual area they can find consistent patterns and organisation. Of course that only matters if you have enough programmers and enough code that people occasionally look at new areas - there's probably a tipping point around 200,000+ lines of code or so. Your approach makes total sense for indie games development, where it's incredibly important for a minimum number of developers to be able to cover as much ground as possible.

From looking at it, there's definitely a case to be made for splitting it up into a number of different libraries which are tied together by an engine core. At least that way it's possible to individually architect and evolve each piece, and you would expect things like Gui or Sound to be very low maintenance. You could even see the move to SunBurn as a renderer as part of a shift of that kind.

One things I would say is that Ox leaves me a little unclear on how to extend it. That might be a problem unique to C# game engines, perhaps because XNA as a games dev system is built more around coding rather than editing or light scripting compared to some other platforms such as Unity or UDK, and so to a certain extent relies on people creating their own workflow as they add code and extend functionality. It's a tough spot to be in, because authoring efficient project workflow is even harder than code architecture or actual detailed feature coding especially when you're working with other people's code. From my initial browse through C# engines, it seems like a problem that not many people are addressing.

 

Coordinator
Aug 6, 2010 at 8:30 AM
Edited Aug 6, 2010 at 8:32 AM
kerome, The ideas I put forth may seem simplistic or niche, but there's some profound merit to reducing the size of any code base in any situation. Profound enough to take years to learn / understand (at least, that's what it took me). So consider keeping what I said in the back of your mind, and don't expect to see it from that perspective right away. It's like zen or something. I *could* be wrong, but I feel confident to ask that you think about it further. As to breaking things up and modularizing them - that's possible, but again, it's just easier to build something new than to re-factor (knowing, of course, what I know now and having the upcoming C# features that permit it). Extending Ox is one of the pain points I'm getting rid of with the new engine. Currently it's a matter of mutating the configuration .xml files and overriding hook methods scattered about the engine. There is some reasonable docs for this in Ox if you've got time to read them, however. Workflow is absolutely huge, and is something Ox didn't get quite right either. It's better than all other open source XNA engines (it at least has editors), but it's still not where it should be. This will be addressed in the subsequent engine. But the essential thrust of the new engine is to make a game that satisfies a client. So, if we can release it, it will have only the features specifically needed for his game. I might round it out with a few tiny features before open sourcing, but it ought to be pretty raw initially.
Coordinator
Aug 6, 2010 at 8:33 AM
...and why in the hell does this forum kill all of my line breaks??? ...i blame you, bubzy! >:D j/k
Editor
Aug 6, 2010 at 11:57 AM
cout << "\n";
Aug 6, 2010 at 6:55 PM

I do hope you didn't take it as criticism - it certainly wasn't intended that way. Overall I totally agree with the philosophy, any software engineer who's been in the business long enough to have read The Mythical Man-Month will agree that the ultimate conclusion is that "smaller code = more readable = more maintanable = better code" in nearly all cases. But it has its limits, there comes a point where you have to admit that the code becomes too large and all the other techniques of software design come into play, and then the game becomes to organise things as well as possible within the general framework of keeping things as small as possible.

Curiously I don't seem to have any problems with carriage returns. Maybe it's a behavior that's reserved for people with the 'Coordinator' tag ;)

Coordinator
Aug 6, 2010 at 9:19 PM
kerome, Not taken as criticism, but criticism is encouraged anyway! (If, by reading this response, you feel I'm repeating back to you what you've already said, I suspect your feeling is right. We might be hard-pressed to disagree, actually.) To the subject at hand, once the code becomes 'large' in that sense, it basically loses its applicability to other problems / domains. The small, lower-level parts can often be factored out (or ripped out, as the case may be), but the big fat sticky 'middle' just can't be generalized or partitioned out with a reasonable amount of work. At that point, the best you can do, as you say, is to keep it from falling in on itself with higher-level modularity idioms. This, in itself is manageable and not a problem if you have the resources. Hopefully, by the time the code base is that large, you've pulling in enough money to justify maintaining it. While I'm not happy to admit it, perhaps this is the right way for things to be in application software. It's horrifying, but I can't think of nor have heard of a better way. Library software, on the other hand, might not be so entirely doomed. Because it's (hopefully) applicable across problems and domains, it doesn't have to deal with that hideous, poorly-defined middle layer that applications do. Nor does it have to be budgeted (and have its development corners cut) in the same way as a project that 'must ship yesterday' to keep the company from bankruptcy. I think that if a piece of software is to achieve reasonable reusability, it will have to be very limited in its aims. Since reusable code is not its own ends, it can be rewritten without the dangers otherwise present. That one can rewrite it, however, doesn't mean one should when the need to do so can be preempted with an easy-to-change code base.
Aug 15, 2010 at 2:19 PM

Hey, I just wanted to say that I think this venture will be a good one for you. I've been debating whether to purchase the basic version of Sunburn for some time now. I've also chosen your Ox framework to actually use, develop, and learn more about xna as far as Engine development is concerned. Although Ox may or may not die out, I'd be interested in hearing more about what it is you'd be working on as a blog or something. Obviously, the gig is contractual and you probably can't release too much information, but it still would be interesting. Best of luck.

Aug 26, 2010 at 3:59 PM

Just a little concerned about basing a redux around a pay for product though :o

Coordinator
Sep 23, 2010 at 7:10 AM

Hey all!

I just asked to see if the client is willing to open source the generic parts of the new engine I'm working on. It's got a lot of functionality now, so if he gives the OK, I should be able to release it in a week or so.

Here's hoping I can get it out!

cheers!

Editor
Sep 24, 2010 at 11:45 AM

yesyesyes!

look forward to seeing your latest work! lets hope he's in a good mood :)

Coordinator
Sep 25, 2010 at 1:02 AM

I hope to hear something from him on Monday. I will release the source as soon as possible, but mind that there are some features that aren't finished yet and will take a week or three to get complete.

cheers!

Editor
Sep 27, 2010 at 10:28 PM

sooooooo..... it's monday, whats the deal? :D

Coordinator
Sep 27, 2010 at 10:43 PM

Sadly, the deal is that people take forever to answer e-mails nowadays, so I have no idea what's going on.

However, I am putting a lot of work into the engine nonetheless. I got the 2D physics and rendering up and running yesterday, so I can tell you that the engine is going to be both 2D and 3D-enabled. This is something I just could not get done with Ox because that code base was too big and I would have had to write the 2D rendering from scratch as well.

So ya.

:)

Coordinator
Oct 1, 2010 at 3:01 PM

I wrote up a feature description list for the new engine -

Xi Game Engine Feature / Motivation List

Game-centric Object Architecture - Having worked on several games, I factored out a common object structure that applies to all games. It consists of an Application object, a Screen objects, ScreenTransition objects (transition in and transition out), ActorGroup objects, and finally, the main object, Actor. All of these objects are serializable at any time, and little to no custom serialization code need be written for user objects that extend these. A big problem with Ox is that it exposed a lot of functionality to the user, but it gave the user no idea of how to structure an actual application. By providing this object model out of the box, the developer gets started clearly and quickly.

Element System (AKA, 'Components') - Many newer game engines use some type of component system to allow actors (AKA Entities in other engines) to be composed at design-time. This allows new actor behaviors to be contrived by cimbining 'components' at design-time with little to no additional code. Only a few elements exist currently, but more will be forthcoming.

Overlay System - When an actor is configured in the editor, it often shares common property values with other actors. The overlay system allows the developer to specify common property values for objects in XML. The overlay system also has an inheritance semantic that brings in property values from one overlay to another. Multiple overlay inheritance is also supported. This is similar to TorqueX's Template system.

Pooling System - Currently, the .Net Compact Framework cannot much handle frame-based object allocation / deallocation. Therefore, a pooling system is put in place to automatically pool objects where specified, resetting properties to the defaults as specified by the overlay system to keep recycled objects uniform.

3D Functionality - Xi allows building and editing 3D games using BEPU physics in real-time. Unlike Ox, the physics are integrated into existing actors and need no extra extension from the user to get an interactive physics scene. Currently only simple box and sphere physics is implemented along with static triangle meshes, but ragdolls and other types are forthcoming. Also forthcoming is skinned model actors, using the sgAnimation library).

2D Functionality - Xi allows building and editing 2D games using farseer physics in real-time. Physics already integrated here as well. Currently only simple sprites are available, but animated sprites as well as composed sprites / physics are forthcoming, as well as tile editing.

UI System - Xi implements a basic UI system (currently only Button and Label UIs are available). More UI actors are forth coming.

Fully-Integrated and Extensible Editor - Unlike Ox, Xi allows editing of 3D, 2D, and UI actors all in the same editor session. There is also a way to extend the editor with custom editing scenarios for a particular game or object (however, this functionality isn't completely exposed yet).

Play-In-Editor - Xi is built to allow the designer to not only design a level in the editor, but to run the level interactively in the editor. This can reduce iteration times.

Message System - In addition to .Net events that are exposed where appropriate, Xi offers a message system which allow messages to be sent from one object to for a given event as specified directly in the editor. Message receivers are specified with custom syntax (currently based on some String.Split symbolics) relative to the sender. The messaging system uses memoization to avoid generating garbage when making use of reflection (memoization is still not fully implemented, but is forthcoming).

Focus-based Input System - Xi keeps track of the object that each player currently has control over, and automatically sends both low and high-level input messages to it. This also automates switching from UI controls with the DPad. Currently only the Xbox controller inputs are processed.

Oct 4, 2010 at 6:27 AM

This sounds pretty sweet, I can't wait to hear if he's ok with this being open sourced. 

Coordinator
Oct 4, 2010 at 4:37 PM

Ok, I got permission to open source it.

I hope to get something up this weekend.

Watch this thread!

cheers!

Oct 5, 2010 at 10:27 PM

That's great news, can't wait to see what you've got.

Editor
Oct 6, 2010 at 9:18 PM

Woooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo!

Coordinator
Oct 8, 2010 at 5:26 PM

Okay, talking to Synapse again, so gonna need another week before making any release decisions.

cheers!

Coordinator
Oct 13, 2010 at 7:32 AM

I'm starting to think that Synapse is going to diverge somewhat from my engine's intent. So it might be best just to open source it.

I hope, again, to have something out by the end of the week. Maybe like a preview version or something ;)

cheers!

Oct 13, 2010 at 4:43 PM

I too am anxiously awaiting this! :)

Coordinator
Oct 15, 2010 at 6:38 AM

I'm just about ready to push out the preview version. The last big thing I have to do is document the whole public API. Seeing as how the engine is about 12 klocs and mostly undocumented, this will take a lot of energy.

And then there's all the external documentation needed...

Oh my car.... :(

Oct 16, 2010 at 2:04 AM

You can do it. Can't wait to try it out

Coordinator
Oct 17, 2010 at 5:45 AM

Okay, I've finally got something up -

Xi Game Engine Preview

If you already have SunBurn or just want to peruse the code, go ahead and pull it down. There are lots of little issues I'm still sorting through, very little documentation, and lots of bugs I'm sure.

If you get a chance to play with it, feel free to post feedback to this thread! But remember, this is a preview for now, so it's not quite up to snuff in many ways.

Oct 17, 2010 at 5:59 PM

Excellent Brian!  Thanks for the updates.  You going to stay on Sourceforge for this one?

Coordinator
Oct 17, 2010 at 6:03 PM

I'm going to keep using SourceForge for Xi's source, but once it becomes mature, I'm going to use a Codeplex front-end. Ox actually does the same. It uses SourceForce for the source repository and Codeplex for front-end. I'd rather use Codeplex for all of it, but last I checked, the SVN support here is inadequate :(

Oct 18, 2010 at 1:13 AM

Inadequate, in what way?, just curious.

Actually something else I'm curious about is why it makes such a great front end...

I'm not doubting, I am just seeking to understand.

Coordinator
Oct 25, 2010 at 8:58 PM
Edited Oct 26, 2010 at 6:59 AM

So, Synapse shut down the Xi Game Engine.

They said that building a competing product that uses SunBurn violates their EULA. Even if it's open source. I didn't realize this until they sent me a letter.

So, now I will have to write an alternative renderer (probably just port Ox's) so I can put Xi back up. This will take some time. If anyone can help, please let me know.

cheers!

Editor
Oct 26, 2010 at 11:30 AM

sorry to hear that, i guess they were scared that it was too good :D

ill help in any way i can of course,unfortunately i've been out of the coding scene for some time now, but as winter approaches ill probably have time to do a bit more.  If you need some documentation or anything written up ill be more than happy to help.

 

-bubzy

Coordinator
Nov 6, 2010 at 5:40 AM

hey all,

Just wanted to say that I'm slowly working on porting the Ox renderer over to Xi. I've a new job, so that takes priority, but I'm working on it when I can ;)

Anyways, just wanted to update ya'll :)

cheers!