I'm now starting coding new game engine. It's called RegEngine. Here is a general design. It's coded in C++ for Windows. This time it will be general game engine, not only renderer. Also, this time I'm going to create an editor (I've never done this before).
I will use wxWidgets 2.9 as GUI library for the editor. It's not an obvious choice though. For example, C++/CLI (which I use at work) is also nice technology as it allows to freely mix native and managed code so one can create GUI with Windows Forms while coding the rest in native C++. But personally I prefer something that is fully native. Qt is also good, looks more mature, stable and, well, "serious" than wxWidgets and it's also free for some time. But wxWidgets is more lightweight than Qt (to a certain degree, as GUI library can be). It also looks like wxWidgets is the only GUI library that has both docking and property grid control built in. (I'm talking about version 2.9 here, because stable 2.8 version of wx doesn't have the property grid yet.)
I want to separate engine from renderer. But I'm not going to make an abstraction to be able to transparently switch between DirectX and OpenGL as it makes no sense for me. I just want to have the ability to use my engine, with all its functionality like scenes, actors, scripts, AI, sound etc. with new renderer I would code some day, let it be DirectX 11 or software one. To do this, my engine will be component based. An actor standing in the scene will just have its name and/or some ID, position, bounding volume and a set of components. If it's visible, its rendering component will be an object from the renderer.
There is also a great idea I want to incorporate into my engine that I know from Krzysiek K. (greetings!). It goes like this: The game, which will be a document for the editor (that's what you can create as New, Open, Save, Save As etc.) is a single, big tree of objects of different type. This tree includes both resources (like textures, meshes, sounds) and scenes with hierarchies of actors inside them, as well as any other entities (like dialogs in an RPG game). Object can have sub-objects and can also have a set of properties of different types (like bool, int, float, string, vector, matrix, color). Framework for these properties (I'll call it "reflection") will automatically serialize them to/from file, show them in property grid in the editor and hopefully make them visible to scripts. This whole idea gives consistent framework to store, serialize and edit all game data.
As creating new dialog boxes is not my favourite activity (no matter if it's coding or clicking :) I've been thinking for a long time about the "ultimate" set of general controls necessary to make a game editor. I currently think that it's sufficient to have:
Some not so necessary and more difficult to code controls are:
RegEngine will consist of modules compiled as static libraries: