Topic: Plugin system concept

On poking around with Maratis, one thing I realised that is missing, in my opinion, is a plugin system. It's not something that is a neccessity, just something which is nice to have.

At the moment, the functionality is focused on the engine itself. There is a lot of good work done in it. If you want to extend it, there's only one dynamic library you can load (Game.{so,dll,dylib}) which means that realistically, that should only be game specific code.

What I would suggest is to also search for libraries in a specific folder (maybe plugins/) and load them in a similar way to the game library. They could then contain wrappers around 3rd party libraries, networking support, maybe AI behaviours. Allowing adding lua hooks would mean that, potentially, you could allow more people to use more of the engine without compiling a single line of code themselves.

iOS support would need compiling I guess, or at the very least, linking the libraries on publishing.

Anyway, just an idea smile

Re: Plugin system concept

+1.
I wanted to mention it in the engine feature request then thought there are lot of thing yet to implement, which are more important than this. Since anael is the only developer. This is a very good idea. Like Dark Basic Pro and Copercube. Than users will be able to use the plug ins without re compiling it.

Regards.

Re: Plugin system concept

Hi,

in fact the functionality is there but I hide it for the moment for two reasons :

- as Maratis is still under a beta phase and evolving a lot, every plugin would have to be rebuild each time MCore or MEngine is modified, so I wanted to reach a more mature release.

- iOS doesn't allow the use of dynamic libraries (forbidden by apple), so I'm still not sure what is the best way to deal with it. The plugin system could just be disabled for iOS, but at least for plugin with open source code it should be possible to include the code inside the xcode project ; The current plugin system doesn't allow that statically because the functions calling the plugin has the same name for every plugin (StartPlugin and EndPlugin).

Re: Plugin system concept

Ahh, so it's iOS which is the main issue again tongue

Is it possible to use static factories for plugins?

#define MPLUGIN_CREATE(type) \
class MPluginConstructor##type \
{ \
MPluginConstructor##type() \
{ \
MEngine* engine = MEngine::getInstance(); \
engine->getPluginManager()->registerPlugin(new type) \
} \
}

I can see why plugins for a beta engine might be a problem, but then again, I think you might be surprised that people will maintain their own plugins and share them. Also, can build in a check when loading it, put a version number in Maratis through a #define and then compare to the one that the plugin is built with, before doing anything else, if it doesn't match, in the editor or somewhere, give a warning saying that it's potentially a problem.

Re: Plugin system concept

hmm yes, doing real cross platform is a real challenge, specially when something like apple use very restrictive decisions, I'm trying to be really smooth with the moves of the engine. That's also why I paused the multi plugin system, I was sure it was ok with a one game plugin, but I was still not 100% sure of the pipeline for a multi plugin system.

More I think about it and more I don't find so important to take iOS into account.
The multi plugin system is more a "tool" and desktop thing.

It can be enabled for the next release.

Re: Plugin system concept

Continuing from another thread. I have a few ideas for the plugins.

NOTE: this is a brain dump. Expect no coherency tongue

Firstly, I guess it's unfair to not have plugin functionality for iOS so it would have to be extended to support static linking. My initial thought is to have a macro for initialising plugins, when compiled with something like M_PLUGIN_STATIC it would add a start(plugin)Plugin prototype otherwise it creates a startPlugin prototype. Likewise, there can be an implementation macro that wraps up both situations. I can't remember what happens with static constructors in staticly linked libraries, so I will have to take a look into that, but potentially that could provide the plugin initialisation.

Secondly, as Anael mentioned, separate plugins for engine/editor might be confusing for users, my suggestion is to, during publish, check whether the plugin is needed by MEngine, and not publish those that aren't. That way users can just unzip plugin packages into plugins/ and not worry about it.

Also, maybe get Maratis/MatatisPlayer to look in another directory for standard plugins as well as the project plugin directory. On Linux, maybe ~/.Maratis/plugins, %APPDATA%/.Maratis/plugins for Windows. This would give a nice central local repository. Maybe have a list of used plugins stored inside the .mproj (although when published, just load all the copied ones)

Finally, and most crazily, maybe having a repository for plugins which Maratis can check and download. There would be no need for the repository itself to be centralised, just a plugin list, probably in xml, with urls and checksums.

Anyway, that's a quick high level brain dump. It would require moving the plugin management into MEngine. Hmmm, I can think of a couple of reasons why my project might benefit from that. Maybe my next task? tongue

Also, if we could scan the computer for gcc-arm (windows at least should have an environment variable if using devkitARM) then could potentially even link an iOS application from the editor... as everything should be interacting with Maratis, there would be little need for the iOS SDK... sorry off topic

Re: Plugin system concept

For some reason, Maratis is running _really_ slowly under Linux for me. It takes about 2 minutes for any interaction to register. Much as it _does_ work, it's really unusable (at 4x2.4GHz that really shouldn't happen...)

I think this is a perfect time for me to try to add the profiler to see what's going on. The first thing that might be useful is to know where you put the sleep to reduce CPU usage. Or maybe we want to put a pause in to lock it at a maximum of, say 30fps?

Secondly, you said that the plugin system was currently hidden. I've found m_plugins hidden in MaratisPlayer (I assume there's a matching one in MaratisEditor) How did you go about populating this? Is this something that I can re-add simply?

EDIT: I've just put a quick check in Maratis::Maratis for now to try to load all the files in ~/.Maratis/plugins. (will be %APPDATA%/Maratis/plugins for Windows I guess when I tidy it up) I guess for game functionality plugins it can also search (projdir)/plugins in Maratis::start and MaratisPlayer::start. Does that sound acceptable?

Last edited by Nistur (2012-01-29 19:53:46)

Re: Plugin system concept

Hi,

- for linux, do you have opengl drivers ? 3d is slow on my linux because there is no nvidia driver available, so it's using mesa (not really accelerated). If it react very slow also when the editor is empty, maybe there is an issue with X11 events ?

- the plugins was initially parsed in maratis folder /plugins/
but yes AppData would be better is a user doesn't own the write authorization of maratis dir.
And the project dir /plugins/ yes.

it works very simple normally, just need to parse the dir and load the plugin using MPlugin m_plugins.
need to be called just before maratis->loadGamePlugin()

Re: Plugin system concept

Hmmm, that's a good point, I potentially don't have 3D set up properly. I thought I had radeon-hd drivers. Worth finding out tongue Maratis Editor itself isn't slow though, until I load a project.

As I said, I had some quick tests in just to get it working on my machine quickly, which worked pretty well, so I can clean that up and add the MProfilerContext interface (which is being implemented in the plugin I'm writing, if there's no profiler attached, it won't do anything). Any issues with me submitting this code? Do we want debug player/editor builds to profile Maratis itself and the release build to only profile the game(/plugins)?

The time profiler is practically done as it will just use shiny, I'll leave memory profiling for now until I have some nice output method. It currently will spew to stdout but should be able to rewrite the OGRE one to draw to a Maratis (child) window pretty easily.

Re: Plugin system concept

3d drivers are sometime a problem on linux,
I'm still searching for my card, I probably need to install an older version of ubuntu (my card is a bit old)

For the commit, just leave me some days to look at the bin mesh thing, Mario (MarKill) also committed a new feature,
he made a custom shortcut system smile ! I'd like to validate this two things before a new contribution.

Re: Plugin system concept

Sure, I'll keep working and clean it up smile It will be here when you want it tongue Along with maybe some little plugins smile

Re: Plugin system concept

smile cool
thanks !

Re: Plugin system concept

No worries, also, do you have any issue with the plugins depending on MGui being a shared library? If I'm going to make a profiler window come up, it sort of makes sense, but if you have problems with it, I will work around it smile

Re: Plugin system concept

I still have mix feeling about it, for compatibility reasons, MGui is used only for tool things,
so we still need to find a good way to separate editor plugins and game plugins.

For example, a plugin code using MGui won't be compatible for embedded system (iOS, Android etc).
I'd like first that we take a decision on the editor/game separation plugins.

But after that, yes it should be dynamic.

Re: Plugin system concept

No worries, I mean, for editor plugins, you can extend MPlugin a little to look for startEditorPlugin or something in order to make sure they don't overlap.

As for plugin code being compatible across embedded systems, I really wouldn't know, maybe it's possible to create a basic MGui iOS wrapper?

Anyway, as I said, I will work on it as it is now, when you give me the go-ahead, I will integrate it with whatever system you have in place then smile

Re: Plugin system concept

Success. Well, partial for now anyway tongue I have some of Maratis profiling smile I stuck profile tags in all the functions in MaratisUI and I'm about 1/2 way through Maratis when I realised that I actually want a different macro for Maratis, so only debug builds of the engine will even consider profiling but games can add profiling to whatever they want.

Anyway, no MGui output yet, just stdout for now, pastebin'd to give a quick idea
http://pastebin.ca/2108407

PS. I don't have a clue why it's giving <unprofiled> 1083546%

Re: Plugin system concept

very nice,
maybe you can also just write to a text file ?
to keep a trace

Re: Plugin system concept

Sure, it already has a stream writer built into Shiny Profiler. I can easily add a file output, maybe before closing. The stdout stuff was just to check that things were being profiled correctly smile

I wasn't planning on doing a MGui output window until I was sure I had Maratis fully tagged and the profiler giving valid information.

Re: Plugin system concept

I've been very busy with various things, but I've been slowly progressing with the profiler. Just a bit fed up with putting tags throughout the Maratis code tongue Time consuming and boring. Almost done though.

Anyway, a couple more things I wanted to ask about. Firstly, the plugins, for ones that aren't game.dll, what do you think about a version number system, so we can add dependencies and ensure they've got the correct features. I can add that to the stuff I've been working on, if it's approved.
Secondly, in terms of drawing things, for MGui things, I should be able to just access MGui if I build it to a dll, right? And I can create a child window? Or should I try to work it into the existing UI layout? This might cause problems though if there are multiple plugins which add their own UI stuff.
Finally, and still on drawing things. What should I do if I want to draw things directly? Would just using GL calls work do you think? In particular, I'm considering integrating the Löve2d engine into a separate plugin which I can then use for a data driven UI solution.

Re: Plugin system concept

Hi,

I understand, I also have a lot of work those days and was not able to finish the level bin thing, I'm hardly find time to respond to mails and clean the forum (lots of bots...).

The version thing can be a good idea to avoid crashing, how do you see it ? MEngine returning a version value ?

For drawing a child window, there is different ways, by making MGui dynamic yes, but I'm not sure the memory management is safe for shared library right now, as it was always static. If you create a new window from a plugin, it won't be deleted the right way when the plugin will be unloaded.

The other way from a plugin is to create a custom MGame class and change the draw() function to write text.

Finally, for drawing things directly it is also using MGame::draw class,
you can simply use MRenderingContext :

    // to draw in 2d

    MRenderingContext * render = MEngine::getInstance()->getRenderingContext();
    render->setViewport(0, 0, width, height);
    
    // set ortho projection
    render->setMatrixMode(M_MATRIX_PROJECTION);
    render->loadIdentity();
    
    render->setOrthoView(0.0f, (float)width, (float)height, 0.0f, 1.0f, -1.0f);
    
    render->setMatrixMode(M_MATRIX_MODELVIEW);
    render->loadIdentity();

    // draw a quad

    MVector2 vertices[8];

    render->disableNormalArray();
    render->disableTexCoordArray();
    render->disableColorArray();
    render->enableVertexArray();

    vertices[0] = MVector2(position.x, position.y);
    vertices[1] = MVector2(position.x, position.y + scale.y);
    vertices[3] = MVector2(position.x + scale.x, position.y + scale.y);
    vertices[2] = MVector2(position.x + scale.x, position.y);

    render->setVertexPointer(M_FLOAT, 2, vertices);
    render->drawArray(M_PRIMITIVE_TRIANGLE_STRIP, 0, 4);

You can use all the core drawing functions of MRenderingContext like that and mix it with the engine drawing.

The problem of including Love2d is that the drawing will not use MRenderingContext, it can work on the current Maratis as it uses OpenGL but will break the portability scheme.

Re: Plugin system concept

Ok, to begin with, the version numbering, yes, maybe something like

// in some plugin
bool StartPlugin(unsigned int maratisVersion)
{
    if(maratisVersion != s_pluginRequirement)
        return false;
//...
    return true;
}

But also, it might be worth allowing for plugin version numbers. I'm not sure if it should be in the same method, maybe filling a version number reference. Not entirely sure. Anyway, then you can have plugins dependant on specific versions of other plugins.

I will have a look at MGui and see whether I can pinpoint any specific memory issues, I think it's worth making dynamic as it's something that plugins will almost definitely want to take advantage of.

The issue with using the MGame::draw virtual is that we can only have one game, which should be implemented inside the game.dll. If there are other plugins, they can't really add to this. I was considering maybe something like registering something in the MPlugin to update and draw. Some plugins can then ignore it when not needed. Not sure.

MRenderingContext looks like exactly what I was looking for. I was intending to integrate Love2d more fully and remove GL calls and replace them with engine specific ones, so MRenderingContext calls should be able to do the job nicely. Quite a bit of work, but worth it in the end, I think. Anyway, I'll keep poking and see what I come up with.

Thanks for the reply

Re: Plugin system concept

For the version number, passing it inside StartPlugin will break the current plugins, it could be either using MEngine or a global variable (a define like M_ENGINE_VERSION ? ; could a macro update it depending on svn version ?).

For MGui, basically we should remove all the new and delete direct calls and replace it, like for MEngine by ::getNew() and ->destroy() class function. MaratisUI.cpp should be updated too to not use "new" but getNew when creating the gui.

Re: Plugin system concept

It could be exposed in MEngine, certainly. I wouldn't base it entirely off the svn version. In fact, I might populate a struct, something like:

struct MVersionNum
{
    unsigned short major;
    unsigned short minor;
    unsigned short revision;
};

That way, you can have something like {3, 2, svnversion}. It should be fairly easy to pull out the information with your scons build system and create a specially formatted header.

When I get around to it for the profiler window, I'll see if I can sort out the MGui new/delete stuff.

Re: Plugin system concept

So, I've finally got around to doing some of this Love plugin/MGui.dll stuff. The obvious first stage is to get MGui building as a DLL, which works fine using scons, but in the VS project, after a bit of rearranging and stuff, I've noticed that MGui.cpp includes MDevILLoader.h, which is in Maratis/Common/MLoaders... Not really a great place for MGui to access. Is there any reason why this is used directly, or why it's not using MEngine::m_imageLoader?

Unfortunately I can't devote an awful lot of time to this right now, as well as work, I sort of have 3-ish other projects :S My actual game, the tutorial series which I have a few more things to write up, and the Maratis3D/Love2D mash up (plus the mini profiler, which has the time profiling done, I'm just waiting for access to MGui properly so I can create a profiler window)

Re: Plugin system concept

Mh, yes about MDevILLoader loader,
MGui don't have dependency on MEngine (and should not have), only MCore, so it doesn't know MEngine::m_imageLoader.
It need to be using a data loader pointer instead of using MDevILLoader directly, so it can be initialised separately.