Topic: package file request

I mean engine runs and loads media files(mesh, srcipt...) in from packed file, like nowadays commercial games do. it's good for publicing.

Re: package file request

My first post!

I found Maratis last week and have spent the past few days looking over the source, the examples, the tutorials and the forums. I've also emailed Anael about using Maratis for my own project. As it stands, this is the only bit which, from what I can see, is missing in order to be ready for publishing a game, is the package format.

Anael, it would seem, has already done a lot of research and has suggested "npk". I'm not sure if he's got it planned for integrating it right away, but I thought I'd just (re)start the conversation here so can discuss how to go ahead with this and possibly take over the integration of this library.

First of all, from what I can see at a quick glance, there are only a couple of places where files are loaded (a quick find/grep shows 8) so it shouldn't be too difficult to keep on top of those and override fopen/fread/fwrite with ones that access npk packages, Falling back to the normal filesystem if there's no package available.

Which then leads to "how to make the packages". NPK comes with a command line utility which allows creation of packages, but that doesn't exactly fit with the flow of the engine, which seems to be "things should just work" and also would cause problems if you updated an original file. What I think would be the way to go is to look for a package (probably named the same as the level?) and then, if it exists, check for a file inside it. If the file doesn't exist, or the timestamp is older than a file in the filesystem, then add the new file to the package, then load and return that.

I probably shouldn't have written such a long post. I hope noone minds. If Anael doesn't mind, I'll start looking into integrating this properly.

Last edited by Nistur (2012-01-07 15:10:40)

Re: package file request

Yes, I think npk is what we need, it look clean and simple to use, and we should be able to add it to the current file system. For the files we read manually it will be easy, like you said by adding it to the fread.

The only thing I'm not sure is how to do it for the image and sound library (DevIL and libsnd),
I'm sure they can read a buffer instead of a file, but I never did it before.

You was talking about how to package the data, my idea was to stay on the current file system while making the game, it's easy to access/delete the files, and if necessary to open the xml manually (to correct levels manually or with a batch).

We can just add a button "Publish" in Maratis editor that will just pack all the data of the opened project in another directory, if necessary convert the meshs in binary, copy the MaratisPlayer etc.

In the future it will be also good to add some automation for the iOS and phone publishing :
    - reduce the textures size, compress in jpg
    - generate a xcode project etc

Re: package file request

When I see the npk sample code, I really like how it looks, clear and simple :
http://code.google.com/p/npk/wiki/SampleCode

Re: package file request

Ahh yes, my pet hatred, libraries that expect a FILE* or const char* to do their dirty work. I found a nice image library a while ago which took a buffer and was using that in my engine. I will take a look and see what I can do with DevIL and libsnd.

You're entirely right with the package button. I had never thought of that because I've not worked with an editor before apart from when I was fiddling with big engines.

I am not suggesting doing it at this stage, but the publish button could also run something like NSIS to build an installer. The same with iOS deployment. So many possibilities. One step at a time though.

I had a look at the sample code and I agree, it looks like a very nice library. Let's see if everything else plays ball tongue

Re: package file request

Hi all!

Publish/export option would be great, especially for iOS.
Later on we could also add support for the optimized geometry through the PowerVR POD format and the PVRTC textures, because JPG is just too slow on iOS devices. I don't know how many changes it would require to the existing code, but I will look at it. PowerVR SDK is freely available at their site and includes some command line tools which could be useful for the export.

Mario

Re: package file request

I looked at the PVRTC headers at work a few months ago when I was integrating a flash player for our frontend GUI system. I don't know what the license for PVRTC is, but it seems very simple to use. I'm currently not overly interested in doing much iOS stuff myself, especially as I believe with my contract, I'm not allowed to work on anything in competition with products developed there, so I am not sure whether I can offer to do much in this line of work. If I get some work done on the publish functionality using npk, I will try my best to leave it as open and easy to extend as possible.

As for publishing to iOS, it shouldn't be a problem at all, as Apple have now practically dropped half of their restrictive clauses in their agreement since Epic released the Citadel demo (technically they can still say no if they are feeling pedantic about interpreted code or being built on anything bar OSX) Especially with the format of Maratis, as long as the engine binary was compiled against the iOS SDK, data could easily be packaged under any OS, and even the game plugin could be cross compiled against the engine. Maybe this is a bit off topic for this thread, but I can't see why it couldn't be done with a little work.

Re: package file request

I've taken a look at DevIL and libsndfile and they both _do_ accept buffer inputs. So that shouldn't be a problem at all. DevIL asks for "Lumps" which is quite straight forward, libsndfile has a virtual file open thing which has some instructions on the API on how to use it, but wasn't immediately apparent. Regardless, they're now upgraded to "nice libraries" in my books smile

I'll maybe start with the npk integration tomorrow.

Re: package file request

Regarding PVRTC license, it seems it's free to use.
http://www.imgtec.com/forum/forum_posts.asp?TID=138
cocos3d and oolong engine which are distributed under MIT license use it.

If if understand it correctly, iOS apps can use any programming language. Apple only forbids downloading code from the network.

I'm looking forward to your implementation smile

Re: package file request

Nisturb > good to hear about DevIL and libsnd buffer loading smile

MarKill > PVRTC is not complex to use, I tried it for a production on iOS one time, the small irritating thing is that the texture has to be a square and a pow2 size, but it saves a lot of power and memory. It is maybe not a priority, but once the package system is done, we could add this to the publishing process just before the packing.

I know a bit the PowerVR POD format, it is not really compatible with Maratis and got a bit too much limitation, as it is optimized for openglES and phones. But Maratis meshs are quite fast on iOS, the indices are coded using shorts when the vertices are under the maximum size and every mesh is instanced.

Re: package file request

Right. This is annoying. My laptop had been playing up with a couple of the packages I had installed as I hadn't updated it for a while. Subversion was expecting some libraries which weren't there and when I got them, more things broke. So I finally did a full update and now X doesn't start at boot and I can't connect to wifi networks. The good news is than svn works fine and I have the scons building maratis... sort of... it's now complaining it can't find XGrabKeyboard. This isn't going well. At least I have internet by tethering my phone...

I will probably give up on this in a bit, and then possibly try to beat it into submission on the train on the way back to London tonight. I have yet to get maratis compiling under any OS nicely sad As soon as it's sorted, I'll be rolling.

EDIT: I didn't give up. It was one of the examples (ManualUse) which didn't have X11 added to the libs, Added that to it and it worked fine. MaratisEditor and MaratisPlayer also needed X11 magic as well as dl. Yay. One step closer

EDIT EDIT: Nope. MaratisEditor now says the following:

[nistur@IMP dev]$ build/linux2/release/Maratis/Editor/MaratisEditor
AL lib: pulseaudio.c:612: Context did not connect: Access denied
GL_VERSION : 1.4 Mesa 7.11.2

and then just closes, whereas MaratisPlayer just segfaults. Woo.

MarKill - If I remember correctly, Apple used to have a total no-go on interpreted scripts for submission to the app store because they couldn't verify the safety of the app properly. They also disallowed any package that wasn't compiled and built on an official version of OSX with XCode. When Epic released their Citadel demo, it was clear that it had scripts and was developed with the Unreal toolchain. They couldn't disallow Epic Citadel as it provided them with a lot of publicity for mobile gaming, so they had to change their app store agreement. What they changed it to was that they would allow interpreted code for minor parts of the application, obviously leaving it vague. I can't remember the exact wording as it's been a while since I read it.
Anyway, the point was that Apple _probably_ wouldn't have anything against publishing the application from Windows. The action iOS package is fairly straight forward if I remember correctly, just a compressed directory with a set structure.

Last edited by Nistur (2012-01-08 14:51:46)

Re: package file request

Is it working when you use a compiled version from the website or is it writing the same error ?
download : http://www.maratis3d.org/?page_id=57

Also, we should make it more clear, but the build is copied with all the needed files and dynamic lib in trunk/dev/prod/
Don't try to run the one in trunk/dev/build/

Re: package file request

Yes! Thanks. The Editor in prod worked fine. The Player still throws the segfault. If I can figure out how to build a debug version I will take a look at why it does that. The precompiled player works though, although on my underpowered netbook with it's Intel 945GM graphics chip, it doesn't like any shaders.

Anyway, I figured out how to add 3rd party libs and have it linking in a build of npk which I built. Now to the fun!

Re: package file request

I've just found out that TinyXML loads files itself... and guess what... you can only give it a filename. I just had a sinking feeling... but then I remembered that you (usually) compile TinyXML yourself for most projects, so I checked Maratis. Yay, that means that overriding fopen etc will work smile

Relief!

Re: package file request

@anael: Good to hear that smile

@Nistur: Yes, before the change C, C++ and Objective-C where the only allowed languages, if I remember correctly.
About packaging - I wrote a few sentences, but wasn't sure if I got it right. Do you mean we could bypass Xcode and submit directly to the itunes connect on any OS?

There's already iOS publishing branch, with an Xcode project, but StandardRenderer is broken on iOS 5 (iPod, simulator), but renders correctly on iOS 4.x (simulator).

Re: package file request

I haven't done any apple submissions myself, so I don't know how it works exactly, but as far as I know you just need to send them an .ipa file. What I meant was that you could develop on windows or Linux (or OS, using the editor, press publish which would then create the ipa. If the computer has libimobiledevice it could potentially even push the app to a connected device.

I don't pretend to know exactly how this would work, or suggesting it would be easy to do. I'm also not entirely sure how much I should say about this subject as it is very close to what my work does. Linking in any game plugin to an engine executable should be able to be done with gcc-arm. The rest is just data and can be packaged easily. The only thing that is missing is signing the app with a valid developer certificate which I believe scripts exist for.

This thread is getting off topic, should we migrate to another thread to continue this?

Update on npk: I am sitting on a train, my laptop booted and awaiting my command, I have the design drawn up for the package management. Time to get busy. I have 1.5hrs and only a bearded dragon to distract me. Wish me luck.

Re: package file request

I just checked UDK distribution article, so it seems it's possible. ipa structure is really simple, the only problem could be signing as you said.
Yea, debugging on other platforms would be the biggest significance.

I think we can stop here, if there will be some interest we can open new thread.

Good luck smile

Re: package file request

Back on topic tongue

I didn't get as much done as I would have liked on the train, but I made a start at least. I've also created a git repository fork of the maratis repository and I'm keeping a private repository for me to fiddle in. Partially because I can do mini-commits then and not break anything, partially because I don't have access to the main repo yet (not a hint, honest wink ) but mainly because, if I make any changes to Maratis for the game I'm working on, it makes sense that I wouldn't polute the main engine.

Anyway, for the time being at least, the only changes maratis will have in it will be the npk stuff, so if anyone wants me to stick it on a public repo, I can either put it on the Maratis google code project, or I can make a project myself on either google code or github or whatever.

Oh, and I've done a little bit of rearranging for the git repo, but any merging to/from the svn that would need to be done, I will take care of. I just got rid of the tags/branches directories and moved dev up one level as it was all implicit in git tongue

Re: package file request

Take your time, you don't need to do it in just 2 hours smile

I will send you a mail about creating you an account on google code for svn so you can commit,
if it's ok with you I prefer we don't have multiple public repository.

Re: package file request

That's ok, I'll just keep git for my work and merge with the svn repo when I've got something working.

Re: package file request

I did a bit of coding during my lunch break...
I actually asked the lead programmer on the project I'm working on how he patched a function in our in house tech, it's a couple of lines of code and he's said I could use it, so that's made life a bit easier. There's a couple of problems with it, in that it will currently only work on Win32 (it uses WriteProcessMemory and patches a different JMP instruction in) I had a look and I'm pretty sure that I can get it working on Linux, and I assume OSX without too much of a problem, but I haven't got a clue how ARM assembly works, and I'm going to have to do a deal more reading to make it work on x64 too.

EDIT: I just realised this is Win7 x64. I guess x86/x64 might just work... hmm...

For iOS it shouldn't be _too_ big a problem, everything is packaged withing the .ipa anyway, I mean, it's always nice to have the data packaged, especially as npk can {en,de}crypt the data which would protect against cracking the game, but I think, for now, for any ARM builds, it would be ok to not patch the functions.

Also, what sort of structures should the npk files have? I can duplicate the directory structure inside each package, are there any requirement to have multiple packages so they can be loaded/unloaded on the fly (frontend, ingame, level1, level2 etc) Does Maratis support this at all?

Also, a very minor thing in the grand scheme of things, but what do I call the files? I mean, for now I'm going to be calling it {projectname}.npk. Would it be preferable for this to be obfuscated in some way? something like PKG00001.maratis?
With regards to the encryption, does anyone want it? Do people want to choose their own key, or should I make it generate random keys for each publish? If so, where should it be stored? The logical place would probably be inside the .mproj file, as I don't think that would be inside the package, but if it's in there, it should be in some way obfuscated, otherwise there's no reason for the encryption.

Personally, I think the .mproj should be able to be embedded inside the player executable on publish, possibly not as xml. That would obviously take more work, but shouldn't be too bad.

I hope noone minded me taking a brain-dump here, I just thought I should write down things to get some sort of feedback what people wanted, so I know what I'm aiming towards. Of course, to begin with, I'm not adding encryption, just supporting one package, no x64 etc.

Last edited by Nistur (2012-01-09 14:33:19)

Re: package file request

What do you mean by "patching a function" ?
(I'm not really a hardcore programmer)

About the package structure, it seems good to keep the directory structure as pak-entity name, so when a file is loaded in lua or in c++, we can find the file in the pak or as a file with the same filename :

In Maratis, a level or a mesh can be loaded using the project-local filename for ex : "mesh/box.mesh"
So, in this case the pak-entity could be named "mesh/box.mesh"

About having one pak or multiple pak, one is the most simple, but is it possible that the loading can be too slow if the pak is too big ? (I don't have any previous experience on packing).

For the encryption, let's think about it, we can always add it later, it can stay an option that developers have to control manually
(for example by setting the password of the pak in c++).

Re: package file request

Basically, when you run the executable, the OS usually copies the program into memory (loading dynamic libraries when needed) and then every function call has an address which the program can get to (for example &fopen will get you the address of the fopen function).

There are 2 ways that you can then intercept library functions, especially ones which are part of the standard C libraries like fopen. The first is to simply wrap it:

FILE* MFOpen(const char* path, const char* mode) 
{
    // some crazy stuff here involving package loading/checking etc
    ...
    else
    {
        return fopen(path, mode);
    }
}

would do the job. The second is to replace the library function with some patching. Basically, when you start the program, you get the address of the original fopen call, replace it with a JMP instruction to your own code. It's usually used for things like debuggers as well as cracking programs where they replace authentication calls on other processes. For what our situation I think it would work quite well though as it would basically replace all calls to fopen in all libraries and route it through our package process, if needed.

Much as the Maratis code could easily be modified to use the first method, it would then also require changing TinyXML and any other libraries which don't have handy buffer loading. On the other hand, the function patching is a little more work, but then the buffer loading would probably be entirely redundant as DevIL almost definitely uses fopen internally, which would, again, be patched through to the custom function.

Having the pak-entity name as the relative path seems most logical. Agreed.

I don't know what the performance penalties might be, but it makes sense that accessing and potentially decrypting/decompressing larger amounts of data might be memory/cpu intensive. Also, usually there's no reason to keep things like frontend data in memory, so maybe worth trashing it on larger projects. Packages could then be mounted on top of each other... with the file loader looking first in the latest package loaded, then the previous one, then the previous one (etc) and falling back to the filesystem if no packages are loaded or the file isn't found in any of them.

Setting the password in C++ seems like a reasonable option, as you load any game plugin in the editor, I should be able to create something I can pull out during the publishing process.

I realise I sometimes have a tendency to over-engineer things. If you think that anything is wrong, or you'd prefer something done in a different way, just say. I chose to patch the function as it would more easily allow addition of libraries later to use the package management, but I don't want to seem like I'm trying to make design decisions on this. Maratis is your baby wink

Re: package file request

Ok, thank you, I was not knowing anything about function patching, interesting. I'm maybe a bit afraid by the cross-platform compatibility, but it's mainly because it's the first time I'm hearing about this. I would have picked the first option (the only one I was knowing) but if the function patching is simpler why not.

Is it possible to check that the same thing exist for unix, as it is a more secured OS ?
It might be forbidden on iOS and Android, even if the ARM support it.

It's good to hear about new things, don't worry, Maratis is starting evolving by itself, and there is a lot of better programmer than me smile

Re: package file request

I did a check and it's definitely available on Linux using ptrace
http://www.linuxmanpages.com/man2/ptrace.2.php
Having a quick google suggests that OS X might be slightly more problematic as Apple don't like people poking around with their things, but vm_write is apparently what I'm looking for, although, I don't have access to a Mac to play around with that.
http://web.mit.edu/darwin/src/modules/x … write.html

In general, patching the function is more difficult than simply wrapping it and would require extra debugging to ensure it's working properly (so you don't accidentally jump to a random area of memory) but once the patch is written, it shouldn't need to change, so you can then extend things, for example the multiple packages, also, every call to the function will then be routed through the patch, so even if you got a closed source library with an evil file loader, it would still probably use fopen, so you could still use the package to contain the data for that library.

I will see if I can find out if it's possible to do this for iOS and Android, I can't see why not. The issue is that things like WriteProcessMemory, ptrace and vm_write all allow access to other processes, but Apple have secured OS X by making some processes un-alterable. Then again, Apple are pedantic about a lot of things. I'll go a-hunting for information.