package file request (Page 1 of 7)

Yes, I think npk is what we need, it look clean and simple to use, and we should be able to add it to the current file system. For the files we read manually it will be easy, like you said by adding it to the fread.

The only thing I'm not sure is how to do it for the image and sound library (DevIL and libsnd),
I'm sure they can read a buffer instead of a file, but I never did it before.

You was talking about how to package the data, my idea was to stay on the current file system while making the game, it's easy to access/delete the files, and if necessary to open the xml manually (to correct levels manually or with a batch).

We can just add a button "Publish" in Maratis editor that will just pack all the data of the opened project in another directory, if necessary convert the meshs in binary, copy the MaratisPlayer etc.

In the future it will be also good to add some automation for the iOS and phone publishing :
    - reduce the textures size, compress in jpg
    - generate a xcode project etc

Ahh yes, my pet hatred, libraries that expect a FILE* or const char* to do their dirty work. I found a nice image library a while ago which took a buffer and was using that in my engine. I will take a look and see what I can do with DevIL and libsnd.

You're entirely right with the package button. I had never thought of that because I've not worked with an editor before apart from when I was fiddling with big engines.

I am not suggesting doing it at this stage, but the publish button could also run something like NSIS to build an installer. The same with iOS deployment. So many possibilities. One step at a time though.

I had a look at the sample code and I agree, it looks like a very nice library. Let's see if everything else plays ball

I looked at the PVRTC headers at work a few months ago when I was integrating a flash player for our frontend GUI system. I don't know what the license for PVRTC is, but it seems very simple to use. I'm currently not overly interested in doing much iOS stuff myself, especially as I believe with my contract, I'm not allowed to work on anything in competition with products developed there, so I am not sure whether I can offer to do much in this line of work. If I get some work done on the publish functionality using npk, I will try my best to leave it as open and easy to extend as possible.

As for publishing to iOS, it shouldn't be a problem at all, as Apple have now practically dropped half of their restrictive clauses in their agreement since Epic released the Citadel demo (technically they can still say no if they are feeling pedantic about interpreted code or being built on anything bar OSX) Especially with the format of Maratis, as long as the engine binary was compiled against the iOS SDK, data could easily be packaged under any OS, and even the game plugin could be cross compiled against the engine. Maybe this is a bit off topic for this thread, but I can't see why it couldn't be done with a little work.

Regarding PVRTC license, it seems it's free to use.
http://www.imgtec.com/forum/forum_posts.asp?TID=138
cocos3d and oolong engine which are distributed under MIT license use it.

If if understand it correctly, iOS apps can use any programming language. Apple only forbids downloading code from the network.

I'm looking forward to your implementation

Right. This is annoying. My laptop had been playing up with a couple of the packages I had installed as I hadn't updated it for a while. Subversion was expecting some libraries which weren't there and when I got them, more things broke. So I finally did a full update and now X doesn't start at boot and I can't connect to wifi networks. The good news is than svn works fine and I have the scons building maratis... sort of... it's now complaining it can't find XGrabKeyboard. This isn't going well. At least I have internet by tethering my phone...

I will probably give up on this in a bit, and then possibly try to beat it into submission on the train on the way back to London tonight. I have yet to get maratis compiling under any OS nicely As soon as it's sorted, I'll be rolling.

EDIT: I didn't give up. It was one of the examples (ManualUse) which didn't have X11 added to the libs, Added that to it and it worked fine. MaratisEditor and MaratisPlayer also needed X11 magic as well as dl. Yay. One step closer

EDIT EDIT: Nope. MaratisEditor now says the following:

[nistur@IMP dev]$ build/linux2/release/Maratis/Editor/MaratisEditor
AL lib: pulseaudio.c:612: Context did not connect: Access denied
GL_VERSION : 1.4 Mesa 7.11.2

and then just closes, whereas MaratisPlayer just segfaults. Woo.

MarKill - If I remember correctly, Apple used to have a total no-go on interpreted scripts for submission to the app store because they couldn't verify the safety of the app properly. They also disallowed any package that wasn't compiled and built on an official version of OSX with XCode. When Epic released their Citadel demo, it was clear that it had scripts and was developed with the Unreal toolchain. They couldn't disallow Epic Citadel as it provided them with a lot of publicity for mobile gaming, so they had to change their app store agreement. What they changed it to was that they would allow interpreted code for minor parts of the application, obviously leaving it vague. I can't remember the exact wording as it's been a while since I read it.
Anyway, the point was that Apple _probably_ wouldn't have anything against publishing the application from Windows. The action iOS package is fairly straight forward if I remember correctly, just a compressed directory with a set structure.

Last edited by Nistur (2012-01-08 14:51:46)

Yes! Thanks. The Editor in prod worked fine. The Player still throws the segfault. If I can figure out how to build a debug version I will take a look at why it does that. The precompiled player works though, although on my underpowered netbook with it's Intel 945GM graphics chip, it doesn't like any shaders.

Anyway, I figured out how to add 3rd party libs and have it linking in a build of npk which I built. Now to the fun!

@anael: Good to hear that

@Nistur: Yes, before the change C, C++ and Objective-C where the only allowed languages, if I remember correctly.
About packaging - I wrote a few sentences, but wasn't sure if I got it right. Do you mean we could bypass Xcode and submit directly to the itunes connect on any OS?

There's already iOS publishing branch, with an Xcode project, but StandardRenderer is broken on iOS 5 (iPod, simulator), but renders correctly on iOS 4.x (simulator).

I just checked UDK distribution article, so it seems it's possible. ipa structure is really simple, the only problem could be signing as you said.
Yea, debugging on other platforms would be the biggest significance.

I think we can stop here, if there will be some interest we can open new thread.

Good luck

Take your time, you don't need to do it in just 2 hours

I will send you a mail about creating you an account on google code for svn so you can commit,
if it's ok with you I prefer we don't have multiple public repository.

I did a bit of coding during my lunch break...
I actually asked the lead programmer on the project I'm working on how he patched a function in our in house tech, it's a couple of lines of code and he's said I could use it, so that's made life a bit easier. There's a couple of problems with it, in that it will currently only work on Win32 (it uses WriteProcessMemory and patches a different JMP instruction in) I had a look and I'm pretty sure that I can get it working on Linux, and I assume OSX without too much of a problem, but I haven't got a clue how ARM assembly works, and I'm going to have to do a deal more reading to make it work on x64 too.

EDIT: I just realised this is Win7 x64. I guess x86/x64 might just work... hmm...

For iOS it shouldn't be _too_ big a problem, everything is packaged withing the .ipa anyway, I mean, it's always nice to have the data packaged, especially as npk can {en,de}crypt the data which would protect against cracking the game, but I think, for now, for any ARM builds, it would be ok to not patch the functions.

Also, what sort of structures should the npk files have? I can duplicate the directory structure inside each package, are there any requirement to have multiple packages so they can be loaded/unloaded on the fly (frontend, ingame, level1, level2 etc) Does Maratis support this at all?

Also, a very minor thing in the grand scheme of things, but what do I call the files? I mean, for now I'm going to be calling it {projectname}.npk. Would it be preferable for this to be obfuscated in some way? something like PKG00001.maratis?
With regards to the encryption, does anyone want it? Do people want to choose their own key, or should I make it generate random keys for each publish? If so, where should it be stored? The logical place would probably be inside the .mproj file, as I don't think that would be inside the package, but if it's in there, it should be in some way obfuscated, otherwise there's no reason for the encryption.

Personally, I think the .mproj should be able to be embedded inside the player executable on publish, possibly not as xml. That would obviously take more work, but shouldn't be too bad.

I hope noone minded me taking a brain-dump here, I just thought I should write down things to get some sort of feedback what people wanted, so I know what I'm aiming towards. Of course, to begin with, I'm not adding encryption, just supporting one package, no x64 etc.

Last edited by Nistur (2012-01-09 14:33:19)

Basically, when you run the executable, the OS usually copies the program into memory (loading dynamic libraries when needed) and then every function call has an address which the program can get to (for example &fopen will get you the address of the fopen function).

There are 2 ways that you can then intercept library functions, especially ones which are part of the standard C libraries like fopen. The first is to simply wrap it:

FILE* MFOpen(const char* path, const char* mode) 
{
    // some crazy stuff here involving package loading/checking etc
    ...
    else
    {
        return fopen(path, mode);
    }
}

would do the job. The second is to replace the library function with some patching. Basically, when you start the program, you get the address of the original fopen call, replace it with a JMP instruction to your own code. It's usually used for things like debuggers as well as cracking programs where they replace authentication calls on other processes. For what our situation I think it would work quite well though as it would basically replace all calls to fopen in all libraries and route it through our package process, if needed.

Much as the Maratis code could easily be modified to use the first method, it would then also require changing TinyXML and any other libraries which don't have handy buffer loading. On the other hand, the function patching is a little more work, but then the buffer loading would probably be entirely redundant as DevIL almost definitely uses fopen internally, which would, again, be patched through to the custom function.

Having the pak-entity name as the relative path seems most logical. Agreed.

I don't know what the performance penalties might be, but it makes sense that accessing and potentially decrypting/decompressing larger amounts of data might be memory/cpu intensive. Also, usually there's no reason to keep things like frontend data in memory, so maybe worth trashing it on larger projects. Packages could then be mounted on top of each other... with the file loader looking first in the latest package loaded, then the previous one, then the previous one (etc) and falling back to the filesystem if no packages are loaded or the file isn't found in any of them.

Setting the password in C++ seems like a reasonable option, as you load any game plugin in the editor, I should be able to create something I can pull out during the publishing process.

I realise I sometimes have a tendency to over-engineer things. If you think that anything is wrong, or you'd prefer something done in a different way, just say. I chose to patch the function as it would more easily allow addition of libraries later to use the package management, but I don't want to seem like I'm trying to make design decisions on this. Maratis is your baby