Re: package file request

anael wrote:

It's good to hear about new things, don't worry, Maratis is starting evolving by itself, and there is a lot of better programmer than me smile

Even if there are better programmers out there, you're the owner and architect. It's your vision.

I've been having a look around and I'm not sure than vm_write is available on the iOS SDK. Which is just great. People suggest that ptrace might work. If I had a Mac here I'd prod around in the iOS SDK and see what's available, but people are using the work Macs... From what I can see though, you should be able to modify the memory within your own application, as long as it doesn't do anything else to violate the developer agreement, should be allowed. I'll try and remember an apple ID so I can read through the developer agreement later.

EDIT: I had a look and it doesn't seem to allow it. Technically there might be some way to hack it to unlock the memory and write it manually, but that seems like an insanely stupid idea. That sucks. Hmmm. Technically you _could_ just not link in -lc and compile something like uclibc into Maratis without file I/O stuff, but that seems a lot like overkill. Plus I'm not sure if that would be allowed either.

Last edited by Nistur (2012-01-09 18:11:33)

  • anael
  • Administrator
  • Offline

Re: package file request

Also, just a taught, what if npk itself uses fread ?

Re: package file request

If it uses fread, fopen etc, it will be intercepted by the overridden functions, which will check to see if any packages are loaded, if they are already (for multiple packages) it will check inside them, otherwise it will load from the normal OS filesystem. Should be fine smile

EDIT: I asked around a bit more. Seems that patching the function will only really work on desktop OSs. There might be some linker black magic that you can do to weakly link, so it won't complain when you try and link a custom lib with the same definitions as libc. Alternatively, you can ignore the iOS libc altogether and link in uclibc or glibc (both LGPL) with modified fopen/fread/fwrite/fclose functions and compiled against the iOS SDK.

Last edited by Nistur (2012-01-09 18:32:11)

  • anael
  • Administrator
  • Offline

Re: package file request

Mh, I feel it will be too much risk for the non-desktop system and I'm a bit afraid of the testing/debugging when I see all the problems we already have when running Maratis on untested hardware.

I understand it will create more work because of the external libs, but if you want to focus on adding npk and creating a layer for fopen/fread I can take care of the DevIL and libsnd loaders as I'm familiar with the libs.

We don't need to modify tinyXML as we should really convert the xml into binary (levels and meshs).

Then there is MBinFontLoader where to replace fread by the layer,
we also don't need to adapt the Freetype loader (it is only needed in the editor, the font is already converted in binary).

I need to check how the script is loaded.
EDIT > we can use "luaL_dostring" or "luaL_dofile"

Re: package file request

I fully understand. I just started working on it under the naive assumption that iOS had a sensible API wink recompiling libc for the sake of that does seem like a bad idea.

Writing the wrappers should be fairly easy and I will look at what lua needs.

EDIT: I've just walked home, thinking about this. I'm pretty sure that, with my luck, I will decide to use some library which will cause problems. So I will do the function wrappers first, and then add the patches, but wrap them in #ifdef M_PATCH_FILE_IO. If I get permission from my work to do some iOS stuff and I splash out to get myself a Mac to do some dev work on, I will look into creating a patched libc. But initially. everything will use wrappers and the patch code won't even be compiled (unless users are feeling brave and/or stupid).

EDIT EDIT: I was also thinking, not that you should replace it now, because you have your xml exporter etc, but have you looked at google's protobuf? It has some nice features including automatic binarisation.

Last edited by Nistur (2012-01-09 19:54:23)

  • anael
  • Administrator
  • Offline

Re: package file request

Hi,

I'm not against the option of it of course, but you don't want to wait to see if you are going to have the problem ? You can always add it the day you want to link a new lib that don't support buffer or doesn't provide source code. I think the good libs are now aware of it, at least for game related things.

Protobuf looks interesting, there is maybe a way to convert xml to it then to binary automatically, it looks to have lot of users.
After, I already have done some time ago the main part of the mesh binarization + reading, it's not yet complete but don't need so much work to be.

Re: package file request

I will focus on getting the files loading through a wrapper first. I'd already mostly written the fopen patch, though. I'm going to try and complete it and test it on Windows and Linux, so it can be included in case someone does come across a case where it's needed. As you said, most good libs are now aware of it, but one that I couldn't get to do this is FBX. Much as there isn't really any need for FBX in Maratis, it shows that a large, widely used lib can still be backwards. I just thought, as the code is there, might make sense to have it as a fall back if someone needs it.

Re: package file request

Research is done. Design is planned. Wrappers are now written.
Task list as follows:
- (probably for lunch during work tomorrow) Replace stdio fopen/fclose/fread/... with new wrappers and test.
- make a test npk manually and get it loaded into the app.
- search npk for files before attempting to load from OS
- ensure fopen, fseek, ftell, fread, fclose all work with package files (no writing using standard file I/O?)
- Add packaging functionality to MaratisEditor. Not sure if this should just add the whole directory.
- Put a publish button into MaratisEditor GUI and create a framework for publishing, including data packaging stage

I hope that I can get the basic reading functionality working by the end of the weekend. With regards to the publishing, is it best to do this in code, or by using python scripts do you think?

  • anael
  • Administrator
  • Offline

Re: package file request

Hi,
nice smile
it's a really big work, thank you for your commitment !

About writing, I don't think we need to be able to write inside the pak ?
At least, the current writing (convert xml, save font as bin) can be done before the packing.

For the publishing, I prefer it's in code as the users normally don't need to have python installed.

Thanks !
Anaël.

Re: package file request

That's what I thought too. I was just playing with the idea of python build scripts because it would add to the flexibility. But I guess that if people have need to change the publishing system in any way, the chances are they will be willing to change code.

I don't think we need to write either. I guess if a file is attempted to open writable, I will try to load it from OS, which would solve the problem of save files as it wouldn't even consider putting them into the package.

I am enjoying working on this and getting to know the engine, so the commitment is no issue smile I will, however, have to start doing some more work on my own game project soon, otherwise I'll get caught up in this and my game won't move ahead wink I am going to try and get this done to an acceptable (albeit maybe quite basic) standard by this weekend, after that I will be focussing mostly on my game, just doing fixes to the package stuff when needed. Once my project is rolling along, I can see if there's anything else I can help with smile

Re: package file request

Ok, so I told a little lie. I didn't test the wrappers yesterday. I ended up talking to some potential artists for my game who wanted me to block out a room so they had something to get started with. So I blocked out a bar and then made a project which they could use to test and work on.

Anyway, the good news. I had time this lunch to test it. It works great and I have DevIL now loading through the wrappers! I guess there's not much point in packaging just the images, so I'll have a play around and get libsndfile to load through them too tonight. Then it's full steam ahead on pulling the data from the packages.

Which leads me onto the next point. What do you want to do about the meshes and levels and stuff? Should I get TinyXML to write them to the package for now? Or leave them outside the package and wait for binary files?

  • anael
  • Administrator
  • Offline

Re: package file request

Wonderful smile

For the levels and meshs xml, don't modify tinyXml, I'll try to start the bin saturday,
but you can try to add the wrapper to MBinFontLoader if you have time ?

Thank you !

Re: package file request

Sure, I had a quick look at it and it doesn't seem like it will be much of a problem.

for the wrappers, I've put them in MFileTools.{h,cpp} and named them M_fopen M_fclose etc. They return MFile* which currently is a base class that handles stdio, but I also have a MFileOpenHook interface which, if a hook is specified, will allow loading from wherever, as long as it returns an MFile* (or inherited class)

I have a skeleton MPackageManager in MEngine which, when initialised, registers it's own MPackageFileOpenHook. The MPackageFile's haven't been written yet, but the plan is to cycle through the loaded packages (even though I'm only going to load one for now, based on the project name) and look for entities in it with the name requested. If they all fall through, it will revert to the MFile behaviour.

I figured MEngine was the correct place to put the package manager as it didn't really contain core functionality, but file I/O itself did. Also, regarding the MFileOpenHook class, as the package manager is in, basically the layer above MCore, seemed sensible to have a generic callback class. Also, as a few people seem to have been talking about a web player, if this ever gets implemented, being able to replace MPackageFileOpenHook with MStreamFileOpenHook should be easy.

Just checking that this design is acceptable before I finish it off. It's still fairly easy to move it around if you're unhappy with anything. I am aiming to submit the basic package code on Saturday, along with instructions on how to create packages manually (as there won't be a publish button yet) and any other useful pieces of information. Then, if I can work out how MGui works (as of yet, I haven't managed to find any data as to how it displays...) then I can try to add the publish button, and make the package... possibly by the end of the weekend.

Actually, that's a point, do you have any preference as to how I specify which files go into a package? I can just add all the subdirectories of the project, but that would quite possibly add source files and raw data, depending on how people's projects are layed out. Maybe another xml file?

Re: package file request

I sorted the BinFontLoader to use the wrapper functions and it compiles, but I can't find any examples which would use it, so I am unable to test it. Everything I can find uses TTF. Can you point me towards something so I can make sure it works?

Also. I just sorted the MSndFileLoader. I got a random track Jamendo recommended, converted it to ogg, tested it. It just worked.
Still only stdio, but through the wrapper. It's nice when things just work though.
The libsndfile interface for buffer reading is a little frustrating, requires a struct with function pointers to what it's going to use to read the data from. A little bulky, but hey, nice and flexible at least. I just remembered about lua, time to dig around with that smile

Re: package file request

Right, I put the code in for lua, and while debugging it, I realised something. I had for some reason been assuming that the paths I was being given were all relative to the project file. Silly me. Well, I have lua working now, but I have to somehow extract the relative path from it.
I think I have 2 choices. The first is to either change directories to the project directory and have all the data files attempt to load from there. The second is, when loading the files using the wrappers, strip the project directory off the beginning first, try to load that, then, if that fails, revert to the full path.

The second option seems a bit hacky. I'm just not sure where to begin with removing the need for absolute paths. I will have a quick looksee tonight, but I think I'm going to call it a night soon. I may have time to sort it during my lunch break tomorrow, which would leave tomorrow evening to try and write the MPackageFile stuff. Sounds like a challenge tongue

(also, I realise that noone is setting this deadline for me but myself)

EDIT: I really don't know what to do right now. The editor expects to be run from it's own directory, so looks for the gui elements within that directory. I'm not sure now if I should change that. Seems like a pretty big change in design :S

Last edited by Nistur (2012-01-12 23:16:38)

  • anael
  • Administrator
  • Offline

Re: package file request

Hi,

yes it's true, I forget to tell that filenames was converted to global when send to data loader,
the gui is an exception and is editor-relative (don't take it in account).

If there is a package detected you can get the local filename :

MEngine * engine = MEngine::getInstance();
MSystemContext * system = engine->getSystemContext();

char localFilename[256];
getLocalFilename(localFilename, system->getWorkingDirectory(), filename);

If the loader received a project-local filename,
the conversion to global would be needed when loading using fopen, so it doesn't change much to modify the path system.

  • anael
  • Administrator
  • Offline

Re: package file request

For the bin font, it is converted by MFontBin,
actually it runs when a font is loaded from editor (for testing), it was not used a lot yet,
we should run it when pressing the publish button, at the same time as converting the xml,
I can take care of that with the button.

[EDIT]
I forget to respond to your first post, the design seems good to me,
for packaging, maybe we can only pack the content/sub-path of the predefined directories ?
- maps/
- meshs/
- fonts/
- levels/
- scripts/
- shaders/
- sounds/

If there is too much chance that users don't use these directories, we can consider copying the files loaded by the editor into them. I was hesitating to do it before to not have double files, but it can be an option, or the user can be alerted.

Re: package file request

So, do you want me to keep the paths absolute within the engine then and only get the relative path for package files?

If I change directory to the project directory (chdir(dir) on *nix, _chdir(dir) on Windows) I can load all files with relative paths, and then load editor files using something like Window's GetModuleFilename. Personally I think this is much neater because it will then allow project files to always be relative paths regardless.

Is it the .font files I'm looking for then? Can I just load them as a font in the editor? Or am I missing something?

  • anael
  • Administrator
  • Offline

Re: package file request

the chdir method was something I was doing in the past, and it occurs to create some problems in some specific cases, for example on iOS and with the editor. But to be honest I don't remember clearly what was the problem.

For now lets not break too much things at the same time, it can create some problems with the editor and with the data system,
I need to think about it, if it's not a problem, I'll do the change later, but I don't want to block you, so the best is to convert to relative path for now.

For the font, if you want to test, load a TTF using Maratis editor, you will see that a .font is created, it is the bin file.
The .font can be load from editor also yes.

Re: package file request

I was actually wondering if iOS was the problem. To be honest, I don't know exactly how iOS treats files in the IPA, what path it expects for them, of course, there's no reason why we should change directory on iOS at all, so we can ignore that, the package can be loaded relatively and all files within that will be using relative paths. Regardless, as you say, it can be changed later. I will strip the path down for packaged files and load them like that.

I will also take a look at the font loading again and make sure that it works properly.
Almost there.

Re: package file request

I tested the font stuff quickly and it loads through the wrapper. So far so good. Everything seems pretty simple.
I made a test .npk for the SponzaFPS example containing maps/ and scripts/.

After work I'll have a go at loading it and pulling data from it smile

Re: package file request

Everything was going so well. Now I get an assert from within the depths of libnpk when trying to open a package file. Meh. Ok. Time to see if I can build libnpk into MEngine instead so I can debug the beast properly

Re: package file request

I added the libnpk files into the Visual Studio solution for the MEngine project and it not only just compiled, but it also just worked. No idea what was wrong with the libs I built. But the package is loading now. I've got to do some minor fixes to the package file wrapper, then a couple more tests, sort scons building........... yeah... almost done tongue

I also realised that I can't really add libnpk-dev to Maratis and libnpk into MaratisPlayer yet as, at the moment, npk is used in MEngine. I think I'll probably submit it as such, it's not a huge amount bigger. What I would like to do is leave just the package manager interface in MEngine and implement them in Maratis/MaratisPlayer.

I guess, when I'm done, I will merge my changes back into the svn tree, then package the whole thing up and email it to you, Anael, if that's ok? Just to be sure that I don't break iOS/OSX builds. It shouldn't do though.

Re: package file request

Tadaa. Ok. MaratisPlayer now loads from an npk if one exists and the file is in a package. Now to move onto scons, then switching to Linux and testing there. Then it's done for now. I'll write a readme/tutorial on how to use the cli tool to make packages. Tomorrow I'll have a quick look at making a new package and populating it. With the system in place, it _should_ be fairly quick to do.
When I'm done with the scons setup, I'll upload a package for people to test and then someone (Anael? me?) to commit to svn smile

Re: package file request

I totally forgot that scons doesn't work with my visual studio on my home machine. I can only really test it on Linux. Speaking of which, I can't figure out how on earth to get MEngine to link to libnpk. I've added what I think should be correct to the correct SConscript.

Can anyone give a quick list of things I need to do to build a third party lib with Maratis?

EDIT: Never mind. I'm just being an idiot. libnpk was empty because I forgot to add the source files to my repository when switching from Windows.

Last edited by Nistur (2012-01-14 20:53:24)