401

(150 replies, posted in Engine)

Everything was going so well. Now I get an assert from within the depths of libnpk when trying to open a package file. Meh. Ok. Time to see if I can build libnpk into MEngine instead so I can debug the beast properly

402

(7 replies, posted in Bug report)

Anael - Yes, I think I had that one too, I think it doesn't like the UTF-8 character in your name, and there was no encoding specified before it or something to that effect.

zester - No problem. Took me a little while of fiddling to get it compiling, no sense in anyone else doing the same smile

403

(3 replies, posted in Engine)

That's fine. Just wanted to clarify it. The difference in size in this case is, for linux, 128k vs 70k. Neither size is huge. I will try and get scons building libnpk, and get it compiling into the VS10 solution too. I will also see if I can find a copy of VC++Express2005 to update the solution for that too.

404

(7 replies, posted in Bug report)

Ok, so there are a couple of SConscript files that need 'X11' adding to them ie:
env_local.AppendUnique(LIBS = ['GL',
                                                             'Xxf86vm',
                                                             'X11'])
MSDK/MGui/SConscript
Maratis/Editor/SConscript
Maratis/Player/SConscript
Examples/ManualUse/SConscript

Also, the editor and player need 'dl' adding to the same list.

Hope that helped. Not sure why, but it compiles fine on one of my machines without it, but one it needs the extra libs.

405

(7 replies, posted in Bug report)

I had this. Should I submit a fix for this with my package stuff (tomorrow hopefully...) Give me 10 mins and I'll reboot into linux and tell you a quick fix

406

(3 replies, posted in Engine)

As I'm getting ready to submit the first pass at the package system, I just thought I'd try and find out what is reccommended for submitting code, and in my case in particular, third party libraries. I made a new topic so that it might be useful to other people if they look.

I am a little confused about third party libraries. I have had to compile libnpk myself but I haven't changed the library myself. Should I include the source for the library into the maratis svn tree or should I just include the libraries and headers? Maybe along with a readme on how to rebuild the library if needed, with link to the svn repo or something? Should I supply debug libraries (with windows pdb files) or release libraries?

Is there any specific directory structure I should be following? At the moment I have:

npk
|---win32
|   |---bin
|   |   `npk.exe
|   `---libnpk.lib
|---linux
|   |---bin
|   |   `npk
|   `---libnpk.a
|---osx
|   |---bin
|   |   `
|   `
`---include
    |---npk.h
    |---npk_base.h
    |---npk_conf.h
    |---npk_dev.h
    `---npk_error.h

Also, I'm aware that I should only submit code which compiles and runs, I, however, don't have access to a Mac, so I cannot build libnpk myself, or test it. Is this acceptable? What are the recommended defines for not compiling untested code per platform?

Another question that's sort of libnpk specific, there is a flag for building libnpk in "dev mode" which allows writing to npk files. If Maratis editor is to publish packages from code, it will need to be built against a dev library. However, there's no need for Maratis player to be built against the larger library, should I therefore make 2 libraries to link to, a libnpk-dev for the editor and libnpk for the player? Or just link to the larger library for both?

407

(150 replies, posted in Engine)

I tested the font stuff quickly and it loads through the wrapper. So far so good. Everything seems pretty simple.
I made a test .npk for the SponzaFPS example containing maps/ and scripts/.

After work I'll have a go at loading it and pulling data from it smile

408

(150 replies, posted in Engine)

I was actually wondering if iOS was the problem. To be honest, I don't know exactly how iOS treats files in the IPA, what path it expects for them, of course, there's no reason why we should change directory on iOS at all, so we can ignore that, the package can be loaded relatively and all files within that will be using relative paths. Regardless, as you say, it can be changed later. I will strip the path down for packaged files and load them like that.

I will also take a look at the font loading again and make sure that it works properly.
Almost there.

409

(150 replies, posted in Engine)

So, do you want me to keep the paths absolute within the engine then and only get the relative path for package files?

If I change directory to the project directory (chdir(dir) on *nix, _chdir(dir) on Windows) I can load all files with relative paths, and then load editor files using something like Window's GetModuleFilename. Personally I think this is much neater because it will then allow project files to always be relative paths regardless.

Is it the .font files I'm looking for then? Can I just load them as a font in the editor? Or am I missing something?

410

(150 replies, posted in Engine)

Right, I put the code in for lua, and while debugging it, I realised something. I had for some reason been assuming that the paths I was being given were all relative to the project file. Silly me. Well, I have lua working now, but I have to somehow extract the relative path from it.
I think I have 2 choices. The first is to either change directories to the project directory and have all the data files attempt to load from there. The second is, when loading the files using the wrappers, strip the project directory off the beginning first, try to load that, then, if that fails, revert to the full path.

The second option seems a bit hacky. I'm just not sure where to begin with removing the need for absolute paths. I will have a quick looksee tonight, but I think I'm going to call it a night soon. I may have time to sort it during my lunch break tomorrow, which would leave tomorrow evening to try and write the MPackageFile stuff. Sounds like a challenge tongue

(also, I realise that noone is setting this deadline for me but myself)

EDIT: I really don't know what to do right now. The editor expects to be run from it's own directory, so looks for the gui elements within that directory. I'm not sure now if I should change that. Seems like a pretty big change in design :S

411

(150 replies, posted in Engine)

I sorted the BinFontLoader to use the wrapper functions and it compiles, but I can't find any examples which would use it, so I am unable to test it. Everything I can find uses TTF. Can you point me towards something so I can make sure it works?

Also. I just sorted the MSndFileLoader. I got a random track Jamendo recommended, converted it to ogg, tested it. It just worked.
Still only stdio, but through the wrapper. It's nice when things just work though.
The libsndfile interface for buffer reading is a little frustrating, requires a struct with function pointers to what it's going to use to read the data from. A little bulky, but hey, nice and flexible at least. I just remembered about lua, time to dig around with that smile

412

(150 replies, posted in Engine)

Sure, I had a quick look at it and it doesn't seem like it will be much of a problem.

for the wrappers, I've put them in MFileTools.{h,cpp} and named them M_fopen M_fclose etc. They return MFile* which currently is a base class that handles stdio, but I also have a MFileOpenHook interface which, if a hook is specified, will allow loading from wherever, as long as it returns an MFile* (or inherited class)

I have a skeleton MPackageManager in MEngine which, when initialised, registers it's own MPackageFileOpenHook. The MPackageFile's haven't been written yet, but the plan is to cycle through the loaded packages (even though I'm only going to load one for now, based on the project name) and look for entities in it with the name requested. If they all fall through, it will revert to the MFile behaviour.

I figured MEngine was the correct place to put the package manager as it didn't really contain core functionality, but file I/O itself did. Also, regarding the MFileOpenHook class, as the package manager is in, basically the layer above MCore, seemed sensible to have a generic callback class. Also, as a few people seem to have been talking about a web player, if this ever gets implemented, being able to replace MPackageFileOpenHook with MStreamFileOpenHook should be easy.

Just checking that this design is acceptable before I finish it off. It's still fairly easy to move it around if you're unhappy with anything. I am aiming to submit the basic package code on Saturday, along with instructions on how to create packages manually (as there won't be a publish button yet) and any other useful pieces of information. Then, if I can work out how MGui works (as of yet, I haven't managed to find any data as to how it displays...) then I can try to add the publish button, and make the package... possibly by the end of the weekend.

Actually, that's a point, do you have any preference as to how I specify which files go into a package? I can just add all the subdirectories of the project, but that would quite possibly add source files and raw data, depending on how people's projects are layed out. Maybe another xml file?

413

(150 replies, posted in Engine)

Ok, so I told a little lie. I didn't test the wrappers yesterday. I ended up talking to some potential artists for my game who wanted me to block out a room so they had something to get started with. So I blocked out a bar and then made a project which they could use to test and work on.

Anyway, the good news. I had time this lunch to test it. It works great and I have DevIL now loading through the wrappers! I guess there's not much point in packaging just the images, so I'll have a play around and get libsndfile to load through them too tonight. Then it's full steam ahead on pulling the data from the packages.

Which leads me onto the next point. What do you want to do about the meshes and levels and stuff? Should I get TinyXML to write them to the package for now? Or leave them outside the package and wait for binary files?

414

(150 replies, posted in Engine)

That's what I thought too. I was just playing with the idea of python build scripts because it would add to the flexibility. But I guess that if people have need to change the publishing system in any way, the chances are they will be willing to change code.

I don't think we need to write either. I guess if a file is attempted to open writable, I will try to load it from OS, which would solve the problem of save files as it wouldn't even consider putting them into the package.

I am enjoying working on this and getting to know the engine, so the commitment is no issue smile I will, however, have to start doing some more work on my own game project soon, otherwise I'll get caught up in this and my game won't move ahead wink I am going to try and get this done to an acceptable (albeit maybe quite basic) standard by this weekend, after that I will be focussing mostly on my game, just doing fixes to the package stuff when needed. Once my project is rolling along, I can see if there's anything else I can help with smile

Something I forgot to add to my source control then. Thanks

416

(150 replies, posted in Engine)

Research is done. Design is planned. Wrappers are now written.
Task list as follows:
- (probably for lunch during work tomorrow) Replace stdio fopen/fclose/fread/... with new wrappers and test.
- make a test npk manually and get it loaded into the app.
- search npk for files before attempting to load from OS
- ensure fopen, fseek, ftell, fread, fclose all work with package files (no writing using standard file I/O?)
- Add packaging functionality to MaratisEditor. Not sure if this should just add the whole directory.
- Put a publish button into MaratisEditor GUI and create a framework for publishing, including data packaging stage

I hope that I can get the basic reading functionality working by the end of the weekend. With regards to the publishing, is it best to do this in code, or by using python scripts do you think?

As it's new trick, Visual Studio has just decided to remove all the "filters"(/directories) in the solution explorer. Now I just have a huge mess of files. GAH! Ahh well. I will sort that later

I've been able to use scons on my work machine. I think it might be a limitation of VC++ Express that I can't use it at home.

I have something working at least. It's not important. I just thought I would write my findings in case other people had problems with it.

419

(150 replies, posted in Engine)

I will focus on getting the files loading through a wrapper first. I'd already mostly written the fopen patch, though. I'm going to try and complete it and test it on Windows and Linux, so it can be included in case someone does come across a case where it's needed. As you said, most good libs are now aware of it, but one that I couldn't get to do this is FBX. Much as there isn't really any need for FBX in Maratis, it shows that a large, widely used lib can still be backwards. I just thought, as the code is there, might make sense to have it as a fall back if someone needs it.

I've been arguing with getting Maratis to build with Visual Studio/C++ 2010. On my work PC (during lunch, of course) it will build using scons VC's 'cl'. At home I only have Visual C++ 2010 Express which doesn't seem to like running cl from the command line.

There appear to be 2 problems with whatever VC2010 does. The first thing is, the post build step for the project Maratis seems to be failing. I assume this would be that one of the steps is failing, possibly due to the file already being there and prompting to overwrite. I haven't really looked into this properly, I copied the files manually and removed the post build step.

The second thing is that VC2010 defaults to using $(TARGET) for debugging whatever you've compiled, whereas the linker saves to build\$(ProjectName)Debug.exe. I assume in previous versions $(TARGET) got updated to whatever the linker spits out. I have tried changing the Output Directory and Target Name options, which gets the debugger to start running the executable, but the actual executable doesn't seem to have been built properly.

Annoyingly, Visual Studio also doesn't seem to save the Debugger properties which is frustrating.
EDITI've been arguing with getting Maratis to build with Visual Studio/C++ 2010. On my work PC (during lunch, of course) it will build using scons VC's 'cl'. At home I only have Visual C++ 2010 Express which doesn't seem to like running cl from the command line.

There appear to be 2 problems with whatever VC2010 does. The first thing is, the post build step for the project Maratis seems to be failing. I assume this would be that one of the steps is failing, possibly due to the file already being there and prompting to overwrite. I haven't really looked into this properly, I copied the files manually and removed the post build step.

The second thing is that VC2010 defaults to using $(TARGET) for debugging whatever you've compiled, whereas the linker saves to build\$(ProjectName)Debug.exe. I assume in previous versions $(TARGET) got updated to whatever the linker spits out. I have tried changing the Output Directory and Target Name options, which gets the debugger to start running the executable, but the actual executable doesn't seem to have been built properly.

Annoyingly, Visual Studio also doesn't seem to save the Debugger properties which is frustrating.
EDIT: If I can get VC2010 to build and debug nicely, I will share my solution. At the moment, most of the changes have to be done every time sad

421

(24 replies, posted in Engine)

Loading screens.

Not sure how feasible this is. Not sure if loading needs to be forked, or if there just needs to be a loading callback. Also not entirely sure how the loading screen would get loaded. Obviously it would be loaded first and separately. Also not sure what kind of metric can be used to measure progress, either relative or absolute.

422

(150 replies, posted in Engine)

I fully understand. I just started working on it under the naive assumption that iOS had a sensible API wink recompiling libc for the sake of that does seem like a bad idea.

Writing the wrappers should be fairly easy and I will look at what lua needs.

EDIT: I've just walked home, thinking about this. I'm pretty sure that, with my luck, I will decide to use some library which will cause problems. So I will do the function wrappers first, and then add the patches, but wrap them in #ifdef M_PATCH_FILE_IO. If I get permission from my work to do some iOS stuff and I splash out to get myself a Mac to do some dev work on, I will look into creating a patched libc. But initially. everything will use wrappers and the patch code won't even be compiled (unless users are feeling brave and/or stupid).

EDIT EDIT: I was also thinking, not that you should replace it now, because you have your xml exporter etc, but have you looked at google's protobuf? It has some nice features including automatic binarisation.

423

(150 replies, posted in Engine)

If it uses fread, fopen etc, it will be intercepted by the overridden functions, which will check to see if any packages are loaded, if they are already (for multiple packages) it will check inside them, otherwise it will load from the normal OS filesystem. Should be fine smile

EDIT: I asked around a bit more. Seems that patching the function will only really work on desktop OSs. There might be some linker black magic that you can do to weakly link, so it won't complain when you try and link a custom lib with the same definitions as libc. Alternatively, you can ignore the iOS libc altogether and link in uclibc or glibc (both LGPL) with modified fopen/fread/fwrite/fclose functions and compiled against the iOS SDK.

424

(150 replies, posted in Engine)

anael wrote:

It's good to hear about new things, don't worry, Maratis is starting evolving by itself, and there is a lot of better programmer than me smile

Even if there are better programmers out there, you're the owner and architect. It's your vision.

I've been having a look around and I'm not sure than vm_write is available on the iOS SDK. Which is just great. People suggest that ptrace might work. If I had a Mac here I'd prod around in the iOS SDK and see what's available, but people are using the work Macs... From what I can see though, you should be able to modify the memory within your own application, as long as it doesn't do anything else to violate the developer agreement, should be allowed. I'll try and remember an apple ID so I can read through the developer agreement later.

EDIT: I had a look and it doesn't seem to allow it. Technically there might be some way to hack it to unlock the memory and write it manually, but that seems like an insanely stupid idea. That sucks. Hmmm. Technically you _could_ just not link in -lc and compile something like uclibc into Maratis without file I/O stuff, but that seems a lot like overkill. Plus I'm not sure if that would be allowed either.

425

(150 replies, posted in Engine)

I did a check and it's definitely available on Linux using ptrace
http://www.linuxmanpages.com/man2/ptrace.2.php
Having a quick google suggests that OS X might be slightly more problematic as Apple don't like people poking around with their things, but vm_write is apparently what I'm looking for, although, I don't have access to a Mac to play around with that.
http://web.mit.edu/darwin/src/modules/x … write.html

In general, patching the function is more difficult than simply wrapping it and would require extra debugging to ensure it's working properly (so you don't accidentally jump to a random area of memory) but once the patch is written, it shouldn't need to change, so you can then extend things, for example the multiple packages, also, every call to the function will then be routed through the patch, so even if you got a closed source library with an evil file loader, it would still probably use fopen, so you could still use the package to contain the data for that library.

I will see if I can find out if it's possible to do this for iOS and Android, I can't see why not. The issue is that things like WriteProcessMemory, ptrace and vm_write all allow access to other processes, but Apple have secured OS X by making some processes un-alterable. Then again, Apple are pedantic about a lot of things. I'll go a-hunting for information.