Re: package file request
Uploading as I type this. [Done]
Maratis + libnpk 59.9MB
Also supplied with npk files for SponzaFPS and YoFrankie (script and image only)
EDIT: UPLOADED!
Last edited by Nistur (2012-01-14 22:40:39)
Re: package file request
Wonderful,
I'll have a look at it tomorrow,
thanks !
Re: package file request
I just realised, I didn't update the VS2005 solution, and I even hunted down a copy of VC++2005 Express especially for that purpose!
I updated the solution and, instead of uploading so much again, I've just packaged the solution up, I had to include a load of libs that weren't included, which was strange :S maybe it's just my PC playing silly buggers with having 2 versions of VS installed
Re: package file request
I started looking at it, it is very nice, good work !
I like the way it is done and it's a lot of commitment, thank you very much !
Before sending it to svn I have to try on every platform to check that everything compile,
there is also the mac project and the iOS publishing project to update.
I might move MPackage manager in another place, as a rule there is no 3rdparty dependencies in MCore and MEngine,
so it will probably be a module like the actual contexts, that's just a small detail to keep a door open in case someone want to use another packaging system, like they can change the physics engine.
I'm really happy to see things evolving like that, I hope we'll continue in the same way, good energy, good sharing 
I have to go right now so I will start this afternoon,
see you later,
Anaël.
Re: package file request
I am going to start working on the package creation/publishing. For that, I realised that I need to have access to the
Re: package file request
Hi,
it looks like your post got a problem,
what do you need ?
Re: package file request
I have no idea why, but it happens quite often where the forum doesn't post my whole message. I usually catch it and edit but I was on the way out this morning too, so I didn't notice.
I was just saying that I'm starting with the publishing stuff now anyway, which involves adding the dev stuff to libnpk. While I'm doing that, I will move the package manager implementation to Maratis/MaratisPlayer and remove the dependancy for libnpk in MEngine. That will also allow for linking to different libraries for each executable.
I also want to try and work out what I can do about not keeping the packages in memory until the end of execution. I think simply loadPackage() loadLevel() unloadPackage() should work fine, especially for simple, trivial cases. However, if there are larger games made, it doesn't make sense at all to keep a whole package loaded the whole time. I think maybe, instead of calling unloadPackage, maybe have an offlinePackage which will read the file list, close the package, and just use the file list later, if more files are requested.
With regards to the code I wrote. I tried to make it as clear as possible, but it just meant to be a first pass, to get a feel for what people thought of the system. Now I know, I can continue
Re: package file request
Look, at some point I will need you to pause a bit so I can commit a version that compile everywhere, and maybe do some minor changes, for now I will wait, tell me when it can be done.
There is also a small thing to clean, there is two copies of zlib in 3rdparty, zlib was already used before and it's also in the npk source. We should try to have only one, but it's not urgent.
I will work on the mesh bin so.
Let me know how it goes 
bye,
Re: package file request
Your work on the mesh binaries was one reason why I wanted to show something, so it can be tested with the package wrappers. The file I/O stuff shouldn't change at all now.
I will sort the zlib stuff, I didn't really think to look whether it was duplicated, sorry.
Re: package file request
No problem, there is no hurry,
please take your time.
I'm using your M_fwrite for the bin, it's perfect,
the mesh file is a bit long to do, I'm not sure when to finish, but I'll work on it on maximum I can.
There is nothing complicated, just time to find.
Re: package file request
That's fine I enjoy doing this, so I'm not hurrying, I'm just excited
I'm glad the wrappers work fine 
Are you expecting to write directly into the package on publishing, or will you be writing the binary files out first, then package afterwards?
Re: package file request
My first idea was to write a temp file and then pack it, to not multiply the techniques, the other files are just imported.
I don't know, if it's simple to write directly to the pack why not, what do you think ?
Re: package file request
I don't know. I don't think that opening a writable file should be too much of a problem. I'm having a look at the npk API right now and I _should_ be able to just write a new entity from a buffer on closing the file. Can we do the same with the fonts?
Although, that raises another question about file extensions. Inside the package, I've been giving them the same entity name as the local file name from the project. If we then put a .font file in to replace the .ttf, we will need to either call it .ttf anyway (and thus maybe confuse people who might want to unpack the data...) or allow the font loader (and possibly the mesh loader) to change the extension to check for binary versions before the raw versions.
Re: package file request
Yes, I have the same taught about the file extension, I'm still not sure what is the best, the .font thing was a temporary test, no need to keep it.
The main problem is the level data, everything inside is linking to the original file.
- We can keep the same extension but it can be confusing and maybe create problem with freetype if it uses the extension.
- Or when converting the level to binary, we can change the extension of all files that has been converted (we can use a small database listing the modified files with their new name).
- Or we can add ".bin" to the original filenames when converting, and in the bin reader, add it to the filename argument.
Changing the extension is probably the best choice as it can be useful in the future to also convert textures in another format depending on the publishing target.
Re: package file request
Yes, I think that makes sense. Will you write the binarised file loading code then or should I try and add it myself?
I have managed to split up the package reading and writing code and it's now in Maratis/Common/MFileManager/MPackageManagerNPK.{cpp,h} whereas the interface (now called MPackageManagerContext) has remained in MEngine.
I am using a define which I can specify at compile time as to whether to add the writing code or not (M_PACKAGE_WRITABLE) and leaving empty functions for those which would require write access.
I'm just looking at how to publish the project. Do you want me to add an option into the File Menu, or maybe create a new menu, Project for example?
EDIT: I just had a thought, if all files to be added to the package had a .bin extension added, then it would make packaging files easy, just search for all files with .bin and add them For some file types, lua scripts for example, it would probably just mean renaming the file someScript.lua.bin
Last edited by Nistur (2012-01-15 17:24:42)
Re: package file request
- If you want to try doing the reading of the bin, I can send you the code of the writing when I finished it and you can tell me if you fell doing it. As I also need to do the writing of the level bin it can help.
- Good to know for the split of the lib
- For the publish button, it can go in the File menu (File > Publish Project).
Later if we need publishing options, we'll just open a new window with additional buttons/menus.
- For the generalized .bin, mh why not... let's think about it, I'm not sure it's so important to do it for all files.
> the bin files need to be also readable outside a package as a normal file.
Like keeping the original extension, it will be less risky than changing the extensions, as users can load data from a c++ plugin,
if they use something like "level->loadFont("fonts/arial.ttf");" it won't be possible to just change extensions as the filename will be in their code.
Just for the previous reason, changing the extension is in fact not an option.
Or we keep the same extension, or we add something after the filename for the modified files or for all the files.
Re: package file request
I've started the publishing work, For now I'm just hard coding a publish event in, but the idea is to be able to specify multiple events, possibly allow events to be added from plugins later (although I don't think I'll be writing this yet). I need to decide how to order the queue, for example if someone writes an nsis installer plugin which automagically creates the installer, we don't want that running before we've packaged the data, moved everything to it's proper location etc. Maybe a list of lists. Each event can specify a priority (say, 1..10) and it sits in it's own list, if we get multiple with the same priority, it _shouldn't_ matter for the majority, if it does, people just need to change the priority. Possibly with an in-engine menu at some point. Sorry. Getting too carried away.
I'm also still not entirely sure what you would want with the file extension.
Re: package file request
The event system is a good idea, it can be useful for some specific production to customize the publishing process.
Lets not touch the file extension, I don't have a better idea that is really safe. We mainly build a package to hide the original files and I checked that all the loaders uses only the file header to recognize the file type.
I checked and Freetype can also use a buffer to read a font. MBinFontLoader is not so useful for desktop publishing.
- the step one is to convert the xml, meshs (in meshs/) and levels (in levels/) to binary in a temporary folder.
- pack the bin files + "scripts/" "fonts/" "maps/" "shaders/" "sounds/"
- delete the temp files
- copy the mproj file and the game plugin (if there is one)
PS : We should also update some functions in MCore to use your fwrite wrapper :
- isFileExist in MFileTool
- readTextFile in MStringTool
Re: package file request
If we're not changing the extension then things get quite simple to load. That's good
I haven't considered shaders, what will need to be done with them to load from the wrapper?
You _should_ be able to get around converting the xml files without temporary files. If we make it part of the publishing process, then the MBinariseMeshEvent can do something like:
- set package manager to writable (otherwise will write to OS files)
- loop through meshes
- create/load file
- write data
- close file
- set package manager to non-writable
That way we don't have to deal with cleaning up after each step.
With regards to the final step of the publishing process, that shouldn't be too difficult to add as a late publish step
I've just looked up those 2 functions, changing them will be no problem, I'll do that now. I also see that readTextFile is how you load shaders. Nice. Another job done
Re: package file request
Yes I was going to say that shaders uses readTextFile in MStringTool
Re: package file request
I'm at work now, so I can't do much until lunch (I have to do programming that will earn me pennies for food ) but last night I wrote the publish event system and tested it. I'm going to have to put the MPublishEvent interface somewhere that plugins could potentially inherit from, but for now it's in Maratis/Editor/MPublish.
I also sorted the 2 functions that you suggested, so now I can add shaders to the list of things which should "just work" from the package
Re: package file request
I almost have the publishing done. I now have 10 separate publish steps, each of which can have multiple events, but not guaranteed as to the order. I have assumed the following steps
0 - clear the published/ directory. Shouldn't be used for anything else [complete]
1-4 - prepare game data. For game/plugins to implement
5 - package game data [complete]
6-8 prepare executable and copy, along with plugins etc [complete]
9 - run install scripts. For plugins to implement
I don't think that there should be any reason why this should be set in stone, but it seems like a reasonable guide for what happens when. I will try to work out writing directly to the package file, but that should be fairly simple. Will I be able to do something like Maratis::getInstance()->writeLevelBin() or something?
EDIT: For now I've just copied the contents of meshs/ and levels/ to the published/ directory and renamed the player executable to the projectname(.exe) I have yet to set up scons to split npk up, but I am confident that I can do that pretty quickly now I know how the build system is set up. I also want to test it quickly on Linux. As for OS X, I could, potentially, use one of the work Mac Mini's, if I have time tomorrow. Otherwise, I'll just upload it all again and let other people test it
Three things which I have come across. The first is something I've already mentioned, having the player renamed, the data packaged all makes for a more professional install, but with the .mproj file still in plaintext, and having to pass it as a parameter to the player executable makes it a little irksome. Maybe, as part of a publish step, we can embed the (binarised?) mproj? As the project is a fixed size (2 strings) we could potentially allocate space for it in the player executable and write the data in during the publishing?
Secondly, maybe allow to change icons for the executable, possibly just look if there are icons files in icons/ and again, write them into the player executable?
Finally, I don't think there's much reason why the "engine" can't be distributed with the player (and libraries) for all OSs/devices. Then from the editor (which would have to obviously be OS specific) when you click publish, it would ask you what you want to publish for (tick boxes?) and then run through all relevant publishing steps
None of those 3 are anything major, just something to think about I might even look into the embedded project, if I haven't got any art back to start work with on my project
Last edited by Nistur (2012-01-17 00:08:28)
Re: package file request
The mproj file need to be a bin for sure,
after, where to put it can be optional.
A simple thing we can do first is to make MaratisPlayer search for a local mproj, to not have to use a command line if there is a project in the same folder.
Or yes, the mproj could be embedded, I personally don't know how to do that (except for mac, to put it inside an app).
Re: package file request
I had a look at embedding data before and, at compile time, it's simple. With GCC you can use objcopy to create an object with the correct symbols to link against. Of course this isn't really realistic as we want to do it when publishing.
There is the possibility of doing something like this:
http://www.daniweb.com/software-develop … ads/228424
I think the easiest way to do this is to make a constant somewhere within the executable, which can be 256 space characters, which can be searched for and replaced. Within the player, it can then first check to see if there's any data there (other than the spaces), then fall back to looking within the current directory for any files ending in .mproj
My USB HDD was playing silly buggers this morning, and I haven't checked what I've done recently into my git repository, so I might not be able to do the tests I was planning to today. I might have a fiddle with the project data and then merge it when I get home tonight.
Re: package file request
Ahh damn, I have found an issue with how I set up the package stuff. As it's in Maratis/Common, it will just be compiled once, which means it will expect to be compiled once and then linked to everything. When I did it in VS, the common filters were copy/pasted into different projects, which means I could specify different #defines at build time which would remove chunks of code, allowing for me to not compile against the dev code when I didn't need to. I'm pretty sure that I can fix this at some point, but for now, I think I'm just going to get it working with scons and one lib. Shouldn't be too bad I think.