126 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

Hi, yes we need to convert all mesh in the mesh directory and sub-directory (i added an option to scan recursive in readDirectory, it returns global filenames if recursive)

You can create a temporary level if you don't want to touch the one in memory and load file by file :
Something like (doing by head, code is not 100% sure) :


MLevel * currentLevel = engine->getLevel();
MLevel * tempLevel = new MLevel();
engine->setLevel(tempLevel);

MMesh * mesh = MMesh::getNew();

(for all meshs)
{
    mesh->clear();
    if(engine->getMeshLoader()->loadData(filename, mesh))
        exportMeshBin(filenameDest, mesh);
    tempLevel->clear();

    (same for armatureAnim, texturesAnim and MaterialsAnim files)
}

mesh->destroy();
engine->setLevel(currentLevel);
SAFE_DELETE(tempLevel);


It will unfortunately load textures also for now,
but I will add an option to the xml mesh loader this week-end to not do it

Re: package file request

Oh, does the mesh loader automatically add the loaded mesh to the level? Ok, Well, I would have made a big mess if I wouldn't have realised that tongue

I'll do the mesh saving at lunch today quickly then smile Doesn't look like much work at all smile

128 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

smile cool

the loader doesn't add the mesh to the level, but it adds the textures.
The main reason to create another level was more to be able to clean it after loading each mesh (to not use too much memory).

And by using engine->getMeshLoader(), if someone adds a custom mesh loader it will still be able to convert it to bin.

Re: package file request

Ok, cool smile I think I can work with that smile Thanks smile

EDIT: I had to mediate a family discussion at lunch time, so I didn't get time to finish this, however, I did manage to get the majority of the groundwork done to get this set up.

Ok if I submit this myself when this is done? I can't see why it would cause any problems with other OSs I can't test.

Last edited by Nistur (2012-01-25 15:43:38)

130 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

yes, no problem, you can submit directly

Re: package file request

Cool, well, I'm going to have to fall back to a slightly less than favourable method for writing to the package, unfortunately npk doesn't allow reading from a buffer (go figure...) so I can't just give it the binarised data, I will need to save it out to a temporary file first, I guess there's no reason in keeping the temporary files, so shall they just be deleted after publishing? Or do you see some reason why we might want to keep them around?

132 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

there is no problem with making a temp, we don't need to keep it after, it can be deleted after send to the package.

Re: package file request

I think I've very nearly almost got the mesh stuff working. For some reason, however, when I call destroy on a MWritablePackageFile, it decides to crash somewhere deep within the windows system, somehow tracing back to DispatchMessage. I guess I've probably trashed the stack somehow, but I'm not overly sure how.  Gah! So close!

EDIT: ignore that. Just me overflowing a buffer... silly me...

Last edited by Nistur (2012-01-26 21:34:17)

Re: package file request

Tadaa! I have it saving the mesh files, and material files, to the npk!
What is strange is, I accidentally clicked on the published executable, without thinking that I would have had to add the bin loaders...
It worked...
I...
I don't know why it would just work...
Did you make it do magic Anael? I can't see the bin loaders added in the player at all.
Oh well. I will add them and do some testing at lunch tomorrow, then submit it probably. Am I ok submitting the VS2010 solution too?

I had a few issues with the "Demos" project, which for some reason kept giving me corrupt anim refs, but I'll see if I can figure out what that is.

135 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

cool smile thanks !
ho, I added the meshBinLoader the last time, that's why it loaded smile
sure you can submit the VS2010, just put it in a directory named VisualStudio2010

Re: package file request

I saw that it was added to Maratis, but I didn't see it being added to MaratisPlayer. Oh well smile

I think I've managed to make it add .mesh .maa and .mta files, but there didn't seem to be a MMaterialAnim::getNew() so I wasn't able to add .mma files. How can I create a new one to write temporary data to?

Re: package file request

Ok, so I've tested it with Visual Studio builds and Windows scons builds and it works fine on almost all the examples, just the Demos project seems to be playing up a little bit. I'm not sure what is special about it, but I cannot export the binary mesh files as the anim refs seem to be corrupt for about half of the meshs. I'm not sure if this is stable enough to submit yet as I have absolutely no idea why it is doing this.

Also, I stuck the 50.8MB model through the exporter, I'm sorry to say that MaratisPlayer opened both binary version and xml version in ~9s. Interestingly, though, the bin is 979KB

138 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

Hi, for the materials anims, it is MMaterialsAnim (with a 's') :

MMaterialsAnim * materialsAnim = MMaterialsAnim::getNew();
...
if(engine->getMaterialsAnimLoader()->loadData(filename, materialsAnim))
        exportMaterialsAnimBin(filenameDest, materialsAnim);

If the loading of your model is 9s for both, it means it's still loading the xml, it cannot be the same time for the bin...
If it's compiling you can still commit it, I can have a look when I have time (today and this week-end is the Angoulême comics festival so I'm not sure when).

cheers,
Anaël.

Re: package file request

I'll try and sort the materials and then submit tonight, maybe after checking what's happening with the loading and why it's taking the same time for both xml and binary versions. Probably after a round of filling in comments everywhere smile

Also, what do you think about the method for calculating a teakey by hashing a passphrase? If it's acceptable, where would you like the hash function kept, and do you have a preference as to which hash function? There are a load here with downloads here. Originally I just chose one of those to test it.

Is there anything else you would like to be added this weekend that comes to mind? I'm going quickly add functionality to change the package extension, if it's desired. Then I think I'm done with it smile

EDIT: It seems that for some reason, the published executable is still referencing the original project location, so yes, it was still picking up the original files. I have no idea how this happened, but shouldn't be too hard to fix.

Last edited by Nistur (2012-01-27 20:06:12)

Re: package file request

I don't know whether to submit it tonight. The mesh file loading was a simple fix, it was a combination of not stripping the project filename to a local filename when embedding it, and embedding it with .exe (it should be .mproj for the embedding to work properly)

The problem is now, I still can't reliably export bin files, and I also can't reliably load them. I guess I will tidy up what I can, then comment out the section which doesn't work, or wrap it in #ifdef, so you can have a look to see if I'm doing anything silly.

141 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

ok, thank you for the effort, if you prefer to not commit it, you can just send me the files and I will look at it,
i also not tested the bin mesh so much, I maybe left a bug or something...

about the hash function, i never dealed with that, what is it for ? to encrypt a pass ?
will it be compatible with potentially another pack system ?
for the where to put it, it could be a new file in MCore I guess, like MHashTools.h / cpp

Re: package file request

I will submit it in a few minutes, I've been tidying up. I don't really understand the binary stuff myself yet, so if you can have a quick look then it will probably be solved much quicker. It's also quite possible that I've done something stupid which I haven't seen. I will take another look at it tomorrow.

Errm, the has functions that I linked to are similar to md5sums, although for a smaller scale. They are usually a trade off between speed and potential clashes. The main use for them is comparing and storing strings, which are large and slow to work with. Most hash functions can hash arbitrary length strings into 32 bits with few clashes. Makes them useful for creating keys for std::maps wink

EDIT: the code is submitted. I hope I haven't broken anything tongue

Last edited by Nistur (2012-01-28 01:02:54)

143 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

Hi, thanks !
I will have a look

Re: package file request

No problem, I just got my linux install finally set up properly, so I can do development here smile The problem is that I built Maratis from svn and it crashed when trying to load the WaterGameDemo, on libsndfile. So I finally had a dig around as to how to build a debug version using scons.

Turns out, for whatever reason, I didn't put a null pointer check on the sound loader... oooops! Fix incoming shortly!

Also, is there any reason why libsndfile might not be able to read a .wav? :S

Last edited by Nistur (2012-01-28 16:13:29)

145 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

in MWritablePackageFile I don't see any delete of the buffer (which is big) :
m_buffer = new char[WRITABLE_BUFFER_SIZE];

I'm starting checking tonight, I'll try to give a review tomorrow or late in the night,
good evening,

Anaël.

Re: package file request

You had me a little worried there for a second, m_buffer is a member of the MPackageFile and _should_ be cleaned up by it at the end of MWritablePackageFile::close (return MPackageFile::close(); )

And yeah, I think I made it about 10MB, which is far too big, but I'm not entirely sure what the best idea for a dynamic size buffer would be when we are expecting to keep writing to it.

147 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

ok perfect smile I didn't see it

148 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

Hi,

I finished the bin mesh publishing, I tested on mac and windows only,
and now the mac version put everything in the app.

There was some problem with the bin mesh :
- The anim data ref was not put to NULL when calling mesh->clean() so as we was re-using it, the mesh bin export was accessing removed refs.
- I forgot something in the anim mesh bin loader, when reading the animation keys (I forgot to init the key type... oups...).

... I spend some hours yesterday to find out hmm

I removed writing to package, I don't think it is necessary, and it was a bit confusing. It's better to just create temp files manually and add the file to the package, I added an entityName argument to it.

Normally it is working, but we should check on linux to be sure.
Only thing remaining are the level files... I'll try this week if I can...

Re: package file request

That's great, I'll sync and check it tonight on Linux smile Great to hear that you found the problem.

I had a look at what you did to add the mesh files and yes, I think that's much cleaner smile

I unfortunately can't do anything at lunch as some recruiter is going to call, and then I want to quickly try to export, import, load etc stuff, maybe give you some pretty things for the showcase courtesy of my friend Umar tongue Using some more of his existing work to make sure I have an art pipeline I understand

150 Reply by anael

  • anael
  • Administrator
  • Offline

Re: package file request

So finally I made a change to tinyXml to read from a buffer,
as the xml levels are not long to load, and might change a lot in the future, it should be ok.

The packing system is normally finished ! It just need some testing, specially on Linux.

cheers smile