Meta-modding question: Mesh and texture file locations

[syscrusher]

I am building a mod that will use multiple resource types, and I am about to completely reorg my development environment to correct the mistakes I made last fall as a rank beginner. I figure the time to do this is now, before I even let my (large) mod get as far as beta testing.

Resource types include:

1. Vanilla game resources -- these are not an issue and are mentioned here only for completeness
   
2. COBL and TESA Clutter resources -- these will use their "standard" .esm and .esp resources and paths, so not an issue here
   
3. My custom resources, including meshes and textures
   
4. Downloaded resource mesh-only objects (that is, they use only vanilla textures)
   
5. Downloaded mesh-plus-texture resources
   

Nothing here is different from everyone else's mods -- I just made the list in order to frame the discussion. I am looking at the best way to organize each of the last three types. Influencing criteria include:

1. Avoiding conflict with other mods that use the same meshes or textures
   
2. Ease of packaging and maintenance for me
   
3. Ease of installation and removal for the user
   
4. Reusability of my resource masters in future projects
   

What I plan to do is to release my mod as a multi-component package with the following files:

1. One BSA file containing all the raw resources that are not in either Bethesda, COBL, or TESA Clutter libraries. The COBL and TESA libraries will be documented as a dependency for my mod, and left to the user to install on their own.
   
2. One ESM file containing all the object types that I view as *not* being specific to this mod. These are objects I create (including some reusable scripts, leveled lists, etc.), but they may rely on meshes and textures that are from other people's resources or from the vanilla game.
   
3. One or more ESP files containing the things specific to my mod.
   
4. Documentation (not particularly under discussion in this thread, but I recognize its importance).
   

The overall goal is that there will be no "stray" meshes or textures outside the BSA.

The intent is that I will provide the BSA+ESM as one downloadable package, and the mod itself as another. The idea is that future mods from me can rely on people having my resource compendium. I intend to follow the precedent of OBSE and guarantee by design that the BSA+ESM package will be 100% upward compatible, so that people downloading a newer version of it can just overwrite the old one. The first of my mods they download will be large, but subsequent ones will be much smaller if they already have my resource bundle.

Again, nothing new here. This is common practice in the software industry. In Windows, for example, you install the .NET "framework" once and it works for hundreds of .NET applications. Hopefully other modders will find my bundle useful and may start building around it, but that's up to the community.

Now, here's the tricky part, where I am asking for community discussion and input: How do I organize the directories (folders) for the underlying resources, and how do I best name my objects' editor IDs. This is going to be a complex discussion, I think, so let me lay out the thoughts I have so far, and ask for comments.

NOTE: In the discussion, please assume that the Oblivion\Data directory is "above" all the other paths discussed.

1. My TESA and Nexus login name is "Syscrusher", but that's kind of long to type as a prefix for every single object (and as a folder path component). Should I abbreviate it as something shorter? "Sys" perhaps, or "SC", which has the added bonus of also being my real-world initials? Or is it very strongly recommended-practice to use the full login name in case someone else happens to have the same initials? For brevity in *this* discussion, I'm going to assume I decide to use "SC".
   
2. For my own fully-custom resources, I will put the meshes in Meshes\SC\(something) and the textures in Textures\SC\(something) so that mine are all together. The (something) part pretty much follows Bethesda's convention, for example a building mesh is in Textures\SC\architecture\foo.nif. When I create the objects, I will take the default editor IDs because they'll automatically begin with my own prefix, and thus are unlikely to conflict with other mods. This seems to me to be pretty much a no-brainer, but did I miss anything? (Hmmmm...the "SC" prefix is less unique than either "Sys" or "Syscrusher", so maybe that argues for using one of the longer names in my folder path...)
   
3. Here's the hard one: What do I do about using other people's resources? Do I keep their directory paths intact, letting my BSA possibly be redundant with someone else's installed version of their mesh (but that is usually okay)? Do I keep their *texture* paths intact but move their mesh directory to be a subfolder under my own? For example, MEO puts all his resources under a "MEO" top directory (very nice). If I keep it that way in my BSA and ESM, though, the default editor IDs will be MEOfooBarBaz, and then my mod conflicts (possibly benignly, but still conflicts) with any other mod that uses the same defaulted editor ID. On one hand, that could lead to installation warning/error messages. On the other hand, it may improve performance for the user by having less total objects in their game.
   
4. If I keep the external resource's directory paths intact, should I prefix all the object IDs with my prefix to make them more unique? In the above example, then, I would override the default editor ID to make my object "SC-MEOfooBarBaz". Downside: More objects in game. Upside: Less conflict.
   
5. If I do move the external resource to my own directory, do I also do the same with its textures? That could mean a *lot* of work in NifSkope changing texture paths.
   
6. There are a few resources out there where the author specifically says they don't want the directory path changed. This complicates things further. If I keep their directory path intact but put it *inside* mine, rather than at the top level, have I complied with the spirit of that license provision? Is there a community consensus on that issue?
   

I appreciate any comments on this; I'm getting to the point where I have several hundred custom objects. I actually am willing at this point to rename/move every single one of them, one at a time if I must, but I want to solve this question and move on before I go much further with this large mod.

Thanks for any input.

Syscrusher

Edited April 26, 2011 by syscrusher

[syscrusher]

By the way, what I have right *now* is a mishmash of all of the possible approaches listed above -- now you know why I am wanting to reorg this, even if it means a lot of work.

[Lady_Nerevar]

I don't think there is an accepted set of rules, but I'll comment with how I do it.

1)I abbreviate in object prefixes but keep folder names full (so textures/syscrusher/SCtexture.dds).

3)Check the readme. MEO wants to have his resources kept in the MEO/ folders to get rid of redundancy, for instance. I like to move everything to my custom folders (meshes/syscrusher/...), simply because its cleaner in the archive and in the CS, and therefore easier to use and pack. I don't use a ton of resources in my mods, and therefore I don't think having a few duplicates is a big deal.

4)I do, and I would. Especially if its nonstatic meshes, the properties of which could differ from other people's versions.

5)I do and I would. Check out texturereplacer, a comandline tool to edit texture paths.

6)Can't move it at all.

[syscrusher]

I don't think there is an accepted set of rules, but I'll comment with how I do it.

1)I abbreviate in object prefixes but keep folder names full (so textures/syscrusher/SCtexture.dds).

3)Check the readme. MEO wants to have his resources kept in the MEO/ folders to get rid of redundancy, for instance. I like to move everything to my custom folders (meshes/syscrusher/...), simply because its cleaner in the archive and in the CS, and therefore easier to use and pack. I don't use a ton of resources in my mods, and therefore I don't think having a few duplicates is a big deal.

4)I do, and I would. Especially if its nonstatic meshes, the properties of which could differ from other people's versions.

5)I do and I would. Check out texturereplacer, a comandline tool to edit texture paths.

6)Can't move it at all.

Thanks, Lady_N. That is an especially good point about the nonstatic meshes; I forgot that they are a tougher situation than the statics.

[DsoS]

I don't know if this would help, but I kinda do what Bethesda did with their folder layouts, such as...

Meshes\DSoS\ModName\Architecture\LowerClass\aaDSoS<Mod Initials>MyMesh.nif

Meshes\DSoS\ModName\Clutter\MiscItem\aaDSoS<Mod Initial>QuillInkWell.nif

I've been thinking about dropping the "oS" off and going with "aaDS" only, some filenames get really long and windows doesn't like long file/folder names too well.

or if its a Modders Resource (such as Meo's Ayleidware)

Meshes\DSoS\ModName\Clutter\MiscItem\MEO\Ayleidware\aaDSoS<Mod Initial>AyleidSpoon.nif

(For this the only reason why I placed my name in the nif filename is to keep everything clean, so instead of it being out of order alphabetically, it would be in alphabetically order.) (Although I didn't do this on Ruins of Malamath, I didn't think about it until after I released it, sorry Meo!)

As for the Textures (if required), I follow the same layout as above but with Textures, such as...

Textures\DSoS\ModName\Architecture\LowerClass\aaDSoS<Mod Initials>MyMesh.DDS

Textures\DSoS\ModName\Clutter\MiscItem\MEO\Ayleidware\aaDSoS<Mod Initial>AyleidSpoon.nif

I do the textures this way so its easier for me to keep up with the file names for both meshes and textures, so if I need to change a texture, all I have to do is follow the same folder path in the texture folder.

----

Hopefully I haven't confused you about my odd OCD folder paths

Good Luck

---

Btw if you need to change texture paths, I have a Tutorial here to change the texture paths a little faster/easier than doing it through NifSkope, its step by step and should be easy to follow.

[syscrusher]

Thanks for the added post, DsoS. You and Lady_N raise an interesting issue: Organize resources by author, or organize resources by type (architecture, clutter, etc.)?

I also am thinking about the distinction of "really prolific resource creators" like MEO, Stroti, and Korana (for instance) versus the ones where the author has created a smaller number of meshes. I use a *lot* of resources in my mod, but I have taken the time to scour the downloads to find the ones I really like and plan to use multiple times, so I want to really do this right so I have a reusable library I can use as a master for future mods.

With something like MEO's static dishes, a huge resource, it probably does make sense to keep it in his directory to minimize redunancy and RAM footprint for the runtime environment. But is that a special case, or do I do that for all resources I didn't create, for consistency? And the other waters-muddying factor is that a few resource creators do really dumb things with really good resources, like putting their NIF at the top level meshes directory. Those I will definitely move, no doubt.

DsoS, your tutorial on using TES4edit to move the mesh directories will save me a lot of time -- thanks! No matter what I finally decide, I'm going to be moving a lot of meshes. I'm also going to look at TES4gecko to see if it might have a batch-mode equivalent, that matches on directory path patterns. If I find that TES4gecko can do that, or some other tool, I'll let you know so you can mention it in the tutorial.

I wish there was a tool to create objects from a directory of meshes, with specified attributes. For instance, "take all the meshes in this directory, and create each one as a static object with the default name prefixed by 'Syscrusher-'" would be really useful.

Keeping the other resources in their own directory, rather than in mine, does offer one other advantage beyond removing redundancy: It makes it really easy for me to know which resources are truly *my* work, versus the ones that are others' work but are used in my mod.

It's clear that there are about ten million ways to do this, and they all have their advantages and drawbacks.

[syscrusher]

...a few resource creators do really dumb things with really good resources, like putting their NIF at the top level meshes directory.

That was unkind. What I should have said was, "Sometimes a talented modeler makes suboptimal choices about directory structures for their meshes."

[donnato]

That was unkind. What I should have said was, "Sometimes a talented modeler makes suboptimal choices about directory structures for their meshes."

Guilty as charged. I`m what you might describe as "file challenged" but I think ...unkind or not you had it right the first time. A poorly organized/constructed file, when DLed by someone like myself can really gum things up and make untangling things a real beast. I have DLed mods that ave almost installed themselves...and ones that I had to hunt down and disect to their component parts or just wade through the swarms of flying wtf boxes and purple meshes. Yes I also tend to overcomplicate things but to be fair alot of problems were caused by my not knowing better. I`ve since learned alot and am now constructing my folders more carefully.

This thread will help to that end. Thank you for posting it.

[donnato]

That was unkind. What I should have said was, "Sometimes a talented modeler makes suboptimal choices about directory structures for their meshes."

Guilty as charged. I`m what you might describe as "file challenged" but I think ...unkind or not you had it right the first time. A poorly organized/constructed file, when DLed by someone like myself can really gum things up and make untangling things a real beast. I have DLed mods that ave almost installed themselves...and ones that I had to hunt down and disect to their component parts or just wade through the swarms of flying wtf boxes and purple meshes. Yes I also tend to overcomplicate things but to be fair alot of problems were caused by my not knowing better. I`ve since learned alot and am now constructing my folders more carefully.

This thread will help to that end. Thank you for posting it.

[syscrusher]

My pet peeve, in modding and elsewhere... Why, oh why, do Windows and Mac users insist on putting spaces and quotation marks in file and directory names? Those make it almost impossible, or at least severely inconvenient, to work with the files from the command line. I recommend "PsychosBrilliantQuestMod.esp" rather than "Psycho's Brilliant Quest Mod.esp".

[donnato]

Wow, ok I`d have never thought of that. Good tip Sysc.

[AndalayBay]

We have a lot of resources from all different areas in the CS. We found when doing armour (or weapons? can't remember) that Oblivion is very picky about the folder structure. Rather than having some resources under our mod name, we've adopted Beth's structure for the top level, with our own directory under that.

So for example: meshes\architecture\DBC... etc.

This is a little less convenient because we can't find all our resources in one directory, but it is organized nicely in the CS. It also allows us to be consistent whether we're dealing with a creature, a piece of furniture or statics. We don't use author names because we have a large team. So everything is organized under DBC, which is the acronym for the mod.

[syscrusher]

We have a lot of resources from all different areas in the CS. We found when doing armour (or weapons? can't remember) that Oblivion is very picky about the folder structure. Rather than having some resources under our mod name, we've adopted Beth's structure for the top level, with our own directory under that.

So for example: meshes\architecture\DBC... etc.

This is a little less convenient because we can't find all our resources in one directory, but it is organized nicely in the CS. It also allows us to be consistent whether we're dealing with a creature, a piece of furniture or statics. We don't use author names because we have a large team. So everything is organized under DBC, which is the acronym for the mod.

That bit about the CS being picky for mesh locations of armor and weapons is good to know. I have only made one weapon so far, and honestly offhand I don't remember where I put the mesh, but it worked okay. :-) I'll keep that tip in mind, though, because I don't want to get bitten later. I may end up putting all my non-combat meshes under meshes/Syscrusher and making the weapons and armor a special case the way you suggest. Luckily, as a modder that's not my area of interest, so I should have very few of those. (Not knocking it...I appreciate the work of others, just not interested in making those items myself.)

[Arthmoor]

#1,2. For things I release with stuff I've done, I follow the convention of:

Meshes\Arthmoor\<modname>\etc...

Textures\Arthmoor\<modname>\etc...

Sound\fx\Arthmoor\<modname>\etc...

I also have a miscellaneous bunch of stuff I use in more than one mod that get stored in

Meshes\Arthmoor\Misc\etc... or whatever is fitting to the particular thing.

#3. For other peoples' resources, I file them into the specific mod they're being used for, even if that creates redundancies later. I NEVER files those under my miscellaneous stuff because I don't want to chance dragging them in and forgetting where they came from.

#4. If it will avoid conflicts, yes, altering the editor ID of the imported model is advised. I usually rename the files prior to import though. To each his own there.

#5. Yep, lots of work if you weren't already following this convention. I've been down that road already and its no fun. I think in the end though it helps keep stuff organized better.

#6. If the resource license explicitly forbids changing the storage paths, don't do it. Though you can always ask the author and see if they'll make an exception.

For my stuff, even though I've long ago reached a point where an ESM and general resource pack would make sense, I haven't done it yet. That's another big pile of work that would need doing to transfer all the used forms to a new ESM file. It would also mean possibly breaking things for people who already have the mods. I may yet do this as more mods get released because down the road I may also want them interacting with each other and the only OTHER viable method for resolving that is a mass merger of several things - which is absolutely out of the question since Gecko WILL change the formIDs on anything that would clash.

Fortunately I realized I needed better organization before I had thousands of things to keep track of. I don't envy you the task of updating that much work. I guess in my case it paid off not having clue 1 how to work Blender and what little I made myself is just retextured vanilla or cheap nifskope hackjobs

Also, I've never had the CS complain about weapons and armors not being in the default locations. Creatures either as long as they're fully self contained and don't need the vanilla skeleton files.

[AndalayBay]

... snipped...

Also, I've never had the CS complain about weapons and armors not being in the default locations. Creatures either as long as they're fully self contained and don't need the vanilla skeleton files.

Just to clarify: it doesn't complain, it just ignores you To be more precise, it will create a new weapons folder under weapons and it makes it hard to find stuff. It also looks really odd and the path won't match what you have as the mesh path, so it gets really confusing.

And also I don't use any modder's resource in which they want their file structure maintained. If that's specified in the license, I just don't use the resource and will build my own instead. Once I pull something into my mod, I want to adhere to my file structure, whatever that may be.

[Arthmoor]

Odd, because when I did the ones for Feldscar it didn't do that to me.

[Khettienna]

I'm with Arthmoor on this. I see no reason to not put my resources in unique umbrella folders. Yes, it's more work for me, but it pays for itself in that I never have to worry about other resources overwriting mine or mine overwriting others.

Some resource makers do ask that you keep the directory structure intact. This is in case they improve the resource later, so the improvements will then show in all mods that use the resource. However, if you are going to change the resource in any way, you should still move it to your own folder so (once again) undesired overwrites will not occur.

As for spaces and punctuation, it doesn't really matter, even in the command line. You can put quotes around any path string that contains spaces or the like, and it will work fine. It is better to have the readability for 99% of users who wouldn't touch the command line with a ten-foot pole.

If you are really dedicated to having others enjoy your mods, you have to consider what will be easiest for the majority, rather than what is easiest for you personally.

I feel a tut coming on...

[AndalayBay]

Odd, because when I did the ones for Feldscar it didn't do that to me.

It might have been armour then. It was really strange because we lost part of the path on one of them and not the other. We just decided to make everything consistent across the board and as I said, now stuff shows up really nicely in the CS. Bit of a pain from a file point of view, because we have to go into all the directories, but nice to work with during development. I use TES4Files to package up my mods and it grabs all the stuff for me anyway.

[Arthmoor]

Didn't have any issue with armors either. *shrug*

[syscrusher]

Some resource makers do ask that you keep the directory structure intact. This is in case they improve the resource later, so the improvements will then show in all mods that use the resource. However, if you are going to change the resource in any way, you should still move it to your own folder so (once again) undesired overwrites will not occur.

That last bit I'm already doing, and noting the original file's source so I can credit the originator even if my version is changed (e.g., "giant snow-gorilla mesh created by Khettienna, edited by me for this mod").

As for spaces and punctuation, it doesn't really matter, even in the command line. You can put quotes around any path string that contains spaces or the like, and it will work fine. It is better to have the readability for 99% of users who wouldn't touch the command line with a ten-foot pole.

The two are not mutually exclusive. "Meshes/Syscrusher/architecture/HouseUpper01Interior.nif" or ".../House-Upper-01-interior.nif" is just as readable as ".../house upper 01 interior.nif". Where spaces *really* become a problem is if you have them in easily-overlooked places, or there are doubles. For example, try correctly typing ".../house upper 01 interior. nif", even with the quotes. (Hint: There are two spaces between "house" and "upper", but you can't easily see that. And the space before "nif" will confuse some file manager software rather badly.)

I understand that a lot of novices are going to want their filenames to have spaces as they find convenient in their word processor, but internal resource files are not word processor files, and I think it's fair to expect a more disciplined approach from modders because modding is by nature at least somewhat technical.

The difference, to me, is between filenames that are meant to be seen by the end user, and filenames that are meant to be seen only by other modders.

All that being said, after doing my first upload on [the other modding web site....{ahem}] this week, I am cutting people a lot more slack. The upload form over there uses the phrase "file name" to identify the title of the upload material as a group, even though you then may attach an arbitrary number of actual files to that page, with each file having its own (true) file name for the download/save-to-disk process. The overloading of the term "file name" is confusing, at least to me. They perhaps should rename the page-level field to "Title" or something like that. What *probably* happened was that the site designer originally meant for the multi-upload feature to be used for newer versions of the same file, and then users started putting related files (the mod, a compatibility patch for each of several other mods, a documentation PDF, and who knows what else) into the same upload page because that makes it easier for users to find it. Good intentions on everyone's part, but nobody could predict how the usage would evolve as the site grew.

[syscrusher]

Very specific question:

My testing so far indicates that if I have a mesh or texture file in a BSA, such as "meshes\foo\bar\baz.nif", and someone else has a file with exactly the same path in *their* BSA, and someone else has a file with exactly the same path as a standalone file (outside any BSA), then one of these is going to "win" and be used by all mods that refer to that mesh, but that the game won't crash or do anything equally dire.

The conclusion I draw from this is that it's okay to put files from some other modder into my BSA using their original path, and that the only downside of this is that users might get a different version of the correct NIF, and that in *most* cases (particularly for static NIFs, which is the bulk of my resource usage) this will at worst be a cosmetic problem until I update my BSA.

Does anyone have test results to the contrary, that is, a serious problem resulting from the game being able to find two copies of the same mesh at the same path?

I am excluding from this the case of two meshes with the same path that were never intended to be the same item. That clearly would be a Very Bad Idea [tm].

[Khettienna]

Tut posted. A lot of your questions are answered here!

[Arthmoor]

Does anyone have test results to the contrary, that is, a serious problem resulting from the game being able to find two copies of the same mesh at the same path?

Basically, don't. There have been enough reports from people who say this crashes the game when you have two different BSAs with the same content to make it a non-trivial event. A BSA file should ALWAYS have ONLY content unique to your mod.

[syscrusher]

Tut posted. A lot of your questions are answered here!

You are my hero. That is one awesome tutorial. Between that one, and the ones by WillieSea and InsanitySorrow, I feel a lot more comfortable with this process now.

I *printed* your tutorial for reference. Anyone who has worked with me IRL knows that I only print things that I find to be extremely valuable. Many thanks!

Syscrusher

[syscrusher]

Basically, don't. There have been enough reports from people who say this crashes the game when you have two different BSAs with the same content to make it a non-trivial event. A BSA file should ALWAYS have ONLY content unique to your mod.

This then seems to point me toward the approach suggested by a couple of people, to put external resources in their own by-creator subdir underneath mine, to guarantee uniqueness, and simply not use anything by people who really mandate that you can't do that. I really don't want any loose files lying around in the meshes or texture directories from my mod, for all the same reasons Khettienna states in her tutorial.