Jump to content

DOWNLOAD MODS

Are you looking for something shiny for your load order? We have many exclusive mods and resources you won't find anywhere else. Start your search now...

LEARN MODDING

Ready to try your hand at making your own mod creations? Visit the Enclave, the original ES/FO modding school, and learn the tricks of the trade from veteran modders...

JOIN THE ALLIANCE

Membership is free and registering unlocks image galleries, project hosting, live chat, unlimited downloads, & more...

Splitting an ESP into ESP and ESM pair -- complex situation


syscrusher
 Share

Recommended Posts

Greetings, all!

 

I need some expert-level help on splitting a rather large ESP into an ESM/ESP pair, and so far neither Google nor tool documentation nor direct searching on TESA has found me a way to do this. I know that TES4Gecko can "split a mod", but it puts way more stuff into the ESM than what I want there, and it doesn't seem to allow me any manual control over that.

 

Background (sorry, this is long, but the information matters):

 

My ESP file defines a couple thousand new base objects, seven interlocking quests, hundreds of custom scripts, and over 3,000 lines of voiced and lip synced dialog. Among those base objects, some use vanilla meshes and textures (e.g., a vanilla door but with a custom script added requiring a base object just to add the script, but no external assets). Other base objects use custom meshes, textures, sound files, music files, or some combination of these. All told, the added meshes and textures total about 250 meg, and the voice files are about 150 meg.

 

I deploy the mod to my alpha test team (several people) as an ESP plus three BSA files (one each for meshes, textures, and sounds which includes voices and lip syncs).

 

Development has reached the point where the meshes and textures are quite seldom changing, but I'm still editing recorded voices and inserting them into the mod. That means that of 400 meg download, about 250 meg of it is the same for the testers every time.

 

Looking ahead, we're just a few weeks away from a public beta release, and hopefully a lot more people downloading. There is no way the voice editing will be 100% done before the beta, so I need to make that part easy to update for testers.

 

Furthermore, the ESM/ESP split is the right thing to do *long term* as well. The reason for this is that the music handling logic for the mod is smart enough to detect Sound Commands, Enhanced Sound and Music, or Better Music System, selecting the best available option, and falls back to vanilla StreamMusic() calls if none of those are installed. The initialization logic builds arrays of the mod's internal cells and which music files should apply to which cells (merchants, taverns, dungeon....etc.). All of that logic is something I don't want to duplicate in the (required by BMS design) separate ESP file that supports BMS. So my code builds the arrays in one place and then shares them with the optional ESP that supports BMS -- and this requires the optional ESP to have my main ESP as a master. I know how to do that, but it's an ugly hack that I'd like to avoid.

 

If I create an ESM, all that shared logic for music handling can go into the ESM.

 

There are other optional ESPs planned, such as one explicitly supporting CM Partners. So getting some of the global logic into an ESM has lots of benefits that have nothing to do with download size.

 

Now, given the above, here's what I need to do:

 

The ESM should contain only one quest, and not one of the existing ones. This quest will contain the music data arrays and initialization logic, so that MyMod-BetterMusicSystem.esp can depend on my ESM and not my ESP. The ESM quest will also have a few declared variables that are shared by the other quests, but other than that it won't have much code.

 

The ESM should define most or all of the unscripted base objects (or base objects that use only vanilla scripts) that use custom meshes, textures, and sound FX files, but *not* voice and music files.

 

For scripted objects such as custom activators, I'll add a dummy unscripted version that has the model attached but no script, just so the models get pulled into the BSA for the ESM file.

 

The ESP will contain all quests except the one mentioned above, all dialog, all interior and exterior cell edits, and almost all of my custom scripts.

 

In short, the ESM should mostly define base objects, and the ESP should have references to those base objects.

 

That last bit is the problem. Here is what I've found so far with the tooling I have, which includes Wrye Bash, Construction Set Extended (very latest version), TES4Edit, TES4Gecko, and TES4Files, among others:

 

  1. TES4Gecko does a perfect job of splitting the mod, but it's way too aggressive in what goes into the ESM. With Gecko, all new records go into the ESM because they can safely load early, and override records stay in the ESP so they can load after the things they override. That is sensible enough for some situations, but not what I want to do here because all those voice files end up in the ESM.
  2. I have tried copying my ESP to a different name, using Wrye Bash to "esmify" one of them, then using either TES4Edit or CSE to delete or mark unmodified the records in each file that I want to use from the other. This basically results in vast numbers of missing records.
  3. I tried making the ESP as a clone of the ESM such that there are intentionally (and temporarily) huge numbers of ITM (identical to master) records in the ESP, the idea being then to delete out of the ESM those that should be in the ESP and vice-versa. The problem here is that when I do this, even though I created the ESM by physically copying the ESP file and esmifying it in place (that is, never opened in the CSE), neither Wrye Bash nor TES4Edit considers the duplication to be ITM records! This completely surprised me.
  4. I have tested with changing just a few objects in the ESP manually (in TES4Edit or the CSE) so that the references point to mod index 00 (implying "load this from whichever master has it defined"), and this works -- but with thousands of base objects, it doesn't scale to my situation.
  5. Using TES4Gecko's display-and-copy functionality doesn't work, either, because the export gets new formIDs for the base objects.
  6. Creating an empty (or nearly so) ESM in the CSE, then making it a master of the ESP, then using TES4Gecko's "merge to master" function doesn't help, because it uses the same decision logic as the "split plugin" feature and puts everything into the ESM that it possibly can.

Looking at the references in the ESP after doing any of the above, it appears what is happening is that the ESP's references are hardwired to itself rather than pointing back to the base object that now exists in an ESM.

 

I think Coda (the programming language embedded in CSE) could do what I want, but try as I might I can't find any documentation for Coda's callable functions. There is documentation for the language syntax but nothing I can find for available functions to do things like change the base object ID of all references of a specific type.

 

I'm sure someone has done this before, but I spent almost an entire day trying to figure out how, and I'm stumped. I don't need (or want) the tooling to pick which objects go where -- that's got to be a manual decision. All I need is something that updates the references in the ESP correctly and lets me do a group of manually-identified base objects in a batch.

 

Ideally, what I want is a tool where I can start with an almost-empty ESM, master it over the ESP, then use a feature like TES4Gecko's "merge to master" but *only on base objects I select* in the UI.

 

Second choice would be more extensive documentation on Coda so I can create my own custom logic to do exactly what I need. I am an IT pro professionally, so writing code doesn't scare me -- but I need API docs and not just syntax docs.

 

I'm sure someone has solved this before. If that person is you, please help a guy out here!

 

As an incentive for replies, I will offer that whoever provides the solution that ultimately works for me, can be listed in the mod's credits as a technical contributor. :-)

 

Many thanks for reading this long post, and for any advice you can offer.

 

--Syscrusher

 

Link to comment
Share on other sites

Update: I have another approach I'm going to try. I will turn my existing ESP into an ESM, create an essentially empty ESP with the ESM as a master, and then try using TES4Edit's "copy as override into...." function on the references but not the base objects. I've never used it before, but it sounds as if it might do what I need.

Link to comment
Share on other sites

Update: I have another approach I'm going to try. I will turn my existing ESP into an ESM, create an essentially empty ESP with the ESM as a master, and then try using TES4Edit's "copy as override into...." function on the references but not the base objects. I've never used it before, but it sounds as if it might do what I need.

 

 

This approach actually seems to be working, but there are complex implications and it requires real care. On the other hand, I have mostly accomplished what I needed to do. If time permits, I may write this up as a tutorial when I'm all done.

Link to comment
Share on other sites

I've abandoned this plan. My method of building the ESM was complex, but it worked, and I got a prototype that was very close to what I wanted. Unfortunately, ESMs don't load BSA files of the same name automatically the way ESPs do. You have to list the BSAs in the Oblivion.ini file. I actually knew this but had, sadly, forgotten.

 

The other problem is that my ESM needs to refer to cells that aren't defined until the ESP. It doesn't need their contents, just the formIDs. I had assumed, naively, that there was a way to runtime lookup a ref from an EditorID string for cells, since their EditorIDs load at runtime (to support the "coc" console command). But I couldn't find any way to do that. The workaround was to have the ESM declare empty cells then the ESP override them to put in their contents and cell-level settings. A lot of trouble to maintain, that.

 

I created a simple little ESP named the same as my ESM, and it made the BSAs load properly. But, honestly, this configuration was getting so complex that I just decided it's more trouble to maintain than what it's worth in benefit.

 

So, thanks to anyone who read this far, but don't spend any time trying to help with this as I've bailed out on the plan. My mod will ship with a couple of ESP files and BSA files, and that's working today.

Link to comment
Share on other sites

  • 1 year later...

Hi! OP needed Coda API docs (me too).

==> Construction Set Extender Coda Script API Documentation <==

In the Construction Set Extender Console window type "DumpCodaDocs". It generates {Oblivion_Directory}\coda_command_doc.html, much in the style of the OBSE docs. See Coda Manual.pdf, section "Executing Scripts".

( I know what thread necromancy is, but since this thread is one of the very few web search hits about Coda ... I mean, even syscrusher couldn't figure out where to get the docs from. ^^ )

Edited by Maxim
ugly line wrap
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...