Jump to content

DOWNLOAD MODS

Are you looking for something shiny for your load order? We have many exclusive mods and resources you won't find anywhere else. Start your search now...

LEARN MODDING

Ready to try your hand at making your own mod creations? Visit the Enclave, the original ES/FO modding school, and learn the tricks of the trade from veteran modders...

JOIN THE ALLIANCE

Membership is free and registering unlocks image galleries, project hosting, live chat, unlimited downloads, & more...

[Guide] Cleaning Skyrim and Skyrim SE Master Files


alt3rn1ty
 Share

Recommended Posts


Cleaning the Official Master ESMs


This guide assumes using TES5Edit on Skyrim Nexus,
Or SSEEdit on Skyrim SE Nexus

Due to this guide being dual purpose ( For Skyrim and Skyrim SE ) for the rest of this guide I will refer to both tools as xEdit.
Screenshots of tools used may be one or the other, or older versions, which does not matter, the images are only to illustrate the method / options used.

 

Why Clean the Master Files ?

 

Firstly because the masters have entries that are identical to the same records in Skyrim.esm or other DLC esms'. They exist because Bethesda may have looked at something in the CK and an unneeded entry was auto included in the plugin even though the item was not altered in any way. The Official Creation Kits are notoriously buggy and randomly create dirty / wild edits, often when the author of the plugin is completely unaware. Wherever that plugin is placed in your load order its records overwrite all the conflicting records from plugins loaded before it ( the rule of one ) resetting the settings back to the values contained in the Official Bethesda DLC. It won't cause crashes, it just changes the values of plugins loaded before it. Which can alter mods that you have for Weapon Damage, Armor, Lighting, Food Effects and so on. The masters are very early in your load order but there is potential for a mod to be made as a fake.esm, and placed among them, and so ITMs in a later loading master may cause problems for that mods esm. Chance is remote that a master will affect another master, and this procedure is best used on all of your mods plugins, but cleaning everything of ITMs ( Identical to Master records ) causes no harm, is more optimal giving the game less to process in your load order, and so it is best to get rid of these completely unnecessary dirty edits.

The Second reason is that Bethesda chose to delete some things that are in the Official DLC. Any mods loaded with references to deleted records from the Official Bethesda DLC will cause your game to crash. This problem particularly affects older mods ( especially mods that were made before newer official patches were released, with more deleted references the old mod did not anticipate - It will also become problematic for the Skyrim Special Edition community where old Original Skyrim mods are being converted to SSE, and Bethesda have deleted even more records from the plugins before they released the newer plugins for that version of the game ). xEdit can restore and properly assign values to these records that will disable them and still allow mods to access them. This is done using the "Undelete and Disable References" option.

For further explanations of why it is still recommended to clean the games masters .. Read on from this post,

Zilav and Arthmoor, most valued technical and vastly experienced modding authors, weigh in on the subject.

 

The following mostly apply to mod authors, but worth knowing about for mod users too :

xEdit will also report when a mod has Deleted NavMeshes as part of the report from Automatic cleaning. Like deleted references, any mod that references a deleted NavMesh will cause Skyrim to Crash. Properly optimizing your mods NavMeshes and checking your mod for Deleted Vanilla NavMeshes ( which can also be caused by a CK wild edit even if you did not do it yourself ) is important. Mods altering the same cell and the same NavMeshes when your mod is not optimized will cause Skyrim to Crash. Poorly optimized NavMeshes with errors reported by the CK will make Skyrim unstable. Instabilities like fast travelling to a location and Skyrim crashes. Note the ones found to be deleted in the games masters, cannot be undeleted. To fix deleted Navmeshes in your mods, Arthmoor has provided a walkthrough in Skyrim - Fixing Navmesh Deletion in TES5Edit

Manually cleaning your mods is also important to remove wild edits. This is mostly down to the experience of Mod Authors to solve such problems, but there are a few noted later in this guide which are in the DLCs which everyone can easily Manually clean. Some mods can have accidental Wild Edits in them caused by the author looking at how Bethesda did something they wish their mod to do as well. These Wild Edits often prevent Skyrim from doing things like advancing quests, spawning NPCs, assigning dialogue to NPCs, preventing NPCs from patrol areas they are assigned to. They can also alter Vanilla Lighting and Triggers that the author wished to use. All of these things affect any plugin with conflicting records loaded before a mod with Wild Edits.

Mod authors - Learn to use xEdit, and ensure the only records in your mod plugins are what you would expect to be in there, its the most important tool the community can make use of when used properly.

Mod Users - Follow this guide...

 

 

 

Before moving on to the Manual cleaning, something everyone should do prior to Manual Cleaning :


Automatic cleaning of Bethesda's ESMs with xEdit

With the games Original esm's installed ( You can use Steam to Verify Integrity of Game Cache of Skyrims files to ensure you have good error free copies of the original master files ), and in accordance with the following wiki article http://www.creationk...ty_Plugins_List :


Load up xEdit.

1. Right click the plugin selection screen and select "none"


large.xEdit1.png.4e327d2849d996c61b870d6


2. Tick the relevant esm to edit, and click okay ( If you have not cleaned any of your Master files yet, the first one to tick will be Skyrims "Update.esm" ), then click Okay


large.xEdit2.png.a538b216ea95a79a5e15ef5


After each of the following actions, wait for a message in the message window that the previous operation has finished / Done :

3. Right click the plugin after you get the "Background Loader : Finished" message,and choose "Apply Filter for Cleaning"



large.xEdit3.png.8392a40f0139e2199b57b4e


Wait until Filtering is finished then ..

4. Right click the plugin and choose Remove Identical to Master Records



large.xEdit4.png.43f1c1457ae7c6d949de243


Wait until it finishes then ..

5. Right click the plugin and choose Undelete and Disable references



large.xEdit5.png.8c600acaeb8e116f182f03b


Wait until it finishes then ..

6. Close xEdit, and it should check with you that you wish to save the plugin ( this only happens if you have made any changes to the plugin to save, if it just closes .. Then you have not cleaned anything )


Rinse and repeat the Automatic cleaning ( steps 1 - 6 above ) for each of the master files.

Working from first to load, to last, not including Skyrim.esm or any unofficial patches
( No point doing Skyrim.esm, and the unofficial patches are already done and should not be cleaned )

So clean in this order

Update.esm

Dawnguard.esm

Dawnguard.esm ( Yes it needs to be done twice )

Hearthfire.esm

Dragonborn.esm


Dawnguard.esm needs to be cleaned twice ( as of xEdit 3.1 onwards - After doing the automatic cleaning routine once on Dawnguard.esm, and saving it, load it up again in xEdit and you will be able to clean a further 6 ITMs )



------------------



Dawnguard.esm needs manual cleaning aswell as automatic cleaning


After the automated cleaning is done, you can also now manually clean a few more Wild edits xEdit will not have touched ...

Recently Arthmoor has brought to the attention of the community additional information regarding manual cleaning of Dawnguard.esm, which everyone needs to do for their own setup same as automatic cleaning ( because nobody can legally upload official master files anywhere, everyone needs to do their own )



First load up xEdit

When the plugin selection comes up, right click and select None
Then put a tick in the box just for Dawnguard.esm, click Okay
After its finished loading, right click Dawnguard.esm and choose "Filter for Cleaning"


1. For "CELL 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm." .... Expand the records as in the following screenshot, and right click the indicated sub-record, and choose Remove



large.58af75b0a7fef_Cleaning1.png.4d9fd9


2. For "CELL 0001FA4C: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af75e78d857_Cleaning2.png.cf5c54


3. For "CELL 0006C3B6: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af761722959_Cleaning3.png.e4213b


NOTE : This guide used to include cleaning instructions for "CELL 00039F67: Wild edit. Remove this record. It's a testing cell" ( The WICourier edit ) - But since the new version of TES5Edit 3.1+ now cleans that as part of the automated cleaning ( which you should have done prior to manual cleaning ), you no longer need to clean it manually afterwards.


-----------------------------------------

 

Now that the Master files are cleaned, you could put them in a zip, and get your mod manager to install them - Maybe at a future date you want to do a refresh of steam cache and it redownloads the masters which are not the same as the originals anymore (because you cleaned them), so then you would need to reclean them again. But beware, Bethesda have started redoing some masters due to Creation Club mods compatability, so make sure any redownloaded masters are not newer than your previously cleaned ones, because in that case you will need to reclean and rezip them again anyway.

 

You can go through the rest of your Load Order using Automatic cleaning of ITMs and UDRs on all your mods plugins. The sequence of cleaning mods plugins should be after you have your Load Order correct, masters are cleaned, then clean them with the last to load being the last to clean. Mod authors should have done them already, so most will probably not need cleaning. Also look out for any mod specific cleaning instructions in the mods description.

 

Prime example = The Unofficial Patches will not need any cleaning, they are already done, and any remaining ITMs in those plugins should be left alone because they do have a purpose .. ( its a very rare occasion when this is true ).



The xEdit Work In Progress Development topic is at the following link https://afkmods.iguanadons.net/index.php?/topic/3750-wipz-tes5edit/

 

Development project is at GitHub https://github.com/TES5Edit/TES5Edit

 

And newer versions of xEdit (3.2.7 +) have a link to Discord top right of xEdit window.

 

 

Edited by alt3rn1ty
Link to comment
Share on other sites

No problem, there have been a couple of changes since this guide was first put up anywhere, due to the evolution of TES5Edit and its automated cleaning becoming better along the way .. There used to be 4 steps to the manual cleaning section.

 

I am keeping an ( occasional ) eye on any wiki updates to the manual cleaning notes.

 

If there are any questions about this guide ..

 

Have a look over what has passed before on AFK Mods topic http://afkmods.iguanadons.net/index.php?/topic/4110-manual-cleaning-skyrim-master-files/

( aswell as the linked references in the guide :coffee: )

 

Thank you  :thumbup:

Edited by alt3rn1ty
Link to comment
Share on other sites

Thank you. In my case, the second round of cleaning Dawnguard.esm returned only 2 new ITM, instead of the 6 you mention.

 

People who wouldn't want to create an account at the Nexus can find TES5Edit (and a lot of other stuff) here: https://onedrive.live.com/?cid=5f2e81535b547400&id=5F2E81535B547400!133

Edited by ladyonthemoon
Link to comment
Share on other sites

The results of each stage have been confirmed by more than just myself, using the same version of TES5Edit which ended up being the release version of 3.1

 

I believe Sharlikran does keep his one drive up to date ish, but personally I would rather use the WIP thread uploads by Zilav to be sure I am using the correct versions for any given test. Second link in the OP.

 

My runs were done with a fresh download of the esms, and confirmed by other users of TES5Edit in the WIP topic. There were no contesting posts to say we were getting the results wrong by anyone reading the WIP topic, or the TES5Edit topics on BGS Forums where similar posts were made and corroborated by other experienced users.

 

I dont know why you are only seeing two new ITM on a second cleaning run. Do the file CRCs match with the CRCs in my last screenshot ?

Did you save the plugin after the first run. Then load up TES5Edit again for a second run, applying filter for cleaning a second time etc etc ?

Or maybe did you try doing the manual cleaning before automatic cleaning ?, that may have resulted in different ITM results

 

Go to about 8 minutes and 40 seconds here

 

 

Pause it where he is talking about cleaning dawnguard twice, and have a look at the results on the right at the bottom with 6 ITMs showing for the second clean. ( I dont understand why he gets less ITMs in the first run than he should, but maybe its a slightly older version of TES5Edit when that screenshot was taken, and was new enough to get the extra 6 - Just prior to version 3.1 release there were a couple of updates to the dev version which increased its ability to catch more ITMs than it did before )

Edited by alt3rn1ty
Link to comment
Share on other sites

  • 2 weeks later...

Thats probably better answered by someone like Arthmoor, but knowing that he cleans his master files and produces the Unofficial patches I would say not ..

 

.. unless your mod for some reason referenced the cleaned records, and everyone else is cleaning them except yourself, then your mod in their setup will fail. So long as your mod does not reference the same records, then no matter what a user does or does not do in regards to cleaning will not matter.

 

Ideally everyone should be conforming to the same cleaning practices ( which is why I have posted this topic everywhere ), not only so that they are all singing from the same hymn sheet, but also because they will have a far more stable game as a result of cleaning up Bethesdas leftover dirty / wild edits ( and the same problems in mods which the Creation Kit occasionally stealth drops into a plugin unknown to its author ).

 

 

Unfortunately a lot of people will probably see this subject as maybe out of their depth : I remember I was the same for a while with TES4Edit use cleaning master files for Oblivion, and also cleaning mods in my load order .. I slowly started to realise the benefits and eventually got around to giving it a bit of study and then had one of those light bulb moments, after that no matter what I setup in my load order, I could always produce a crash free game. Its been the same story for Skyrim.

Edited by alt3rn1ty
Link to comment
Share on other sites

Then they will be no worse off than they were before using your mod, because you have referenced dirty / wild edits they unfortunately still have in their games setup aswell as yourself.

 

 

If you as an author clean, or do not clean, makes no difference to someone who is already running with underlying problems before they start adding mods into the mix

 

If you as an author do not clean, but all users do, then your mod is going to be the possible cause of problems.

Link to comment
Share on other sites

  • 3 months later...
  • 1 month later...

Yey TESA is back ..

The latest being TES5Edit_3_1_1?

W.I.P = Work In Progress, the one that will currently become 3.1.2, link is in the first post, third line.

Look at Zilav's first post, the WIP is linked at the bottom ( currently "xEdit 091015" )

 

I have just edited my OP so that the links are clearer on this new forum software.

If you have never used the WIP before, from the archive all you need to extract and manually install are the following highlighted files ..

sBnQisd.png

You only need the relevant game hardcoded file

( You can also rename TES5Edit.exe as say TES4Edit.exe and you now have the latest version for Oblivion :), same applies to FNVEdit .. Just also include only the relevant hardcoded file )

 

Its your choice if you wish to participate in testing WIPs though, but so far they have all been very reliable. Zilav is one extremely careful, competent and methodical coder.

Edited by alt3rn1ty
  • Upvote 1
Link to comment
Share on other sites

  • 1 month later...

Updated the OP - To reflect Release of TES5Edit 3.1.2 on Nexus
 
Also there was a discrepancy of how many records were processed after the last edit noting 68 ITMs instead of 64 on Dragonborn.esm, the processed records of UDRs now reads 214318 processed records instead of 214322 ( minus the 4 additional records removed prior to doing UDRs )

  • Upvote 1
Link to comment
Share on other sites

  • 4 weeks later...

To anyone who still needs to follow this guide :

 

I am currently updating it ( some old direct links to wiki paragraphs have been lost due to Bethesda re-organising their wiki's )

 

So to become less dependant upon other links and information, I am fleshing it out a bit with more images

 

It also needs an update to file CRCs ( at the end of the first post ) due to Zilavs recent changes to xEdit

 

So you now need to be a bit more up to date and use the Work In Progress version of xEdit available from Fallout 4 Nexus ( link for this in the first post has also been updated )

Link to comment
Share on other sites

Just downloaded and installed Old Skyrim again to check with TES5Edit v3.2 
 
The Results are a little different for the Dawnguard.esm
 
All the auto cleaned and Manual cleaned stages of that plugin now have new CRC / MD5 checksums because the new TES5Edit corrects something it did not do before.
 
 
Edit : And if anyone is wondering - I can confirm the same routine needs to be applied to Skyrim Special Edition plugins, even the manual cleaning, you will get a few different checksum results and amount of ITMs / UDRs cleaned because Bethesda managed to add a few more, but the routine is the same.

Link to comment
Share on other sites

For those like me, who do not want to have an account on the Nexus but want to clean their Skyrim SE master files and mods, download this version: https://github.com/TES5Edit/TES5Edit/releases/tag/FO4Edit-fa3cf0b

Once downloaded, extract it, open the folder and proceed as following:

  1. delete "Oblivion.Hardcoded.keep.this.with.the.exe.and.otherwise.ignore.it.I.really.mean.it",
  2. delete "Fallout3.Hardcoded.keep.this.with.the.exe.and.otherwise.ignore.it.I.really.mean.it",
  3. delete "Fallout4.Hardcoded.keep.this.with.the.exe.and.otherwise.ignore.it.I.really.mean.it",
  4. delete "FalloutNV.Hardcoded.keep.this.with.the.exe.and.otherwise.ignore.it.I.really.mean.it",
  5. rename "FO4Edit.exe" into "SSEEdit.exe".

There you go! You can now edit your Special Edition mods and master files! ;) 

Link to comment
Share on other sites

  • 7 months later...

If anyone is wondering when an update will come to this guide .. When the next official version of xEdit comes out of beta on Nexus

.. But note I can only now support updating the info for Skyrim Special Edition : I no longer have Skyrim LE

Currently I am using the latest development version on afk mods (at the bottom of the first post is a "Download" link).

There are a couple of differences to the results, but I will not be updating the OP until the official release because if I do, then there will be a train of questions why people still using the older but public release are getting different results .. So I need to wait until it goes live .. However I realise like yourselves that the dev version is more than likely the better version to support SSE since the Creation Club update to SSE has already gone live

 

With that in mind here are my results, which I have carefully checked using the dev version of 3.2.1 - But only for Skyrim Special Edition (as explained above).

 

Update.esm
[Removing "Identical to Master" records done]  Processed Records: 17403, Removed Records: 234, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 17169, Undeleted Records: 91, Elapsed Time: 00:00

 

Dawnguard.esm
1st clean :
[Removing "Identical to Master" records done]  Processed Records: 98676, Removed Records: 621, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 98055, Undeleted Records: 82, Elapsed Time: 00:02
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
2nd clean
[Removing "Identical to Master" records done]  Processed Records: 98054, Removed Records: 6, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 98048, Undeleted Records: 0, Elapsed Time: 00:02
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
3rd clean

 

Hearthfires.esm
[Removing "Identical to Master" records done]  Processed Records: 17830, Removed Records: 178, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 17652, Undeleted Records: 11, Elapsed Time: 00:00
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>

 

Dragonborn.esm
[Removing "Identical to Master" records done]  Processed Records: 214498, Removed Records: 69, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 214429, Undeleted Records: 8, Elapsed Time: 00:02
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>
 

The highlighted results are the changes from previous results

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...