Mod Merging: Is it troublesome? Are there solutions?

9 posts / 0 new
Last post
Mod Merging: Is it troublesome? Are there solutions?

Hey Folks!

jt2 messaged me the other day about some of the difficulties he saw in making mods work together (thanks jt2!), and I wanted to expand the conversation to see what other's experiences have been like.

How have you been finding the process of making your mods work with others'? Do you spend much time sorting out overrides and loading order? And if so, what takes the most time?

Also, if any of you have experience modding other games, how do other games handle compatibility between mods?

Dan Fedor - Founder, Blue Bottle Games

The only other game I modded was "Don't Starve".
What I understood from it is that every mod loaded was loaded independently, and then there's a menu that allows you to "activate" mods, but some of them are incompatible with each other. The language was LUA, though.

The Science&Sorcery Mod deal with nanobots. travel to the astral realm.

The biggest problem, beyond mutually incompatible overwrites of vanilla objects, is merging the lists. If one mod adds an item to a treasure table, and another comes along and adds something to the same treasure table, only the one which is loaded last counts.

I've modded Fallout 3 and New Vegas extensively, and this was a problem there, too. One thing which modders used to get around this was scripted events which added various objects to the lists instead of editing the tables directly. That worked well enough. For the other issues, player-made 3rd party programs were made which could read the various entries (which were not too dissimilar to those of NEO Scavenger) and allow the changes to be reconciled. See FO3Edit for a great example of that.

If the modding base were more willing to make tools like FO3Edit, then we wouldn't have any problems. Hard to do with such a small base, mainly working with Notepad++ and WinMerge.

The override merges is something that my Mod Editor tool could handle, but I'm concerned that this might be a little to cumbersome for the end user

If all Mods used a scripting language setup for modifying the override values, then the end user would need to read in each of the merge scripts, perform the merge, and output it to a common 0-override file.

Is that what people had to do with the FE3Edit tool? That's not too much of a difficulty for the end users to handle?

What I am thinking of for the 'scripting language' is a 0-Override modding language that modders can utilize to provide compatibility, and have my NEO_Data Modding Editor provide the merge functionality for the end user. The language would have IfExists, Add, Replace, and Delete commands to merge the mod. Add would simply add to the end of the field. Delete would delete the first instance (left to right) of the supplied string, if found. Replace would replace the first found instance with the desired replacement.

For example, here is my over-ride of encounter 1500:
original encounter:

90.69x1=1520x1x0x0x0,90.66x1=1519x1x0x0x0,90.68x1=1518x1x0x0x0,90.10x1=1525x1x0x0x0, 90.66x1=1546x1x0x0x0,90.69x1=1547x1x0x0x0,90.20x1=1883x1x0x0x0,=1500x1x0x0x0

my override encounter:

90.69x1=1520x1x0x0x0,90.66x1=1519x1x0x0x0,90.68x1=1518x1x0x0x0,90.23x1=TRN:3000x1x0x0x0,90.12x1=TRN:3500x1x0x0x0,90.10x1=1525x1x0x0x0, 90.66x1=1546x1x0x0x0,90.69x1=1547x1x0x0x0,90.20x1=1883x1x0x0x0,=1500x1x0x0x0

The table entry to change one to the other in the Override mod file would have:

<table name="merge"> <column name="tabletype">encounters</column> <column name="id">1500</column> <column name="aResponses">IfExists("90.10", replace("90.10", "90.23x1=TRN:3000x1x0x0x0,90.12x1=TRN:3500x1x0x0x0,90.10"), add("90.23x1=TRN:3000x1x0x0x0,90.12x1=TRN:3500x1x0x0x0"))</column> </table>

The IfExists would first check if 90.10 exists, and then insert the new fields by the replace operation. If the IfExists fails, then the new fields are simply added onto the end of the field.

The end user who wants to run 3 different Mods would need to use the tool to come up with one consolidated override XML file. They would open the NEO_data tool, and add the first Mod's 0_merge.xml file and use it to produce a neogame.xml file that would reside in the 0 folder. They would then need to load the game to make sure that override merge doesn't break the game. they could then load and process the 2nd mod, and merge this into the 0/neogame.xml file, and then retest the merge.

Modders would need to test their 0_merge.xml file with other mods, and perhaps have to figure out ways to resolve differences, but this would only require work on one file, rather than having to provide overrides and merges with all the other popular mods.

I can't see any other way around the merging problem other than a scripting language. Has anyone encountered a scripting language for producing file merges used in other applications? If there is already a well thought out merge language then perhaps we could adopt that.

i'm not fmailiar with all that, and i had a hard time understanding and following what you if it seems like what i'm about to say is what you just said...well sorry

i merge NSE and a bunch of the smaller mods together. i haven't modded for other games, but at least i can say something about the specific topic of merging

1) on it's own merging mods isn't actually painful, even when merging a bunch of them. it becomes a problem though when you try to give people "options" as in, maybe not everyone wants BigBadCheater merged with Overhaul + NSE + Shouldered. this means if i decide to make it easy on people who can't mod and make a no BBC versoin then i have to essentially make an entirely seperate mod. this is a problem because when nse, overhaul, shoulder, or neo scavenger gets an update i have to update two seperate mods instead of one. the more "options" i add the more mods i need to update, and it gets out of hand faster than you would think, much faster.

there is one other pain, but it's more of just an annoyence than anything else. if two mods have a similiar item in their new folders you have to "merge" the "new" .xml as well for the specific items in question instead of just the override .xml

as for what to do about it i don't know. for issue 1) i guess if a program of somesorts could remove, comment out, or copy and paste pre-determined code at spots pre-determined by the merging author then people could just tell the program what mods they want, and with actions already determined by the author do them to make the mod fit the players needs. all while keeping it in one 0 .xml file

OverHaul Mod
DevKit is an upgraded BBC mod.
Improve your mods.

You're misunderstanding what I meant with the scripting language. The game had a basic script setup. Every mod would have a new script added which would have instructions adding entries to a shared list, rather than editing the list directly.

And I have seen your tool, and it's definitely a good thing, but not all of us are on Windows. Excel scripts are not cross platform.

Thanks for the input so far! I totally didn't think of that cross-platform point, so that's one very important consideration already gained by this thread.

I guess I'm not too surprised to hear this issue also plagues FO3. There isn't a hard and fast rule that applies to all mod-merging situations. Each modded field requires special handling depending on which mods are being loaded.

I seem to remember the STALKER series having sort of "curated mod combinations" as the community developed. E.g. mod #1, #2, and #3 were all popular, but incompatible. And a fourth modder would come along and figure out how to make them all work together, releasing it as a 4th mod.

I suppose what they did was to just tweak the loading order of the mods, and then add a final override mod which patched any data collisions. It wouldn't be immune to mod updates in #s 1-3, but keeping #4 (the reconciling mod) updated might be easier.

Dan Fedor - Founder, Blue Bottle Games

That's precisely what the tools and many of my mods from FO3 did, ie, compare the modified list to the original list, see what was added and what was removed in each mod, combine the changes, and spit out an override file to be loaded after all the rest. It worked alright, but when talking about multiple mods which someone may or may not want, it became impossible to have compatibility files for all entries on really big mods, as the complexity of the mod merging grew combinatorically.

This could be side stepped by having a special attribute line for mods which doesn't replace a list, but adds entries to it instead. Maybe like:

<table name="treasuretable"> <column name="id">110</column> <column name="strName">Store rubble</column> <column name="aTreasures" [b]add="true"[/b]>151x0.5x1</column> <column name="bNested">0</column> <column name="bSuppress">0</column> <column name="bIdentify">0</column> </table>

It's also worth noting that while FO3 used a proprietary mod file format (esp and esm files, for "patch" and "master" files), fundamentally they were just lists of objects with various properties, and additional image or model files stored separately, almost identical to the concept of the xml file and images file that NEO Scavenger uses. They just used a non-standard binary format to store the records, instead of a human readable xml file.

Also worth noting, NEO Scavenger's in-built support for referencing entries from different mods is already quite a bit more advanced than FO3 ever was. As far as FO3 was concerned, there was only ever one set of data, never multiple concurrent mods. The patches were all loaded in and then the entire mass processed as one monolithic database. The only thing done better in that modding community was the critical mass of people modding the game and the numerous third party tools created, not because of the openness of the development, but rather in spite of the closed source nature and proprietary corporate formats.

Another thing that would be excellent would be to change the current behavior of loading each mod's neogame.xml in its entirety and then processing it all, but instead loading the file into memory, performing the loading logic only so far as it required to link the objects together, perform overrides, and the like, and then saving the finalization steps for the very end. This would also decrease loading time for really heavily modded games, as each neogame.xml seems to take, at some level, a fixed amount of time to be processed, unrelated to it's overall complexity, with a one-entry mod taking about as long to process as a megamod.

And finally, it would be nice to have the game load all xml files in a mod directory, regardless of their names. Combined with the method described above, one could then split their mod files into various pieces, like weapons.xml and encounters.xml, purely for sanitary purposes.

I don't know, just a few things that have been going through my head about this game re: mods.

A bit late to the table, I know, but wanted to share my thoughts as well.

While I never managed to create any worthwhile mod before, I was creeping around two modding communities for quite some time - Mount and Blade Warband and Civilization 4.

I think that both of those games are good examples here, since they both work on a text-file modifying base, with Civ being .xml based while M&B having a "module system" which is similar, but operates on several files for several things (scenes, dialogs, items are all separate files) which are compiled by a special program afterwards, to get the mod working. Both of those games are also very open and with many things that can be changed, like NEO. And for both, the modding scene multiplied their life-span bu many, many times :D

And both of those game don't have any mod-merging at all. Only one mod can be loaded at a time, and in order to have two (or more) different modifications running, modder needs to create a mod that includes both, as a one new mod.

In the both games I mention, experienced modders tend to create the "modding resources" which the creators of finished mods can incorporate in their work. Those tend to be tailored to be as modular and as none-intrusive as possible, in order to make them work with as many other modifications as possible. There are many modders, in both of those communities, concentrated entirely on creating resources, for others to use.

Examples of those are the famous "Polished Landscapes" or "Diplomacy" packs for the M&B, which are the collections of scripts and mechanical and graphical changes, greatly upgrading the core game and are, by now, considered a basis, from which a person can start creating their own new mod.

So, in essence, modding Civ 4 or Warband, really starts by selecting which "resource packs" you want to use, and then building your own stuff upon them, rather than creating everything from grounds up by yourself. That saves a lot of work, for everyone involved.

That way of thinking was, actually, why I released the Fields of Dead and Sage's Pages as a separate mini-mods. I decided that since I was going to make some very specialized modifications, I may as well make them as a small and separated "packs", including the descriptions of which parts of the vanilla game are being changed, so that other modders can use them to "upgrade" their own mods. It feels like I just assumed it will go this way here too...

And a word about tools - while there were, in both cases mentioned, some tools developed by the community to make it easier for modders to merge those resources together, they were still concentrated at fusing the different things into a one, working mod, rather than trying to make different ones work simultaneously.

<--Mighty (mini)Mod of Doom-->
DeviantArt Gallery of MoD Sprites