Wherever you go, there you are.

Fallout 3 Modding - Primer

1

In this post I want to spend some time talking about the fundamentals of Fallout 3 modding; the engine and how it loads mods, the relevant files and file types involved, and the various tools and their purposes. This primer is not intended to serve as a practical modding guide; it will not necessarily teach you how to do anything just yet; we will not be building any tutorial mod in this post or even opening any of the tools. What it will do is set the ground-work for understanding how modding works, all of the files and locations involved, and should provide a much better place to start from when approaching all of the many practical tutorials already available on the web. This guide will also, as an added bonus, provide a relatively comprehensive list of the tools you will need (and explanations of why you will need them), as well as links to additional resources where you can get your feet wet with practical applications.

Engines, Masters, and Plugins, Oh My!

Fallout guy holding survival kitThe engine Bethesda used for Fallout 3 is the Gamebryo engine. This is not the only game to use this engine; in addition to other games out there, Bethesda has used progressive versions of this engine for Morrowind and Oblivion as well as Fallout 3. The details I will discuss in this series are most relevant to Fallout 3 modding, but a large host of details would work equally well for Oblivion modding, which uses a very similar version of the Gamebryo engine.

The first thing to know about modding Fallout 3 is how the engine loads the game world. There are two types of data files; "Master Files" with the suffix ".esm", and "Plugin Files" with the suffix ".esp". In general, the engine usually loads only ONE master file, and then loads any additional plugin files that are available and configured. Modders are usually working with the plugin ".esp" files. It is possible for the engine to load more than one master file, and some mods do need to be masters (such as the case of a total conversion), but these cases are few and far between so I'm not going to discuss them here.

In Fallout 3, the entire game world is loaded from a master file that came with the game called "Fallout3.esm", which is located in the "Data" directory of your Fallout 3 installation (probably something like C:\Program Files\Bethesda Softworks\Fallout 3). This file contains or references everything necessary to load the game world; all the layouts, NPCs, quests, dialog, models, items, etc. After this master file has been loaded, the engine can then load any additional plugins that it finds in this same "Data" directory (files ending in ".esp"). You can choose which plugin files the game should load using the "Data Files" option in the Fallout 3 launcher.

list of Fallout launcher data files

In essence, plugins modify the data in the master file that is first loaded. The plugin can add new elements, remove existing elements, or modify existing elements. We'll go into more detail about how this is done later, because it is important, but it suffices right now to know that the master file contains the original game world, and the plugin file you create contains all of your changes to the original game world.

The last thing to note is that the master file and the plugin files are essentially identical in content and structure. The concept of a "master file" is more logistical than technical; it just indicates that the file has a self-contained and complete description of a world, and it is the file which should be loaded first to form the base against which all additional plugins make modifications. A plugin file can also be this "self-contained" if so desired, though realistically we often want to use plugins to make changes to the existing world rather than defining an all-new world, because, hey, that would be a lot of work :)

Resource Data

You may have noticed earlier that I said the master file "contains or references" everything needed for the game world. What a master file or plugin file "contains" is the logical description of the world and its elements, but all of the "heavy hitting" items such as game models, textures, sound resources, and so on are actually stored in a different place. So the plugin file (and the original master file) may store information about, for example, an NPC -- that it exists, what its name is, where it is located -- but some of the information about the NPC in the plugin file will be references to external elements, such as which model and texture to use and which voice file to use for its spoken dialog.

These references are defined in the plugin file, but the actual model data, texture data, and sound data live in one of two places. The first place they may live is in a ".bsa" file, which probably stands for "Bethesda Softworks Archive" or something equally witty. These ".bsa" files are also located in the "Data" directory alongside the master and plugin files, and you will notice that the original Fallout 3 game comes with several of these archive files. These contain all of the original game resources, and are significantly larger than the master or plugin files.

The second place that this data can live is in subdirectories within the "Data" directory. These subdirectories contain individual resource files, but you won't see any in a fresh Fallout 3 installation because all of the original game resources live in the ".bsa" archives instead. However, you can put new resources, such as new models or textures, in subdirectories within the "Data" directory for your plugin to reference, and the game will load these just as it does the resources in the archives, allowing you to add completely new content in this way.

example BSA resource list

In fact, if you were to look inside the ".bsa" archive files that come with Fallout 3, you will see that the archives are really just a collection of resource files organized in a directory tree. It is possible to extract these archives into a separate location so that you can look at all of the resources available in the game, and even use them as a basis to make new resources for you plugin, e.g. by taking an existing model and applying a new texture. You would then put the new resource into a subdirectory within the "Data" directory and reference it in your plugin. The engine would then load up the referenced resource and display it in game!

Further, it is even possible to override the resource files that are contained within the original game archive files, by placing an identically named resource file in an identically named directory tree within the "Data" directory. To make sure this is what you intend, Fallout 3 additionally requires a special ArchiveInvalidation.txt file be created in the "Data" directory that lists all of the things in the archive that you want to override with copies in the local directory tree. However, there is a tool that makes this unnecessary, so I won't describe the details of messing with the ArchiveInvalidation.txt file, but it's useful to know at least that much about that process when it comes to troubleshooting.

Finally, you might be interested in what type of resources are available. Links to tools for working with these will be mentioned later in the tools section:

  • sound formats include wav, mp3, and ogg; additionally, lip files are used for dialog lip synching
  • all textures are stored in the standard dds, using DXT1 compression
  • models are stored in the nif format, which is also used for save-games and some other things
  • lots of other things, like skeletons and animations and other bits of data that I'm not going to enumerate here

Plugin File Structure

In theory, you rarely have to worry about the underlying structure of the plugin or master files, because this is all handled transparently by the editor tools for you. In practice, unfortunately, there are often conflicts and issues that can be most simply and easily discovered and resolved by low-level inspection and editing of the plugin files. Knowing about these fundamentals will very likely save you (and your users) hours of frustration in the future.

As mentioned earlier, master and plugin files are quite similar. In fact, when it comes to the underlying structure, they are basically identical, except for a few flags that mark a master file as a master. Simply put, these files are collections of records. A record is an entry in the file that has a type, some data, and possibly some child (or sub) records. So a master file or plugin file is simply a collection of records and subrecords, nested as deeply as needed, each containing data about the element it describes.

Every item, NPC, quest, script, location, effect, and any other element of the game world is contained and described by one of these records. The type defines what the data for that record contains; the SCPT record data contains a script, the QUST record data contains information about a quest, and so on. Some record types are quite complex, and they need additional subrecords in order to completely describe the content because the record data itself is not sufficient or dynamic enough. This makes implicit sense; if you have a record that describes a complex entity like a room, for example, you would expect that it would have a bunch of subrecords, one for each object contained inside that room. And it does.

example of plugin record entries

Reaching back to the discussion on plugin loading, you may remember that first the master file is loaded, and then all plugin files are loaded. At a low-level, what this means is that all of the records in the master file are loaded, and then all of the records and modifications to existing records are loaded from the plugin files. This is how a plugin is able to modify or delete existing content, rather than just adding new content; the plugin can instruct the engine to modify or even delete records that have been loaded previously.

This record-based approach gives plugins their power and flexibility, but it can also cause some problems. If you make modifications to the same records (i.e. elements within the master file) as a different plugin, you may have conflicts with that plugin. It is also very easy to accidentally modify or delete records from the master file unintentionally, which is generally referred to as a "dirty" mod. This happens routinely when you are in the editor looking at existing content from the master file, and accidentally delete or move an item without noticing. Not only can this cause annoyances within the original game world completely unrelated to the scope of your plugin, but it can also cause unintentional conflicts with other plugins that would not ordinarily conflict with your mod.

When this happens, the only way to undo your unintentional changes are to find the DELE records in your plugin that tell the engine to delete the other record loaded from the master file, or else to find the records in your plugin that are making modifications to the master file records that you didn't really want to change (such as giving it a new position or rotation, a ridiculously easy thing to do by accident in the editor).

To track down records and items for these (and other) troubleshooting purposes, it's helpful to know that every record has a FormID. The FormID is a unique hex identifier for that record. You will use this quite often in scripts or when looking through the records of your plugin for specific changes. These FormIDs are created automatically by the editor tool, though you will have to set a "prefix" so that the FormIDs in your mod are unique from the FormIDs in the master file and everyone else's plugin.

In addition to the FormID, many elements in a plugin also have an EditorID. The EditorID is something that can be set within the editor and is a useful way to unique identify items within scripts and references without having to figure out the FormID. I mention this mainly because it is important to know the difference when troubleshooting, but some tutorials use the terms interchangeably, which can make that confusing. Interestingly enough, EditorIDs are in fact defined by an EDID record, which will be a subrecord of the item to which it applies.

Tools and Resources

One thing to note before going into the tools and resources is that the Fallout 3 engine is just a newer version of the Oblivion engine. If you cannot find a specific resource, tool, or tutorial about how to do something in Fallout 3, it may be possible to find equivalent information for doing that same task in Oblivion. Since it is (mostly) the same engine and even a similar editor, the information you learn from an Oblivion tutorial transposes quite nicely to Fallout 3.

Before installing the tools, you may want to re-install Fallout 3 to a simple base directory, like "C:\Fallout3". This makes your file paths much simpler and easier to find, and it can also prevent some known-issues with Vista and UAC when operating out of the Program Files directory. This is entirely up to your discretion, however.

The Garden of Eden Construction Kit (G.E.C.K.)

Get it here.

The G.E.C.K. is the official editor for Fallout 3 and is the modder's bread and butter tool. It is where you do everything you need to edit the plugin files; create and modify items, quests, NPCs, rooms, layouts, etc.

FO3 Archive Utility

Get it here.

This archive utility lets you inspect and extract files from the ".bsa" archives. This allows you to extract resources from the original archives so that you can re-use original game components. You will almost certainly want to do this extraction before starting on your mod. Even if all you want to do is re-use existing resources, the editor does not know how to select these items out of the ".bsa" archives. Instead, you must select it from the extracted resources, and the editor then strips off the root of the resource path (everything before the "Data" directory part). When the engine loads, it then uses this relative path to lookup the resource in the ".bsa" files (or on disk, if a copy exists in one of the subdirectories in the "Data" directory and the necessary override has been configured). So the engine loads the resource from the archive, but the editor cannot select it from the archive, hence the need to extract even for simple re-use.

It is not recommended that you extract these to the Fallout 3 "Data" directory, as you will then be potentially overriding resources within the ".bsa" files, which you don't want to do (except in the specific cases when you do want to do that, but you don't want to be doing it in the general sense for every single resource). Since I installed Fallout 3 to "C:\Fallout3", and I put all of my tools in "C:\Fallout3Tools", it made sense for me to extract the original game archives to "C:\Fallout3Tools\Data" to mirror the original structure, but you can put them in any location that makes sense for you.

This tool will also let you create new ".bsa" archive files, which is a great way to distribute any new resources for your mod in a single self-contained way, instead of requiring the end-user to extract all your resources into individual subdirectories within the "Data" directory.

ArchiveInvalidation Invalidator

Get it here.

As you may recall during the discussion on resources, it was necessary to use an ArchiveInvalidation.txt file if you wanted to override resources within the ".bsa" archives with local copies in the "Data" directory. This tool makes maintaining the ArchiveInvalidation.txt file unnecessary, and instead forces Fallout 3 to check in the "Data" directory first and use anything it finds there to override the ".bsa" archive versions.

Remember that if your mod overrides built-in resources, the end-user will also need to install this tool (or update their ArchiveInvalidation.txt file) in order to use your mod properly.

Games for Windows Disabler

Get it here.

Bethesda chose to include Games for Windows Live as part of the Fallout 3 package, which has been something of a controversial issue. I'll remain safely neutral on the point here, except to point out that during mod development you will be potentially loading and reloading Fallout 3 dozens or hundreds of times. This tool will disable the use of Games for Windows Live within Fallout 3, making startup faster and trouble-shooting potentially easier.

NifSkope

Get it here.

The NIF model format is somewhat proprietary to the Gamebryo engine, though there are plugins to work with it in Blender and other tools. That said, the de-facto tool for viewing and editing NIF files directly is NifSkope, which lets you preview model files and even make simple changes (such as re-texturing).

DDS plugins

For Gimp.
For Photoshop.

The Direct-Draw Surface format is relatively common, and the above links provide plugins for the two most common graphics packages that you would use to edit these textures, which is the most likely use-case. However, there are many more DDS tools available out there; viewers, editors, etc., if you need to do something else with them.

Fallout Mod Manager (FOMM)

Get it here.

Installing and managing mods is not intensely difficult, but many users like to have a tool that handles the task for them. The FOMM is probably the de-facto standard mod manager that end-users have, so it makes sense to be familiar with its features and to make your mod FOMM-friendly.

FO3Edit

Get it here.

The G.E.C.K. editor is a great tool, but when you need to get into the nitty-gritty record-level details of your mod to resolve conflicts or clean up dirty modifications, you'll want to use the FO3Edit tool to manage the low-level details.

Resources

Posts in this series:

All of my posts about Fallout 3 modding.

Fallout 3 Modding - Introduction

1

Fallout 3 logoI have recently decided to create a mod for Fallout 3. Since I am just starting out, I don't plan to talk about the details of the mod just yet. However, the project is relatively ambitious and there's no guarantee that it will ever see the light of day, so I wanted to write a couple of posts about the experience so that some utility may be gleaned from the endeavor, regardless of the eventual outcome. These posts will probably vary between simple curiosities to full-fledged tutorials; hopefully they will all share at least one thing in common, and that is to be in some way be helpful to other Fallout 3 modders and modder-hopefuls.

This will not be my first experience modding this particular engine, since it is the same engine used by the game Oblivion. I have already written a mod for Oblivion, which was in many ways a much more technically challenging mod due to what it was trying to accomplish and the ways in which it pushed the engine. Its coolest feature was arguable, but its second-coolest feature was definitely the fact that the mannequins introduced by the mod would disappear from the game after a few days time, taking all of your best armor and weapons with them. I like to pretend this was intentional; that my Oblivion character was sitting on a hoard of stolen loot inside her castle keep, cackling gleefully under the moonlight as armies of magical mannequins quietly converged on her lair in the depths of the night, dumbly ferrying the stolen booty to their evil master. In reality, of course, it was just a bug, and one I never got around to fixing at that.

The point of this digression is not just to reminisce about my past glories, but also to plan out at least a few of these posts. While I should have known what I was doing going into this mod, I found myself struggling to remember most of the important details. Further, while there were many practical "how-to" tutorials on specific topics, there were very few tutorials discussing the basic details of how mods function and interact with the engine, which are things that I had to learn the hard way and which are just now slowly percolating back into my working memory. While these practical tutorials are probably very useful for some people, I find myself to be more of an abstract, theoretical learner. A step-by-step guide does very little to help me out if it does not explain the "why" behind each step.

Keeping this in mind, I want to write a primer that tackles two main points. First, I want to talk about the fundamentals behind modding this engine, because I feel this is an area that is most lacking within the existing knowledge-base. Knowing about the basics of the file layout, structure, and data resources used by the engine make a lot of common problems much easier to understand and resolve. This will be somewhat technical and abstract, and is intended to introduce all of the basics to someone who knows nothing about modding Oblivion or Fallout 3. Second, I want to build on that basis to provide a comprehensive list of tools required to work with each of these fundamental areas. Instead of just randomly downloading tools and blundering through a "My First Dungeon" tutorial, I hope that this combination will give you a firm understanding of what each of the tools does and why you will need it.

After that, we'll just see where things go. As I make progress on the mod I will probably be making additional posts about the various things I run into. I'm sure at least some of them will be interesting :)

Posts in this series:

All of my posts about Fallout 3 modding.