MikaelKalms Does Not Recommend Mod Tools for Frostbite 1.5
From the electronicarts.co.uk forums:
Zh1nt0 and you folks have asked about it, so here's a piece on the modtools situation for BC2 PC.
Frostbite 1.5 consists of these components:
The game runtime is distributed outside of EA, but the editor + pipeline + Maya plugins are not.
So let's take a look at some things that would need to be solved before we'd be ready to distribute the editor + pipeline. <!--more-->
Pipeline operation
Let's say that you tell the pipeline to build level MP_003.
MP_003 is represented by an XML file, which references a bunch of other files. These in turn reference other files. If you follow this graph of references, you will find the level layout, heightmap, characters, weapons, vehicles, and all the content that you can see in-game. (The in-game HUD and related stuff might also be in the graph.)
When the pipeline is about to build MP_003, it will first perform a consistency check on all content, and yell if any file that is referenced by any other is not present.
If all files are present, the pipeline will attempt to convert all files referenced by MP_003. It uses the file system journal to determine which files have changed on-disk. Also, and any files that have already been converted have info on which files depend on it (so it has info like: "if file X changes, then files Y,Z,W will also need to be rebuilt").
Building all content for BC2 from scratch takes something like 48-72 hours on a normal workstation. Half that time is spent building common content (such as character animations), half builds level-specific content.
In addition, there's a caching mechanism: if the pipeline wants to build a specific bit of content, it will first check if the pre-built content is already available on a cache server and take the result directly from the cache server instead. The pipeline can also populate the cache if it builds something new.
Pipeline issues
So how does this work in practice? It's not ideal, but it's good enough for us to ship games on it.
The pipeline is a bit overzealous with regards to rebuilding assets - sometimes it rebuilds stuff that it shouldn't need to.
The pipeline will normally crash about 2-3 times during a full rebuild.
You need to have Maya 8.5 (32-bit version) installed in order to convert any meshes.
Any content in the cache expires after 3 weeks. After 3 weeks have passed, that content will need to be rebuilt and re-uploaded by a machine running the pipeline. The effect that this has on day-to-day development is minimized by having one or two machines dedicated to running the pipeline every time any content change is done. By running the pipeline, those machines will populate the cache, thereby speeding up the build process for everyone else. (The output form those content build steps is discarded.)
In short: the pipeline + cache setup works better the more people are using it simultaneously.
If there are content errors, you need to know a lot about the internals of the game engine to figure out what's wrong.
Finally, in its current form, the pipeline + editor expects some specific IT infrastructure in place (most notably the cache server and a Perforce server).
If it's not there then the pipeline + editor will behave strangely.
The first time I tried, it took me about one week to get the full editor + pipeline setup to work properly outside of the DICE office. And that was when I had the option to call any of the other developers to ask for help.
... does this sound bad to you?
Truth be told, this is approximately where the industry average is at for game studios' internal game engines. One of FB 1.5's weaknesses is specifically that its content processing is flaky, and the flakiness gets more problematic as the amount of content goes up. FB 2.0 is much improved in this regard, but FB 1.5 is what we're using for BC2 and that's what relevant in the current discussion (or monologue if you prefer).
Content
Both the pipeline and the editor takes in all content in its raw, original form. Anyone who is to build any content needs the full 80GB of raw data on their machine. We are not comfortable giving out all our animations, meshes etc in raw form.
We are comfortable giving out the processed data - after all, that's what on the game disc - but that data does not plug into the editor/pipeline at all.
Licenses
The game, editor and pipeline all use commercial middleware. It is developed by Havok and several other companies.
The licensing agreement for the middleware allows us to use that code in specific products, on specific platforms.
If we want to release editor + pipeline, we need to license the middleware specifically for this. How much would that be? Perhaps $1M-$3M. I'm guessing wildly here.
Stripping out that middleware would seriously hamper the functionality especially of the pipeline. We use Havok Physics, for instance. Without Havok Physics, the pipeline wouldn't be able to convert any of the physics meshes. We also use Granny. Without Granny, the pipeline will not be able to convert any of the character animations. Etc.
Re-implementing the necessary functionality of the middleware ourselves ("let's make our own physics engine / let's plug in an open-source physics engine") would take literally man-years. Licensing is cheaper in pure $ cost and faster (it works now instead of by 2012).
The pipeline also uses some code that is under GPL. Given that we do not want to release the full source code for the editor + pipeline, we would need to replace the GPLed code with other implementations.
The GPLed code is less of a problem than the proprietary middleware.
Editor
The editor itself is reasonably stable and well-behaving. It is far from obvious how to set up the game logic for a level, but that is easily covered by releasing some example levels which contain the logic setup for the common gamemodes.
Test-running levels
First the level needs to be successfully processed by the pipeline. Then you'd want to be able to test it locally. That involves having a listen server around. We don't have a listen server neatly packaged. There's probably a piracy angle here too but I'm not going to discuss that.
Distribution of levels
Getting levels onto the RSPs server machines would likely not be any problem. However there's need for checksumming levels, so that game clients can know whether or not they have the correct version of level X on their machines. There's a whole bunch of other things (mainly UI-related) which will need cleaning up as well. Not difficult to do, just takes time and I'm listing it for the sake of completeness.
Also, there are some complications wrt when we release patches that affect the base game's content. Whenever we release a patch, all existing levels will need to be rebuilt with a new set of original data. This is because some level-common data is stored inside of the level archives. I'm not sure at the time of writing, but that probably means that the only manageable way for us would be to invalidate any user-made levels when we release a patch of that form.
Then creators of any user-generated levels would be required to run their levels again through the pipeline with the new base content supplied.
So how about just a map editor?
If it doesn't plug into the ecosystem above, then getting it to work involves some serious wrangling. Either it is a light-weight replacement for our existing editor - in which case all the challenges with the pipeline still remain - or it is a separate mode (think Forge for Halo). Developing an extra mod-layer that is sandwiched into the game would easily take 6-12 months.
Synergy effects between FB 1.5 and FB 2.0
So let's say that we would go through the procedure of making mod tools for FB 1.5. How much of that work would be reusable for FB 2.0?
I don't have any firm figures, but the differences between FB 1.5 and FB 2.0 are pretty large by now. Given this and the fact that a fair bit of the FB 1.5-specific problems (where the devil often is in the details) don't apply to FB 2.0, I'd guess that less than half of the work would port over to FB 2.0.
Conclusion
In conclusion, my recommendation to the rest of DICE is not to develop mod tools for BC2 PC. There are too many hurdles to overcome. That energy is better spent elsewhere, be that on BC2 or other titles.
So there you have it. A lot of issues and problems to make these tools work. Maybe Frostbite 2 will be more stable.
Source
Zh1nt0 and you folks have asked about it, so here's a piece on the modtools situation for BC2 PC.
Frostbite 1.5 consists of these components:
- The game runtime
- The editor runtime
- The content processing runtime (aka "the pipeline")
- and some plugins for Maya
The game runtime is distributed outside of EA, but the editor + pipeline + Maya plugins are not.
So let's take a look at some things that would need to be solved before we'd be ready to distribute the editor + pipeline. <!--more-->
Pipeline operation
Let's say that you tell the pipeline to build level MP_003.
MP_003 is represented by an XML file, which references a bunch of other files. These in turn reference other files. If you follow this graph of references, you will find the level layout, heightmap, characters, weapons, vehicles, and all the content that you can see in-game. (The in-game HUD and related stuff might also be in the graph.)
When the pipeline is about to build MP_003, it will first perform a consistency check on all content, and yell if any file that is referenced by any other is not present.
If all files are present, the pipeline will attempt to convert all files referenced by MP_003. It uses the file system journal to determine which files have changed on-disk. Also, and any files that have already been converted have info on which files depend on it (so it has info like: "if file X changes, then files Y,Z,W will also need to be rebuilt").
Building all content for BC2 from scratch takes something like 48-72 hours on a normal workstation. Half that time is spent building common content (such as character animations), half builds level-specific content.
In addition, there's a caching mechanism: if the pipeline wants to build a specific bit of content, it will first check if the pre-built content is already available on a cache server and take the result directly from the cache server instead. The pipeline can also populate the cache if it builds something new.
Pipeline issues
So how does this work in practice? It's not ideal, but it's good enough for us to ship games on it.
The pipeline is a bit overzealous with regards to rebuilding assets - sometimes it rebuilds stuff that it shouldn't need to.
The pipeline will normally crash about 2-3 times during a full rebuild.
You need to have Maya 8.5 (32-bit version) installed in order to convert any meshes.
Any content in the cache expires after 3 weeks. After 3 weeks have passed, that content will need to be rebuilt and re-uploaded by a machine running the pipeline. The effect that this has on day-to-day development is minimized by having one or two machines dedicated to running the pipeline every time any content change is done. By running the pipeline, those machines will populate the cache, thereby speeding up the build process for everyone else. (The output form those content build steps is discarded.)
In short: the pipeline + cache setup works better the more people are using it simultaneously.
If there are content errors, you need to know a lot about the internals of the game engine to figure out what's wrong.
Finally, in its current form, the pipeline + editor expects some specific IT infrastructure in place (most notably the cache server and a Perforce server).
If it's not there then the pipeline + editor will behave strangely.
The first time I tried, it took me about one week to get the full editor + pipeline setup to work properly outside of the DICE office. And that was when I had the option to call any of the other developers to ask for help.
... does this sound bad to you?
Truth be told, this is approximately where the industry average is at for game studios' internal game engines. One of FB 1.5's weaknesses is specifically that its content processing is flaky, and the flakiness gets more problematic as the amount of content goes up. FB 2.0 is much improved in this regard, but FB 1.5 is what we're using for BC2 and that's what relevant in the current discussion (or monologue if you prefer).
Content
Both the pipeline and the editor takes in all content in its raw, original form. Anyone who is to build any content needs the full 80GB of raw data on their machine. We are not comfortable giving out all our animations, meshes etc in raw form.
We are comfortable giving out the processed data - after all, that's what on the game disc - but that data does not plug into the editor/pipeline at all.
Licenses
The game, editor and pipeline all use commercial middleware. It is developed by Havok and several other companies.
The licensing agreement for the middleware allows us to use that code in specific products, on specific platforms.
If we want to release editor + pipeline, we need to license the middleware specifically for this. How much would that be? Perhaps $1M-$3M. I'm guessing wildly here.
Stripping out that middleware would seriously hamper the functionality especially of the pipeline. We use Havok Physics, for instance. Without Havok Physics, the pipeline wouldn't be able to convert any of the physics meshes. We also use Granny. Without Granny, the pipeline will not be able to convert any of the character animations. Etc.
Re-implementing the necessary functionality of the middleware ourselves ("let's make our own physics engine / let's plug in an open-source physics engine") would take literally man-years. Licensing is cheaper in pure $ cost and faster (it works now instead of by 2012).
The pipeline also uses some code that is under GPL. Given that we do not want to release the full source code for the editor + pipeline, we would need to replace the GPLed code with other implementations.
The GPLed code is less of a problem than the proprietary middleware.
Editor
The editor itself is reasonably stable and well-behaving. It is far from obvious how to set up the game logic for a level, but that is easily covered by releasing some example levels which contain the logic setup for the common gamemodes.
Test-running levels
First the level needs to be successfully processed by the pipeline. Then you'd want to be able to test it locally. That involves having a listen server around. We don't have a listen server neatly packaged. There's probably a piracy angle here too but I'm not going to discuss that.
Distribution of levels
Getting levels onto the RSPs server machines would likely not be any problem. However there's need for checksumming levels, so that game clients can know whether or not they have the correct version of level X on their machines. There's a whole bunch of other things (mainly UI-related) which will need cleaning up as well. Not difficult to do, just takes time and I'm listing it for the sake of completeness.
Also, there are some complications wrt when we release patches that affect the base game's content. Whenever we release a patch, all existing levels will need to be rebuilt with a new set of original data. This is because some level-common data is stored inside of the level archives. I'm not sure at the time of writing, but that probably means that the only manageable way for us would be to invalidate any user-made levels when we release a patch of that form.
Then creators of any user-generated levels would be required to run their levels again through the pipeline with the new base content supplied.
So how about just a map editor?
If it doesn't plug into the ecosystem above, then getting it to work involves some serious wrangling. Either it is a light-weight replacement for our existing editor - in which case all the challenges with the pipeline still remain - or it is a separate mode (think Forge for Halo). Developing an extra mod-layer that is sandwiched into the game would easily take 6-12 months.
Synergy effects between FB 1.5 and FB 2.0
So let's say that we would go through the procedure of making mod tools for FB 1.5. How much of that work would be reusable for FB 2.0?
I don't have any firm figures, but the differences between FB 1.5 and FB 2.0 are pretty large by now. Given this and the fact that a fair bit of the FB 1.5-specific problems (where the devil often is in the details) don't apply to FB 2.0, I'd guess that less than half of the work would port over to FB 2.0.
Conclusion
In conclusion, my recommendation to the rest of DICE is not to develop mod tools for BC2 PC. There are too many hurdles to overcome. That energy is better spent elsewhere, be that on BC2 or other titles.
So there you have it. A lot of issues and problems to make these tools work. Maybe Frostbite 2 will be more stable.
Source
Comment