Advanced Renderer Documentation / Question - Answer thread

[Schmackbolzen]

This text refers to my OpenGL renderer (https://the-starport.com/forums/post/65081). You can try it out yourself with our mod: http://www.flnu.net/

The text is a work in progress, so more will probably be added from time to time. Feel free to ask questions. Comments are also welcome

General questions
Will the renderer be usable for everyone / can I use it in my mod?
Can I make changes to the shaders?
Will you release all the tools you wrote?
Why did it take so long?
What DirectX Version does Freelancer originally use?
What does OpenGL 3.3 mean?
What exactly does PBR renderer mean?

Technical background questions
What exactly does the renderer change?
What optimizations did you add?
Why did you chose OpenGL and not Direct3D?
Will there be Vulkan support?
How much reverse engineering did you have to do?

Documentation for the renderer
Storing of the additional model data
Location of the additional texture maps
Supported kinds of texture maps
Unique texture names
Generation of the self shadowing normal maps
Material system
Format of normal/roughness/metalness maps
Changes needed to make DPI scaling work

General questions

Will the renderer be usable for everyone / can I use it in my mod?
If your mod does not use any stolen assets (meaning you have permission to use any of the 3D models / textures etc. and the authors are also mentioned in the credits of your mod) then yes, you are free to use my renderer in your mod. You just have to mention me in the credits (and also tolerate the small text overlay displaying the renderer version on game startup). Exact details are in the top section of the supplied “FLAR licenses.txt” file along with the renderer.

Can I make changes to the shaders?
Yes, you are free to do so under the following constraint: You must write in the top section and as comment in the code of each shader file you modified what you did change and that you are the author. If you implemented a technique from a paper / other author you have to mention it / the author and where you got it from. Exact details for this are in the top comment section of each shader file.
The graphics development scene from game development is very open and has a scientific approach of doing it (meaning many papers and presentations including citations). I fully support this approach and wrote down any source / paper I used including the author.

Will you release all the tools you wrote?
Yes, I will release all the tools I wrote/modified (including the source code under GPL3 license). This includes tools for generating the additional textures, converting a mod to only have unique texture names, the modified UTF editor for recalculating normals and adding tangent space data to models.

Why did it take so long?
I developed it during my spare time and when I was in the mood. This is basically a hobby of mine and was (mostly) a lot of fun. Developing it was not only just the graphics, but also solving the problems of how to convert an old renderer to a modern physical based render (e.g. it has to be gamma correct). This also included writing an algorithm for recalculating the 3D model normals (not ideally solved currently), modifying the TBN data calculation for models, upscaling the textures so they are not so pixelated and automatically generating additional texture maps, so that it looks at least good, while not being completely correct. I have a background in image processing (besides 3D rendering), so this helped a lot.
On top of that there is no source code of Freelancer, so I had to reverse engineer a lot of stuff. I tried to keep it at a minimum, because this always costs the most time. But still this caused a lot of slowdown.
More details about reverse engineering are in the “How much reverse engineering did you have to do?” section.

What DirectX Version does Freelancer originally use?
Freelancer originally uses DirectX 8.0.

What does OpenGL 3.3 mean?
Think of it like Direct3D 10 (which is the rendering API of DirectX 10), but open and not owned by anyone. Thus also Mac OS and Linux support it. For phones there is OpenGL ES, which is a cut down version.

What exactly does PBR renderer mean?
For reference: The renderer uses the metalness workflow.
Here are some a nice explanations:
https://marmoset.co/posts/physically-based-rendering-and-you-can-too/
https://marmoset.co/posts/pbr-texture-conversion/
https://www.a23d.co/blog/different-maps-in-pbr-textures/

Technical background questions

What exactly does the renderer change?
The renderer is a full PBR renderer, meaning it can use modern 3D models with all the additional texture maps. The formulas used for lighting are what modern games use. Additionally it adds shadows to the game and also post processing effects like tone mapping and light scattering. The background (currently the starsphere) also is used for the ambient lighting and the roughness of the material directly influences its sharpness (you can create e.g. a perfect mirror).
For each frame all render calls are being stored, newly sorted and then rendered. This allows for fixing transparency bugs (e.g. the nomads now are being rendered correctly), use instancing as optimization and also render the scene multiple time for special effects like light scattering.
Also if this is not necessary (e.g. when docked) just directly rendering without saving (translated to OpenGL) is being done.

What optimizations did you add?
Basically the complete pipeline is a modern rendering pipeline now. Where it is possible instancing is being used. The renderer also has an own state tracker which reduces unnecessary state switches for the graphics card. All the render calls are being sorted by texture usage so that unnecessary texture switches are avoided. Transparent objects are rendered at the end, fixing some weird transparency bugs from the original game. There also is an added vertex and index buffer cache, since Freelancer likes to unnecessarily render geometry without using a constant buffer on the graphics card.

Why did you chose OpenGL and not Direct3D?
Basically OpenGL makes it easier to run on other operating systems like Linux and Mac OS. Plus I can dynamically use extensions if the hardware supports it. In DirectX terms this means I am not bound to just use Direct3D 10, but also features from Direct3D 11. In fact I already am doing this for optimizations. While the renderer uses OpenGL 3.3 (which means the same hardware features as Direct3D 10) if the hardware supports it features from Direct3D 11 are used.
Another plus is that a transition to Vulkan is not as difficult which also makes it easer for running it on other operating systems besides Windows. Vulkan would allow to include ray tracing support.

Will there be Vulkan support?
This is another advantage of using OpenGL. Internally most of the renderer is already similar written to modern pipelines, and the shaders are nearly the same, so a transition would not be as difficult. Whether I will do it depends on many factors (e.g. how many mods/players use it etc.).
Vulkan would allow to include ray tracing support.

How much reverse engineering did you have to do?
Since adoxa had not done any graphics rendering, there was not much to start with. FriendlyFire asked him back then, but he only could give us one function. This was at least a good starting point. FriendlyFire and me also exchanged some code, but since the Freeworlds renderer works a lot different than mine, there was not so much overlap.
For supporting shadows, you have to track all the objects in the game and know their geometry. This also means you have to know when they are being destroyed (meaning memory has been freed). This is not as easy as it sounds.
For the additional texture maps you have to know which textures are currently being used.
Also to render objects correctly (select the right shader etc.), you have to know which object is currently being rendered (3DB, CMP etc.).
I have reversed lot of the materials (sadly each type is an own class) and know which one is currently being used, but I had not the time to reverse all of them, so this is not implemented currently.
Dynamic asteroids are directly read from memory and rendered by me (not by the game). This allows instancing (meaning only one draw call and not one for each asteroid like originally). Reversing static asteroids is on my list and are hopefully similarly implemented in Freelancer. Since I save the render calls at least instancing is used here, too (but it costs more CPU time due to the overhead).
There still was a lot other stuff which was needed and I might add here later on.

Documentation for the renderer

Storing of the additional model data
For some of the additional texture maps you need to do all the computation in tangent space. For this not only normal, but also binormal and tangent vectors are needed. This is documented very well, so I won’t explain it any further here. I use the same format as the UTF Editor already supports (additional texture coordinates), with the exception that I save one additional vector with 4 components instead two additional vectors with 3 components. You can calculate the binormal vector just with the cross product of tangent and normal vector in the shader. You just have to store the orientation of the binormal vector, meaning whether it is flipped or not. For this the tangent vector gets one addition component (the 4.) which stores either 1 or -1. This technique is also very well documented and nothing new. It is more graphics card friendly, meaning it is faster, despite doing more computations. This is because for graphics cards memory accesses are slower than computing a value (unless the computation is very slow, which is not the case here).
I will release a modified UTF editor (with source) which is able to generate the additional vector and also contains my normal vector recalculation algorithm (more on that later).

Supported kinds of texture maps
In addition to the original textures Freelancer supports, currently supported are: Normal, height, roughness and metalness maps. If there is demand for more they can easily be added. The height maps are used for parallax occlusion mapping with offset limiting. Also they act as a base for the ssbump normal maps (see “Generation of the self shadowing normal maps”).

Location of additional texture maps
Additional texture maps (like normal maps) are loaded according to the name of the base texture. Meaning they have to have the same name as the texture they are made to be used with. The type of texture map is determined by the folder they are placed in. E.g. normals maps are located at “FLARDATA\Textures\Normalmaps”. The names of all folders are:
• OverrideTextures
• NormalMaps
• RoughnesMaps
• MetalnessMaps
• HeightMaps
Textures located in OverrideTextures folder will be loaded instead of the one in a .mat, .txm etc. file. The other names are self explanatory.
The renderer can load lz4 compressed dds texture files, but also directly load dds textures. It will prefer lz4 and if that is not found it will look for the dds file. For compressing / decompressing lz4 files you can use e.g. https://github.com/mcmilk/7-Zip-zstd.

Unique texture names
Since additional texture maps are loaded according to the name of the base texture, this means that each texture in the game has to have a unique name. I developed tools for making sure of this, but it still takes some effort to convert a mod.
There might be a better solution in the future. This was the fastest solution to implement and also has the plus side that textures can easily be reused.
For our mod I wrote a tool which replaced all the textures in the .mat files with 1x1 pixel textures and I just use the one from the override folder. Overall this saved a lot of disk space and also you easily notice if textures are duplicated / have the wrong name. The workaround with the 1x1 pixel textures is not ideal, but for the time being other features were more important.

Generation of the self shadowing normal maps
The self shadow normal maps are based on Valve’s ssbump paper (Efficient self-shadowed radiosity normal mapping, DOI: https://doi.org/10.1145/1281500.1281664) and a forum post I found with nice c++ code (https://www.gamedev.net/forums/topic.asp?topic_id=557465) which applies this to tangent space normal maps. They work really well in Freelancer. I cleaned up the code, added png texture loading and also made it multi threaded. Of course I will release it.

Material system
Note that the material system currently is more of a temporal solution and might be improved later on. You can store additional information about a material (currently based on the texture name or multiple texture names) in the materials.json file located in the FLARDATA directory. You can specify constant values for metalness and roughness (textures will override this). If you want to modify the roughness value of a texture you can use “RoughnessTextureBias”, which will be added in the shader (you can use negative values). Note that the final values in the shader are clamped between 0 and 1.
The “AlphaTestValue” property is used for shadows and allows to have transparency enabled for textures. As the name suggest this is only an alpha test currently (so only full transparent or opaque).

Format of normal/roughness/metalness maps
The normal map is stored as ati2n dds, whereas the other maps use ati1n dds.
Make sure that the normal map is properly normalized (not all tools do that before converting).

Changes needed to make DPI scaling work
You need to add a manifest entry to your Freelancer.exe. The content is:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3">
  <asmv3:application>
    <asmv3:windowsSettings>
      <dpiAware xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">true</dpiAware>
      <dpiAwareness xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">PerMonitorV2</dpiAwareness>
    </asmv3:windowsSettings>
  </asmv3:application>
</assembly>