Release: Advanced Renderer v. 1.1 beta 1

Schmackbolzen

I finished version 1.1 beta 1: Download

Preview image:

Example video: Link

This version got a lot larger than originally planned. The renderer now supports clustered shading which allows for thousands of point lights. I plan to extend this for spot lights as well in the future. Note that performance will degrade the more lights have overlapping ranges.

I also added the option to replace those fake lights from FL with real point lights and a much better glow effect using real sphere geometry and PBR bloom. The range of such a point light is three times the glow size. The size of the sphere is the bulb size. You can find those values in DATA\EQUIPMENT\light_equip.ini.

Additionally I implemented automatic camera exposure (was needed, because the scenes are now darker because of all the other fixes). This is an early version and probably will need some more improvements. But it is quite usable already.

There have been many internal changes (more reversed stuff, optimizations and bug fixes, some rewrites (I even wrote test cases for verification (!)), etc.), which will be very helpful for future versions. For that reason I decided to increase the version number to 1.1.

Because of all the changes the graphics again look a lot better than before (you should notice it especially when comparing with older versions).

As always I am looking forward to your feedback

List of changes:

Added clustered shading which allows thousands of point lights (will hopefully be extended to spot lights later on)
Added option to replace fake lights from FL with real point lights
Added much better glow effect when replacing fake lights with real lights using sphere geometry and PBR bloom
Added automatic camera exposure (early version for testing)
Added the option to use precise SRGB colour conversion which improves colours a lot (for transparency this is now always enabled since it caused wrong colours)
Special lights e.g. from explosions etc. are now recognized and will be used automatically
Many internal changes which will be helping a lot in future versions (e.g. used material types etc. are now known)
When using glass materials without textures glass PBR settings are now applied automatically
Improved appearance of light scattering
Light range is now more similar to original FL light range (was too bright previously)
Various optimizations and bug fixes

Timmy51m

Good stuff. Looks nice in the video. Thanks for continuing to work on this, much appreciated.

SWAT_OP-R8R

I did a short test and it is indeed a big improvement over the last version.
The replace fake lights is really nice and adds much more detail to the game. There seem to be a few minor issues that I notices (such as dock lights being way too big and some other lights on the contrary even disappearing) but I guess that is a matter of what settings the lights use. I am a bit short on time lately but will take a close look at later.´
The clustered shading itself is extremely good compared to the previous version. Lights and reflections fit much better now. Just the fake light replacement in some cases feels a bit strange… but like I said… probably just a matter of creating correct settings for it.

The SRGB color Conversation looks perfect the way it is. Much more realistic.

The automatic camera exposure indeed helps with the darker scenes, especially on some decks and bars. In some cases it maybe felt a bit too bright already but maybe I just simply got used to the darker look.
A problem maybe is the exposure in space when it switches from bright to dark environment. e.g. when looking at a bright planet surface/atmosphere and then tilting the ship away to a dark starsphere.
The exposure transition is too abrupt and feels a bit disturbing (almost buggy). Maybe a longer fade time would help here.

Also during my test I stumbled over a some very weird lighting issues in an asteroid field. I am not sure if this is new in 1.1 beta 1 or if this existed before. I’ll make a video next week so you can see what I mean. Maybe something that needs to be fixed, maybe just simple something that needs to be changed on my end.

w0dk4

Great work, keep it up

Schmackbolzen

Thanks guys, very much appreciated!

@SWAT_OP-R8R: This is really helpful. thanks! I think we basically are on the same page. And yes, you will need to adapt the light bulb sizes for some fake lights. I already started to do this in our mod.
The automatic exposure definitely needs some more tuning and increased transition time was also feedback I got from other players.
Regarding the asteroid field lighting issues: Yes, a video would be very helpful. I also got a report from testing with Discovery mod which also has problems in some fields. There is some bug I was able to mostly fix, but it is still there in some places: Distant dynamic asteroids can quickly appear and disappear when e.g. you rotate the view quickly.

SWAT_OP-R8R

Here is the vid:
https://www.youtube.com/watch?v=cj3j3bZDjpA

Maybe an issue with the light settings (the system actually is meant to be relative dark). Dunno.

I still have not had much time testing the new version and therefore can not tell if certain stuff/issues from the old version still are valid or not. I wanted to collect and discuss them with you all at once but then RL came between.
I just went through my notes and well, the latest update covered a significant amount of topics I wanted to discuss.
However, a few issues I noticed in the last version were:

Dxt1 1bit alpha textures were not rendering correctly (the alpha channel was interpreted as black)
Material type “BtDetailMapMaterial” and its variations were not rendering correctly (mostly observed on station interiors); the tilemaps were missing completely

I don’t know if these issues were known already or are still in the current version of the renderer. Like I mention, I am very short on time lately and could not test it yet.

I have a few additional questions and suggestions in my notes.

the ingame slider for metalness, roughness etc. … I was wondering if there is a way to make a texture completely matte with these sliders (I struggled to find working settings); if not, it’s still ok and probably better using dds textures instead; I was just wondering.
In the original description you mentioned to use ATI2 for the normalmaps and certainly everyone working with textures could eventually figure this out by testing, but maybe you can include the info about the channel layout into that description post. There is ATI2 and swizzeled ATI2 where red and green channels are swapped. Depending on what tools and exporters are used this info can be helpful.
Is it planned to apply the full feature set of the renderer also to static asteroids? Dynamic asteroids already make use of it but static ones so far do not make use of normalmaps etc.
I believe that this specifically would highly improve how the game looks since asteroid fields and debris fill large parts of the game. Converting the static ones into dynamic ones might be a solution but has disadvantages in other areas.
Do you have any information if fog settings in cutscenes or bases are rendered differently now? I noticed that specifically where fog is used in the scripts the lighting looks very different. I at this point would assume that in DX the direction lighting that often is used in the scripts had a very different impact on this setting. The fog now seems to be stronger.
I used to have a question about “tone mapping” but since thats the only keyword I wrote down to this topic months ago… dunno… I’ll ask again when I remember the question I had.
Last but not least something that might be a bit specific (maybe not). Many mods these days get updated by some kind of autoupdate feature that is either hash based and/or have a version based way to identify if files need to be updated.
If the renderer files are distributed via such an autoupdate the config.cfg file is at risk to be overwritten on the client each time the autoupdate runs (for some mods this means with every launch of the game). That in return means that players can not save any settings because next time the game launches the config file get overwritten.
In my mod I have some features which allow players to store their personal settings in files but also allow me to update such files if really required by using a version index.
e.g.:

[VersionHistory]
Version=1.1beta1

[Settings]
MSAAMode=0
AnisotropicFilteringMode=0
ToneMappingType=0
ShadowNumSplits=4
ShadowmapSize=4096
ShadowCastingLights=2
EnableNewLighting=1
EnableShadows=1
EnableEnvMaps=1
EnablePostProcessing=1
EnableLightScattering=1
EnableParallaxMaps=1
EnableHQDiffuse=1
EnableShadowPCF=1
EnableAsteroidShadows=1
LightScatteringSunLimit=8
EnableClusteredShading=1
MaxNumberOfLightsIndex=1
ReplaceFakeLights=1
EnablePointLights=0
EnableAutomaticExposure=1
AlwaysUseAccurateSRGBConversion=1

So players can store their settings without the config.cfg being overwritten as long as the version of the file on the update server and on the client are identical. If version 1.1beta2 is available then an updated config file (with the addiontal settings) could be downloaded.
This would allow online distribution of updates of the renderer via mods without blocking players from accessing the renderer menu and storing their settings.
I don’t know exactly how other modders distribute their updates when they have an autoupdate. I can only guess there.
The example above is how I can imagine a working solution (since I have multiple similar config files players already can store data to but also can be updated once a new version is available). Important would be that when storing the settings in game the [VersionHistory] remains untouched while only the [Settings] part gets updated.
Does that make sense? Yes, no, maybe?

A final note: I kept the renderer running with the Here is the vid:
https://www.youtube.com/watch?v=cj3j3bZDjpA

Maybe an issue with the light settings (the system actually is meant to be relative dark). Dunno.

I still have not had much time testing the new version and therefore can not tell if certain stuff/issues from the old version still are valid or not. I wanted to collect and discuss them with you all at once but then RL came between.
I just went through my notes and well, the latest update covered a significant amount of topics I wanted to discuss.
However, a few issues I noticed in the last version were:

Dxt1 1bit alpha textures were not rendering correctly (the alpha channel was interpreted as black)
Material type “BtDetailMapMaterial” and its variations were not rendering correctly (mostly observed on station interiors); the tilemaps were missing completely

I don’t know if these issues were known already or are still in the current version of the renderer. Like I mentioned, I am very short on time lately and could not test it yet.

I have a few additional questions and suggestions in my notes.

the ingame slider for metalness, roughness etc. … I was wondering if there is a way to make a texture completely matte with these sliders (I struggled to find working settings); if not, it’s still ok and probably better using dds textures instead; I was just wondering.
In the original description you mentioned to use ATI2 for the normalmaps and certainly everyone working with textures could eventually figure this out by testing, but maybe you can include the info about the channel layout into that description post. There is ATI2 and swizzeled ATI2 where red and green channels are swapped. Depending on what tools and exporters are used this info can be helpful.
Is it planned to apply the full feature set of the renderer also to static asteroids? Dynamic asteroids already make use of it but static ones so far do not make use of normalmaps etc.
I believe that this specifically would highly improve how the game looks since asteroid fields and debris fill large parts of the game. Converting the static ones into dynamic ones might be a solution but has disadvantages in other areas.
Do you have any information if fog settings in cutscenes or bases are rendered differently now? I noticed that specifically where fog is used in the scripts the lighting looks very different. I at this point would assume that in DX the direction lighting that often is used in the scripts had a very different impact on this setting. The fog now seems to be stronger.
I used to have a question about “tone mapping” but since thats the only keyword I wrote down to this topic months ago… dunno… I’ll ask again when I remember the question I had.
Last but not least something that might be a bit specific (maybe not). Many mods these days get updated by some kind of autoupdate feature that is either hash based and/or have a version based way to identify if files need to be updated.
If the renderer files are distributed via such an autoupdate the config.cfg file is at risk to be overwritten on the client each time the autoupdate runs (for some mods this means with every launch of the game). That in return means that players can not save any settings because next time the game launches the config file get overwritten.
In my mod I have some features which allow players to store their personal settings in files but also allow me to update such files if really required by using a version index.
e.g.:

[VersionHistory]
Version=1.1beta1

[Settings]
MSAAMode=0
AnisotropicFilteringMode=0
ToneMappingType=0
ShadowNumSplits=4
ShadowmapSize=4096
ShadowCastingLights=2
EnableNewLighting=1
EnableShadows=1
EnableEnvMaps=1
EnablePostProcessing=1
EnableLightScattering=1
EnableParallaxMaps=1
EnableHQDiffuse=1
EnableShadowPCF=1
EnableAsteroidShadows=1
LightScatteringSunLimit=8
EnableClusteredShading=1
MaxNumberOfLightsIndex=1
ReplaceFakeLights=1
EnablePointLights=0
EnableAutomaticExposure=1
AlwaysUseAccurateSRGBConversion=1

So players can store their settings without the config.cfg being overwritten as long as the version of the file on the update server and on the client are identical. If version 1.1beta2 is available then an updated config file (with the addiontal settings) could be downloaded.
This would allow online distribution of updates of the renderer via mods without blocking players from accessing the renderer menu and storing their settings.
I don’t know exactly how other modders distribute their updates when they have an autoupdate. I can only guess there.
The example above is how I can imagine a working solution (since I have multiple similar config files players already can store data to but also can be updated once a new version is available). Important would be that when storing the settings in game the [VersionHistory] remains untouched while only the [Settings] part gets updated.
Does that make sense? Yes, no, maybe?

On a final note: I kept the renderer running with the large address aware flag, that we discussed earlier and have not had a single crash in the past months nor have I received any reports of any issues from other players. That one crash issue, I reported earlier certainly was related to that. The issue can be considered resolved and maybe it might be worth adding info to the initial instruction post how to apply LLA. It might prevent some headaches if somebody else runs into such an issue.

Alucard_01

Two questions, as I am having difficulty to get them to work. I’ve started moving all textures to the Texture override folder, normals, etc, to the corresponding folders. But when I try and edit them, they dont show up. Even using lz4 files. Also, when setting the lighting, when I hit save setting, they only save for the time I’m in game. As soon as I resatrt FL, they all revert back default. Am I missing something? Sprry if these have been addressed already, thanks.

Alucard_01

Also I am using the UTF from here to add tangent/bionormal data.

SWAT_OP-R8R

I personally stayed aways from using lz4 after doing a test with it.
In one very specific case a texture caused crashes when starting the game when compressed to lz4. Overall I noticed random instabilities using lz4.

The settings not saving could be related to what I meantioned with autoupdates and version indexes. I could be wrong as I dont know the exact conditions under which you try to run it.

Alucard_01

OP, ok ill just use the normal textures and not use the lz4. Could you screenshot a few of your files to see if mine match? Because mine do not show up under the texture modifier in game.

SWAT_OP-R8R

Im not at home this week so I cant take any screenshots atm.
But I have a few vids from a few weeks ago.

It’s not the latest version but probably good enough.
https://www.youtube.com/watch?v=1GggMG3fehw
https://www.youtube.com/watch?v=1YPTdY2E9aY
https://www.youtube.com/watch?v=VUCgl7SUvPw
https://www.youtube.com/watch?v=hPz-_TDcJwM
https://www.youtube.com/watch?v=v4WKOhzVN_s

Alucard_01

Still nothing showing up. I’m not sure it’s looking in the right folders. Am I supposed to enter the names in the materials file? (json) And what do I add?

{
“TextureNames”: [
“tree_64.tga”,“Sprauge_Grass.tga”
],
“Roughness” : 0.2,
“Metalness” : 0.1
},

I’m guessing this tells what files to look at, but am not sure how to code for the other folder i.e Height, normal, etc.

Or am I wrong?

SWAT_OP-R8R

\Freelancer\FLARDATA\Textures
should be the folder structure

inside you have the PBR texture folders e.g. NormalMaps
Creating normal maps would be the starting point, Metalness and Roughness maps later on if you need them. HeightMaps would be a special case that you wont need that often actually.
Create the DDS files and put them into the folders using the original texture names that you can find via UTF. If you work based on original tga textures don’t forget to flip the texture before storing them as dds.
Normalmaps need to be stored as ATI2 and all others as ATI1 dds. Overwrite textures can be dxt1-5.
You dont need to manually edit the material file (but you can). You can use the ingame dev menu to set roughness and metalness values. These values get stored inside the material config file.
I personally edit the material config manually once I know the settings. This way I can group textures which have similar settings and keep everything lightweight that way.

Alucard_01

Sorry for the late response, may I can show you when I got everything to help explain this better:
https://ibb.co/0tcz1TM
https://ibb.co/BqNQVd2
https://ibb.co/zPK2MRZ
https://ibb.co/wLkh42P
so the normals are coverted correctly, in correct folder, override textures as well, but only get the one editor to come up on screen. Nothing else was added or changed. Any ideas? Also added binomal data from utf

SWAT_OP-R8R

I can only come up with some wild guesses at this point.
Since the UI is working in game the renderer seems to be active and running normally, therefore I would assume this is not the error source.
Paths look good, therefore also not the error.
Leaving the textures as potential error source. At this point I would suggest checking if the exported textures were correctly exported as ATI2/ATI1.
You can do this by simply loading a random mat file from FL into UTF. Go to a MIPS node and import one of your textures into it (no need to save the file). The right info window should display the DDS type now. IF it does not show ATI1 or ATI2 there then the export did not work correctly.

Also keep in mind that if the texture node in UTF is called eg. “mytexture.tga” (this very often is the case) then the exported file has to be named “mytexture.tga.dds”. Do not drop the .tga anywhere in the export progress.