Neural Frame Rate Upscaling (NFRU)

Neural Frame Rate Upscaling (NFRU) is an efficient network for frame rate upscaling on mobile devices. Content rendered at lower frame rates can be upscaled by up to 2x, which yields improved power efficiency and perceived fluidity.

With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

🎥 Neural Frame Rate Upscaling Demo

Model Details

Neural Frame Rate Upscaling (NFRU) is a model that predicts the correct parameters needed to generate Frame T using information from Frame T-1 and Frame T+1. The model, developed by Arm, is optimized for execution on Neural Accelerators (NX) in mobile GPUs. NFRU is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

NFRU is under active development with regular updates planned. As we increase the size and diversity of the training dataset, we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases.

The model is released under Arm's permissive AI Model Community License, designed to foster innovation in the graphics industry and provide differentiation to content creators. It allows NFRU to be retrained on datasets captured from your own content.

Future releases of the Neural Graphics Model Gym will provide the tools to capture and convert content for use in (re)training.

Integration

NFRU can be directly integrated into graphics pipelines using ML extensions for Vulkan®.

The recommended way of integrating the model into a graphics pipeline is by using the VGF Library from the ML SDK for Vulkan.

Integration in Game Engines

NFRU has been integrated into Unreal Engine via the NFRU Plugin for Unreal Engine. See the Learning Path for step-by-step instructions on how to use NFRU in Unreal Engine.

It is also available to use as part of the Neural Graphics SDK for Game Engines. This is designed to help integrate NFRU across a wide variety of game engines.

Getting Started

This repository contains pre-trained weights and a compiled NFRU model in VGF format, ready for integration with Vulkan applications. The VGF model is unshaped, so will require developers to shape-specialize it before use. An example of how to do this can be seen in the Neural Graphics SDK for Game Engines.

See the included ML SDK for Vulkan scenarios for the simplest way to evaluate the model. The scenario includes the necessary pre- and post-processing fragment shaders along with a single frame worth of input data.

It demonstrates full execution of NFRU on a Vulkan compute-capable system. The Emulation Layer can be used to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

1. Clone the Repository

These instructions will work on both Windows and Linux. On Windows, we strongly recommend the use of PowerShell over cmd.exe ("Command Prompt").

1. Ensure that Git LFS is installed. See the project's website for more details.

2. Clone the NFRU model repository from Hugging Face:

git clone https://huggingface.co/Arm/neural-frame-rate-upscaling
cd neural-frame-rate-upscaling
git lfs pull

2. Install the Scenario Runner and Emulation Layers

1. Follow the steps in the Python documentation's "Create a new virtual environment" guide. We will assume below that you installed this environment into a .venv subdirectory.

2. Follow the steps in the Python documentation's "Activate a virtual environment" guide.

   Note for Windows users: It is not necessary to determine the location of the Python interpreter as described in this guide. For reference, however, the correct PowerShell command to do this is gcm python rather than the given where python.

3. Install Arm's Scenario Runner and Emulation Layer into your new virtual environment:

   pip install ai-ml-sdk-scenario-runner ai-ml-emulation-layer-for-vulkan
   

4. The Emulation Layer package provides a helper, that prints the commands to set environment variables for your current platform and virtual environment:

   • Enter the following command:

   emulation_layer --help
   

   Copy and run the commands it prints in the same shell you will use for scenario-runner.

   On Linux this will set LD_LIBRARY_PATH, VK_LAYER_PATH, and VK_INSTANCE_LAYERS. On Windows it will set the equivalent PowerShell $env:VK_LAYER_PATH and $env:VK_INSTANCE_LAYERS variables.

   For more help, the Emulation Layer's documentation explains how its environment variables should be configured on Linux and on Windows.

3. Run the Scenario

1. Change your working directory to the scenario folder:

   cd scenario
   

2. Execute the Scenario Runner with the example scenario:

   scenario_runner --scenario scenario.json --output out
   

3. Use RenderDoc to view the output images. Output images are encoded as B10G11R11_UFLOAT. This format is common for framebuffers but not widely supported by image viewers.

Training and Evaluation

For background on NFRU architecture and training please watch our walkthrough presentation.

Training and evaluation details, including model architecture code, training pipeline, and test configurations, are available at:

• Model training code: https://github.com/arm/neural-graphics-model-gym
• Examples and tutorials: https://github.com/arm/neural-graphics-model-gym-examples
• Sample dataset: https://huggingface.co/datasets/Arm/neural-graphics-dataset

🔎 Model Explorer VGF Extension

The VGF extension to Model Explorer provides a simple interface to visualize and analyse the NFRU model and any other VGF model.

Biases, Risks, Limitations and Out-of-Scope Use

• All motion between Frame T-1 and Frame T+1 is presumed to be linear
• Optical flow is required for interpolation of features without geometry (e.g. shadows, particles, etc) because these features do not have corresponding motion vectors. The included NFRU scenario will calculate these optical flow vectors via calls to VK_ARM_data_graph_optical_flow.
• Not suitable for (re)training on non-deterministic scenes

Recommendations

When collecting training/testing data, two separate render passes are required to capture both the low-frame-rate dataset and the ground-truth high-frame-rate dataset. Due to this, the scenes must be deterministic to ensure that the datasets align properly.

License

• The license for the model source code can be found here.
• The license for the content of this repository can be found here

More Information

🧑‍🔬 More technical details about the model can be found in the NFRU Paper.

👩🏽‍💻 Our Neural Graphics Development Kit contains engine plugins, model training tools, code examples and extensive developer documentation.

🙋🏻‍♀️ For questions or feedback please start a discussion

Trademark Notice

• Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere.
• Python® is a registered trademark of the Python Software Foundation.
• Unreal® is a trademark or registered trademark of Epic Games, Inc. in the United States of America and elsewhere.
• Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc..
• Windows® is a trademark of the Microsoft group of companies.