Inverse rendering tutorials

Mitsuba 3 can be used to solve inverse problems involving light using a technique known as differentiable rendering. This enables us to estimate physical attributes of a scene, e.g., reflectance, geometry, and lighting, from images. The following tutorials will teach you how to use Mitsuba 3 in such applications:

• Gradient-based optimization
  • Overview
  • Setup
  • Scene loading
  • Reference image
  • Initial state
  • Optimization
  • Results
  • See also
• Forward inverse rendering
  • Overview
  • Setup
  • Preparing the scene
  • Rendering
  • Visualizing the gradient image
  • Using latent variables (advanced)
  • See also
• Caustics optimization
  • Overview
  • 0. Setup
  • 1. Choosing a configuration
  • 2. Creating the scene
    • Creating the lens mesh
    • Creating the emitter
    • Creating the integrator
    • Assembling the scene
  • 3. Loading the reference image
  • 4. Creating the displacement texture
  • 5. Applying the displacement texture
  • 6. Running the optimization
  • 7. Visualizing the results
  • See also
• Object pose estimation
  • Overview
  • Setup
  • direct_reparam and scene construction
  • Reference image
  • Optimizer and latent variables
  • Visualizing the results
  • See also
• Volumetric inverse rendering
  • Overview
  • Setup
  • Creating multiple sensors
  • Rendering synthetic reference images
  • Setting up the optimization scene
  • Optimization
  • Intermediate results
  • Volume upsampling
  • Continuing the optimization
  • Final results
  • See also
• Shape optimization
  • Overview
  • Setup
  • Batched rendering
  • Scene construction
  • Naive optimization
  • Large Steps Optimization
  • Intermediate result
  • Remeshing
  • Final result
  • See also
• Radiance Field Reconstruction (NeRF-like)
  • Setup
  • Parameters
  • Creating multiple sensors
  • Rendering synthetic reference images
  • NeRF-like PRB integrator
  • Setting up the optimization scene
  • Optimization
  • Results
  • See also
• Polarizer optimization
  • Overview
  • Reference image
  • Setup optimization
  • Optimization
  • Results
  • See also
• PyTorch and Mitsuba interoperability
  • Overview
  • Setup
  • Load texture dataset
  • Scene construction
  • Wrap the rendering code
  • Instantiate a fully connected layer
  • Optimization loop
  • Results
  • See also

Gradient-based optimization

Forward inverse rendering

Caustics optimization

Object pose estimation

Volumetric inverse rendering

Shape optimization

Radiance Field Reconstruction (NeRF-like)

Polarizer optimization

PyTorch and Mitsuba interoperability