Can't render volume based project with MtoA-4.2.4-2022

[futuremotionrecovery]

I have a project that just refuses to GPU render a scene I'm working on. It's a huge pain, because everything so much more slow on the CPU. I have a good GPU (RTX 2080 TI) and I want to use it.

I've uploaded the project here:
https://www.dropbox.com/s/rh2pfdvka6iwxx4/Volume%20Disp%20Troubleshooting.rar?dl=0

Here is the error I get in the script editor after failure:

# Warning: /aiVolume/aiVolumeShape: could not set BOOL parameter "disable_ray_extents" # 
// Error: [gpu] an error happened during rendering. OptiX error is: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: cudaDriver().CuEventSynchronize( m_event ) returned (700): Illegal address, file: <internal>, line: 0)
    GPU 0 had 10271MB free before rendering started and 7353MB free when crash occurred

Here's the Arnold log file:

00:00:00   834MB         | log started Sun Sep 26 19:46:02 2021
00:00:00   834MB         | Arnold 6.2.1.1 [fbf3b21a] windows clang-10.0.1 oiio-2.3.2 osl-1.12.0 vdb-7.1.1 clm-2.0.0.235 rlm-14.1.3 optix-6.6.0 2021/06/04 18:35:03
00:00:00   834MB         | host application: MtoA 4.2.4 3c94e997 (fix-4.2.4) Maya 2022
00:00:00   834MB         | running on DESKTOP-AIKP4FP, pid=42092
00:00:00   834MB         |  1 x Intel(R) Core(TM) i9-9900KF CPU @ 3.60GHz (8 cores, 16 logical) with 65467MB
00:00:00   834MB         |  NVIDIA driver version 471.68 (Optix 60801)
00:00:00   834MB         |  GPU 0: NVIDIA GeForce RTX 2080 Ti @ 1650MHz (compute 7.5) with 11264MB (10294MB available) (NVLink:0)
00:00:00   834MB         |  Windows 10 (version 10.0, build 19043)
00:00:00   834MB         |  soft limit for open files is set at 2048
00:00:00   834MB         |  
00:00:00   834MB         | [metadata] loading metadata file: C:/Program Files/Autodesk/Arnold/maya2022/plug-ins/mtoa.mtd
00:00:00   836MB         | [volume] searching OpenVDB file: C:/Users/futur/Desktop/Volume Displacement Lite/cache/VDBs/cloud_1_sim_050.vdb
00:00:00   836MB         |  [volume] found grid "density"
00:00:00   836MB         |  [volume] found grid "fuel"
00:00:00   836MB         |  [volume] found grid "heat"
00:00:00   836MB         |  [volume] found grid "temperature"
00:00:00   836MB         |  [volume] found grid "vel.x"
00:00:00   836MB         |  [volume] found grid "vel.y"
00:00:00   836MB         |  [volume] found grid "vel.z"
00:00:02  1332MB         | [bifrost] C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost\arnold-6.2.0.0
00:00:02  1332MB         | loading plugin: C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost/arnold-6.2.0.0/arnold_bifrost.dll ...
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_graph uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_object uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_points uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_polymesh uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_blocks uses Arnold 6.2.0.0
00:00:02  1333MB         | loaded 9 plugins from 1 lib(s) in 0:00.00
00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenSplineTranslator.dll
00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenTranslator.dll
00:00:04  1470MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/usdProxyShapeTranslator.dll
00:00:23  1506MB         |  
00:00:23  1506MB         | releasing resources
00:00:23  1504MB         |  unloading 14 plugins
00:00:23  1504MB         |   closing hairPhysicalShader_shaders.dll ...
00:00:23  1504MB         |   closing lookdevkit_shaders.dll ...
00:00:23  1504MB         |   closing mtoa_shaders.dll ...
00:00:23  1503MB         |   closing renderSetup_shaders.dll ...
00:00:23  1503MB         |   closing xgenSpline_shaders.dll ...
00:00:23  1503MB         |   closing mtoa_ParticleInstancer_proc.dll ...
00:00:23  1503MB         |   closing mtoa_ParticleVolume_proc.dll ...
00:00:23  1503MB         |   closing xgenSpline_procedural.dll ...
00:00:23  1503MB         |   closing xgen_procedural.dll ...
00:00:23  1503MB         |   closing alembic_proc.dll ...
00:00:23  1503MB         |   closing cryptomatte.dll ...
00:00:23  1503MB         |   closing synColor_shaders.dll ...
00:00:23  1502MB         |   closing usd_proc.dll ...
00:00:23  1500MB         |   closing arnold_bifrost.dll ...
00:00:23  1500MB         |  unloading plugins done
00:00:23  1500MB         | Arnold shutdown

The most relevant bits of information I think is the CUDA error:

OptiX error is: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: cudaDriver().CuEventSynchronize( m_event ) returned (700): Illegal address, file: <internal>, line: 0)

The versions of loaded bifrost dll's are on an older version of the Arnold core:

00:00:02  1332MB         | loading plugin: C:/Program Files/Autodesk/Bifrost/Maya2022/2.3.0.1/bifrost/arnold-6.2.0.0/arnold_bifrost.dll ...
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_graph uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_object uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_multires_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_volume uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_points uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_implicit uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_polymesh uses Arnold 6.2.0.0
00:00:02  1333MB         |  arnold_bifrost.dll: bifrost_blocks uses Arnold 6.2.0.0

I have the latest MtoA 4.2.4 (6.2.1.1)
https://docs.arnoldrenderer.com/display/A5AFMUG/4.2.4
Shouldn't bifrost be using the new version?

And finally the delayed loading of the extensions:

00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenSplineTranslator.dll
00:00:04  1384MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/xgenTranslator.dll
00:00:04  1470MB         | [mtoa] Delayed loading Extension: C:/Program Files/Autodesk/Arnold/maya2022/extensions/usdProxyShapeTranslator.dll

I would be overjoyed if someone could help me troubleshoot and fix this problem.

Thanks so much.

[Stephen.Blair]

This is a generic error. Could be a bug, in Arnold or in Optix, or something weird in the scene/vdb. Impossible to tell from just this error:

// Error: [gpu] an error happened during rendering. OptiX error is: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: cudaDriver().CuEventSynchronize( m_event ) returned (700): Illegal address, file: <internal>, line: 0)

[futuremotionrecovery]

Yeah. I created a new scene with some volumes and a volume shader and GPU rendering worked this time around. I think it must be some kind of quirk with the scene.

This, however, highlighted another problem. When I render a volume on the GPU, it looks way different than the CPU render. I've tried adjusting samples, turned on adaptive sampling, etc. but the GPU render is extremely noisy and looks horrible.

If I switch back to CPU, all is well. I was excited that my volume rendering was going to be much quicker, but I guess that's not the case.

Any tips on how to match the CPU in volume quality?

[Stephen.Blair]

Can you post screenshots that show the differences? Thanks

[futuremotionrecovery]

@Stephen Blair

CPU Render (~15 seconds):

GPU Render (~15 seconds)

Looks pretty horrible. I can share the file if that will help.

[Stephen.Blair]

Hi @

That looks similar to the scene you sent to support, right?

Unfortunately, volume displacement has some issues on the GPU (internal ticket #ARNOLD-9160), and this looks like an example of that issue. A mesh as a volume (the head) should match fairly well, but the VDB cloud won't.

[futuremotionrecovery]

@Stephen Blair

Well that's a bummer. Any idea on when that ticket will be resolved?