Arnold GPU crahes when rendering

jean_leoni · ‎01-08-2020

Hello everyone, im using arnold 4.0.1 for maya 2019 when i'm receiving the next message when i try to render some image on arnold gpu:

I'm using geforce gtx 1060

the verbosity log:

00:02:15 6894MB ERROR | [gpu] an error happened during GPU rendering : Insufficient device memory

00:02:20 7062MB ERROR | signal caught: error C0000005 -- access violation

****

* Arnold 5.4.0.2 [30d7daaa] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/10/09 17:19:01

* CRASHED in optixQueryFunctionTable at 00:00:00

* signal caught: error C0000005 -- access violation

*

* backtrace:

* 0 0x00007ff9342c61ce [ai ]

* 1 0x00007ff9342c546f [ai ]

* 2 0x00007ff9c186f67a [KERNELBASE ] UnhandledExceptionFilter

* 3 0x00007ff9c3cc4af2 [ntdll ] memset

* 4 0x00007ff9c3cac6d6 [ntdll ] _C_specific_handler

* 5 0x00007ff9c3cc11ff [ntdll ] _chkstk

* 6 0x00007ff9c3c8a289 [ntdll ] RtlRaiseException

* 7 0x00007ff9c3cbfe6e [ntdll ] KiUserExceptionDispatcher

>> 8 0x00007ff92124cfef [nvoptix ] optixQueryFunctionTable

* 9 0x00007ff921240ec1 [nvoptix ] optixQueryFunctionTable

* 10 0x00007ff920c201ae [nvoptix ] optixQueryFunctionTable

* 11 0x00007ff920c0a0e1 [nvoptix ] optixQueryFunctionTable

* 12 0x00007ff920c3153e [nvoptix ] optixQueryFunctionTable

* 13 0x00007ff920c32af2 [nvoptix ] optixQueryFunctionTable

* 14 0x00007ff920c30e37 [nvoptix ] optixQueryFunctionTable

* 15 0x00007ff920c47c10 [nvoptix ] optixQueryFunctionTable

* 16 0x00007ff921208647 [nvoptix ] optixQueryFunctionTable

* 17 0x00007ff9211f1913 [nvoptix ] optixQueryFunctionTable

* 18 0x00007ff920a554b0 [nvoptix ] optixQueryFunctionTable

* 19 0x00007ff9209e8871 [nvoptix ] optixQueryFunctionTable

* 20 0x00007ff93413b753 [ai ] AiNodeEntryGetCount

* 21 0x00007ff9342a983f [ai ] AiNodeSetVec

* 22 0x00007ff9b6c41030 [VCRUNTIME140 ]

* 23 0x00007ff9b6c432e8 [VCRUNTIME140 ] is_exception_typeof

* 24 0x00007ff9c3cc0646 [ntdll ] RtlCaptureContext

* 25 0x00007ff9342a942b [ai ] AiNodeSetVec

* 26 0x00007ff9342d6ba0 [ai ]

* 27 0x00007ff93410a0b0 [ai ] AiVolumeSampleFltFunc

* 28 0x00007ff934322560 [ai ] AiRenderResume

* 29 0x00007ff9343221b9 [ai ] AiRenderResume

* 30 0x00007ff933fde555 [ai ] AiRender

* 31 0x00007ff9489ca68d [mtoa ] CRenderSwatchGenerator::operator=

* 32 0x00007ff94fd42a98 [OpenMaya ] THcommandObject::doIt

* 33 0x00007ff968d77ae8 [CommandEngine ] TmetaCommand::doCommand

* 34 0x00007ff968dd0063 [CommandEngine ] TpythonInterpreter::dispatchMayaCommand

* 35 0x00007ff968dd2483 [CommandEngine ] TpythonInterpreter::genericDispatchFunction

* 36 0x000000005650dbb6 [python27 ] PyCFunction_Call

* 37 0x00000000564caaf6 [python27 ] PyObject_Call

* 38 0x00000000565703dc [python27 ] PyEval_CallObjectWithKeywords

* 39 0x0000000056565281 [python27 ] PyAST_FromNode

* 40 0x000000005650dba9 [python27 ] PyCFunction_Call

* 41 0x00000000565707f4 [python27 ] PyEval_GetFuncDesc

* 42 0x000000005656e08c [python27 ] PyEval_EvalFrameEx

* 43 0x000000005656f3ef [python27 ] PyEval_EvalCodeEx

* 44 0x00000000564fc3d8 [python27 ] PyFunction_SetClosure

* 45 0x00000000564caaf6 [python27 ] PyObject_Call

* 46 0x00000000565710f5 [python27 ] PyEval_GetFuncDesc

* 47 0x000000005656e132 [python27 ] PyEval_EvalFrameEx

* 48 0x00000000565709a8 [python27 ] PyEval_GetFuncDesc

* 49 0x0000000056570868 [python27 ] PyEval_GetFuncDesc

* 50 0x000000005656e08c [python27 ] PyEval_EvalFrameEx

* 51 0x000000005656f3ef [python27 ] PyEval_EvalCodeEx

* 52 0x000000005656b2b9 [python27 ] PyEval_EvalCode

* 53 0x00007ff968dce4c8 [CommandEngine ] TpythonInterpreter::executeCommand

* 54 0x00007ff968de2bd2 [CommandEngine ] TpythonFlagException::cache

* 55 0x00007ff968d78da2 [CommandEngine ] TmetaCommandPtrArray::catenate

* 56 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 57 0x00007ff968dc63e4 [CommandEngine ] melScanDouble

* 58 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 59 0x00007ff968d9c814 [CommandEngine ] TminMelArgList::nextDigit

* 60 0x00007ff968dc3e37 [CommandEngine ] SophiaExecutable::evaluate

* 61 0x00007ff968d806ae [CommandEngine ] TcommandEngine::executeCommand

* 62 0x00007ff9683e8e1b [ExtensionLayer] QmayaMenuItem::version

* 63 0x00007ff968d78da2 [CommandEngine ] TmetaCommandPtrArray::catenate

* 64 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 65 0x00007ff968dc5d71 [CommandEngine ] melScanDouble

* 66 0x00007ff968dc544a [CommandEngine ] melScanDouble

* 67 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 68 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 69 0x00007ff968dc54f7 [CommandEngine ] melScanDouble

* 70 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 71 0x00007ff968dc54f7 [CommandEngine ] melScanDouble

* 72 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 73 0x00007ff968dc58a5 [CommandEngine ] melScanDouble

* 74 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 75 0x00007ff968dc63e4 [CommandEngine ] melScanDouble

* 76 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 77 0x00007ff968dc63e4 [CommandEngine ] melScanDouble

* 78 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 79 0x00007ff968dc5d71 [CommandEngine ] melScanDouble

* 80 0x00007ff968dc552a [CommandEngine ] melScanDouble

* 81 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 82 0x00007ff968d96ca7 [CommandEngine ] TminMelArgList::nextDigit

* 83 0x00007ff968d9c814 [CommandEngine ] TminMelArgList::nextDigit

* 84 0x00007ff968dc3e37 [CommandEngine ] SophiaExecutable::evaluate

* 85 0x00007ff968d81bf2 [CommandEngine ] TcommandEngine::sourceFile

* 86 0x00007ff767dd357b [mayabatch ] TiteratorWrapperFwd<TmultiIndexDependNodeSet::NodesOfTypeConstIteratorDescriptor,TiteratorWrapper<TmultiIndexDependNodeSet::NodesOfTypeConstIteratorDescriptor> >::operator=

* 87 0x00007ff767dd2f46 [mayabatch ] TiteratorWrapperFwd<TmultiIndexDependNodeSet::NodesOfTypeConstIteratorDescriptor,TiteratorWrapper<TmultiIndexDependNodeSet::NodesOfTypeConstIteratorDescriptor> >::operator=

* 88 0x00007ff9683ba2e7 [ExtensionLayer] Tapplication::start

* 89 0x00007ff767dd2101 [mayabatch ]

* 90 0x00007ff767de59a1 [mayabatch ] TiteratorWrapperFwd<TscenePartitions::ConstIteratorDescriptor,TiteratorWrapper<TscenePartitions::ConstIteratorDescriptor> >::operator++

* 91 0x00007ff9c2d97bd4 [KERNEL32 ] BaseThreadInitThunk

* 92 0x00007ff9c3c8ced1 [ntdll ] RtlUserThreadStart

*

* loaded modules:

* 0x00007ff933f70000 ai

* 0x00007ff9c1770000 KERNELBASE

* 0x00007ff9c3c20000 ntdll

* 0x00007ff920690000 nvoptix

* 0x00007ff9b6c40000 VCRUNTIME140

* 0x00007ff948970000 mtoa

* 0x00007ff94fd00000 OpenMaya

* 0x00007ff968d60000 CommandEngine

* 0x0000000056420000 python27

* 0x00007ff9683a0000 ExtensionLayer

* 0x00007ff767dd0000 mayabatch

* 0x00007ff9c2d80000 KERNEL32

****

Stack trace:

ntdll.dll!KiUserExceptionDispatcher

nvoptix.dll!optixQueryFunctionTable

ai.dll!AiNodeEntryGetCount

ai.dll!AiNodeSetVec

VCRUNTIME140.dll!<7FF9B6C41030>

VCRUNTIME140.dll!_is_exception_typeof

ntdll.dll!RtlCaptureContext

ai.dll!AiNodeSetVec

ai.dll!AiCreateAtStringData_private

ai.dll!AiVolumeSampleFltFunc

ai.dll!AiRenderResume

ai.dll!AiRender

mtoa.mll!CRenderSwatchGenerator::operator=

OpenMaya.dll!THcommandObject::doIt

CommandEngine.dll!TmetaCommand::doCommand

CommandEngine.dll!TpythonInterpreter::dispatchMayaCommand

CommandEngine.dll!TpythonInterpreter::genericDispatchFunction

python27.dll!PyCFunction_Call

python27.dll!PyObject_Call

python27.dll!PyEval_CallObjectWithKeywords

python27.dll!PyAST_FromNode

python27.dll!PyCFunction_Call

python27.dll!PyEval_GetFuncDesc

python27.dll!PyEval_EvalFrameEx

python27.dll!PyEval_EvalCodeEx

python27.dll!PyFunction_SetClosure

python27.dll!PyObject_Call

python27.dll!PyEval_GetFuncDesc

Fatal Error. Attempting to save in C:/Users/ead1/AppData/Local/Temp/ead1.20200108.0959.ma

// Maya exited with status -1073741819

C:\Users\ead1>

Stephen.Blair · ‎01-08-2020

Hi

Can you post the log header, where it shows what GPUs you have, and how much memory?

Does this happen with any scene?

If you ignore textures (Render Settings > Diagnostics) do you still get "Insufficient device memory" ?

Is there a lot of geometry in the scene?

// Stephen Blair
// Arnold Renderer Support

jean_leoni · ‎01-09-2020

Hello, Stephen.

I ignored the textures in the diagnostics menu and after that the scene did not crashed again, but the render is stucked in 0%.

here is the log head:

00:00:00 3099MB | log started Wed Jan 8 12:01:31 2020

00:00:00 3099MB | Arnold 5.4.0.2 [30d7daaa] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/10/09 17:19:01

00:00:00 3099MB | host application: MtoA 3.3.0.2 f5753062 (master) Maya 2019

00:00:00 3099MB | running on MNEAD_001, pid=11948

00:00:00 3099MB | 1 x Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (4 cores, 8 logical) with 15310MB

00:00:00 3099MB | NVIDIA driver version 441.28 (Optix 60600)

00:00:00 3099MB | GPU 0: GeForce GTX 1060 3GB @ 1771MHz (compute 6.1) with 3072MB (783MB available) (NVLink:0)

00:00:00 3099MB | Windows 8 Professional (version 6.2, build 9200)

00:00:00 3099MB | soft limit for open files raised from 2048 to 2048

yes, this is happening with any scene in this project, i can use just cpu render.

and yes, theres a lot of geometry in this scene.

Thank you and advance!

thiago.ize · ‎01-10-2020

I suspect that you don't have enough space on your GPU. It's a 3GB card and apparently only 783MB is available (Maya and other apps are probably using most of it), which is likely not enough for a scene with lots of geometry and possibly not even for a trivially simple scene since we need quite a bit of memory to do all our calculations (on my P6000 card, over a GB is needed just to store all the computations done by the many threads of this card).

Also, it's possible that once you get an out of memory error, arnold/cuda gets stuck in an invalid state and so you might need to restart Maya or else Arnold will stay in that invalid state and not be able to properly render.

My suggestion is to reboot Maya and try to render a simple scene (maybe just a quad). If that works, try increasingly more complex scenes.

artoboi · ‎05-23-2020

I have a similar trouble with gtx 1060 and using the arnold render preview in maya 2019. The scene get freeze, white o distort like minecraft style. I dont know what to do. I reinstall maya an drivers for nvidia bur nothing happened.

Arnold GPU crahes when rendering

Arnold GPU crahes when rendering

Forums Links

Post to forums