C0000005 crashes in HtoA while generating ass files

[koaleksa]

Hi,

We are randomly getting C0000005 crashes when generating ass files for hair curves (hairgen) over the farm.

Hardware varies, but in general, all machines have 64gb ram. GPUs and CPUs vary.

OS: Windows 8/10

Farm Manager: Deadline

Houdini: 17.5.173

HtoA: 4.0.1

Crash log (relevant part):

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | log started Sun May 19 23:24:19 2019

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | Arnold 5.3.0.1 [567ba37e] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/03/26 14:03:19

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | running on [redacted]

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | 1 x Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (6 cores, 12 logical) with 65339MB

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | NVIDIA driver version 417.71

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | Windows 8 Professional (version 6.2, build 9200)

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | soft limit for open files raised from 2048 to 2048

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB |

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | loading plugins from [redacted] ...

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | loading plugins from [redacted]...

2019-05-19 23:24:19: 0: STDOUT: 00:00:00 1159MB | loaded 6 plugins from 4 lib(s) in 0:00.03

2019-05-20 00:24:08: 0: STDOUT: 00:59:49 40256MB ERROR | signal caught: error C0000005 -- access violation

2019-05-20 00:24:08: 0: STDOUT: ****

2019-05-20 00:24:08: 0: STDOUT: * Arnold 5.3.0.1 [567ba37e] windows icc-17.0.2 oiio-2.1.0 osl-1.11.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.4.2 optix-6.0.0 2019/03/26 14:03:19

2019-05-20 00:24:08: 0: STDOUT: * CRASHED in tbbmalloc (0x000000002231acab) at 00:00:03, pixel (0, 0)

2019-05-20 00:24:08: 0: STDOUT: * signal caught: error C0000005 -- access violation

2019-05-20 00:24:08: 0: STDOUT: *

2019-05-20 00:24:08: 0: STDOUT: * backtrace:

2019-05-20 00:24:08: 0: STDOUT: * 0 0x00007ff8a862b76e [ai ]

2019-05-20 00:24:08: 0: STDOUT: * 1 0x00007ff8a862aa0f [ai ]

2019-05-20 00:24:08: 0: STDOUT: * 2 0x00007ff90c425fa7 [KERNELBASE] UnhandledExceptionFilter

2019-05-20 00:24:08: 0: STDOUT: * 3 0x00007ff90ff1ed1b [ntdll ] memset

2019-05-20 00:24:08: 0: STDOUT: * 4 0x00007ff90ff06bd6 [ntdll ] _C_specific_handler

2019-05-20 00:24:08: 0: STDOUT: * 5 0x00007ff90ff1ab9d [ntdll ] _chkstk

2019-05-20 00:24:08: 0: STDOUT: * 6 0x00007ff90feb9913 [ntdll ] RtlImageNtHeaderEx

2019-05-20 00:24:08: 0: STDOUT: * 7 0x00007ff90ff19cba [ntdll ] KiUserExceptionDispatcher

2019-05-20 00:24:08: 0: STDOUT: >> 8 0x000000002231acab [tbbmalloc ]

2019-05-20 00:24:08: 0: STDOUT: * 9 0x000000002231c993 [tbbmalloc ]

2019-05-20 00:24:08: 0: STDOUT: * 10 0x000000002231d3fb [tbbmalloc ] scalable_malloc

2019-05-20 00:24:08: 0: STDOUT: * 11 0x0000000017950779 [libSOP ] SOP_BoneDeformParms::getBoneTransformPath

2019-05-20 00:24:08: 0: STDOUT: * 12 0x0000000017afdf3a [libSOP ] SOP_HairClumpParms::operator!=

2019-05-20 00:24:08: 0: STDOUT: * 13 0x0000000017b06732 [libSOP ] SOP_HairClumpParms::copyFrom

2019-05-20 00:24:08: 0: STDOUT: * 14 0x00000000224ef236 [tbb ] tbb::interface7::internal::task_arena_base::internal_max_concurrency

2019-05-20 00:24:08: 0: STDOUT: * 15 0x00000000224e90c0 [tbb ] tbb::task_scheduler_init::default_num_threads

2019-05-20 00:24:08: 0: STDOUT: * 16 0x00000000224e8656 [tbb ] tbb::task_scheduler_init::default_num_threads

2019-05-20 00:24:08: 0: STDOUT: * 17 0x00000000224e393c [tbb ] tbb::internal::thread_sleep_v3

2019-05-20 00:24:08: 0: STDOUT: * 18 0x00000000224e38bb [tbb ] tbb::internal::thread_sleep_v3

2019-05-20 00:24:08: 0: STDOUT: * 19 0x00007ff90d21cab0 [ucrtbase ] o__realloc_base

2019-05-20 00:24:08: 0: STDOUT: * 20 0x00007ff90fdc8364 [KERNEL32 ] BaseThreadInitThunk

2019-05-20 00:24:08: 0: STDOUT: * 21 0x00007ff90fed70d1 [ntdll ] RtlUserThreadStart

2019-05-20 00:24:08: 0: STDOUT: *

2019-05-20 00:24:08: 0: STDOUT: * loaded modules:

2019-05-20 00:24:08: 0: STDOUT: * 0x00007ff8a8310000 ai

2019-05-20 00:24:08: 0: STDOUT: * 0x00007ff90c3d0000 KERNELBASE

2019-05-20 00:24:08: 0: STDOUT: * 0x00007ff90fe70000 ntdll

2019-05-20 00:24:08: 0: STDOUT: * 0x0000000022310000 tbbmalloc

2019-05-20 00:24:08: 0: STDOUT: * 0x00000000125a0000 libSOP

2019-05-20 00:24:08: 0: STDOUT: * 0x00000000224d0000 tbb

2019-05-20 00:24:08: 0: STDOUT: * 0x00007ff90d200000 ucrtbase

2019-05-20 00:24:08: 0: STDOUT: * 0x00007ff90fdc0000 KERNEL32

2019-05-20 00:24:08: 0: STDOUT: ****

I cant figure out any pattern - the same machine sometimes manages to generate ass file for a given frame, and sometimes it crashes.

Any suggestions?