Hello,
I'm working for a VFX company and we are facing a problem with the Houdini Arnold ROP.
Basically, we render Arnold frames on our render farm, so HtoA is writing temporary ass files in /tmp/houdini_temp/htoa_{$USER} .
From what I've seen, the ASS files are automatically cleaned up after every render which is very useful to maintain a clean render farm.
But recently we submitted jobs that are exporting big ass files ~100Gb, those files are filling up all the Hard drive of the render node until Houdini crash (out of disk space), and the partially generated ass files are left in /tmp/houdini_temp and the render node become unusable.
The logline emitted is (I replaced the username):
[2018-07-27 08:00:16] 00:32:53 79383MB ERROR | [ass] Problem writing to ass-file: '/tmp/houdini_temp/htoa_<USERNAME>/6007_8b1ff627-3f5d-4aa1-8ec6-29e71959e04d.ass'
So I'm wondering if this is a bug for this particular case?
Is there a way to force Arnold or the Houdini's Arnold ROP to make the cleanup even on a crash for insufficient disk space?
Thanks,
Solved! Go to Solution.
Solved by frederic.servant. Go to Solution.
HtoA deletes the old ass files before exporting a new ass file. You can see the code in scripts\python\htoa\session.py
Assuming the old ass files are in the same temporary folder, they should be removed when we call os.remove
Can you run strace and see what's happening? Does HtoA find those ass files? Is there some sort of access/permissions error?
Temporary .ass file older that 5 minutes will get cleaned up automatically with the next run. You can customize this timeout with the HTOA_TMP_TIMEOUT environment variable.
Note that you can also customize the default location of the temporary .ass files with the HTOA_TMP environment variable if you want to switch to a location that has more space available.
See also https://support.solidangle.com/display/A5AFHUG/Environment+Variables
One last solution is to do your render in 2 steps: export .ass file from the ROP, then render it with kick on the farm. Some render managers support this workflow natively.
Hello,
Thanks a lot for your replies!
So I think I'm gonna try to set the environment variable to a network mount with more space. That might do the trick!
That said, just to be sure I understood clearly, the cleanup is not happening at the end of the render, but at the beginning of the next run? So if a new Arnold job start on the machine and a previous job left some temp files in the /tmp folder, they will be cleared only when the new one starts ?