Community
Arnold for Houdini Forum
Rendering with Arnold in Houdini and Solaris using the HtoA plug-in.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

[Resolved] Houdini ass files cleanup on render failure

3 REPLIES 3
SOLVED
Reply
Message 1 of 4
amaury_matu
1588 Views, 3 Replies

[Resolved] Houdini ass files cleanup on render failure

Hello,

I'm working for a VFX company and we are facing a problem with the Houdini Arnold ROP.

Basically, we render Arnold frames on our render farm, so HtoA is writing temporary ass files in /tmp/houdini_temp/htoa_{$USER} .
From what I've seen, the ASS files are automatically cleaned up after every render which is very useful to maintain a clean render farm.

But recently we submitted jobs that are exporting big ass files ~100Gb, those files are filling up all the Hard drive of the render node until Houdini crash (out of disk space), and the partially generated ass files are left in /tmp/houdini_temp and the render node become unusable.

The logline emitted is (I replaced the username):

[2018-07-27 08:00:16] 00:32:53 79383MB ERROR   |  [ass] Problem writing to ass-file: '/tmp/houdini_temp/htoa_<USERNAME>/6007_8b1ff627-3f5d-4aa1-8ec6-29e71959e04d.ass'

So I'm wondering if this is a bug for this particular case?

Is there a way to force Arnold or the Houdini's Arnold ROP to make the cleanup even on a crash for insufficient disk space?

Thanks,

Tags (3)
Labels (3)
3 REPLIES 3
Message 2 of 4
Stephen.Blair
in reply to: amaury_matu

HtoA deletes the old ass files before exporting a new ass file. You can see the code in scripts\python\htoa\session.py

Assuming the old ass files are in the same temporary folder, they should be removed when we call os.remove

Can you run strace and see what's happening? Does HtoA find those ass files? Is there some sort of access/permissions error?



// Stephen Blair
// Arnold Renderer Support
Message 3 of 4

Temporary .ass file older that 5 minutes will get cleaned up automatically with the next run. You can customize this timeout with the HTOA_TMP_TIMEOUT environment variable.

Note that you can also customize the default location of the temporary .ass files with the HTOA_TMP environment variable if you want to switch to a location that has more space available.

See also https://support.solidangle.com/display/A5AFHUG/Environment+Variables

One last solution is to do your render in 2 steps: export .ass file from the ROP, then render it with kick on the farm. Some render managers support this workflow natively.

Message 4 of 4
amaury_matu
in reply to: amaury_matu

Hello,

Thanks a lot for your replies!

So I think I'm gonna try to set the environment variable to a network mount with more space. That might do the trick!

That said, just to be sure I understood clearly, the cleanup is not happening at the end of the render, but at the beginning of the next run? So if a new Arnold job start on the machine and a previous job left some temp files in the /tmp folder, they will be cleared only when the new one starts ?

Can't find what you're looking for? Ask the community or share your knowledge.

Post to forums  

Autodesk Design & Make Report