question

amaury matu avatar image
amaury matu asked ·

[Resolved] Houdini ass files cleanup on render failure

Hello,

I'm working for a VFX company and we are facing a problem with the Houdini Arnold ROP.

Basically, we render Arnold frames on our render farm, so HtoA is writing temporary ass files in /tmp/houdini_temp/htoa_{$USER} .
From what I've seen, the ASS files are automatically cleaned up after every render which is very useful to maintain a clean render farm.

But recently we submitted jobs that are exporting big ass files ~100Gb, those files are filling up all the Hard drive of the render node until Houdini crash (out of disk space), and the partially generated ass files are left in /tmp/houdini_temp and the render node become unusable.

The logline emitted is (I replaced the username):

[2018-07-27 08:00:16] 00:32:53 79383MB ERROR   |  [ass] Problem writing to ass-file: '/tmp/houdini_temp/htoa_<USERNAME>/6007_8b1ff627-3f5d-4aa1-8ec6-29e71959e04d.ass'

So I'm wondering if this is a bug for this particular case?

Is there a way to force Arnold or the Houdini's Arnold ROP to make the cleanup even on a crash for insufficient disk space?

Thanks,

htoaassrender farm
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Frederic Servant avatar image
Frederic Servant answered ·

Temporary .ass file older that 5 minutes will get cleaned up automatically with the next run. You can customize this timeout with the HTOA_TMP_TIMEOUT environment variable.

Note that you can also customize the default location of the temporary .ass files with the HTOA_TMP environment variable if you want to switch to a location that has more space available.

See also https://support.solidangle.com/display/A5AFHUG/Environment+Variables

One last solution is to do your render in 2 steps: export .ass file from the ROP, then render it with kick on the farm. Some render managers support this workflow natively.

Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Stephen Blair avatar image
Stephen Blair answered ·

HtoA deletes the old ass files before exporting a new ass file. You can see the code in scripts\python\htoa\session.py

Assuming the old ass files are in the same temporary folder, they should be removed when we call os.remove

Can you run strace and see what's happening? Does HtoA find those ass files? Is there some sort of access/permissions error?

Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

amaury matu avatar image
amaury matu answered ·

Hello,

Thanks a lot for your replies!

So I think I'm gonna try to set the environment variable to a network mount with more space. That might do the trick!

That said, just to be sure I understood clearly, the cleanup is not happening at the end of the render, but at the beginning of the next run? So if a new Arnold job start on the machine and a previous job left some temp files in the /tmp folder, they will be cleared only when the new one starts ?

Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Welcome to the Arnold Answers community.

This is the place for Arnold renderer users everywhere to ask and answer rendering questions, and share knowledge about using Arnold, Arnold plugins, workflows and developing tools with Arnold.

If you are a new user to Arnold Answers, please first check out our FAQ and User Guide for more information.

When posting questions, please be sure to select the appropriate Space for your Arnold plugin and include the plugin version you are using.

Please include images, scene and log files whenever possible as this helps the community answer your questions.

Instructions for generating full verbosity log files are available for MtoA, MaxtoA, C4DtoA, HtoA, KtoA, and Kick.

If you are looking for Arnold Documentation and Support please visit the Arnold Support site.

To try Arnold please visit the Arnold Trial page.

Bottom No panel present for this section.