RTX 3090 Out Of Memory

Question

question

Lennart Oberscheidt asked · Dec 01, 2020 at 05:37 PM

RTX 3090 Out Of Memory

Hey, unfortunately I don't have much time right now to give all the needed detail information (Arnold log etc) but I wanted to make you aware of an issue (maybe that's already a known problem) I have noticed while recording longer video tutorials (due to covid) for a course I am giving. I've recorded a short video. Arnold 4.0.4.1.

https://www.youtube.com/watch?v=j1q5Msd9qRw

In short: A (not too complicated) asset renders fine on RTX 2080 and 1080 machines (8GB), and the IPR can always be closed an restarted without problems (part 1 of the video labeled computer 1). This is true for 4 machines with either RTX 2080 or GTX 1080).

Now we do have two new machines with RTX 3090s, which I was using for the tutorial (it's a TeamViewer connection for all machines, so the recording is not interfering with anything here). Once you close the IPR and then reopen it again, arnold will exit with an Out of Device Memory (the asset clearly fits into the 24 GB if it renders fine on 8GB cards ;)) That's the computer 2 part. This was actually tested on two machines, same behavior. After that, Maya will hard crash if you open the IPR again, which, as you know ;) it always does once Arnold had a GPU error and you try that.

(As a note: The thin filme warning comes from a thin film node that is used in the CPU setup of that file. The node is not actually evaluated as it's connected to an AI switch which currently just reads a noise as a GPU alternative. The texture warning just comes from the fact that UDIM tile 1005 doesn't exist, although I don't know why it has to warn about that. 1001-1004 exist. I expect there to be not relation to the error on the 3090 but still thought I'd mention it.)

I'm sorry I can't post a log right now because I need to finish a few things here. You will most likely ask if we would update to 4.1.0 (reverted back to 4.0.4.1 because of another issue), which I'd gladly test out if I could! Unfortunately, I need those machines for rendering over night, and I can't update them right now, because I'd have to roll back to 4.0.4.1 since that's what all other machines currently use, and you can't have 4.1.0 and 4.0.4.1 render the same asset at the same time from the same Maya project on a network drive, as the older arnold versions will randomly loose their textures on some frames when doing so (I assume that is because of the changes to the .tx files / texture memory in 4.1.0) . Not an issue – just noticed it and that's basically the reason I unfortunately can't run some test you might ask me to.

I still hope that is enough information for now to investigate the issue, if its not an already known problem in either 4.0.4.1 or already fixed in newer versions.

Thanks and cheers,

Lennart

gpu

10 |600 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Answer 1 · 2020-12-01T19:23:04Z

Lennart Oberscheidt answered · Dec 01, 2020 at 07:23 PM

Thank you Thiago. Since i had a few mins (and kinda wanted to use the faster 3090 to finally get that shading overview for my students done without constant random crashes, since well… there's a pretty big difference in "snappyness" between the 3090 and the 2080 ;)) i did both of these and now have to revert back on the machine. The IPR can be restarted "more often" now, which does unfortunately not completely solve the problem. While it may restart 4-5 times, i do get random "Optix stack overflow encountered" errors. After that, it may, or may not, work again. Again, the whole thing runs without any kind of glitch basically for hours on the older cards. Once I have the time, i will get some logs for you.

I'd have a quick random different question though: Isolate select. They behave differently on the GPU and CPU when you isolate shading nodes. While the CPU of course basically just displays the shading nodes output that you selected, and everything else turns black (which is practical) the GPU will isolate the thing correctly, but still display the rest of the scene and render it. (It doesn't when you select objects instead of shading nodes). Since i assume the way the CPU does it is the intended one – is that just a GPU limitation? Or maybe even a design decision (since it could be argued the GPU way might be desirable, although I think isolating the way the CPU does it is way more useful). Any plans to change that to CPU behaviour in the near future? I just ask because while recording the tutorial and isolating things, i reverted back to the CPU and 1-2 samples, because having black around everything that you have just isolated makes things just a lot clearer ;).

Thanks again

Lennart

1

10 |600 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Thiago Ize · Dec 05, 2020 at 07:40 PM

Could you post the isolate select isssue as a new question so it can get the proper attention it deserves? It also helps other users who might be searching for the similar issue.

As for the stack overflow issue, that is something we are actively working on. We've reproduced some of these issues and we suspect we're close to having a fix. Hopefully that will fix things for you once/if it comes out, but as usual, no promises.

0 ·

Answer 2 · 2020-12-01T18:56:46Z

Thiago Ize answered · Dec 01, 2020 at 06:56 PM

When it comes to GPU issues, a good first step is upgrading your driver. You mention using 456.38. I see there's a 457.30 you can use. And after that, yeah, try upgrading arnold :)

10 |600 characters needed characters left characters exceeded

Attachments: Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

question