question

G W avatar image
G W asked ·

Arnold GPU for Maya - Error

Hello guys

I bought two used 2080ti for rendering and Nvlink.

so I did some tests on the company's project. the asset has 14x albedo + 14x specular + 14x roughness + 14x displacement, all 4k UDIM. and it has 60k+ poly faces, subdivision set up to 4.

and the end is not what I hoped for, it's so sad, the red line made me to doubt myself whether this investment is correct? should I sell the cards CPU and MB?


anyway, I record a video from beginning ( I forgot to set log files, when I record the video, so I did again after I upload the video)

https://www.youtube.com/watch?v=uDa6Wv0kOSg

2x 2080Ti FE + Maya 2020.1 + Arnold Core 6.0.2.1 1.


Error - 1

// Error: [gpu] an error happened during GPU update : Insufficient device memory //  


Error - 2

// Error: [gpu] an error happened during rendering : Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Assertion failed: "!targets.empty() : Buffer abandoned but no other devices are allowed to hold a copy", file: <internal>, line: 676) //

arnold.1 - Error.zip

gpu
arnold1-error.zip (4.6 KiB)
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Thiago Ize avatar image
Thiago Ize answered ·

It looks like you did 5 subdivision iterations? Does it run out of memory if you do fewer subdivisions? From your video, I see that the CPU memory goes from 8.8GB before the render starts, to a max of 18GB before the error occurs (So Arnold used about 10GB of CPU memory). My guess is that the bulk of this memory could be Arnold doing the subdivision on the CPU and then when it tries to copy it to GPU it finds that the GPU won't have space for all the data and so reports this error. As for how much memory the GPU has, it has close to 9GB free at the start of the render, so if CPU arnold has 10GB of geo, then it wouldn't surprise me if GPU arnold will run out. Keep in mind that nvlink does not currently double your gpu memory capacity for geometry.

Also, no need to pre-populate the cache before rendering (I actually never use it).

What would be helpful is to try the same render on the CPU, with at least info verbosity, and then look at the stats printed out at the end of the log file. How much memory was used by the CPU? If it's just a GB or so, then yeah, something is wrong. If the CPU used about 5 or more GB of RAM, then the GPU probably would have failed since at the moment GPU doesn't store data as efficiently as CPU.

5 comments Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

in the first video, I tested twice. the first one I set subdivision iterations directly to 5, I got Error - 1. Then I restarted the software and tested the subdivision iterations from 2 to 4, as you can see, in subdivision 3, Arnold used 4.8G GPU memory, so when I change to 4, I got another Error.

I know GeForce NVLink doesn't double GPU memory, but I remember in some specific software, it does.

In my test, it just hit the wall over 10GB, so 11gb+11gb=11gb in Arnold?

0 Likes 0 · ·

Thanks for confirming you were running out of memory. Each time you subdivide, you get 4 times as much geometry, so if 3 iterations was using 4.8GB of GPU memory, then 4 iterations would use up to 19GB of memory, which is more than each 11GB card can handle. nvlink does allow the sharing of some memory, but for now the rtx hardware acceleration requires that this data be replicated on each GPU. The non-rtx accelerated geometry, such as curves and volumes, should be shared across gpus, so there you will see a benefit from nvlink. Likewise, textures will hopefully soon also be shared.

1 Like 1 · ·

100% understand that. Thank you for your patience.

now I know It's not a good time to invest GPU rendering at present unless you have 48GB of GPU memory in one card.

I decide to sell my GPUs, CPU, and Motherboard. Then I'm gonna buy a 3970x, and switch back to CPU rendering.


0 Likes 0 · ·
Show more comments
G W avatar image
G W answered ·

projects.zip (120.2 KiB)
Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Thiago Ize avatar image
Thiago Ize answered ·

Followup: Right now it looks like Arnold might have a bug that is causing us to report an out of memory error without factoring in the sharing done by nvlink. What's more, while some of the data can't be shared across nvlink, other parts can be, so when we fix this I suspect there's a chance your test might just pass (it'd still be close).

Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

James Grimes avatar image
James Grimes answered ·

Hi G W,

I am in the same boat as you. I have 4 2080 Supers that I bought for use with Arnold GPU. I did some tests with all 4 GPUs and with 2 GPUs only with NVLink. I too ran into the Insufficient device memory error. My test consisted of twelve trees at a total of about 3 million polys. Arnold GPU choked, even with NVLink. The 2080 Super GPUs have 8GB of VRAM. I understood that Arnold GPU pooled the memory with NVLink. I guess that isn't true?

I did the same test with Redshift. Redshift handled it with no problem. I then increased the number of trees up to 48 for a total of approx. 12 million polys. Redshift still had no problem.

I really want to use Arnold GPU. But it isn't clear that the dev team is close to solving this problem.

2 comments Share
10 |600 characters needed characters left characters exceeded

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Like I said for G W, try this test on the CPU and see how much CPU memory is used. That might help to pinpoint where the memory is going. If you are using texturing, keep in mind that at the moment GPU arnold uses significantly more texture memory than CPU arnold (we expect that that to be fixed soonish).

2 Likes 2 · ·

Arnold GPU does pool the memory with NVlink. Its worth noting though that you can only pool the memory between the cards that are NVlinked together. So say we have you case with 4 2080 supers broken up into two pairs connected with NVlink. Then in total the maximum memory you have available in theory is 16GB, but in practice it's lower because not everything can be shared.

1 Like 1 · ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 5 attachments (including images) can be used with a maximum of 2.0 MiB each and 9.8 MiB total.

Welcome to the Arnold Answers community.

This is the place for Arnold renderer users everywhere to ask and answer rendering questions, and share knowledge about using Arnold, Arnold plugins, workflows and developing tools with Arnold.

If you are a new user to Arnold Answers, please first check out our FAQ and User Guide for more information.

When posting questions, please be sure to select the appropriate Space for your Arnold plugin and include the plugin version you are using.

Please include images, scene and log files whenever possible as this helps the community answer your questions.

Instructions for generating full verbosity log files are available for MtoA, MaxtoA, C4DtoA, HtoA, KtoA, and Kick.

If you are looking for Arnold Documentation and Support please visit the Arnold Support site.

To try Arnold please visit the Arnold Trial page.

Bottom No panel present for this section.