Community
Arnold GPU Forum
General discussions about GPU rendering with Arnold.
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Arnold GPU for Maya - Error

11 REPLIES 11
SOLVED
Reply
Message 1 of 12
gw950619
3353 Views, 11 Replies

Arnold GPU for Maya - Error

Hello guys

I bought two used 2080ti for rendering and Nvlink.

so I did some tests on the company's project. the asset has 14x albedo + 14x specular + 14x roughness + 14x displacement, all 4k UDIM. and it has 60k+ poly faces, subdivision set up to 4.

and the end is not what I hoped for, it's so sad, the red line made me to doubt myself whether this investment is correct? should I sell the cards CPU and MB?


anyway, I record a video from beginning ( I forgot to set log files, when I record the video, so I did again after I upload the video)

https://www.youtube.com/watch?v=uDa6Wv0kOSg

2x 2080Ti FE + Maya 2020.1 + Arnold Core 6.0.2.1 1.


Error - 1

// Error: [gpu] an error happened during GPU update : Insufficient device memory //  


Error - 2

// Error: [gpu] an error happened during rendering : Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Assertion failed: "!targets.empty() : Buffer abandoned but no other devices are allowed to hold a copy", file: <internal>, line: 676) //

arnold.1 - Error.zip

Tags (1)
Labels (1)
  • gpu
11 REPLIES 11
Message 2 of 12
James_Grimes
in reply to: gw950619

Hi G W,

I am in the same boat as you. I have 4 2080 Supers that I bought for use with Arnold GPU. I did some tests with all 4 GPUs and with 2 GPUs only with NVLink. I too ran into the Insufficient device memory error. My test consisted of twelve trees at a total of about 3 million polys. Arnold GPU choked, even with NVLink. The 2080 Super GPUs have 8GB of VRAM. I understood that Arnold GPU pooled the memory with NVLink. I guess that isn't true?

I did the same test with Redshift. Redshift handled it with no problem. I then increased the number of trees up to 48 for a total of approx. 12 million polys. Redshift still had no problem.

I really want to use Arnold GPU. But it isn't clear that the dev team is close to solving this problem.

Message 3 of 12
thiago.ize
in reply to: gw950619

It looks like you did 5 subdivision iterations? Does it run out of memory if you do fewer subdivisions? From your video, I see that the CPU memory goes from 8.8GB before the render starts, to a max of 18GB before the error occurs (So Arnold used about 10GB of CPU memory). My guess is that the bulk of this memory could be Arnold doing the subdivision on the CPU and then when it tries to copy it to GPU it finds that the GPU won't have space for all the data and so reports this error. As for how much memory the GPU has, it has close to 9GB free at the start of the render, so if CPU arnold has 10GB of geo, then it wouldn't surprise me if GPU arnold will run out. Keep in mind that nvlink does not currently double your gpu memory capacity for geometry.

Also, no need to pre-populate the cache before rendering (I actually never use it).

What would be helpful is to try the same render on the CPU, with at least info verbosity, and then look at the stats printed out at the end of the log file. How much memory was used by the CPU? If it's just a GB or so, then yeah, something is wrong. If the CPU used about 5 or more GB of RAM, then the GPU probably would have failed since at the moment GPU doesn't store data as efficiently as CPU.

Message 4 of 12
thiago.ize
in reply to: James_Grimes

Like I said for G W, try this test on the CPU and see how much CPU memory is used. That might help to pinpoint where the memory is going. If you are using texturing, keep in mind that at the moment GPU arnold uses significantly more texture memory than CPU arnold (we expect that that to be fixed soonish).

Message 5 of 12

Arnold GPU does pool the memory with NVlink. Its worth noting though that you can only pool the memory between the cards that are NVlinked together. So say we have you case with 4 2080 supers broken up into two pairs connected with NVlink. Then in total the maximum memory you have available in theory is 16GB, but in practice it's lower because not everything can be shared.

Message 6 of 12
gw950619
in reply to: gw950619

Message 7 of 12
gw950619
in reply to: thiago.ize

in the first video, I tested twice. the first one I set subdivision iterations directly to 5, I got Error - 1. Then I restarted the software and tested the subdivision iterations from 2 to 4, as you can see, in subdivision 3, Arnold used 4.8G GPU memory, so when I change to 4, I got another Error.

I know GeForce NVLink doesn't double GPU memory, but I remember in some specific software, it does.

In my test, it just hit the wall over 10GB, so 11gb+11gb=11gb in Arnold?

Message 8 of 12
thiago.ize
in reply to: gw950619

Thanks for confirming you were running out of memory. Each time you subdivide, you get 4 times as much geometry, so if 3 iterations was using 4.8GB of GPU memory, then 4 iterations would use up to 19GB of memory, which is more than each 11GB card can handle. nvlink does allow the sharing of some memory, but for now the rtx hardware acceleration requires that this data be replicated on each GPU. The non-rtx accelerated geometry, such as curves and volumes, should be shared across gpus, so there you will see a benefit from nvlink. Likewise, textures will hopefully soon also be shared.

Message 9 of 12
gw950619
in reply to: thiago.ize

100% understand that. Thank you for your patience.

now I know It's not a good time to invest GPU rendering at present unless you have 48GB of GPU memory in one card.

I decide to sell my GPUs, CPU, and Motherboard. Then I'm gonna buy a 3970x, and switch back to CPU rendering.


Message 10 of 12
thiago.ize
in reply to: gw950619

Yeah, if you really do need hundreds of millions of triangles and lots of textures, then 8GB likely won't work. We do plan on reducing the amount of memory that triangles occupy on the GPU so it matches better the CPU, but even if we get a 2x reduction, that won't prevent you from doubling the number of triangles and being back to running out of memory.

If you add a third card for general purpose usage and then only use your dual 2080s for Arnold, that will give you an extra 3GB/card which might make the difference here. Try rendering on 1 GPU (the unused one) to see if it helps.

Message 11 of 12
thiago.ize
in reply to: gw950619

Followup: Right now it looks like Arnold might have a bug that is causing us to report an out of memory error without factoring in the sharing done by nvlink. What's more, while some of the data can't be shared across nvlink, other parts can be, so when we fix this I suspect there's a chance your test might just pass (it'd still be close).

Message 12 of 12
gw950619
in reply to: thiago.ize

Yes, 2080Ti is not enough even 3080Ti, so I already sold my CPU MB and GPU, 3970X is on.

Can't find what you're looking for? Ask the community or share your knowledge.

Post to forums  

Autodesk Design & Make Report