Hey,
This is less of a question and maybe more feedback on the shadowmatte shader. Here are the issues I currently see with the shader. Later I'll try to outline how we are overcoming these issues.
Here are our workarounds:
1. Combine all shadow catching geometry and disable shelf shadows. This works but has the disadvantage of destroying your object pivots, and its makes it hard to update.
2. You can either cut the geometry to the camera frustum, or pipe the shadowmatte output into a standard shader, and use the opacity to cut to the camera frustum.
3. You can use various rayswitching techniques to make the background black to the camera.
Although all of these issues have workarounds, I would like to see the shadowmatte improved to make plate integration easier.
I'm really interested in how other people work with the shadowmatte shader as well.
2. Does Offscreen Color help with this?
3. Background doesn't help this?
Offscreen color can help in certain cases if you setup a texture to mirror/wrap.
I would argue that its better to "see" through to the scene, so the lights in the scene can contribute to the image.
Here is an example of what the difference is when you "cut" the shadowmatte geometry to the camera frustum. This is done by piping the output of the shadowmatte to the emission of a standard surface and using opacity to "cut" the geometry.
Default shadowmatte
Shadowmatte through standard surface
Apart from the reflections, notice the lack of light contribution on screen-left of the white sphere. This is hard to troubleshoot unless you know what is going on offscreen.
When an object gets closer to the edge of the camera frustum, this darkening becomes more apparent.
Not entirely sure what workflow you are suggesting.
If you are referring to switching the shadowmatte "Background" mode to "background_color" and turning this black, then this is issue:
With background
Black background
Difference
Notice the indirect lighting that is missing on the white sphere and the reflection on the chrome sphere.
As I've said you can work around this with rayswitching, but thought it might better to have it built into the shadowmatte shader.