Thanks JJMack for the answer! I was hoping there was a less messy way to deal with it.
What I'm doing currently is: my workflow involves trimming the layers to their contents. I take the bounds before and after trimming and calculate the correct bounds.