buffer_cache: Delay buffer destructions

Delay buffer destruction some extra frames to avoid destroying buffers
that are still being used from older frames. This happens on Nvidia's
driver with mailbox.