video_core/memory_manager: Default the destructor within the cpp file

Makes the class less surprising when it comes to forward declaring the
type, and also prevents inlining the destruction code of the class,
given it contains non-trivial types.