This should speed up compile times a bit, as well as enable more liberal
use of forward declarations. (Due to SharedPtr not trying to emit the
destructor anymore.)

This is to support the removal of GetHandle soon