kernel/physical_core: Remove unused kernel reference member variable

This isn't used within the class, so it can be removed to simplify the
overall interface.

While we're in the same area, we can simplify a unique_ptr reset() call.