hid/controller: Remove unused header inclusions

swap.h only needs to be present in the header for the type aliases and
definitions, it's not actually needed in the cpp files though. input.h
is just unused entirely in xpad.h