applets/controller: Introduce additional checks for mode and caller

Some games like Cave Story+ set invalid values in the ControllerPrivateArg's mode and caller fields.
Use other fields to determine the appropriate mode and caller should either or both fields be invalid.