service/vi: Unstub IApplicationDisplayService' SetLayerScalingMode()

This appears to only check if the scaling mode can actually be
handled, rather than actually setting the scaling mode for the layer.

This implements the same error handling performed on the passed in
values.