cmake: fix fmt linking when found

This is a new attempt at #4206 that shouldn't break windows builds.

If someone else could test on windows, it would be much appreciated.
Previously, the build bot passed but the actual builds failed.