CMakeLists: Move compilation flags into the src directory

We generally shouldn't be hijacking CMAKE_CXX_FLAGS, etc as a means to
append flags to the targets, since this adds the compilation flags to
everything, including our externals, which can result in weird issues
and makes the build hierarchy fragile.

Instead, we want to just apply these compilation flags to our targets,
and let those managing external libraries to properly specify their
compilation flags.

This also results in us not getting as many warnings, as we don't raise
the warning level on every external target.