host_memory: Fix fastmem crashes in debug builds

It is possible for virtual_offset to not be 0 when the iterator is at the beginning, and thus, std::prev(it) may be evaluated, leading to a crash in debug mode.

Co-Authored-By: Fernando S. <1731197+FernandoS27@users.noreply.github.com>