Refactor ScanDirectoryTreeAndCallback to separate errors and retvals

ScanDirectoryTreeAndCallback, before this change, coupled error/return
codes and actual return values (number of entries found). This caused
confusion and difficulty interpreting the precise way the function
worked.

Supersedes, and closes #1255.