Clean up conditional, don't set load binaries twice

Follow Alex Langford's feedback to my patch from
https://reviews.llvm.org/D145547 , and fix a
side issue I noticed while testing this, where
binaries loaded via LC_NOTE metadata were loaded
in the Target twice unnecessarily.