Revert "[InferAttributes] Materialize all infered attributes for declaration"

Breaks check-clang, see comments on D100400

Also revert follow-up "[NFC] Move a recently added utility into a location to enable reuse"

This reverts commit 3ce61fb6.
This reverts commit 61a85da8.