[Attributor] Use known alignment as lower bound to avoid work

If we know already more than available from a use, we don't need to
invest time on it.