[Attributes] Avoid duplicate hasAttribute() query (NFC)

removeAttribute() already performs a hasAttribute() check, so no
need to also do it in the caller. Instead check whether the
attribute set was changed.

This makes the implementations in line with removeAttributesAtIndex().