[AttrBuilder] Make handling of int attribtues more generifc (NFC)

This is basically the same change as 42cc7f3c
but for integer attributes. Rather than treating each attribute
individually, handle them all the same way. The only thing that
needs to be done per attribute is specify how get/add convert
from/to the raw representation.