Revert "Support polynomial attributes with floating point coefficients (#91137)" (#92001)

This reverts commit 91a14dbf.

Not sure how to fix the build error this introduced, so reverting until
I can figure it out

https://lab.llvm.org/buildbot/#/builders/264/builds/10468



Co-authored-by: Jeremy Kun <j2kun@users.noreply.github.com>