[libc++] Revert the std::to_address change to avoid relying on element_type.

This reverts commit da456167, which broke the Clang build. I'm able to
reproduce it but I want to give myself a bit more time to investigate.

Differential Revision: https://reviews.llvm.org/D101638