Require existing typeinfo for direct conversions

This avoid a hashmap lookup since the pointer to the list of
direct converters is now cached in the typeinfo.