Fix tsan problem where the per-thread shared_ptr() can be locked right before...

Fix tsan problem where the per-thread shared_ptr() can be locked right before the cache is destroyed causing a race where it tries to remove an entry from a destroyed cache.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D142394