Fix tsan problem where the per-thread shared_ptr() can be locked right before...

Fix tsan problem where the per-thread shared_ptr() can be locked right before the cache is destroyed causing a race where it tries to remove an entry from a destroyed cache.

This is a rollforward with fixes of https://reviews.llvm.org/rGbcc10817d5569172ee065015747e226280e9b698 (originally https://reviews.llvm.org/D142394). The original patch exposed an asan problem on aarch64, which is fixed by simply calling the context destructors properly.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D143294