[libclang] Add API to override preamble storage path

TempPCHFile::create() calls llvm::sys::fs::createTemporaryFile() to
create a file named preamble-*.pch in a system temporary directory. This
commit allows overriding the directory where these often many and large
preamble-*.pch files are stored.

The referenced bug report requests the ability to override the temporary
directory path used by libclang. However, overriding the return value of
llvm::sys::path::system_temp_directory() was rejected during code review
as improper and because it would negatively affect multithreading
performance. Finding all places where libclang uses the temporary
directory is very difficult. Therefore this commit is limited to
override libclang's single known use of the temporary directory.

This commit allows to override the preamble storage path only during
CXIndex construction to avoid multithreading issues and ensure that all
preambles are stored in the same directory. For the same multithreading
and consistency reasons, this commit deprecates
clang_CXIndex_setGlobalOptions() and
clang_CXIndex_setInvocationEmissionPathOption() in favor of specifying
these options during CXIndex construction.

Adding a new CXIndex constructor function each time a new initialization
argument is needed leads to either a large number of function parameters
unneeded by most libclang users or to an exponential number of overloads
that support different usage requirements. Therefore this commit
introduces a new extensible struct CXIndexOptions and a general function
clang_createIndexWithOptions().

A libclang user passes a desired preamble storage path to
clang_createIndexWithOptions(), which stores it in
CIndexer::PreambleStoragePath. Whenever
clang_parseTranslationUnit_Impl() is called, it passes
CIndexer::PreambleStoragePath to ASTUnit::LoadFromCommandLine(), which
stores this argument in ASTUnit::PreambleStoragePath. Whenever
ASTUnit::getMainBufferWithPrecompiledPreamble() is called, it passes
ASTUnit::PreambleStoragePath to PrecompiledPreamble::Build().
PrecompiledPreamble::Build() forwards the corresponding StoragePath
argument to TempPCHFile::create(). If StoragePath is not empty,
TempPCHFile::create() stores the preamble-*.pch file in the directory at
the specified path rather than in the system temporary directory.

The analysis below proves that this passing around of the
PreambleStoragePath string is sufficient to guarantee that the libclang
user override is used in TempPCHFile::create(). The analysis ignores API
uses in test code.

TempPCHFile::create() is called only in PrecompiledPreamble::Build().
PrecompiledPreamble::Build() is called only in two places: one in
clangd, which is not used by libclang, and one in
ASTUnit::getMainBufferWithPrecompiledPreamble().
ASTUnit::getMainBufferWithPrecompiledPreamble() is called in 3 places:

ASTUnit::LoadFromCompilerInvocation() [analyzed below].
ASTUnit::Reparse(), which in turn is called only from
clang_reparseTranslationUnit_Impl(), which in turn is called only from
clang_reparseTranslationUnit(). clang_reparseTranslationUnit() is never
called in LLVM code, but is part of public libclang API. This function's
documentation requires its translation unit argument to have been built
with clang_createTranslationUnitFromSourceFile().
clang_createTranslationUnitFromSourceFile() delegates its work to
clang_parseTranslationUnit(), which delegates to
clang_parseTranslationUnit2(), which delegates to
clang_parseTranslationUnit2FullArgv(), which delegates to
clang_parseTranslationUnit_Impl(), which passes
CIndexer::PreambleStoragePath to the ASTUnit it creates.

ASTUnit::CodeComplete() passes AllowRebuild = false to
ASTUnit::getMainBufferWithPrecompiledPreamble(), which makes it return
nullptr before calling PrecompiledPreamble::Build().

Both ASTUnit::LoadFromCompilerInvocation() overloads (one of which
delegates its work to another) call
ASTUnit::getMainBufferWithPrecompiledPreamble() only if their argument
PrecompilePreambleAfterNParses > 0. LoadFromCompilerInvocation() is
called in:

ASTBuilderAction::runInvocation() keeps the default parameter value
of PrecompilePreambleAfterNParses = 0, meaning that the preamble file is
never created from here.

ASTUnit::LoadFromCommandLine().
ASTUnit::LoadFromCommandLine() is called in two places:

CrossTranslationUnitContext::ASTLoader::loadFromSource() keeps the
default parameter value of PrecompilePreambleAfterNParses = 0, meaning
that the preamble file is never created from here.

clang_parseTranslationUnit_Impl(), which passes
CIndexer::PreambleStoragePath to the ASTUnit it creates.

Therefore, the overridden preamble storage path is always used in
TempPCHFile::create().

TempPCHFile::create() uses PreambleStoragePath in the same way as
LibclangInvocationReporter() uses InvocationEmissionPath. The existing
documentation for clang_CXIndex_setInvocationEmissionPathOption() does
not specify ownership, encoding, separator or relative vs absolute path
requirements. So the documentation for
CXIndexOptions::PreambleStoragePath doesn't either. The assumptions are:

no ownership transfer;
UTF-8 encoding;
native separators.
Both relative and absolute paths are supported.

The added API works as expected in KDevelop:
https://invent.kde.org/kdevelop/kdevelop/-/merge_requests/283

Fixes: https://github.com/llvm/llvm-project/issues/51847
Differential Revision: https://reviews.llvm.org/D143418

1 files changed, 164 insertions, 1 deletions

diff --git a/clang/unittests/libclang/LibclangTest.cpp b/clang/unittests/libclang/LibclangTest.cpp
index 11a729f..18d0fc1 100644
--- a/clang/unittests/libclang/LibclangTest.cpp
+++ b/clang/unittests/libclang/LibclangTest.cpp
@@ -6,6 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "TestUtils.h"
 #include "clang-c/Index.h"
 #include "clang-c/Rewrite.h"
 #include "llvm/ADT/StringRef.h"
@@ -14,7 +15,7 @@
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include "gtest/gtest.h"
-#include "TestUtils.h"
+#include <cstring>
 #include <fstream>
 #include <functional>
 #include <map>
@@ -355,6 +356,168 @@ TEST(libclang, ModuleMapDescriptor) {
   clang_ModuleMapDescriptor_dispose(MMD);
 }
 
+TEST_F(LibclangParseTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index), CXGlobalOpt_None);
+}
+
+class LibclangIndexOptionsTest : public LibclangParseTest {
+  virtual void AdjustOptions(CXIndexOptions &Opts) {}
+
+protected:
+  void CreateIndex() override {
+    CXIndexOptions Opts;
+    memset(&Opts, 0, sizeof(Opts));
+    Opts.Size = sizeof(CXIndexOptions);
+    AdjustOptions(Opts);
+    Index = clang_createIndexWithOptions(&Opts);
+    ASSERT_TRUE(Index);
+  }
+};
+
+TEST_F(LibclangIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index), CXGlobalOpt_None);
+}
+
+class LibclangIndexingEnabledIndexOptionsTest
+    : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangIndexingEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForIndexing);
+}
+
+class LibclangIndexingDisabledEditingEnabledIndexOptionsTest
+    : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Disabled;
+    Opts.ThreadBackgroundPriorityForEditing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangIndexingDisabledEditingEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForEditing);
+}
+
+class LibclangBothEnabledIndexOptionsTest : public LibclangIndexOptionsTest {
+  void AdjustOptions(CXIndexOptions &Opts) override {
+    Opts.ThreadBackgroundPriorityForIndexing = CXChoice_Enabled;
+    Opts.ThreadBackgroundPriorityForEditing = CXChoice_Enabled;
+  }
+};
+
+TEST_F(LibclangBothEnabledIndexOptionsTest, GlobalOptions) {
+  EXPECT_EQ(clang_CXIndex_getGlobalOptions(Index),
+            CXGlobalOpt_ThreadBackgroundPriorityForAll);
+}
+
+class LibclangPreambleStorageTest : public LibclangParseTest {
+  std::string Main = "main.cpp";
+
+protected:
+  std::string PreambleDir;
+  void InitializePreambleDir() {
+    llvm::SmallString<128> PathBuffer(TestDir);
+    llvm::sys::path::append(PathBuffer, "preambles");
+    namespace fs = llvm::sys::fs;
+    ASSERT_FALSE(fs::create_directory(PathBuffer, false, fs::perms::owner_all));
+
+    PreambleDir = static_cast<std::string>(PathBuffer);
+    FilesAndDirsToRemove.insert(PreambleDir);
+  }
+
+public:
+  void CountPreamblesInPreambleDir(int PreambleCount) {
+    // For some reason, the preamble is not created without '\n' before `int`.
+    WriteFile(Main, "\nint main() {}");
+
+    TUFlags |= CXTranslationUnit_CreatePreambleOnFirstParse;
+    ClangTU = clang_parseTranslationUnit(Index, Main.c_str(), nullptr, 0,
+                                         nullptr, 0, TUFlags);
+
+    int FileCount = 0;
+
+    namespace fs = llvm::sys::fs;
+    std::error_code EC;
+    for (fs::directory_iterator File(PreambleDir, EC), FileEnd;
+         File != FileEnd && !EC; File.increment(EC)) {
+      ++FileCount;
+
+      EXPECT_EQ(File->type(), fs::file_type::regular_file);
+
+      const auto Filename = llvm::sys::path::filename(File->path());
+      EXPECT_EQ(Filename.size(), std::strlen("preamble-%%%%%%.pch"));
+      EXPECT_TRUE(Filename.startswith("preamble-"));
+      EXPECT_TRUE(Filename.endswith(".pch"));
+
+      const auto Status = File->status();
+      ASSERT_TRUE(Status);
+      if (false) {
+        // The permissions assertion below fails, because the .pch.tmp file is
+        // created with default permissions and replaces the .pch file along
+        // with its permissions. Therefore the permissions set in
+        // TempPCHFile::create() don't matter in the end.
+        EXPECT_EQ(Status->permissions(), fs::owner_read | fs::owner_write);
+      }
+    }
+
+    EXPECT_EQ(FileCount, PreambleCount);
+  }
+};
+
+class LibclangNotOverriddenPreambleStoragePathTest
+    : public LibclangPreambleStorageTest {
+protected:
+  void CreateIndex() override {
+    InitializePreambleDir();
+    LibclangPreambleStorageTest::CreateIndex();
+  }
+};
+
+class LibclangSetPreambleStoragePathTest : public LibclangPreambleStorageTest {
+  virtual const char *PreambleStoragePath() = 0;
+
+protected:
+  void CreateIndex() override {
+    InitializePreambleDir();
+
+    CXIndexOptions Opts = {sizeof(CXIndexOptions)};
+    Opts.PreambleStoragePath = PreambleStoragePath();
+    Index = clang_createIndexWithOptions(&Opts);
+    ASSERT_TRUE(Index);
+  }
+};
+
+class LibclangNullPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return nullptr; }
+};
+class LibclangEmptyPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return ""; }
+};
+class LibclangPreambleDirPreambleStoragePathTest
+    : public LibclangSetPreambleStoragePathTest {
+  const char *PreambleStoragePath() override { return PreambleDir.c_str(); }
+};
+
+TEST_F(LibclangNotOverriddenPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangNullPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangEmptyPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(0);
+}
+TEST_F(LibclangPreambleDirPreambleStoragePathTest, CountPreambles) {
+  CountPreamblesInPreambleDir(1);
+}
+
 TEST_F(LibclangParseTest, AllSkippedRanges) {
   std::string Header = "header.h", Main = "main.cpp";
   WriteFile(Header,