| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This should not include targets that aren't enabled in cmake.
6ccf97674b2deaa03e271725306b18a712a56113
|
|
|
|
I did this improperly in
4ef68008b61b5d8d3297836c328cc72ee29efbdb not realizing those headers
already had libraries setup with the correct deps.
|
|
|
|
# What
This PR adds a shared library `MLIRPythonSupport` which contains all of
the CRTP classes ike `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute`, as well as other useful code like `Defaulting*`
and etc enabling their reuse in downstream projects. Downstream projects
can now do
```c++
struct PyTestType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyTestType> {
...
};
class PyTestAttr : public mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteAttribute<PyTestAttr> {
...
}
NB_MODULE(_mlirPythonTestNanobind, m) {
PyTestType::bind(m);
PyTestAttr::bind(m);
}
```
instead of using the discordant alternative
`mlir_type_subclass`/`mlir_attr_subclass` (same goes for
`PyConcreteValue`/`mlir_value_subclass`).
# Why
This PR is mostly code motion (along with CMake) but before I describe
the changes I want to state the goals/benefits:
1. Currently upstream "core" extensions and "dialect" extensions ([all
of the `Dialect*` extensions
here](https://github.com/llvm/llvm-project/tree/d7c734b5a14bd91e1c76e2ce0014c19f9deef487/mlir/lib/Bindings/Python))
are a two-tier system;
**a**. [core
extensions](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Bindings/Python/IRTypes.cpp#L361)
enjoy first class support as far as type inference[^3], type stub
generation, and ease of implementation, while dialect extensions [have
poorer support](https://reviews.llvm.org/D150927), incorrect type stub
generation much more tedious (boilerplate) implementation;
**b**. Crucially, this two-tiered system is reflected in the fact that
**the two sets of types/attributes are not in the same Python object
hierarchy**. To wit: `isinstance(..., Type)` and `isinstance(...,
Attribute)` are not supported for the dialect extensions[^2];
**c**. Since these types are not exposed in public headers, downstream
users (dialect extensions or not) cannot write functions that overload
on e.g. `PyFloat8*Type` - that's quite a [useful
feature](https://github.com/nod-ai/PI/blob/fdbee98df8376f47818e6b47e1cf089528c9d48d/cpp_ext/TorchOps.cpp#L29-L69)!
2. The dialect extensions incur a sizeable performance penalty relative
to the core extensions in that every single trip across the wire (either
`python->cpp` or `cpp->python`) requires work in addition to nanobind's
own casting/construction pipeline;
**a**. When going from `python->cpp`, [we extract the capsule object
from the Python
object](https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L219C24-L219C46)
and then extract from the capsule the `Mlir*` opaque struct/ptr. This
side isn't so onerous;
**b**. When going from `cpp->python` we call long-hand call Python
`import` APIs and construct the Python object using `_CAPICreate`. Note,
there at least 2 `attr` calls incurred in addition to `_CAPICreate`;
this is already much more [efficiently handled by nanobind
itself](https://github.com/wjakob/nanobind/blob/4ba51fcf795971c5d603d875ae4184bc0c9bd8e6/src/nb_internals.h#L381-L382)!
3. This division blocks various features: in some configurations[^1] we
trigger a circular import bug because "dialect" types and attributes
perform an [import of the root `_mlir`
module](https://github.com/llvm/llvm-project/blob/bd9651bf78f2b1713a8203e0bd5b97f7ff199924/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L585)
when they are created (the types themselves, not even instances of those
types). This blocks type stub generation for dialect extensions (i.e.,
the reason we currently only generate type stubs for `_mlir`).
# How
Prior this was not done/possible because of "ODR" issues but I have
resolved those issues; the basic idea for how we solve this is "move
things we want to share into shared libraries":
1. Move IRCore (stuff like `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute`) into `MLIRPythonSupport`;
- Note, we move the rest of the things in `IRModule.h` (renamed to
`IRCore.h`) because `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute` depend on them. This makes for a bigger PR than
one would hope for but ultimately I think we should give people access
to these classes to use as they see fit (specifically inherit from, but
also liberally use in bindings signatures instead of the opaque `Mlir*`
struct wrappers).
2. Put all of this code into a nested namespace
`MLIR_BINDINGS_PYTHON_DOMAIN` which is determined by a compile time
define (and tied to `MLIR_BINDINGS_PYTHON_NB_DOMAIN`). This is necessary
in order to prevent conflicts on both symbol name **and** typeid
(necessary for nanobind to not double register binded types) between
multiple bindings libraries (e.g., `torch-mlir`, and `jax`). Note
[nanobind doesn't support `module_local` like
pybind11](https://nanobind.readthedocs.io/en/latest/porting.html#removed-features).
It does support `NB_DOMAIN` but that is not sufficient for
disambiguating typeids across projects (to wit: we currently define
`NB_DOMAIN` and it was still necessary to move everything to a nested
namespace);
3. Build the [nanobind library itself as a shared
object](https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake#L127)
(and link it to both the extensions and `MLIRPythonSupport`).
4. CMake to make this work, in-tree, out-of-tree, downstream, upstream,
etc.
# Testing
Three tests are added here
1. `PythonTestModuleNanobind` is ported to use
`PyConcreteType<PyTestType>` instead of `mlir_type_subclass` and
`PyConcreteAttribute<PyTestAttr>` instead of `mlir_atrr_subclass`,
verifying this works for non-core extensions in-tree;
2. `StandaloneExtensionNanobind` is ported to use `struct PyCustomType :
mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyCustomType>`
instead of `mlir_type_subclass` verifying this works for non-core
extensions out-of-tree;
3. `StandaloneExtensionNanobind`'s `smoketest` is extended to also load
another bindings package (namely `mlir`) verifying
`MLIR_BINDINGS_PYTHON_DOMAIN` successfully disambiguates symbols and
typeids.
I have also tested this downstream:
https://github.com/llvm/eudsl/pull/287 as well run the following builder
bots:
mlir-nvidia-gcc7:
https://lab.llvm.org/buildbot/#/buildrequests/6654424?redirect_to_build=true
I have also tested against IREE:
https://github.com/iree-org/iree/pull/21916
# Integration
It is highly recommended to set the CMake var
`MLIR_BINDINGS_PYTHON_NB_DOMAIN` (which will also determine
`MLIR_BINDINGS_PYTHON_DOMAIN`) to something unique for each downstream.
This can also be passed explicitly to `add_mlir_python_modules` if your
project builds multiple bindings packages. I added a `WARNING` to this
effect in `AddMLIRPython.cmake`.
[^3]: Python values being typed correctly when exiting from cpp;
[^1]: Specifically when the modules are imported using `importlib`,
which occurs with nanobind's
[stubgen](https://github.com/wjakob/nanobind/blob/master/src/stubgen.py#L965);
[^2]: The workaround we implemented was a class method for the dialect
bindings called `Class.isinstance(...)`;
|
|
Co-authored-by: Pranav Kant <prka@google.com>
|
|
502c34250420eff47318568ab154f0a66c4a2324 fixed the build, but did not
order the dependencies correctly, so this broke buildifier.
|
|
Co-authored-by: Pranav Kant <prka@google.com>
|
|
|
|
|
|
|
|
Co-authored-by: Pranav Kant <prka@google.com>
|
|
It hasn't been built by default in CMake side.
|
|
Co-authored-by: Pranav Kant <prka@google.com>
|
|
1.
https://github.com/llvm/llvm-project/commit/f54df0d09e19ec6b205cb0af45c7ecea2fd8aeff
moved PassPlugin.h from llvm/Passes to llvm/Plugins.
2. https://github.com/llvm/llvm-project/pull/173104 updated clib dep for
__support_alloc_checker
This bazel rules accomodates the changes
|
|
src/__support/math folder. (#162131)
Part of #147386
in preparation for: https://discourse.llvm.org/t/rfc-make-clang-builtin-math-functions-constexpr-with-llvm-libc-to-support-c-23-constexpr-math-functions/86450
|
|
Just adds some missing configs
|
|
This is pointing to the same thing but without this subdomain which is
currently down. I'm not sure that is temporary or not but I think this
is the more canonical URL
|
|
|
|
|
|
|
|
|
|
fix #172479
|
|
Otherwise users of mlir::populateGpuToROCDLConversionPatterns cannot
call it, because they can't access the definition of the
mlir::gpu::amd::Runtime enum, which is in Runtimes.h.
|
|
Added in #162362 by mistake.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Added a new preprocessor macro to llvm-config.h which needs to be
reflected on the bazel side.
|
|
As in the description.
|
|
|
|
|
|
Was missing a dep on the bazel side.
|
|
Enables verification of attributes, independent of op, that references symbols.
This enables verifying Attribute with symbol usage independent of operation
attached to (e.g., the validity is on the Attribute independent of the operation).
---------
Co-authored-by: Mehdi Amini <joker.eph@gmail.com>
|
|
Adds a couple additional deps to OpenACCUtils.
|
|
This patch removed some source files that were explicitly enumerated in
the bazel files. Remove them so that the build passes.
|
|
|
|
Move some headers into the common __support_time library now that they
are no longer platform specific.
|
|
|
|
|
|
fenv_t. (#165015)
|
|
dialect interfaces (#170884)
|
|
(#170869)
|
|
deps (#170866)
|
|
|
|
function selection (#165613) (#170738)
[Previous commit had an incorrect default case when
FIND_FIRST_CHARACTER_WIDE_READ_IMPL was not specified in config.json.
This PR is identical to that one with one line fixed.]
As we implement more high-performance string-related functions, we have
found a need for better control over their selection than the big-hammer
LIBC_CONF_STRING_LENGTH_WIDE_READ. For example, I have a memchr
implementation coming, and unless I implement it in every variant, a
simple binary value doesn't work.
This PR makes gives finer-grained control over high-performance
functions than the generic LIBC_CONF_UNSAFE_WIDE_READ option. For any
function they like, the user can now select one of four implementations
at build time:
1. element, which reads byte-by-byte (or wchar by wchar)
2. wide, which reads by unsigned long
3. generic, which uses standard clang vector implemenations, if
available
4. arch, which uses an architecture-specific implemenation
(Reading the code carefully, you may note that a user can actually
specify any namespace they want, so we aren't technically limited to
those 4.)
We may also want to switch from command-line #defines as it is currently
done, to something more like
llvm-project/llvm/include/llvm/Config/llvm-config.h.cmake, and
complexity out of the command-line. But that's a future problem.
|
