aboutsummaryrefslogtreecommitdiff
path: root/clang/lib/CodeGen/CodeGenModule.cpp
AgeCommit message (Collapse)AuthorFilesLines
2024-11-19[PAC][clang] Add signed GOT cc1 flag (#96160)Daniil Kovalev1-0/+3
Add `-fptrauth-elf-got` clang cc1 flag and set `ptrauth_elf_got` preprocessor feature and `PointerAuthELFGOT` LangOption correspondingly. No additional checks like ensuring OS binary format is ELF are performed: it should be done on clang driver level when a pauth-enabled environment implying signed GOT enabled is requested. If the cc1 flag is passed, "ptrauth-elf-got" IR module flag is set.
2024-11-16[CodeGen] Remove unused includes (NFC) (#116459)Kazu Hirata1-4/+0
Identified with misc-include-cleaner.
2024-11-14[clang codegen] Add CreateRuntimeFunction overload that takes a clang type. ↵Eli Friedman1-15/+49
(#113506) Correctly computing the LLVM types/attributes is complicated in general, so add a variant which does that for you.
2024-10-30[C++20] [Modules] Fix the duplicated static initializer problem (#114193)Chuanqi Xu1-2/+2
Reproducer: ``` //--- a.cppm export module a; int func(); static int a = func(); //--- a.cpp import a; ``` The `func()` should only execute once. However, before this patch we will somehow import `static int a` from a.cppm incorrectly and initialize that again. This is super bad and can introduce serious runtime behaviors. And also surprisingly, it looks like the root cause of the problem is simply some oversight choosing APIs.
2024-10-23[clang codegen] avoid to crash when emit init func for global variable with ↵Congcong Cai1-3/+5
flexible array init (#113336) Fixes: #113187 Avoid to create init function since clang does not support global variable with flexible array init. It will cause assertion failure later.
2024-10-18[clang] Deduplicate the logic that only warns once when stack is almost full ↵Boaz Brickner1-10/+2
(#112552) Zero diff in behavior.
2024-10-17[HLSL] Add handle initialization for simple resource declarations (#111207)Helena Kotas1-0/+3
Adds `@_init_resource_bindings()` function to module initialization that includes `handle.fromBinding` intrinsic calls for simple resource declarations. Arrays of resources or resources inside user defined types are not supported yet. While this unblocks our progress on [Compile a runnable shader from clang](https://github.com/llvm/wg-hlsl/issues/7) milestone, this is probably not the way we would like to handle resource binding initialization going forward. Ideally, it should be done via the resource class constructors in order to support dynamic resource binding or unbounded arrays if resources. Depends on PRs #110327 and #111203. Part 1 of #105076
2024-10-16[HLSL][SPIRV] Use Spirv target codegen (#112573)Steven Perron1-0/+1
When the arch in the triple in "spirv", the default target codegen is currently used. We should be using the spir-v target codegen. This will be used to have SPIR-V specific lowering of the HLSL types.
2024-10-14[clang] Fix segmentation fault caused by stack overflow on deeply nested ↵Boaz Brickner1-0/+14
expressions (#111701) Done by calling clang::runWithSufficientStackSpace(). Added CodeGenModule::runWithSufficientStackSpace() method similar to the one in Sema to provide a single warning when this triggers Fixes: #111699
2024-10-14[LLVM] [Clang] Support for Gentoo `*t64` triples (64-bit time_t ABIs) (#111302)Michał Górny1-4/+1
Gentoo is planning to introduce a `*t64` suffix for triples that will be used by 32-bit platforms that use 64-bit `time_t`. Add support for parsing and accepting these triples, and while at it make clang automatically enable the necessary glibc feature macros when this suffix is used. An open question is whether we can backport this to LLVM 19.x. After all, adding new triplets to Triple sounds like an ABI change — though I suppose we can minimize the risk of breaking something if we move new enum values to the very end.
2024-10-11[NFC] Rename `Intrinsic::getDeclaration` to `getOrInsertDeclaration` (#111752)Rahul Joshi1-2/+2
Rename the function to reflect its correct behavior and to be consistent with `Module::getOrInsertFunction`. This is also in preparation of adding a new `Intrinsic::getDeclaration` that will have behavior similar to `Module::getFunction` (i.e, just lookup, no creation).
2024-10-08Recommit "[RISCV][FMV] Support target_version" (#111096)" (#111333)Piyou Chen1-1/+6
Fix the buildbot failure caused by heap use-after-free error. Origin message: This patch enable `target_version` attribute for RISC-V target. The proposal of `target_version` syntax can be found at the https://github.com/riscv-non-isa/riscv-c-api-doc/pull/48 (which has landed), as modified by the proposed https://github.com/riscv-non-isa/riscv-c-api-doc/pull/85 (which adds the priority syntax). `target_version` attribute will trigger the function multi-versioning feature and act like `target_clones` attribute. See https://github.com/llvm/llvm-project/pull/85786 for the implementation of `target_clones`.
2024-10-04Revert "[RISCV][FMV] Support target_version" (#111096)Piyou Chen1-6/+1
Reverts llvm/llvm-project#99040 due to https://lab.llvm.org/buildbot/#/builders/190/builds/7052
2024-10-04[RISCV][FMV] Support target_version (#99040)Piyou Chen1-1/+6
This patch enable `target_version` attribute for RISC-V target. The proposal of `target_version` syntax can be found at the https://github.com/riscv-non-isa/riscv-c-api-doc/pull/48 (which has landed), as modified by the proposed https://github.com/riscv-non-isa/riscv-c-api-doc/pull/85 (which adds the priority syntax). `target_version` attribute will trigger the function multi-versioning feature and act like `target_clones` attribute. See https://github.com/llvm/llvm-project/pull/85786 for the implementation of `target_clones`.
2024-09-29[cuda][HIP] `__constant__` should imply constant (#110182)Alex Voicu1-2/+3
Currently, `__constant__` variables do not get unconditionally marked as `constant` in IR, which seems a bit odd given their definition. This is generally inconsequential for NVPTX/AMDGPU, since said variables get emitted in the constant address space for those BEs. However, it is potentially significant for e.g. HIP-on-SPIR-V cases, as SPIR-V does not allow casts to/from the constant AS (`UniformConstant`), which forces `__constant__` variables to be emitted in the global AS, thus making IR constness meaningful.
2024-09-26[clang][RISCV] Introduce command line options for RISC-V Zicfilp CFIMing-Yi Lai1-0/+10
This patch enables the following command line flags for RISC-V targets: + `-fcf-protection=branch` turns on forward-edge control-flow integrity conditioning + `-mcf-branch-label-scheme=unlabeled|func-sig` selects the label scheme used in the forward-edge CFI conditioning
2024-09-24[codegen][NFC] add static mark for internal usage variable and function ↵Congcong Cai1-4/+5
(#109431) Detect by clang-tidy misc-use-internal-linkage
2024-09-17Reapply "[HLSL] set alwaysinline on HLSL functions (#106588)"Thurston Dang1-6/+14
This reverts commit 4a63f4d301c0e044073e1b1f8f110015ec1778a1. It was reverted because of a buildbot breakage, but the fix-forward has landed (https://github.com/llvm/llvm-project/pull/109023).
2024-09-17Revert "[HLSL] set alwaysinline on HLSL functions (#106588)"Thurston Dang1-14/+6
This reverts commit a729e706de3fc6ebee49ede3c50afb47f2e29191. Reason:bBuildbot failure (https://lab.llvm.org/buildbot/#/builders/25/builds/2541): 'Clang :: CodeGenHLSL/builtins/StructuredBuffer-subscript.hlsl' failed
2024-09-17[HLSL] set alwaysinline on HLSL functions (#106588)Greg Roth1-6/+14
HLSL inlines all its functions by default. This uses the alwaysinline attribute to make the alwaysinliner pass inline any function not explicitly marked noinline by the user or autogeneration. The alwayslinline marking takes place in `SetLLVMFunctionAttributesForDefinitions` where all other inlining interactions are determined. The outermost entry function is marked noinline because there's no reason to inline it. Any user calls to an entry function will instead call the internal mangled version of the entry function. Adds tests for function and constructor inlining and augments some existing tests to verify correct inlining of implicitly created functions as well. Incidentally restore RUN line that I believe was mistakenly removed as part of #88918 Fixes #89282
2024-09-13[RISCV][FMV] Support target_clones (#85786)Piyou Chen1-1/+4
This patch enable the function multiversion(FMV) and `target_clones` attribute for RISC-V target. The proposal of `target_clones` syntax can be found at the https://github.com/riscv-non-isa/riscv-c-api-doc/pull/48 (which has landed), as modified by the proposed https://github.com/riscv-non-isa/riscv-c-api-doc/pull/85 (which adds the priority syntax). It supports the `target_clones` function attribute and function multiversioning feature for RISC-V target. It will generate the ifunc resolver function for the function that declared with target_clones attribute. The resolver function will check the version support by runtime object `__riscv_feature_bits`. For example: ``` __attribute__((target_clones("default", "arch=+ver1", "arch=+ver2"))) int bar() { return 1; } ``` the corresponding resolver will be like: ``` bar.resolver() { __init_riscv_feature_bits(); // Check arch=+ver1 if ((__riscv_feature_bits.features[0] & BITMASK_OF_VERSION1) == BITMASK_OF_VERSION1) { return bar.arch=+ver1; } else { // Check arch=+ver2 if ((__riscv_feature_bits.features[0] & BITMASK_OF_VERSION2) == BITMASK_OF_VERSION2) { return bar.arch=+ver2; } else { // Default return bar.default; } } } ```
2024-09-10[DirectX] Add DirectXTargetCodeGenInfo (#104856)Helena Kotas1-0/+2
Adds target codegen info class for DirectX. For now it always translates `__hlsl_resource_t` handle to `target("dx.TypedBuffer", i32, 1, 0, 1)` (`RWBuffer<int>`). More work is needed to determine the actual target exp type and parameters based on the resource handle attributes. Part 1/2 of #95952
2024-08-28[CodeGen] Create IFUNCs in the program address space, not hard-coded 0 (#105726)Jessica Clarke1-12/+12
Commit 0d527e56a5ee ("GlobalIFunc: Make ifunc respect function address spaces") added support for this within LLVM, but Clang does not properly honour the target's address spaces when creating IFUNCs, crashing with RAUW and verifier assertion failures when compiling C code on a target with a non-zero program address space, so fix this.
2024-08-21[clang-repl] [codegen] Reduce the state in TBAA. NFC for static compilation. ↵Vassil Vassilev1-6/+9
(#98138) In incremental compilation clang works with multiple `llvm::Module`s. Our current approach is to create a CodeGenModule entity for every new module request (via StartModule). However, some of the state such as the mangle context needs to be preserved to keep the original semantics in the ever-growing TU. Fixes: llvm/llvm-project#95581. cc: @jeaye
2024-08-20Fix KCFI types for generated functions with integer normalization (#104826)Sami Tolvanen1-0/+5
With -fsanitize-cfi-icall-experimental-normalize-integers, Clang appends ".normalized" to KCFI types in CodeGenModule::CreateKCFITypeId, which changes type hashes also for functions that don't have integer types in their signatures. However, llvm::setKCFIType does not take integer normalization into account, which means LLVM generated functions with KCFI types, e.g. sanitizer constructors, will fail KCFI checks when integer normalization is enabled in Clang. Add a cfi-normalize-integers module flag to indicate integer normalization is used, and append ".normalized" to KCFI types also in llvm::setKCFIType to fix the type mismatch.
2024-08-19[PAC][ELF][AArch64] Encode several ptrauth features in PAuth core info (#102508)Daniil Kovalev1-2/+8
For llvm_linux platform, define the following meaning for bits 9, 10, 11: - bit 9: set if indirect gotos signing is enabled; - bit 10: set if type info vtable pointer discrimination is enabled; - bit 11: set if function pointer type discrimination is enabled.
2024-08-09[DebugInfo][RemoveDIs] Use iterator-inserters in clang (#102006)Jeremy Morse1-5/+5
As part of the LLVM effort to eliminate debug-info intrinsics, we're moving to a world where only iterators should be used to insert instructions. This isn't a problem in clang when instructions get generated before any debug-info is inserted, however we're planning on deprecating and removing the instruction-pointer insertion routines. Scatter some calls to getIterator in a few places, remove a deref-then-addrof on another iterator, and add an overload for the createLoadInstBefore utility. Some callers passes a null insertion point, which we need to handle explicitly now.
2024-08-06[PAC][ELF][AArch64] Encode signed GOT flag in PAuth core info (#96159)Daniil Kovalev1-5/+6
Treat 8th bit of version value for llvm_linux platform as signed GOT flag. - clang: define `PointerAuthELFGOT` LangOption and set 8th bit of `aarch64-elf-pauthabi-version` LLVM module flag correspondingly; - llvm-readobj: print `PointerAuthELFGOT` or `!PointerAuthELFGOT` in version description of llvm_linux platform depending on whether the flag is set.
2024-08-06Reapply "Finish deleting the le32/le64 targets" (#99079) (#101983)Aaron Ballman1-2/+0
This reverts commit d3f8105c65046173e20c4c59394b4a7f1bbe7627. Halide no longer relies on this target: https://github.com/llvm/llvm-project/pull/98497#issuecomment-2253358685
2024-08-06[PAC][AArch64] Support init/fini array signing (#96478)Daniil Kovalev1-24/+43
If both `-fptrauth-init-fini` and `-fptrauth-calls` are passed, sign function pointers in `llvm.global_ctors` and `llvm.global_dtors` with constant discriminator 0xD9D4 (`ptrauth_string_discriminator("init_fini")`). Additionally, if `-fptrauth-init-fini-address-discrimination` is passed, address discrimination is used for signing (otherwise, just constant discriminator is used). For address discrimination, we use it's special form since uses of `llvm.global_{c|d}tors` are disallowed (see `Verifier::visitGlobalVariable`) and we can't emit `getelementptr` expressions referencing these special arrays. A signed ctor/dtor pointer with special address discrimination applied looks like the following: ``` ptr ptrauth (ptr @foo, i32 0, i64 55764, ptr inttoptr (i64 1 to ptr)) ```
2024-08-01[C++20] [Modules] Always emit the inline builtins (#101278)Chuanqi Xu1-5/+5
See the attached test for the motivation example. If we're too greedy to not emit the definition for inline builtins, we may meet a middle end crash. And it should be good to emit inline builtins always.
2024-07-31[Clang] [NFC] Fix potential dereferencing of nullptr (#101405)smanna121-1/+1
This patch replaces getAs with castAs and dyn_cast with cast to ensure type safety and prevents potential null pointer dereferences. These changes enforce compile-time checks for correct type casting in ASTContext and CodeGenModule.
2024-07-30Revert "[C++20] [Modules] Always emit the inline builtins"Haowei Wu1-5/+5
This reverts commit ca8a4111f796fe8533e0af95557875b15becff06. This patch broke clang test on Windows x64 msvc
2024-07-30[clang][OpenMP] Rename `varlists` to `varlist`, NFC (#101058)Krzysztof Parzyszek1-1/+1
It returns a range of variables (via Expr*), not a range of lists.
2024-07-30[C++20] [Modules] Always emit the inline builtinsChuanqi Xu1-5/+5
See the attached test for the motivation example. If we're too greedy to not emit the definition for inline builtins, we may meet a middle end crash. And it should be good to emit inline builtins always.
2024-07-22[PAC][Driver] Support `pauthtest` ABI for AArch64 Linux triples (#97237)Daniil Kovalev1-0/+2
When `pauthtest` is either passed as environment part of AArch64 Linux triple or passed via `-mabi=`, enable the following ptrauth flags: - `intrinsics`; - `calls`; - `returns`; - `auth-traps`; - `vtable-pointer-address-discrimination`; - `vtable-pointer-type-discrimination`; - `init-fini`. Some related stuff is still subject to change, and the ABI itself might be changed, so end users are not expected to use this and the ABI name has 'test' suffix. If `-mabi=pauthtest` option is used, it's normalized to effective triple. When the environment part of the effective triple is `pauthtest`, try to use `aarch64-linux-pauthtest` as multilib directory. The following is not supported: - combination of `pauthtest` ABI with any branch protection scheme except BTI; - explicit set of environment part of the triple to a value different from `pauthtest` in combination with `-mabi=pauthtest`; - usage on non-Linux OS. --------- Co-authored-by: Anatoly Trosinenko <atrosinenko@accesssoftek.com>
2024-07-18[FMV][AArch64] Do not emit ifunc resolver on use. (#97761)Alexandros Lamprineas1-47/+50
It was raised in https://github.com/llvm/llvm-project/issues/81494 that we are not generating correct code when there is no TU-local caller. The suggestion was to emit a resolver: * Whenever there is a use in the TU. * When the TU has a definition of the default version. See the comment for more details: https://github.com/llvm/llvm-project/issues/81494#issuecomment-1985963497 This got addressed with https://github.com/llvm/llvm-project/pull/84405. Generating a resolver on use means that we may end up with multiple resolvers across different translation units. Those resolvers may not be the same because each translation unit may contain different version declarations (user's fault). Therefore the order of linking the final image determines which of these weak symbols gets selected, resulting in non consisted behavior. I am proposing to stop emitting a resolver on use and only do so in the translation unit which contains the default definition. This way we guarantee the existence of a single resolver. Now, when a versioned function is used we want to emit a declaration of the function symbol omitting the multiversion mangling. I have added a requirement to ACLE mandating that all the function versions are declared in the translation unit which contains the default definition: https://github.com/ARM-software/acle/pull/328
2024-07-16Revert "Finish deleting the le32/le64 targets" (#99079)Aaron Ballman1-0/+2
Reverts llvm/llvm-project#98497 We're reverting this for approx 30 days so that the Halide project has time to transition off the target.
2024-07-15[CodeGen] Set attributes on resolvers emitted after ifuncsFangrui Song1-6/+11
Visiting the ifunc calls `GetOrCreateLLVMFunction` with `NotForDefinition` while visiting the resolver calls `GetOrCreateLLVMFunction` with `ForDefinition`. When an ifunc is emitted before its resolver, the `ForDefinition` call does not call `SetFunctionAttributes`, because the function prematurely returns due to `(Entry->getValueType() == Ty)` and `llvm::GlobalIFunc::getResolverFunctionType(DeclTy)`. This leads to missing `!kcfi_type` with -fsanitize=kcfi. ``` extern void ifunc0(void) __attribute__ ((ifunc("resolver0"))); void *resolver0(void) { return 0; } // SetFunctionAttributes not called extern void ifunc1(void) __attribute__ ((ifunc("resolver1"))); static void *resolver1(void) { return 0; } // SetFunctionAttributes not called extern void ifunc2(void) __attribute__ ((ifunc("resolver2"))); static void *resolver2(void*) { return 0; } ``` Ensure `SetFunctionAttributes` is called by calling `GetOrCreateLLVMFunction` with a dummy non-function type. Now that the `F->takeName(Entry)` code path may be taken, the `DisableSanitizerInstrumentation` code (https://reviews.llvm.org/D150262) should be moved to `checkAliases`, when the resolver function is finalized. Pull Request: https://github.com/llvm/llvm-project/pull/98832
2024-07-12[CUDA][HIP] Fix template static member (#98580)Yaxun (Sam) Liu1-1/+2
Should check host/device attributes before emitting static member of template instantiation. Fixes: https://github.com/llvm/llvm-project/issues/98151
2024-07-12Finish deleting the le32/le64 targets (#98497)Aaron Ballman1-2/+0
This is a revert of ef5e7f90ea4d5063ce68b952c5de473e610afc02 which was a temporary partial revert of 77ac823fd285973cfb3517932c09d82e6a32f46d. The le32 and le64 targets are no longer necessary to retain, so this removes them entirely.
2024-07-11[CUDA][HIP][NFC] add CodeGenModule::shouldEmitCUDAGlobalVar (#98543)Yaxun (Sam) Liu1-20/+24
Extract the logic whether to emit a global var based on CUDA/HIP host/device related attributes to CodeGenModule::shouldEmitCUDAGlobalVar to be used by other places.
2024-07-10[Coverage] Suppress covmap and profdata for system headers. (#97952)NAKAMURA Takumi1-0/+3
With `system-headers-coverage=false`, functions defined in system headers were not instrumented but corresponding covmaps were emitted. It caused wasting covmap and profraw. This change improves: - Reduce object size (due to reduced covmap) - Reduce size of profraw (uninstrumented system headers occupied counters) - Smarter view of coverage report. Stubs of uninstrumented system headers will be no longer seen.
2024-07-08[PowerPC] Diagnose musttail instead of crash inside backend (#93267)Chen Zheng1-0/+15
musttail is not often possible to be generated on PPC targets as when calling to a function defined in another module, PPC needs to restore the TOC pointer. To restore the TOC pointer, compiler needs to emit a nop after the call to let linker generate codes to restore TOC pointer. Tail call cannot generate expected call sequence for this case. To avoid the crash inside the compiler backend, a diagnosis is added in the frontend. Fixes #63214
2024-07-05[BPF] Fix linking issues in static map initializers (#91310)Nick Zavaritsky1-2/+17
When BPF object files are linked with bpftool, every symbol must be accompanied by BTF info. Ensure that extern functions referenced by global variable initializers are included in BTF. The primary motivation is "static" initialization of PROG maps: ```c extern int elsewhere(struct xdp_md *); struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY); __uint(max_entries, 1); __type(key, int); __type(value, int); __array(values, int (struct xdp_md *)); } prog_map SEC(".maps") = { .values = { elsewhere } }; ``` BPF backend needs debug info to produce BTF. Debug info is not normally generated for external variables and functions. Previously, it was solved differently for variables (collecting variable declarations in ExternalDeclarations vector) and functions (logic invoked during codegen in CGExpr.cpp). This patch generalises ExternalDefclarations to include both function and variable declarations. This change ensures that function references are not missed no matter the context. Previously external functions referenced in constant expressions lacked debug info.
2024-07-03[OpenCL] Emit opencl.cxx.version metadata for C++ (#92140)Sven van Haastregt1-11/+19
Currently there is no way to tell whether an IR module was generated using `-cl-std=cl3.0` or `-cl-std=clc++2021`, i.e., whether the origin was a OpenCL C or C++ for OpenCL source. Add new `opencl.cxx.version` named metadata when compiling C++. Keep the `opencl.ocl.version` metadata to convey the compatible OpenCL C version. Fixes https://github.com/llvm/llvm-project/issues/91912
2024-06-25[clang][Driver] Add HIPAMD Driver support for AMDGCN flavoured SPIR-V (#95061)Alex Voicu1-1/+2
This patch augments the HIPAMD driver to allow it to target AMDGCN flavoured SPIR-V compilation. It's mostly straightforward, as we re-use some of the existing SPIRV infra, however there are a few notable additions: - we introduce an `amdgcnspirv` offload arch, rather than relying on using `generic` (this is already fairly overloaded) or simply using `spirv` or `spirv64` (we'll want to use these to denote unflavoured SPIRV, once we bring up that capability) - initially it is won't be possible to mix-in SPIR-V and concrete AMDGPU targets, as it would require some relatively intrusive surgery in the HIPAMD Toolchain and the Driver to deal with two triples (`spirv64-amd-amdhsa` and `amdgcn-amd-amdhsa`, respectively) - in order to retain user provided compiler flags and have them available at JIT time, we rely on embedding the command line via `-fembed-bitcode=marker`, which the bitcode writer had previously not implemented for SPIRV; we only allow it conditionally for AMDGCN flavoured SPIRV, and it is handled correctly by the Translator (it ends up as a string literal) Once the SPIRV BE is no longer experimental we'll switch to using that rather than the translator. There's some additional work that'll come via a separate PR around correctly piping through AMDGCN's implementation of `printf`, for now we merely handle its flags correctly.
2024-06-24[clang][AArch64][FMV] Stop emitting alias to ifunc. (#96221)Alexandros Lamprineas1-2/+2
Long story short the interaction of two optimizations happening in GlobalOpt results in a crash. For more details look at the issue https://github.com/llvm/llvm-project/issues/96197. I will be fixing this in GlobalOpt but it is a conservative solution since it won't allow us to optimize resolvers which return a pointer to a function whose definition is in another TU when compiling without LTO: ``` __attribute__((target_version("simd"))) void bar(void); __attribute__((target_version("default"))) void bar(void); int foo() { bar(); } ``` fixes: #96197
2024-06-10[clang] Fix loss of `dllexport` for exported template specialization (#94664)Andrew Ng1-4/+15
When dropping DLL attributes, ensure that the most recent declaration is being checked.
2024-06-07[ARM] r11 is reserved when using -mframe-chain=aapcs (#86951)Oliver Stannard1-0/+3
When using the -mframe-chain=aapcs or -mframe-chain=aapcs-leaf options, we cannot use r11 as an allocatable register, even if -fomit-frame-pointer is also used. This is so that r11 will always point to a valid frame record, even if we don't create one in every function.