diff options
author | Martin Liska <mliska@suse.cz> | 2022-11-13 21:59:29 +0100 |
---|---|---|
committer | Martin Liska <mliska@suse.cz> | 2022-11-14 09:35:06 +0100 |
commit | d77de738290156fafe079182888e5e03a2f835f1 (patch) | |
tree | 0fa1501804778de28e5323a1ecc0d39073b4045c /libgomp | |
parent | 40a39381063fdd83c4cbf5eacebfc50a2201308b (diff) | |
download | gcc-d77de738290156fafe079182888e5e03a2f835f1.zip gcc-d77de738290156fafe079182888e5e03a2f835f1.tar.gz gcc-d77de738290156fafe079182888e5e03a2f835f1.tar.bz2 |
Revert "sphinx: remove texinfo files"
This reverts commit 54ca4eef58661a7d7a511e2bbbe309bde1732abf.
Diffstat (limited to 'libgomp')
-rw-r--r-- | libgomp/libgomp.texi | 4884 |
1 files changed, 4884 insertions, 0 deletions
diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi new file mode 100644 index 0000000..10fefa9 --- /dev/null +++ b/libgomp/libgomp.texi @@ -0,0 +1,4884 @@ +\input texinfo @c -*-texinfo-*- + +@c %**start of header +@setfilename libgomp.info +@settitle GNU libgomp +@c %**end of header + + +@copying +Copyright @copyright{} 2006-2022 Free Software Foundation, Inc. + +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License, Version 1.3 or +any later version published by the Free Software Foundation; with the +Invariant Sections being ``Funding Free Software'', the Front-Cover +texts being (a) (see below), and with the Back-Cover Texts being (b) +(see below). A copy of the license is included in the section entitled +``GNU Free Documentation License''. + +(a) The FSF's Front-Cover Text is: + + A GNU Manual + +(b) The FSF's Back-Cover Text is: + + You have freedom to copy and modify this GNU Manual, like GNU + software. Copies published by the Free Software Foundation raise + funds for GNU development. +@end copying + +@ifinfo +@dircategory GNU Libraries +@direntry +* libgomp: (libgomp). GNU Offloading and Multi Processing Runtime Library. +@end direntry + +This manual documents libgomp, the GNU Offloading and Multi Processing +Runtime library. This is the GNU implementation of the OpenMP and +OpenACC APIs for parallel and accelerator programming in C/C++ and +Fortran. + +Published by the Free Software Foundation +51 Franklin Street, Fifth Floor +Boston, MA 02110-1301 USA + +@insertcopying +@end ifinfo + + +@setchapternewpage odd + +@titlepage +@title GNU Offloading and Multi Processing Runtime Library +@subtitle The GNU OpenMP and OpenACC Implementation +@page +@vskip 0pt plus 1filll +@comment For the @value{version-GCC} Version* +@sp 1 +Published by the Free Software Foundation @* +51 Franklin Street, Fifth Floor@* +Boston, MA 02110-1301, USA@* +@sp 1 +@insertcopying +@end titlepage + +@summarycontents +@contents +@page + + +@node Top, Enabling OpenMP +@top Introduction +@cindex Introduction + +This manual documents the usage of libgomp, the GNU Offloading and +Multi Processing Runtime Library. This includes the GNU +implementation of the @uref{https://www.openmp.org, OpenMP} Application +Programming Interface (API) for multi-platform shared-memory parallel +programming in C/C++ and Fortran, and the GNU implementation of the +@uref{https://www.openacc.org, OpenACC} Application Programming +Interface (API) for offloading of code to accelerator devices in C/C++ +and Fortran. + +Originally, libgomp implemented the GNU OpenMP Runtime Library. Based +on this, support for OpenACC and offloading (both OpenACC and OpenMP +4's target construct) has been added later on, and the library's name +changed to GNU Offloading and Multi Processing Runtime Library. + + + +@comment +@comment When you add a new menu item, please keep the right hand +@comment aligned to the same column. Do not use tabs. This provides +@comment better formatting. +@comment +@menu +* Enabling OpenMP:: How to enable OpenMP for your applications. +* OpenMP Implementation Status:: List of implemented features by OpenMP version +* OpenMP Runtime Library Routines: Runtime Library Routines. + The OpenMP runtime application programming + interface. +* OpenMP Environment Variables: Environment Variables. + Influencing OpenMP runtime behavior with + environment variables. +* Enabling OpenACC:: How to enable OpenACC for your + applications. +* OpenACC Runtime Library Routines:: The OpenACC runtime application + programming interface. +* OpenACC Environment Variables:: Influencing OpenACC runtime behavior with + environment variables. +* CUDA Streams Usage:: Notes on the implementation of + asynchronous operations. +* OpenACC Library Interoperability:: OpenACC library interoperability with the + NVIDIA CUBLAS library. +* OpenACC Profiling Interface:: +* OpenMP-Implementation Specifics:: Notes specifics of this OpenMP + implementation +* Offload-Target Specifics:: Notes on offload-target specific internals +* The libgomp ABI:: Notes on the external ABI presented by libgomp. +* Reporting Bugs:: How to report bugs in the GNU Offloading and + Multi Processing Runtime Library. +* Copying:: GNU general public license says + how you can copy and share libgomp. +* GNU Free Documentation License:: + How you can copy and share this manual. +* Funding:: How to help assure continued work for free + software. +* Library Index:: Index of this documentation. +@end menu + + +@c --------------------------------------------------------------------- +@c Enabling OpenMP +@c --------------------------------------------------------------------- + +@node Enabling OpenMP +@chapter Enabling OpenMP + +To activate the OpenMP extensions for C/C++ and Fortran, the compile-time +flag @command{-fopenmp} must be specified. This enables the OpenMP directive +@code{#pragma omp} in C/C++ and @code{!$omp} directives in free form, +@code{c$omp}, @code{*$omp} and @code{!$omp} directives in fixed form, +@code{!$} conditional compilation sentinels in free form and @code{c$}, +@code{*$} and @code{!$} sentinels in fixed form, for Fortran. The flag also +arranges for automatic linking of the OpenMP runtime library +(@ref{Runtime Library Routines}). + +A complete description of all OpenMP directives may be found in the +@uref{https://www.openmp.org, OpenMP Application Program Interface} manuals. +See also @ref{OpenMP Implementation Status}. + + +@c --------------------------------------------------------------------- +@c OpenMP Implementation Status +@c --------------------------------------------------------------------- + +@node OpenMP Implementation Status +@chapter OpenMP Implementation Status + +@menu +* OpenMP 4.5:: Feature completion status to 4.5 specification +* OpenMP 5.0:: Feature completion status to 5.0 specification +* OpenMP 5.1:: Feature completion status to 5.1 specification +* OpenMP 5.2:: Feature completion status to 5.2 specification +@end menu + +The @code{_OPENMP} preprocessor macro and Fortran's @code{openmp_version} +parameter, provided by @code{omp_lib.h} and the @code{omp_lib} module, have +the value @code{201511} (i.e. OpenMP 4.5). + +@node OpenMP 4.5 +@section OpenMP 4.5 + +The OpenMP 4.5 specification is fully supported. + +@node OpenMP 5.0 +@section OpenMP 5.0 + +@unnumberedsubsec New features listed in Appendix B of the OpenMP specification +@c This list is sorted as in OpenMP 5.1's B.3 not as in OpenMP 5.0's B.2 + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item Array shaping @tab N @tab +@item Array sections with non-unit strides in C and C++ @tab N @tab +@item Iterators @tab Y @tab +@item @code{metadirective} directive @tab N @tab +@item @code{declare variant} directive + @tab P @tab @emph{simd} traits not handled correctly +@item @emph{target-offload-var} ICV and @code{OMP_TARGET_OFFLOAD} + env variable @tab Y @tab +@item Nested-parallel changes to @emph{max-active-levels-var} ICV @tab Y @tab +@item @code{requires} directive @tab P + @tab complete but no non-host devices provides @code{unified_address}, + @code{unified_shared_memory} or @code{reverse_offload} +@item @code{teams} construct outside an enclosing target region @tab Y @tab +@item Non-rectangular loop nests @tab Y @tab +@item @code{!=} as relational-op in canonical loop form for C/C++ @tab Y @tab +@item @code{nonmonotonic} as default loop schedule modifier for worksharing-loop + constructs @tab Y @tab +@item Collapse of associated loops that are imperfectly nested loops @tab N @tab +@item Clauses @code{if}, @code{nontemporal} and @code{order(concurrent)} in + @code{simd} construct @tab Y @tab +@item @code{atomic} constructs in @code{simd} @tab Y @tab +@item @code{loop} construct @tab Y @tab +@item @code{order(concurrent)} clause @tab Y @tab +@item @code{scan} directive and @code{in_scan} modifier for the + @code{reduction} clause @tab Y @tab +@item @code{in_reduction} clause on @code{task} constructs @tab Y @tab +@item @code{in_reduction} clause on @code{target} constructs @tab P + @tab @code{nowait} only stub +@item @code{task_reduction} clause with @code{taskgroup} @tab Y @tab +@item @code{task} modifier to @code{reduction} clause @tab Y @tab +@item @code{affinity} clause to @code{task} construct @tab Y @tab Stub only +@item @code{detach} clause to @code{task} construct @tab Y @tab +@item @code{omp_fulfill_event} runtime routine @tab Y @tab +@item @code{reduction} and @code{in_reduction} clauses on @code{taskloop} + and @code{taskloop simd} constructs @tab Y @tab +@item @code{taskloop} construct cancelable by @code{cancel} construct + @tab Y @tab +@item @code{mutexinoutset} @emph{dependence-type} for @code{depend} clause + @tab Y @tab +@item Predefined memory spaces, memory allocators, allocator traits + @tab Y @tab Some are only stubs +@item Memory management routines @tab Y @tab +@item @code{allocate} directive @tab N @tab +@item @code{allocate} clause @tab P @tab Initial support +@item @code{use_device_addr} clause on @code{target data} @tab Y @tab +@item @code{ancestor} modifier on @code{device} clause + @tab Y @tab See comment for @code{requires} +@item Implicit declare target directive @tab Y @tab +@item Discontiguous array section with @code{target update} construct + @tab N @tab +@item C/C++'s lvalue expressions in @code{to}, @code{from} + and @code{map} clauses @tab N @tab +@item C/C++'s lvalue expressions in @code{depend} clauses @tab Y @tab +@item Nested @code{declare target} directive @tab Y @tab +@item Combined @code{master} constructs @tab Y @tab +@item @code{depend} clause on @code{taskwait} @tab Y @tab +@item Weak memory ordering clauses on @code{atomic} and @code{flush} construct + @tab Y @tab +@item @code{hint} clause on the @code{atomic} construct @tab Y @tab Stub only +@item @code{depobj} construct and depend objects @tab Y @tab +@item Lock hints were renamed to synchronization hints @tab Y @tab +@item @code{conditional} modifier to @code{lastprivate} clause @tab Y @tab +@item Map-order clarifications @tab P @tab +@item @code{close} @emph{map-type-modifier} @tab Y @tab +@item Mapping C/C++ pointer variables and to assign the address of + device memory mapped by an array section @tab P @tab +@item Mapping of Fortran pointer and allocatable variables, including pointer + and allocatable components of variables + @tab P @tab Mapping of vars with allocatable components unsupported +@item @code{defaultmap} extensions @tab Y @tab +@item @code{declare mapper} directive @tab N @tab +@item @code{omp_get_supported_active_levels} routine @tab Y @tab +@item Runtime routines and environment variables to display runtime thread + affinity information @tab Y @tab +@item @code{omp_pause_resource} and @code{omp_pause_resource_all} runtime + routines @tab Y @tab +@item @code{omp_get_device_num} runtime routine @tab Y @tab +@item OMPT interface @tab N @tab +@item OMPD interface @tab N @tab +@end multitable + +@unnumberedsubsec Other new OpenMP 5.0 features + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item Supporting C++'s range-based for loop @tab Y @tab +@end multitable + + +@node OpenMP 5.1 +@section OpenMP 5.1 + +@unnumberedsubsec New features listed in Appendix B of the OpenMP specification + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item OpenMP directive as C++ attribute specifiers @tab Y @tab +@item @code{omp_all_memory} reserved locator @tab Y @tab +@item @emph{target_device trait} in OpenMP Context @tab N @tab +@item @code{target_device} selector set in context selectors @tab N @tab +@item C/C++'s @code{declare variant} directive: elision support of + preprocessed code @tab N @tab +@item @code{declare variant}: new clauses @code{adjust_args} and + @code{append_args} @tab N @tab +@item @code{dispatch} construct @tab N @tab +@item device-specific ICV settings with environment variables @tab Y @tab +@item @code{assume} directive @tab Y @tab +@item @code{nothing} directive @tab Y @tab +@item @code{error} directive @tab Y @tab +@item @code{masked} construct @tab Y @tab +@item @code{scope} directive @tab Y @tab +@item Loop transformation constructs @tab N @tab +@item @code{strict} modifier in the @code{grainsize} and @code{num_tasks} + clauses of the @code{taskloop} construct @tab Y @tab +@item @code{align} clause/modifier in @code{allocate} directive/clause + and @code{allocator} directive @tab P @tab C/C++ on clause only +@item @code{thread_limit} clause to @code{target} construct @tab Y @tab +@item @code{has_device_addr} clause to @code{target} construct @tab Y @tab +@item Iterators in @code{target update} motion clauses and @code{map} + clauses @tab N @tab +@item Indirect calls to the device version of a procedure or function in + @code{target} regions @tab N @tab +@item @code{interop} directive @tab N @tab +@item @code{omp_interop_t} object support in runtime routines @tab N @tab +@item @code{nowait} clause in @code{taskwait} directive @tab Y @tab +@item Extensions to the @code{atomic} directive @tab Y @tab +@item @code{seq_cst} clause on a @code{flush} construct @tab Y @tab +@item @code{inoutset} argument to the @code{depend} clause @tab Y @tab +@item @code{private} and @code{firstprivate} argument to @code{default} + clause in C and C++ @tab Y @tab +@item @code{present} argument to @code{defaultmap} clause @tab N @tab +@item @code{omp_set_num_teams}, @code{omp_set_teams_thread_limit}, + @code{omp_get_max_teams}, @code{omp_get_teams_thread_limit} runtime + routines @tab Y @tab +@item @code{omp_target_is_accessible} runtime routine @tab Y @tab +@item @code{omp_target_memcpy_async} and @code{omp_target_memcpy_rect_async} + runtime routines @tab Y @tab +@item @code{omp_get_mapped_ptr} runtime routine @tab Y @tab +@item @code{omp_calloc}, @code{omp_realloc}, @code{omp_aligned_alloc} and + @code{omp_aligned_calloc} runtime routines @tab Y @tab +@item @code{omp_alloctrait_key_t} enum: @code{omp_atv_serialized} added, + @code{omp_atv_default} changed @tab Y @tab +@item @code{omp_display_env} runtime routine @tab Y @tab +@item @code{ompt_scope_endpoint_t} enum: @code{ompt_scope_beginend} @tab N @tab +@item @code{ompt_sync_region_t} enum additions @tab N @tab +@item @code{ompt_state_t} enum: @code{ompt_state_wait_barrier_implementation} + and @code{ompt_state_wait_barrier_teams} @tab N @tab +@item @code{ompt_callback_target_data_op_emi_t}, + @code{ompt_callback_target_emi_t}, @code{ompt_callback_target_map_emi_t} + and @code{ompt_callback_target_submit_emi_t} @tab N @tab +@item @code{ompt_callback_error_t} type @tab N @tab +@item @code{OMP_PLACES} syntax extensions @tab Y @tab +@item @code{OMP_NUM_TEAMS} and @code{OMP_TEAMS_THREAD_LIMIT} environment + variables @tab Y @tab +@end multitable + +@unnumberedsubsec Other new OpenMP 5.1 features + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item Support of strictly structured blocks in Fortran @tab Y @tab +@item Support of structured block sequences in C/C++ @tab Y @tab +@item @code{unconstrained} and @code{reproducible} modifiers on @code{order} + clause @tab Y @tab +@item Support @code{begin/end declare target} syntax in C/C++ @tab Y @tab +@item Pointer predetermined firstprivate getting initialized +to address of matching mapped list item per 5.1, Sect. 2.21.7.2 @tab N @tab +@item For Fortran, diagnose placing declarative before/between @code{USE}, + @code{IMPORT}, and @code{IMPLICIT} as invalid @tab N @tab +@end multitable + + +@node OpenMP 5.2 +@section OpenMP 5.2 + +@unnumberedsubsec New features listed in Appendix B of the OpenMP specification + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item @code{omp_in_explicit_task} routine and @emph{explicit-task-var} ICV + @tab Y @tab +@item @code{omp}/@code{ompx}/@code{omx} sentinels and @code{omp_}/@code{ompx_} + namespaces @tab N/A + @tab warning for @code{ompx/omx} sentinels@footnote{The @code{ompx} + sentinel as C/C++ pragma and C++ attributes are warned for with + @code{-Wunknown-pragmas} (implied by @code{-Wall}) and @code{-Wattributes} + (enabled by default), respectively; for Fortran free-source code, there is + a warning enabled by default and, for fixed-source code, the @code{omx} + sentinel is warned for with with @code{-Wsurprising} (enabled by + @code{-Wall}). Unknown clauses are always rejected with an error.} +@item Clauses on @code{end} directive can be on directive @tab N @tab +@item Deprecation of no-argument @code{destroy} clause on @code{depobj} + @tab N @tab +@item @code{linear} clause syntax changes and @code{step} modifier @tab Y @tab +@item Deprecation of minus operator for reductions @tab N @tab +@item Deprecation of separating @code{map} modifiers without comma @tab N @tab +@item @code{declare mapper} with iterator and @code{present} modifiers + @tab N @tab +@item If a matching mapped list item is not found in the data environment, the + pointer retains its original value @tab N @tab +@item New @code{enter} clause as alias for @code{to} on declare target directive + @tab Y @tab +@item Deprecation of @code{to} clause on declare target directive @tab N @tab +@item Extended list of directives permitted in Fortran pure procedures + @tab N @tab +@item New @code{allocators} directive for Fortran @tab N @tab +@item Deprecation of @code{allocate} directive for Fortran + allocatables/pointers @tab N @tab +@item Optional paired @code{end} directive with @code{dispatch} @tab N @tab +@item New @code{memspace} and @code{traits} modifiers for @code{uses_allocators} + @tab N @tab +@item Deprecation of traits array following the allocator_handle expression in + @code{uses_allocators} @tab N @tab +@item New @code{otherwise} clause as alias for @code{default} on metadirectives + @tab N @tab +@item Deprecation of @code{default} clause on metadirectives @tab N @tab +@item Deprecation of delimited form of @code{declare target} @tab N @tab +@item Reproducible semantics changed for @code{order(concurrent)} @tab N @tab +@item @code{allocate} and @code{firstprivate} clauses on @code{scope} + @tab Y @tab +@item @code{ompt_callback_work} @tab N @tab +@item Default map-type for @code{map} clause in @code{target enter/exit data} + @tab Y @tab +@item New @code{doacross} clause as alias for @code{depend} with + @code{source}/@code{sink} modifier @tab Y @tab +@item Deprecation of @code{depend} with @code{source}/@code{sink} modifier + @tab N @tab +@item @code{omp_cur_iteration} keyword @tab Y @tab +@end multitable + +@unnumberedsubsec Other new OpenMP 5.2 features + +@multitable @columnfractions .60 .10 .25 +@headitem Description @tab Status @tab Comments +@item For Fortran, optional comma between directive and clause @tab N @tab +@item Conforming device numbers and @code{omp_initial_device} and + @code{omp_invalid_device} enum/PARAMETER @tab Y @tab +@item Initial value of @emph{default-device-var} ICV with + @code{OMP_TARGET_OFFLOAD=mandatory} @tab N @tab +@item @emph{interop_types} in any position of the modifier list for the @code{init} clause + of the @code{interop} construct @tab N @tab +@end multitable + + +@c --------------------------------------------------------------------- +@c OpenMP Runtime Library Routines +@c --------------------------------------------------------------------- + +@node Runtime Library Routines +@chapter OpenMP Runtime Library Routines + +The runtime routines described here are defined by Section 3 of the OpenMP +specification in version 4.5. The routines are structured in following +three parts: + +@menu +Control threads, processors and the parallel environment. They have C +linkage, and do not throw exceptions. + +* omp_get_active_level:: Number of active parallel regions +* omp_get_ancestor_thread_num:: Ancestor thread ID +* omp_get_cancellation:: Whether cancellation support is enabled +* omp_get_default_device:: Get the default device for target regions +* omp_get_device_num:: Get device that current thread is running on +* omp_get_dynamic:: Dynamic teams setting +* omp_get_initial_device:: Device number of host device +* omp_get_level:: Number of parallel regions +* omp_get_max_active_levels:: Current maximum number of active regions +* omp_get_max_task_priority:: Maximum task priority value that can be set +* omp_get_max_teams:: Maximum number of teams for teams region +* omp_get_max_threads:: Maximum number of threads of parallel region +* omp_get_nested:: Nested parallel regions +* omp_get_num_devices:: Number of target devices +* omp_get_num_procs:: Number of processors online +* omp_get_num_teams:: Number of teams +* omp_get_num_threads:: Size of the active team +* omp_get_proc_bind:: Whether theads may be moved between CPUs +* omp_get_schedule:: Obtain the runtime scheduling method +* omp_get_supported_active_levels:: Maximum number of active regions supported +* omp_get_team_num:: Get team number +* omp_get_team_size:: Number of threads in a team +* omp_get_teams_thread_limit:: Maximum number of threads imposed by teams +* omp_get_thread_limit:: Maximum number of threads +* omp_get_thread_num:: Current thread ID +* omp_in_parallel:: Whether a parallel region is active +* omp_in_final:: Whether in final or included task region +* omp_is_initial_device:: Whether executing on the host device +* omp_set_default_device:: Set the default device for target regions +* omp_set_dynamic:: Enable/disable dynamic teams +* omp_set_max_active_levels:: Limits the number of active parallel regions +* omp_set_nested:: Enable/disable nested parallel regions +* omp_set_num_teams:: Set upper teams limit for teams region +* omp_set_num_threads:: Set upper team size limit +* omp_set_schedule:: Set the runtime scheduling method +* omp_set_teams_thread_limit:: Set upper thread limit for teams construct + +Initialize, set, test, unset and destroy simple and nested locks. + +* omp_init_lock:: Initialize simple lock +* omp_set_lock:: Wait for and set simple lock +* omp_test_lock:: Test and set simple lock if available +* omp_unset_lock:: Unset simple lock +* omp_destroy_lock:: Destroy simple lock +* omp_init_nest_lock:: Initialize nested lock +* omp_set_nest_lock:: Wait for and set simple lock +* omp_test_nest_lock:: Test and set nested lock if available +* omp_unset_nest_lock:: Unset nested lock +* omp_destroy_nest_lock:: Destroy nested lock + +Portable, thread-based, wall clock timer. + +* omp_get_wtick:: Get timer precision. +* omp_get_wtime:: Elapsed wall clock time. + +Support for event objects. + +* omp_fulfill_event:: Fulfill and destroy an OpenMP event. +@end menu + + + +@node omp_get_active_level +@section @code{omp_get_active_level} -- Number of parallel regions +@table @asis +@item @emph{Description}: +This function returns the nesting level for the active parallel blocks, +which enclose the calling call. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_active_level(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_active_level()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_level}, @ref{omp_get_max_active_levels}, @ref{omp_set_max_active_levels} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.20. +@end table + + + +@node omp_get_ancestor_thread_num +@section @code{omp_get_ancestor_thread_num} -- Ancestor thread ID +@table @asis +@item @emph{Description}: +This function returns the thread identification number for the given +nesting level of the current thread. For values of @var{level} outside +zero to @code{omp_get_level} -1 is returned; if @var{level} is +@code{omp_get_level} the result is identical to @code{omp_get_thread_num}. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_ancestor_thread_num(int level);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_ancestor_thread_num(level)} +@item @tab @code{integer level} +@end multitable + +@item @emph{See also}: +@ref{omp_get_level}, @ref{omp_get_thread_num}, @ref{omp_get_team_size} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.18. +@end table + + + +@node omp_get_cancellation +@section @code{omp_get_cancellation} -- Whether cancellation support is enabled +@table @asis +@item @emph{Description}: +This function returns @code{true} if cancellation is activated, @code{false} +otherwise. Here, @code{true} and @code{false} represent their language-specific +counterparts. Unless @env{OMP_CANCELLATION} is set true, cancellations are +deactivated. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_cancellation(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_get_cancellation()} +@end multitable + +@item @emph{See also}: +@ref{OMP_CANCELLATION} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.9. +@end table + + + +@node omp_get_default_device +@section @code{omp_get_default_device} -- Get the default device for target regions +@table @asis +@item @emph{Description}: +Get the default device for target regions without device clause. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_default_device(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_default_device()} +@end multitable + +@item @emph{See also}: +@ref{OMP_DEFAULT_DEVICE}, @ref{omp_set_default_device} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.30. +@end table + + + +@node omp_get_device_num +@section @code{omp_get_device_num} -- Return device number of current device +@table @asis +@item @emph{Description}: +This function returns a device number that represents the device that the +current thread is executing on. For OpenMP 5.0, this must be equal to the +value returned by the @code{omp_get_initial_device} function when called +from the host. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_device_num(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_device_num()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_initial_device} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.0}, Section 3.2.37. +@end table + + + +@node omp_get_dynamic +@section @code{omp_get_dynamic} -- Dynamic teams setting +@table @asis +@item @emph{Description}: +This function returns @code{true} if enabled, @code{false} otherwise. +Here, @code{true} and @code{false} represent their language-specific +counterparts. + +The dynamic team setting may be initialized at startup by the +@env{OMP_DYNAMIC} environment variable or at runtime using +@code{omp_set_dynamic}. If undefined, dynamic adjustment is +disabled by default. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_dynamic(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_get_dynamic()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_dynamic}, @ref{OMP_DYNAMIC} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.8. +@end table + + + +@node omp_get_initial_device +@section @code{omp_get_initial_device} -- Return device number of initial device +@table @asis +@item @emph{Description}: +This function returns a device number that represents the host device. +For OpenMP 5.1, this must be equal to the value returned by the +@code{omp_get_num_devices} function. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_initial_device(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_initial_device()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_num_devices} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.35. +@end table + + + +@node omp_get_level +@section @code{omp_get_level} -- Obtain the current nesting level +@table @asis +@item @emph{Description}: +This function returns the nesting level for the parallel blocks, +which enclose the calling call. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_level(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_level()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_active_level} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.17. +@end table + + + +@node omp_get_max_active_levels +@section @code{omp_get_max_active_levels} -- Current maximum number of active regions +@table @asis +@item @emph{Description}: +This function obtains the maximum allowed number of nested, active parallel regions. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_max_active_levels(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_max_active_levels()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_max_active_levels}, @ref{omp_get_active_level} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.16. +@end table + + +@node omp_get_max_task_priority +@section @code{omp_get_max_task_priority} -- Maximum priority value +that can be set for tasks. +@table @asis +@item @emph{Description}: +This function obtains the maximum allowed priority number for tasks. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_max_task_priority(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_max_task_priority()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.29. +@end table + + +@node omp_get_max_teams +@section @code{omp_get_max_teams} -- Maximum number of teams of teams region +@table @asis +@item @emph{Description}: +Return the maximum number of teams used for the teams region +that does not use the clause @code{num_teams}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_max_teams(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_max_teams()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_num_teams}, @ref{omp_get_num_teams} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 3.4.4. +@end table + + + +@node omp_get_max_threads +@section @code{omp_get_max_threads} -- Maximum number of threads of parallel region +@table @asis +@item @emph{Description}: +Return the maximum number of threads used for the current parallel region +that does not use the clause @code{num_threads}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_max_threads(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_max_threads()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_num_threads}, @ref{omp_set_dynamic}, @ref{omp_get_thread_limit} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.3. +@end table + + + +@node omp_get_nested +@section @code{omp_get_nested} -- Nested parallel regions +@table @asis +@item @emph{Description}: +This function returns @code{true} if nested parallel regions are +enabled, @code{false} otherwise. Here, @code{true} and @code{false} +represent their language-specific counterparts. + +The state of nested parallel regions at startup depends on several +environment variables. If @env{OMP_MAX_ACTIVE_LEVELS} is defined +and is set to greater than one, then nested parallel regions will be +enabled. If not defined, then the value of the @env{OMP_NESTED} +environment variable will be followed if defined. If neither are +defined, then if either @env{OMP_NUM_THREADS} or @env{OMP_PROC_BIND} +are defined with a list of more than one value, then nested parallel +regions are enabled. If none of these are defined, then nested parallel +regions are disabled by default. + +Nested parallel regions can be enabled or disabled at runtime using +@code{omp_set_nested}, or by setting the maximum number of nested +regions with @code{omp_set_max_active_levels} to one to disable, or +above one to enable. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_nested(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_get_nested()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_max_active_levels}, @ref{omp_set_nested}, +@ref{OMP_MAX_ACTIVE_LEVELS}, @ref{OMP_NESTED} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.11. +@end table + + + +@node omp_get_num_devices +@section @code{omp_get_num_devices} -- Number of target devices +@table @asis +@item @emph{Description}: +Returns the number of target devices. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_num_devices(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_num_devices()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.31. +@end table + + + +@node omp_get_num_procs +@section @code{omp_get_num_procs} -- Number of processors online +@table @asis +@item @emph{Description}: +Returns the number of processors online on that device. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_num_procs(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_num_procs()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.5. +@end table + + + +@node omp_get_num_teams +@section @code{omp_get_num_teams} -- Number of teams +@table @asis +@item @emph{Description}: +Returns the number of teams in the current team region. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_num_teams(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_num_teams()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.32. +@end table + + + +@node omp_get_num_threads +@section @code{omp_get_num_threads} -- Size of the active team +@table @asis +@item @emph{Description}: +Returns the number of threads in the current team. In a sequential section of +the program @code{omp_get_num_threads} returns 1. + +The default team size may be initialized at startup by the +@env{OMP_NUM_THREADS} environment variable. At runtime, the size +of the current team may be set either by the @code{NUM_THREADS} +clause or by @code{omp_set_num_threads}. If none of the above were +used to define a specific value and @env{OMP_DYNAMIC} is disabled, +one thread per CPU online is used. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_num_threads(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_num_threads()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_max_threads}, @ref{omp_set_num_threads}, @ref{OMP_NUM_THREADS} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.2. +@end table + + + +@node omp_get_proc_bind +@section @code{omp_get_proc_bind} -- Whether theads may be moved between CPUs +@table @asis +@item @emph{Description}: +This functions returns the currently active thread affinity policy, which is +set via @env{OMP_PROC_BIND}. Possible values are @code{omp_proc_bind_false}, +@code{omp_proc_bind_true}, @code{omp_proc_bind_primary}, +@code{omp_proc_bind_master}, @code{omp_proc_bind_close} and @code{omp_proc_bind_spread}, +where @code{omp_proc_bind_master} is an alias for @code{omp_proc_bind_primary}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{omp_proc_bind_t omp_get_proc_bind(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer(kind=omp_proc_bind_kind) function omp_get_proc_bind()} +@end multitable + +@item @emph{See also}: +@ref{OMP_PROC_BIND}, @ref{OMP_PLACES}, @ref{GOMP_CPU_AFFINITY}, + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.22. +@end table + + + +@node omp_get_schedule +@section @code{omp_get_schedule} -- Obtain the runtime scheduling method +@table @asis +@item @emph{Description}: +Obtain the runtime scheduling method. The @var{kind} argument will be +set to the value @code{omp_sched_static}, @code{omp_sched_dynamic}, +@code{omp_sched_guided} or @code{omp_sched_auto}. The second argument, +@var{chunk_size}, is set to the chunk size. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_get_schedule(omp_sched_t *kind, int *chunk_size);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_get_schedule(kind, chunk_size)} +@item @tab @code{integer(kind=omp_sched_kind) kind} +@item @tab @code{integer chunk_size} +@end multitable + +@item @emph{See also}: +@ref{omp_set_schedule}, @ref{OMP_SCHEDULE} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.13. +@end table + + +@node omp_get_supported_active_levels +@section @code{omp_get_supported_active_levels} -- Maximum number of active regions supported +@table @asis +@item @emph{Description}: +This function returns the maximum number of nested, active parallel regions +supported by this implementation. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_supported_active_levels(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_supported_active_levels()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_max_active_levels}, @ref{omp_set_max_active_levels} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.0}, Section 3.2.15. +@end table + + + +@node omp_get_team_num +@section @code{omp_get_team_num} -- Get team number +@table @asis +@item @emph{Description}: +Returns the team number of the calling thread. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_team_num(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_team_num()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.33. +@end table + + + +@node omp_get_team_size +@section @code{omp_get_team_size} -- Number of threads in a team +@table @asis +@item @emph{Description}: +This function returns the number of threads in a thread team to which +either the current thread or its ancestor belongs. For values of @var{level} +outside zero to @code{omp_get_level}, -1 is returned; if @var{level} is zero, +1 is returned, and for @code{omp_get_level}, the result is identical +to @code{omp_get_num_threads}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_team_size(int level);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_team_size(level)} +@item @tab @code{integer level} +@end multitable + +@item @emph{See also}: +@ref{omp_get_num_threads}, @ref{omp_get_level}, @ref{omp_get_ancestor_thread_num} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.19. +@end table + + + +@node omp_get_teams_thread_limit +@section @code{omp_get_teams_thread_limit} -- Maximum number of threads imposed by teams +@table @asis +@item @emph{Description}: +Return the maximum number of threads that will be able to participate in +each team created by a teams construct. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_teams_thread_limit(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_teams_thread_limit()} +@end multitable + +@item @emph{See also}: +@ref{omp_set_teams_thread_limit}, @ref{OMP_TEAMS_THREAD_LIMIT} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 3.4.6. +@end table + + + +@node omp_get_thread_limit +@section @code{omp_get_thread_limit} -- Maximum number of threads +@table @asis +@item @emph{Description}: +Return the maximum number of threads of the program. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_thread_limit(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_thread_limit()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_max_threads}, @ref{OMP_THREAD_LIMIT} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.14. +@end table + + + +@node omp_get_thread_num +@section @code{omp_get_thread_num} -- Current thread ID +@table @asis +@item @emph{Description}: +Returns a unique thread identification number within the current team. +In a sequential parts of the program, @code{omp_get_thread_num} +always returns 0. In parallel regions the return value varies +from 0 to @code{omp_get_num_threads}-1 inclusive. The return +value of the primary thread of a team is always 0. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_get_thread_num(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function omp_get_thread_num()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_num_threads}, @ref{omp_get_ancestor_thread_num} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.4. +@end table + + + +@node omp_in_parallel +@section @code{omp_in_parallel} -- Whether a parallel region is active +@table @asis +@item @emph{Description}: +This function returns @code{true} if currently running in parallel, +@code{false} otherwise. Here, @code{true} and @code{false} represent +their language-specific counterparts. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_in_parallel(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_in_parallel()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.6. +@end table + + +@node omp_in_final +@section @code{omp_in_final} -- Whether in final or included task region +@table @asis +@item @emph{Description}: +This function returns @code{true} if currently running in a final +or included task region, @code{false} otherwise. Here, @code{true} +and @code{false} represent their language-specific counterparts. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_in_final(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_in_final()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.21. +@end table + + + +@node omp_is_initial_device +@section @code{omp_is_initial_device} -- Whether executing on the host device +@table @asis +@item @emph{Description}: +This function returns @code{true} if currently running on the host device, +@code{false} otherwise. Here, @code{true} and @code{false} represent +their language-specific counterparts. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_is_initial_device(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_is_initial_device()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.34. +@end table + + + +@node omp_set_default_device +@section @code{omp_set_default_device} -- Set the default device for target regions +@table @asis +@item @emph{Description}: +Set the default device for target regions without device clause. The argument +shall be a nonnegative device number. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_default_device(int device_num);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_default_device(device_num)} +@item @tab @code{integer device_num} +@end multitable + +@item @emph{See also}: +@ref{OMP_DEFAULT_DEVICE}, @ref{omp_get_default_device} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.29. +@end table + + + +@node omp_set_dynamic +@section @code{omp_set_dynamic} -- Enable/disable dynamic teams +@table @asis +@item @emph{Description}: +Enable or disable the dynamic adjustment of the number of threads +within a team. The function takes the language-specific equivalent +of @code{true} and @code{false}, where @code{true} enables dynamic +adjustment of team sizes and @code{false} disables it. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_dynamic(int dynamic_threads);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_dynamic(dynamic_threads)} +@item @tab @code{logical, intent(in) :: dynamic_threads} +@end multitable + +@item @emph{See also}: +@ref{OMP_DYNAMIC}, @ref{omp_get_dynamic} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.7. +@end table + + + +@node omp_set_max_active_levels +@section @code{omp_set_max_active_levels} -- Limits the number of active parallel regions +@table @asis +@item @emph{Description}: +This function limits the maximum allowed number of nested, active +parallel regions. @var{max_levels} must be less or equal to +the value returned by @code{omp_get_supported_active_levels}. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_max_active_levels(int max_levels);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_max_active_levels(max_levels)} +@item @tab @code{integer max_levels} +@end multitable + +@item @emph{See also}: +@ref{omp_get_max_active_levels}, @ref{omp_get_active_level}, +@ref{omp_get_supported_active_levels} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.15. +@end table + + + +@node omp_set_nested +@section @code{omp_set_nested} -- Enable/disable nested parallel regions +@table @asis +@item @emph{Description}: +Enable or disable nested parallel regions, i.e., whether team members +are allowed to create new teams. The function takes the language-specific +equivalent of @code{true} and @code{false}, where @code{true} enables +dynamic adjustment of team sizes and @code{false} disables it. + +Enabling nested parallel regions will also set the maximum number of +active nested regions to the maximum supported. Disabling nested parallel +regions will set the maximum number of active nested regions to one. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_nested(int nested);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_nested(nested)} +@item @tab @code{logical, intent(in) :: nested} +@end multitable + +@item @emph{See also}: +@ref{omp_get_nested}, @ref{omp_set_max_active_levels}, +@ref{OMP_MAX_ACTIVE_LEVELS}, @ref{OMP_NESTED} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.10. +@end table + + + +@node omp_set_num_teams +@section @code{omp_set_num_teams} -- Set upper teams limit for teams construct +@table @asis +@item @emph{Description}: +Specifies the upper bound for number of teams created by the teams construct +which does not specify a @code{num_teams} clause. The +argument of @code{omp_set_num_teams} shall be a positive integer. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_num_teams(int num_teams);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_num_teams(num_teams)} +@item @tab @code{integer, intent(in) :: num_teams} +@end multitable + +@item @emph{See also}: +@ref{OMP_NUM_TEAMS}, @ref{omp_get_num_teams}, @ref{omp_get_max_teams} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 3.4.3. +@end table + + + +@node omp_set_num_threads +@section @code{omp_set_num_threads} -- Set upper team size limit +@table @asis +@item @emph{Description}: +Specifies the number of threads used by default in subsequent parallel +sections, if those do not specify a @code{num_threads} clause. The +argument of @code{omp_set_num_threads} shall be a positive integer. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_num_threads(int num_threads);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_num_threads(num_threads)} +@item @tab @code{integer, intent(in) :: num_threads} +@end multitable + +@item @emph{See also}: +@ref{OMP_NUM_THREADS}, @ref{omp_get_num_threads}, @ref{omp_get_max_threads} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.1. +@end table + + + +@node omp_set_schedule +@section @code{omp_set_schedule} -- Set the runtime scheduling method +@table @asis +@item @emph{Description}: +Sets the runtime scheduling method. The @var{kind} argument can have the +value @code{omp_sched_static}, @code{omp_sched_dynamic}, +@code{omp_sched_guided} or @code{omp_sched_auto}. Except for +@code{omp_sched_auto}, the chunk size is set to the value of +@var{chunk_size} if positive, or to the default value if zero or negative. +For @code{omp_sched_auto} the @var{chunk_size} argument is ignored. + +@item @emph{C/C++} +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_schedule(omp_sched_t kind, int chunk_size);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_schedule(kind, chunk_size)} +@item @tab @code{integer(kind=omp_sched_kind) kind} +@item @tab @code{integer chunk_size} +@end multitable + +@item @emph{See also}: +@ref{omp_get_schedule} +@ref{OMP_SCHEDULE} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.2.12. +@end table + + + +@node omp_set_teams_thread_limit +@section @code{omp_set_teams_thread_limit} -- Set upper thread limit for teams construct +@table @asis +@item @emph{Description}: +Specifies the upper bound for number of threads that will be available +for each team created by the teams construct which does not specify a +@code{thread_limit} clause. The argument of +@code{omp_set_teams_thread_limit} shall be a positive integer. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_teams_thread_limit(int thread_limit);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_teams_thread_limit(thread_limit)} +@item @tab @code{integer, intent(in) :: thread_limit} +@end multitable + +@item @emph{See also}: +@ref{OMP_TEAMS_THREAD_LIMIT}, @ref{omp_get_teams_thread_limit}, @ref{omp_get_thread_limit} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 3.4.5. +@end table + + + +@node omp_init_lock +@section @code{omp_init_lock} -- Initialize simple lock +@table @asis +@item @emph{Description}: +Initialize a simple lock. After initialization, the lock is in +an unlocked state. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_init_lock(omp_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_init_lock(svar)} +@item @tab @code{integer(omp_lock_kind), intent(out) :: svar} +@end multitable + +@item @emph{See also}: +@ref{omp_destroy_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.1. +@end table + + + +@node omp_set_lock +@section @code{omp_set_lock} -- Wait for and set simple lock +@table @asis +@item @emph{Description}: +Before setting a simple lock, the lock variable must be initialized by +@code{omp_init_lock}. The calling thread is blocked until the lock +is available. If the lock is already held by the current thread, +a deadlock occurs. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_lock(omp_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_lock(svar)} +@item @tab @code{integer(omp_lock_kind), intent(inout) :: svar} +@end multitable + +@item @emph{See also}: +@ref{omp_init_lock}, @ref{omp_test_lock}, @ref{omp_unset_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.4. +@end table + + + +@node omp_test_lock +@section @code{omp_test_lock} -- Test and set simple lock if available +@table @asis +@item @emph{Description}: +Before setting a simple lock, the lock variable must be initialized by +@code{omp_init_lock}. Contrary to @code{omp_set_lock}, @code{omp_test_lock} +does not block if the lock is not available. This function returns +@code{true} upon success, @code{false} otherwise. Here, @code{true} and +@code{false} represent their language-specific counterparts. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_test_lock(omp_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_test_lock(svar)} +@item @tab @code{integer(omp_lock_kind), intent(inout) :: svar} +@end multitable + +@item @emph{See also}: +@ref{omp_init_lock}, @ref{omp_set_lock}, @ref{omp_set_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.6. +@end table + + + +@node omp_unset_lock +@section @code{omp_unset_lock} -- Unset simple lock +@table @asis +@item @emph{Description}: +A simple lock about to be unset must have been locked by @code{omp_set_lock} +or @code{omp_test_lock} before. In addition, the lock must be held by the +thread calling @code{omp_unset_lock}. Then, the lock becomes unlocked. If one +or more threads attempted to set the lock before, one of them is chosen to, +again, set the lock to itself. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_unset_lock(omp_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_unset_lock(svar)} +@item @tab @code{integer(omp_lock_kind), intent(inout) :: svar} +@end multitable + +@item @emph{See also}: +@ref{omp_set_lock}, @ref{omp_test_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.5. +@end table + + + +@node omp_destroy_lock +@section @code{omp_destroy_lock} -- Destroy simple lock +@table @asis +@item @emph{Description}: +Destroy a simple lock. In order to be destroyed, a simple lock must be +in the unlocked state. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_destroy_lock(omp_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_destroy_lock(svar)} +@item @tab @code{integer(omp_lock_kind), intent(inout) :: svar} +@end multitable + +@item @emph{See also}: +@ref{omp_init_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.3. +@end table + + + +@node omp_init_nest_lock +@section @code{omp_init_nest_lock} -- Initialize nested lock +@table @asis +@item @emph{Description}: +Initialize a nested lock. After initialization, the lock is in +an unlocked state and the nesting count is set to zero. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_init_nest_lock(omp_nest_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_init_nest_lock(nvar)} +@item @tab @code{integer(omp_nest_lock_kind), intent(out) :: nvar} +@end multitable + +@item @emph{See also}: +@ref{omp_destroy_nest_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.1. +@end table + + +@node omp_set_nest_lock +@section @code{omp_set_nest_lock} -- Wait for and set nested lock +@table @asis +@item @emph{Description}: +Before setting a nested lock, the lock variable must be initialized by +@code{omp_init_nest_lock}. The calling thread is blocked until the lock +is available. If the lock is already held by the current thread, the +nesting count for the lock is incremented. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_set_nest_lock(omp_nest_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_set_nest_lock(nvar)} +@item @tab @code{integer(omp_nest_lock_kind), intent(inout) :: nvar} +@end multitable + +@item @emph{See also}: +@ref{omp_init_nest_lock}, @ref{omp_unset_nest_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.4. +@end table + + + +@node omp_test_nest_lock +@section @code{omp_test_nest_lock} -- Test and set nested lock if available +@table @asis +@item @emph{Description}: +Before setting a nested lock, the lock variable must be initialized by +@code{omp_init_nest_lock}. Contrary to @code{omp_set_nest_lock}, +@code{omp_test_nest_lock} does not block if the lock is not available. +If the lock is already held by the current thread, the new nesting count +is returned. Otherwise, the return value equals zero. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int omp_test_nest_lock(omp_nest_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{logical function omp_test_nest_lock(nvar)} +@item @tab @code{integer(omp_nest_lock_kind), intent(inout) :: nvar} +@end multitable + + +@item @emph{See also}: +@ref{omp_init_lock}, @ref{omp_set_lock}, @ref{omp_set_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.6. +@end table + + + +@node omp_unset_nest_lock +@section @code{omp_unset_nest_lock} -- Unset nested lock +@table @asis +@item @emph{Description}: +A nested lock about to be unset must have been locked by @code{omp_set_nested_lock} +or @code{omp_test_nested_lock} before. In addition, the lock must be held by the +thread calling @code{omp_unset_nested_lock}. If the nesting count drops to zero, the +lock becomes unlocked. If one ore more threads attempted to set the lock before, +one of them is chosen to, again, set the lock to itself. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_unset_nest_lock(omp_nest_lock_t *lock);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_unset_nest_lock(nvar)} +@item @tab @code{integer(omp_nest_lock_kind), intent(inout) :: nvar} +@end multitable + +@item @emph{See also}: +@ref{omp_set_nest_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.5. +@end table + + + +@node omp_destroy_nest_lock +@section @code{omp_destroy_nest_lock} -- Destroy nested lock +@table @asis +@item @emph{Description}: +Destroy a nested lock. In order to be destroyed, a nested lock must be +in the unlocked state and its nesting count must equal zero. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_destroy_nest_lock(omp_nest_lock_t *);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_destroy_nest_lock(nvar)} +@item @tab @code{integer(omp_nest_lock_kind), intent(inout) :: nvar} +@end multitable + +@item @emph{See also}: +@ref{omp_init_lock} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.3.3. +@end table + + + +@node omp_get_wtick +@section @code{omp_get_wtick} -- Get timer precision +@table @asis +@item @emph{Description}: +Gets the timer precision, i.e., the number of seconds between two +successive clock ticks. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{double omp_get_wtick(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{double precision function omp_get_wtick()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_wtime} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.4.2. +@end table + + + +@node omp_get_wtime +@section @code{omp_get_wtime} -- Elapsed wall clock time +@table @asis +@item @emph{Description}: +Elapsed wall clock time in seconds. The time is measured per thread, no +guarantee can be made that two distinct threads measure the same time. +Time is measured from some "time in the past", which is an arbitrary time +guaranteed not to change during the execution of the program. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{double omp_get_wtime(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{double precision function omp_get_wtime()} +@end multitable + +@item @emph{See also}: +@ref{omp_get_wtick} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 3.4.1. +@end table + + + +@node omp_fulfill_event +@section @code{omp_fulfill_event} -- Fulfill and destroy an OpenMP event +@table @asis +@item @emph{Description}: +Fulfill the event associated with the event handle argument. Currently, it +is only used to fulfill events generated by detach clauses on task +constructs - the effect of fulfilling the event is to allow the task to +complete. + +The result of calling @code{omp_fulfill_event} with an event handle other +than that generated by a detach clause is undefined. Calling it with an +event handle that has already been fulfilled is also undefined. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void omp_fulfill_event(omp_event_handle_t event);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine omp_fulfill_event(event)} +@item @tab @code{integer (kind=omp_event_handle_kind) :: event} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.0}, Section 3.5.1. +@end table + + + +@c --------------------------------------------------------------------- +@c OpenMP Environment Variables +@c --------------------------------------------------------------------- + +@node Environment Variables +@chapter OpenMP Environment Variables + +The environment variables which beginning with @env{OMP_} are defined by +section 4 of the OpenMP specification in version 4.5, while those +beginning with @env{GOMP_} are GNU extensions. + +@menu +* OMP_CANCELLATION:: Set whether cancellation is activated +* OMP_DISPLAY_ENV:: Show OpenMP version and environment variables +* OMP_DEFAULT_DEVICE:: Set the device used in target regions +* OMP_DYNAMIC:: Dynamic adjustment of threads +* OMP_MAX_ACTIVE_LEVELS:: Set the maximum number of nested parallel regions +* OMP_MAX_TASK_PRIORITY:: Set the maximum task priority value +* OMP_NESTED:: Nested parallel regions +* OMP_NUM_TEAMS:: Specifies the number of teams to use by teams region +* OMP_NUM_THREADS:: Specifies the number of threads to use +* OMP_PROC_BIND:: Whether theads may be moved between CPUs +* OMP_PLACES:: Specifies on which CPUs the theads should be placed +* OMP_STACKSIZE:: Set default thread stack size +* OMP_SCHEDULE:: How threads are scheduled +* OMP_TARGET_OFFLOAD:: Controls offloading behaviour +* OMP_TEAMS_THREAD_LIMIT:: Set the maximum number of threads imposed by teams +* OMP_THREAD_LIMIT:: Set the maximum number of threads +* OMP_WAIT_POLICY:: How waiting threads are handled +* GOMP_CPU_AFFINITY:: Bind threads to specific CPUs +* GOMP_DEBUG:: Enable debugging output +* GOMP_STACKSIZE:: Set default thread stack size +* GOMP_SPINCOUNT:: Set the busy-wait spin count +* GOMP_RTEMS_THREAD_POOLS:: Set the RTEMS specific thread pools +@end menu + + +@node OMP_CANCELLATION +@section @env{OMP_CANCELLATION} -- Set whether cancellation is activated +@cindex Environment Variable +@table @asis +@item @emph{Description}: +If set to @code{TRUE}, the cancellation is activated. If set to @code{FALSE} or +if unset, cancellation is disabled and the @code{cancel} construct is ignored. + +@item @emph{See also}: +@ref{omp_get_cancellation} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.11 +@end table + + + +@node OMP_DISPLAY_ENV +@section @env{OMP_DISPLAY_ENV} -- Show OpenMP version and environment variables +@cindex Environment Variable +@table @asis +@item @emph{Description}: +If set to @code{TRUE}, the OpenMP version number and the values +associated with the OpenMP environment variables are printed to @code{stderr}. +If set to @code{VERBOSE}, it additionally shows the value of the environment +variables which are GNU extensions. If undefined or set to @code{FALSE}, +this information will not be shown. + + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.12 +@end table + + + +@node OMP_DEFAULT_DEVICE +@section @env{OMP_DEFAULT_DEVICE} -- Set the device used in target regions +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Set to choose the device which is used in a @code{target} region, unless the +value is overridden by @code{omp_set_default_device} or by a @code{device} +clause. The value shall be the nonnegative device number. If no device with +the given device number exists, the code is executed on the host. If unset, +device number 0 will be used. + + +@item @emph{See also}: +@ref{omp_get_default_device}, @ref{omp_set_default_device}, + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.13 +@end table + + + +@node OMP_DYNAMIC +@section @env{OMP_DYNAMIC} -- Dynamic adjustment of threads +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Enable or disable the dynamic adjustment of the number of threads +within a team. The value of this environment variable shall be +@code{TRUE} or @code{FALSE}. If undefined, dynamic adjustment is +disabled by default. + +@item @emph{See also}: +@ref{omp_set_dynamic} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.3 +@end table + + + +@node OMP_MAX_ACTIVE_LEVELS +@section @env{OMP_MAX_ACTIVE_LEVELS} -- Set the maximum number of nested parallel regions +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies the initial value for the maximum number of nested parallel +regions. The value of this variable shall be a positive integer. +If undefined, then if @env{OMP_NESTED} is defined and set to true, or +if @env{OMP_NUM_THREADS} or @env{OMP_PROC_BIND} are defined and set to +a list with more than one item, the maximum number of nested parallel +regions will be initialized to the largest number supported, otherwise +it will be set to one. + +@item @emph{See also}: +@ref{omp_set_max_active_levels}, @ref{OMP_NESTED} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.9 +@end table + + + +@node OMP_MAX_TASK_PRIORITY +@section @env{OMP_MAX_TASK_PRIORITY} -- Set the maximum priority +number that can be set for a task. +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies the initial value for the maximum priority value that can be +set for a task. The value of this variable shall be a non-negative +integer, and zero is allowed. If undefined, the default priority is +0. + +@item @emph{See also}: +@ref{omp_get_max_task_priority} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.14 +@end table + + + +@node OMP_NESTED +@section @env{OMP_NESTED} -- Nested parallel regions +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Enable or disable nested parallel regions, i.e., whether team members +are allowed to create new teams. The value of this environment variable +shall be @code{TRUE} or @code{FALSE}. If set to @code{TRUE}, the number +of maximum active nested regions supported will by default be set to the +maximum supported, otherwise it will be set to one. If +@env{OMP_MAX_ACTIVE_LEVELS} is defined, its setting will override this +setting. If both are undefined, nested parallel regions are enabled if +@env{OMP_NUM_THREADS} or @env{OMP_PROC_BINDS} are defined to a list with +more than one item, otherwise they are disabled by default. + +@item @emph{See also}: +@ref{omp_set_max_active_levels}, @ref{omp_set_nested} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.6 +@end table + + + +@node OMP_NUM_TEAMS +@section @env{OMP_NUM_TEAMS} -- Specifies the number of teams to use by teams region +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies the upper bound for number of teams to use in teams regions +without explicit @code{num_teams} clause. The value of this variable shall +be a positive integer. If undefined it defaults to 0 which means +implementation defined upper bound. + +@item @emph{See also}: +@ref{omp_set_num_teams} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 6.23 +@end table + + + +@node OMP_NUM_THREADS +@section @env{OMP_NUM_THREADS} -- Specifies the number of threads to use +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Specifies the default number of threads to use in parallel regions. The +value of this variable shall be a comma-separated list of positive integers; +the value specifies the number of threads to use for the corresponding nested +level. Specifying more than one item in the list will automatically enable +nesting by default. If undefined one thread per CPU is used. + +@item @emph{See also}: +@ref{omp_set_num_threads}, @ref{OMP_NESTED} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.2 +@end table + + + +@node OMP_PROC_BIND +@section @env{OMP_PROC_BIND} -- Whether theads may be moved between CPUs +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies whether threads may be moved between processors. If set to +@code{TRUE}, OpenMP theads should not be moved; if set to @code{FALSE} +they may be moved. Alternatively, a comma separated list with the +values @code{PRIMARY}, @code{MASTER}, @code{CLOSE} and @code{SPREAD} can +be used to specify the thread affinity policy for the corresponding nesting +level. With @code{PRIMARY} and @code{MASTER} the worker threads are in the +same place partition as the primary thread. With @code{CLOSE} those are +kept close to the primary thread in contiguous place partitions. And +with @code{SPREAD} a sparse distribution +across the place partitions is used. Specifying more than one item in the +list will automatically enable nesting by default. + +When undefined, @env{OMP_PROC_BIND} defaults to @code{TRUE} when +@env{OMP_PLACES} or @env{GOMP_CPU_AFFINITY} is set and @code{FALSE} otherwise. + +@item @emph{See also}: +@ref{omp_get_proc_bind}, @ref{GOMP_CPU_AFFINITY}, +@ref{OMP_NESTED}, @ref{OMP_PLACES} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.4 +@end table + + + +@node OMP_PLACES +@section @env{OMP_PLACES} -- Specifies on which CPUs the theads should be placed +@cindex Environment Variable +@table @asis +@item @emph{Description}: +The thread placement can be either specified using an abstract name or by an +explicit list of the places. The abstract names @code{threads}, @code{cores}, +@code{sockets}, @code{ll_caches} and @code{numa_domains} can be optionally +followed by a positive number in parentheses, which denotes the how many places +shall be created. With @code{threads} each place corresponds to a single +hardware thread; @code{cores} to a single core with the corresponding number of +hardware threads; with @code{sockets} the place corresponds to a single +socket; with @code{ll_caches} to a set of cores that shares the last level +cache on the device; and @code{numa_domains} to a set of cores for which their +closest memory on the device is the same memory and at a similar distance from +the cores. The resulting placement can be shown by setting the +@env{OMP_DISPLAY_ENV} environment variable. + +Alternatively, the placement can be specified explicitly as comma-separated +list of places. A place is specified by set of nonnegative numbers in curly +braces, denoting the hardware threads. The curly braces can be omitted +when only a single number has been specified. The hardware threads +belonging to a place can either be specified as comma-separated list of +nonnegative thread numbers or using an interval. Multiple places can also be +either specified by a comma-separated list of places or by an interval. To +specify an interval, a colon followed by the count is placed after +the hardware thread number or the place. Optionally, the length can be +followed by a colon and the stride number -- otherwise a unit stride is +assumed. Placing an exclamation mark (@code{!}) directly before a curly +brace or numbers inside the curly braces (excluding intervals) will +exclude those hardware threads. + +For instance, the following specifies the same places list: +@code{"@{0,1,2@}, @{3,4,6@}, @{7,8,9@}, @{10,11,12@}"}; +@code{"@{0:3@}, @{3:3@}, @{7:3@}, @{10:3@}"}; and @code{"@{0:2@}:4:3"}. + +If @env{OMP_PLACES} and @env{GOMP_CPU_AFFINITY} are unset and +@env{OMP_PROC_BIND} is either unset or @code{false}, threads may be moved +between CPUs following no placement policy. + +@item @emph{See also}: +@ref{OMP_PROC_BIND}, @ref{GOMP_CPU_AFFINITY}, @ref{omp_get_proc_bind}, +@ref{OMP_DISPLAY_ENV} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.5 +@end table + + + +@node OMP_STACKSIZE +@section @env{OMP_STACKSIZE} -- Set default thread stack size +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Set the default thread stack size in kilobytes, unless the number +is suffixed by @code{B}, @code{K}, @code{M} or @code{G}, in which +case the size is, respectively, in bytes, kilobytes, megabytes +or gigabytes. This is different from @code{pthread_attr_setstacksize} +which gets the number of bytes as an argument. If the stack size cannot +be set due to system constraints, an error is reported and the initial +stack size is left unchanged. If undefined, the stack size is system +dependent. + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.7 +@end table + + + +@node OMP_SCHEDULE +@section @env{OMP_SCHEDULE} -- How threads are scheduled +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Allows to specify @code{schedule type} and @code{chunk size}. +The value of the variable shall have the form: @code{type[,chunk]} where +@code{type} is one of @code{static}, @code{dynamic}, @code{guided} or @code{auto} +The optional @code{chunk} size shall be a positive integer. If undefined, +dynamic scheduling and a chunk size of 1 is used. + +@item @emph{See also}: +@ref{omp_set_schedule} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Sections 2.7.1.1 and 4.1 +@end table + + + +@node OMP_TARGET_OFFLOAD +@section @env{OMP_TARGET_OFFLOAD} -- Controls offloading behaviour +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Specifies the behaviour with regard to offloading code to a device. This +variable can be set to one of three values - @code{MANDATORY}, @code{DISABLED} +or @code{DEFAULT}. + +If set to @code{MANDATORY}, the program will terminate with an error if +the offload device is not present or is not supported. If set to +@code{DISABLED}, then offloading is disabled and all code will run on the +host. If set to @code{DEFAULT}, the program will try offloading to the +device first, then fall back to running code on the host if it cannot. + +If undefined, then the program will behave as if @code{DEFAULT} was set. + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.0}, Section 6.17 +@end table + + + +@node OMP_TEAMS_THREAD_LIMIT +@section @env{OMP_TEAMS_THREAD_LIMIT} -- Set the maximum number of threads imposed by teams +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies an upper bound for the number of threads to use by each contention +group created by a teams construct without explicit @code{thread_limit} +clause. The value of this variable shall be a positive integer. If undefined, +the value of 0 is used which stands for an implementation defined upper +limit. + +@item @emph{See also}: +@ref{OMP_THREAD_LIMIT}, @ref{omp_set_teams_thread_limit} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v5.1}, Section 6.24 +@end table + + + +@node OMP_THREAD_LIMIT +@section @env{OMP_THREAD_LIMIT} -- Set the maximum number of threads +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies the number of threads to use for the whole program. The +value of this variable shall be a positive integer. If undefined, +the number of threads is not limited. + +@item @emph{See also}: +@ref{OMP_NUM_THREADS}, @ref{omp_get_thread_limit} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.10 +@end table + + + +@node OMP_WAIT_POLICY +@section @env{OMP_WAIT_POLICY} -- How waiting threads are handled +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Specifies whether waiting threads should be active or passive. If +the value is @code{PASSIVE}, waiting threads should not consume CPU +power while waiting; while the value is @code{ACTIVE} specifies that +they should. If undefined, threads wait actively for a short time +before waiting passively. + +@item @emph{See also}: +@ref{GOMP_SPINCOUNT} + +@item @emph{Reference}: +@uref{https://www.openmp.org, OpenMP specification v4.5}, Section 4.8 +@end table + + + +@node GOMP_CPU_AFFINITY +@section @env{GOMP_CPU_AFFINITY} -- Bind threads to specific CPUs +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Binds threads to specific CPUs. The variable should contain a space-separated +or comma-separated list of CPUs. This list may contain different kinds of +entries: either single CPU numbers in any order, a range of CPUs (M-N) +or a range with some stride (M-N:S). CPU numbers are zero based. For example, +@code{GOMP_CPU_AFFINITY="0 3 1-2 4-15:2"} will bind the initial thread +to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to +CPU 2, the fifth to CPU 4, the sixth through tenth to CPUs 6, 8, 10, 12, +and 14 respectively and then start assigning back from the beginning of +the list. @code{GOMP_CPU_AFFINITY=0} binds all threads to CPU 0. + +There is no libgomp library routine to determine whether a CPU affinity +specification is in effect. As a workaround, language-specific library +functions, e.g., @code{getenv} in C or @code{GET_ENVIRONMENT_VARIABLE} in +Fortran, may be used to query the setting of the @code{GOMP_CPU_AFFINITY} +environment variable. A defined CPU affinity on startup cannot be changed +or disabled during the runtime of the application. + +If both @env{GOMP_CPU_AFFINITY} and @env{OMP_PROC_BIND} are set, +@env{OMP_PROC_BIND} has a higher precedence. If neither has been set and +@env{OMP_PROC_BIND} is unset, or when @env{OMP_PROC_BIND} is set to +@code{FALSE}, the host system will handle the assignment of threads to CPUs. + +@item @emph{See also}: +@ref{OMP_PLACES}, @ref{OMP_PROC_BIND} +@end table + + + +@node GOMP_DEBUG +@section @env{GOMP_DEBUG} -- Enable debugging output +@cindex Environment Variable +@table @asis +@item @emph{Description}: +Enable debugging output. The variable should be set to @code{0} +(disabled, also the default if not set), or @code{1} (enabled). + +If enabled, some debugging output will be printed during execution. +This is currently not specified in more detail, and subject to change. +@end table + + + +@node GOMP_STACKSIZE +@section @env{GOMP_STACKSIZE} -- Set default thread stack size +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Set the default thread stack size in kilobytes. This is different from +@code{pthread_attr_setstacksize} which gets the number of bytes as an +argument. If the stack size cannot be set due to system constraints, an +error is reported and the initial stack size is left unchanged. If undefined, +the stack size is system dependent. + +@item @emph{See also}: +@ref{OMP_STACKSIZE} + +@item @emph{Reference}: +@uref{https://gcc.gnu.org/ml/gcc-patches/2006-06/msg00493.html, +GCC Patches Mailinglist}, +@uref{https://gcc.gnu.org/ml/gcc-patches/2006-06/msg00496.html, +GCC Patches Mailinglist} +@end table + + + +@node GOMP_SPINCOUNT +@section @env{GOMP_SPINCOUNT} -- Set the busy-wait spin count +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +Determines how long a threads waits actively with consuming CPU power +before waiting passively without consuming CPU power. The value may be +either @code{INFINITE}, @code{INFINITY} to always wait actively or an +integer which gives the number of spins of the busy-wait loop. The +integer may optionally be followed by the following suffixes acting +as multiplication factors: @code{k} (kilo, thousand), @code{M} (mega, +million), @code{G} (giga, billion), or @code{T} (tera, trillion). +If undefined, 0 is used when @env{OMP_WAIT_POLICY} is @code{PASSIVE}, +300,000 is used when @env{OMP_WAIT_POLICY} is undefined and +30 billion is used when @env{OMP_WAIT_POLICY} is @code{ACTIVE}. +If there are more OpenMP threads than available CPUs, 1000 and 100 +spins are used for @env{OMP_WAIT_POLICY} being @code{ACTIVE} or +undefined, respectively; unless the @env{GOMP_SPINCOUNT} is lower +or @env{OMP_WAIT_POLICY} is @code{PASSIVE}. + +@item @emph{See also}: +@ref{OMP_WAIT_POLICY} +@end table + + + +@node GOMP_RTEMS_THREAD_POOLS +@section @env{GOMP_RTEMS_THREAD_POOLS} -- Set the RTEMS specific thread pools +@cindex Environment Variable +@cindex Implementation specific setting +@table @asis +@item @emph{Description}: +This environment variable is only used on the RTEMS real-time operating system. +It determines the scheduler instance specific thread pools. The format for +@env{GOMP_RTEMS_THREAD_POOLS} is a list of optional +@code{<thread-pool-count>[$<priority>]@@<scheduler-name>} configurations +separated by @code{:} where: +@itemize @bullet +@item @code{<thread-pool-count>} is the thread pool count for this scheduler +instance. +@item @code{$<priority>} is an optional priority for the worker threads of a +thread pool according to @code{pthread_setschedparam}. In case a priority +value is omitted, then a worker thread will inherit the priority of the OpenMP +primary thread that created it. The priority of the worker thread is not +changed after creation, even if a new OpenMP primary thread using the worker has +a different priority. +@item @code{@@<scheduler-name>} is the scheduler instance name according to the +RTEMS application configuration. +@end itemize +In case no thread pool configuration is specified for a scheduler instance, +then each OpenMP primary thread of this scheduler instance will use its own +dynamically allocated thread pool. To limit the worker thread count of the +thread pools, each OpenMP primary thread must call @code{omp_set_num_threads}. +@item @emph{Example}: +Lets suppose we have three scheduler instances @code{IO}, @code{WRK0}, and +@code{WRK1} with @env{GOMP_RTEMS_THREAD_POOLS} set to +@code{"1@@WRK0:3$4@@WRK1"}. Then there are no thread pool restrictions for +scheduler instance @code{IO}. In the scheduler instance @code{WRK0} there is +one thread pool available. Since no priority is specified for this scheduler +instance, the worker thread inherits the priority of the OpenMP primary thread +that created it. In the scheduler instance @code{WRK1} there are three thread +pools available and their worker threads run at priority four. +@end table + + + +@c --------------------------------------------------------------------- +@c Enabling OpenACC +@c --------------------------------------------------------------------- + +@node Enabling OpenACC +@chapter Enabling OpenACC + +To activate the OpenACC extensions for C/C++ and Fortran, the compile-time +flag @option{-fopenacc} must be specified. This enables the OpenACC directive +@code{#pragma acc} in C/C++ and @code{!$acc} directives in free form, +@code{c$acc}, @code{*$acc} and @code{!$acc} directives in fixed form, +@code{!$} conditional compilation sentinels in free form and @code{c$}, +@code{*$} and @code{!$} sentinels in fixed form, for Fortran. The flag also +arranges for automatic linking of the OpenACC runtime library +(@ref{OpenACC Runtime Library Routines}). + +See @uref{https://gcc.gnu.org/wiki/OpenACC} for more information. + +A complete description of all OpenACC directives accepted may be found in +the @uref{https://www.openacc.org, OpenACC} Application Programming +Interface manual, version 2.6. + + + +@c --------------------------------------------------------------------- +@c OpenACC Runtime Library Routines +@c --------------------------------------------------------------------- + +@node OpenACC Runtime Library Routines +@chapter OpenACC Runtime Library Routines + +The runtime routines described here are defined by section 3 of the OpenACC +specifications in version 2.6. +They have C linkage, and do not throw exceptions. +Generally, they are available only for the host, with the exception of +@code{acc_on_device}, which is available for both the host and the +acceleration device. + +@menu +* acc_get_num_devices:: Get number of devices for the given device + type. +* acc_set_device_type:: Set type of device accelerator to use. +* acc_get_device_type:: Get type of device accelerator to be used. +* acc_set_device_num:: Set device number to use. +* acc_get_device_num:: Get device number to be used. +* acc_get_property:: Get device property. +* acc_async_test:: Tests for completion of a specific asynchronous + operation. +* acc_async_test_all:: Tests for completion of all asynchronous + operations. +* acc_wait:: Wait for completion of a specific asynchronous + operation. +* acc_wait_all:: Waits for completion of all asynchronous + operations. +* acc_wait_all_async:: Wait for completion of all asynchronous + operations. +* acc_wait_async:: Wait for completion of asynchronous operations. +* acc_init:: Initialize runtime for a specific device type. +* acc_shutdown:: Shuts down the runtime for a specific device + type. +* acc_on_device:: Whether executing on a particular device +* acc_malloc:: Allocate device memory. +* acc_free:: Free device memory. +* acc_copyin:: Allocate device memory and copy host memory to + it. +* acc_present_or_copyin:: If the data is not present on the device, + allocate device memory and copy from host + memory. +* acc_create:: Allocate device memory and map it to host + memory. +* acc_present_or_create:: If the data is not present on the device, + allocate device memory and map it to host + memory. +* acc_copyout:: Copy device memory to host memory. +* acc_delete:: Free device memory. +* acc_update_device:: Update device memory from mapped host memory. +* acc_update_self:: Update host memory from mapped device memory. +* acc_map_data:: Map previously allocated device memory to host + memory. +* acc_unmap_data:: Unmap device memory from host memory. +* acc_deviceptr:: Get device pointer associated with specific + host address. +* acc_hostptr:: Get host pointer associated with specific + device address. +* acc_is_present:: Indicate whether host variable / array is + present on device. +* acc_memcpy_to_device:: Copy host memory to device memory. +* acc_memcpy_from_device:: Copy device memory to host memory. +* acc_attach:: Let device pointer point to device-pointer target. +* acc_detach:: Let device pointer point to host-pointer target. + +API routines for target platforms. + +* acc_get_current_cuda_device:: Get CUDA device handle. +* acc_get_current_cuda_context::Get CUDA context handle. +* acc_get_cuda_stream:: Get CUDA stream handle. +* acc_set_cuda_stream:: Set CUDA stream handle. + +API routines for the OpenACC Profiling Interface. + +* acc_prof_register:: Register callbacks. +* acc_prof_unregister:: Unregister callbacks. +* acc_prof_lookup:: Obtain inquiry functions. +* acc_register_library:: Library registration. +@end menu + + + +@node acc_get_num_devices +@section @code{acc_get_num_devices} -- Get number of devices for given device type +@table @asis +@item @emph{Description} +This function returns a value indicating the number of devices available +for the device type specified in @var{devicetype}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_get_num_devices(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{integer function acc_get_num_devices(devicetype)} +@item @tab @code{integer(kind=acc_device_kind) devicetype} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.1. +@end table + + + +@node acc_set_device_type +@section @code{acc_set_device_type} -- Set type of device accelerator to use. +@table @asis +@item @emph{Description} +This function indicates to the runtime library which device type, specified +in @var{devicetype}, to use when executing a parallel or kernels region. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_set_device_type(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_set_device_type(devicetype)} +@item @tab @code{integer(kind=acc_device_kind) devicetype} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.2. +@end table + + + +@node acc_get_device_type +@section @code{acc_get_device_type} -- Get type of device accelerator to be used. +@table @asis +@item @emph{Description} +This function returns what device type will be used when executing a +parallel or kernels region. + +This function returns @code{acc_device_none} if +@code{acc_get_device_type} is called from +@code{acc_ev_device_init_start}, @code{acc_ev_device_init_end} +callbacks of the OpenACC Profiling Interface (@ref{OpenACC Profiling +Interface}), that is, if the device is currently being initialized. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_device_t acc_get_device_type(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_get_device_type(void)} +@item @tab @code{integer(kind=acc_device_kind) acc_get_device_type} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.3. +@end table + + + +@node acc_set_device_num +@section @code{acc_set_device_num} -- Set device number to use. +@table @asis +@item @emph{Description} +This function will indicate to the runtime which device number, +specified by @var{devicenum}, associated with the specified device +type @var{devicetype}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_set_device_num(int devicenum, acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_set_device_num(devicenum, devicetype)} +@item @tab @code{integer devicenum} +@item @tab @code{integer(kind=acc_device_kind) devicetype} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.4. +@end table + + + +@node acc_get_device_num +@section @code{acc_get_device_num} -- Get device number to be used. +@table @asis +@item @emph{Description} +This function returns which device number associated with the specified device +type @var{devicetype}, will be used when executing a parallel or kernels +region. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_get_device_num(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_get_device_num(devicetype)} +@item @tab @code{integer(kind=acc_device_kind) devicetype} +@item @tab @code{integer acc_get_device_num} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.5. +@end table + + + +@node acc_get_property +@section @code{acc_get_property} -- Get device property. +@cindex acc_get_property +@cindex acc_get_property_string +@table @asis +@item @emph{Description} +These routines return the value of the specified @var{property} for the +device being queried according to @var{devicenum} and @var{devicetype}. +Integer-valued and string-valued properties are returned by +@code{acc_get_property} and @code{acc_get_property_string} respectively. +The Fortran @code{acc_get_property_string} subroutine returns the string +retrieved in its fourth argument while the remaining entry points are +functions, which pass the return value as their result. + +Note for Fortran, only: the OpenACC technical committee corrected and, hence, +modified the interface introduced in OpenACC 2.6. The kind-value parameter +@code{acc_device_property} has been renamed to @code{acc_device_property_kind} +for consistency and the return type of the @code{acc_get_property} function is +now a @code{c_size_t} integer instead of a @code{acc_device_property} integer. +The parameter @code{acc_device_property} will continue to be provided, +but might be removed in a future version of GCC. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{size_t acc_get_property(int devicenum, acc_device_t devicetype, acc_device_property_t property);} +@item @emph{Prototype}: @tab @code{const char *acc_get_property_string(int devicenum, acc_device_t devicetype, acc_device_property_t property);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_get_property(devicenum, devicetype, property)} +@item @emph{Interface}: @tab @code{subroutine acc_get_property_string(devicenum, devicetype, property, string)} +@item @tab @code{use ISO_C_Binding, only: c_size_t} +@item @tab @code{integer devicenum} +@item @tab @code{integer(kind=acc_device_kind) devicetype} +@item @tab @code{integer(kind=acc_device_property_kind) property} +@item @tab @code{integer(kind=c_size_t) acc_get_property} +@item @tab @code{character(*) string} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.6. +@end table + + + +@node acc_async_test +@section @code{acc_async_test} -- Test for completion of a specific asynchronous operation. +@table @asis +@item @emph{Description} +This function tests for completion of the asynchronous operation specified +in @var{arg}. In C/C++, a non-zero value will be returned to indicate +the specified asynchronous operation has completed. While Fortran will return +a @code{true}. If the asynchronous operation has not completed, C/C++ returns +a zero and Fortran returns a @code{false}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_async_test(int arg);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_async_test(arg)} +@item @tab @code{integer(kind=acc_handle_kind) arg} +@item @tab @code{logical acc_async_test} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.9. +@end table + + + +@node acc_async_test_all +@section @code{acc_async_test_all} -- Tests for completion of all asynchronous operations. +@table @asis +@item @emph{Description} +This function tests for completion of all asynchronous operations. +In C/C++, a non-zero value will be returned to indicate all asynchronous +operations have completed. While Fortran will return a @code{true}. If +any asynchronous operation has not completed, C/C++ returns a zero and +Fortran returns a @code{false}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_async_test_all(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_async_test()} +@item @tab @code{logical acc_get_device_num} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.10. +@end table + + + +@node acc_wait +@section @code{acc_wait} -- Wait for completion of a specific asynchronous operation. +@table @asis +@item @emph{Description} +This function waits for completion of the asynchronous operation +specified in @var{arg}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_wait(arg);} +@item @emph{Prototype (OpenACC 1.0 compatibility)}: @tab @code{acc_async_wait(arg);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_wait(arg)} +@item @tab @code{integer(acc_handle_kind) arg} +@item @emph{Interface (OpenACC 1.0 compatibility)}: @tab @code{subroutine acc_async_wait(arg)} +@item @tab @code{integer(acc_handle_kind) arg} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.11. +@end table + + + +@node acc_wait_all +@section @code{acc_wait_all} -- Waits for completion of all asynchronous operations. +@table @asis +@item @emph{Description} +This function waits for the completion of all asynchronous operations. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_wait_all(void);} +@item @emph{Prototype (OpenACC 1.0 compatibility)}: @tab @code{acc_async_wait_all(void);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_wait_all()} +@item @emph{Interface (OpenACC 1.0 compatibility)}: @tab @code{subroutine acc_async_wait_all()} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.13. +@end table + + + +@node acc_wait_all_async +@section @code{acc_wait_all_async} -- Wait for completion of all asynchronous operations. +@table @asis +@item @emph{Description} +This function enqueues a wait operation on the queue @var{async} for any +and all asynchronous operations that have been previously enqueued on +any queue. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_wait_all_async(int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_wait_all_async(async)} +@item @tab @code{integer(acc_handle_kind) async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.14. +@end table + + + +@node acc_wait_async +@section @code{acc_wait_async} -- Wait for completion of asynchronous operations. +@table @asis +@item @emph{Description} +This function enqueues a wait operation on queue @var{async} for any and all +asynchronous operations enqueued on queue @var{arg}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_wait_async(int arg, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_wait_async(arg, async)} +@item @tab @code{integer(acc_handle_kind) arg, async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.12. +@end table + + + +@node acc_init +@section @code{acc_init} -- Initialize runtime for a specific device type. +@table @asis +@item @emph{Description} +This function initializes the runtime for the device type specified in +@var{devicetype}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_init(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_init(devicetype)} +@item @tab @code{integer(acc_device_kind) devicetype} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.7. +@end table + + + +@node acc_shutdown +@section @code{acc_shutdown} -- Shuts down the runtime for a specific device type. +@table @asis +@item @emph{Description} +This function shuts down the runtime for the device type specified in +@var{devicetype}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_shutdown(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_shutdown(devicetype)} +@item @tab @code{integer(acc_device_kind) devicetype} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.8. +@end table + + + +@node acc_on_device +@section @code{acc_on_device} -- Whether executing on a particular device +@table @asis +@item @emph{Description}: +This function returns whether the program is executing on a particular +device specified in @var{devicetype}. In C/C++ a non-zero value is +returned to indicate the device is executing on the specified device type. +In Fortran, @code{true} will be returned. If the program is not executing +on the specified device type C/C++ will return a zero, while Fortran will +return @code{false}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_on_device(acc_device_t devicetype);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_on_device(devicetype)} +@item @tab @code{integer(acc_device_kind) devicetype} +@item @tab @code{logical acc_on_device} +@end multitable + + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.17. +@end table + + + +@node acc_malloc +@section @code{acc_malloc} -- Allocate device memory. +@table @asis +@item @emph{Description} +This function allocates @var{len} bytes of device memory. It returns +the device address of the allocated memory. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{d_void* acc_malloc(size_t len);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.18. +@end table + + + +@node acc_free +@section @code{acc_free} -- Free device memory. +@table @asis +@item @emph{Description} +Free previously allocated device memory at the device address @code{a}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_free(d_void *a);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.19. +@end table + + + +@node acc_copyin +@section @code{acc_copyin} -- Allocate device memory and copy host memory to it. +@table @asis +@item @emph{Description} +In C/C++, this function allocates @var{len} bytes of device memory +and maps it to the specified host address in @var{a}. The device +address of the newly allocated device memory is returned. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a +variable or array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_copyin(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{void *acc_copyin_async(h_void *a, size_t len, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_copyin(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_copyin(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_copyin_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_copyin_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.20. +@end table + + + +@node acc_present_or_copyin +@section @code{acc_present_or_copyin} -- If the data is not present on the device, allocate device memory and copy from host memory. +@table @asis +@item @emph{Description} +This function tests if the host data specified by @var{a} and of length +@var{len} is present or not. If it is not present, then device memory +will be allocated and the host memory copied. The device address of +the newly allocated device memory is returned. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +Note that @code{acc_present_or_copyin} and @code{acc_pcopyin} exist for +backward compatibility with OpenACC 2.0; use @ref{acc_copyin} instead. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_present_or_copyin(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{void *acc_pcopyin(h_void *a, size_t len);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_present_or_copyin(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_present_or_copyin(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_pcopyin(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_pcopyin(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.20. +@end table + + + +@node acc_create +@section @code{acc_create} -- Allocate device memory and map it to host memory. +@table @asis +@item @emph{Description} +This function allocates device memory and maps it to host memory specified +by the host address @var{a} with a length of @var{len} bytes. In C/C++, +the function returns the device address of the allocated device memory. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_create(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{void *acc_create_async(h_void *a, size_t len, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_create(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_create(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_create_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_create_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.21. +@end table + + + +@node acc_present_or_create +@section @code{acc_present_or_create} -- If the data is not present on the device, allocate device memory and map it to host memory. +@table @asis +@item @emph{Description} +This function tests if the host data specified by @var{a} and of length +@var{len} is present or not. If it is not present, then device memory +will be allocated and mapped to host memory. In C/C++, the device address +of the newly allocated device memory is returned. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +Note that @code{acc_present_or_create} and @code{acc_pcreate} exist for +backward compatibility with OpenACC 2.0; use @ref{acc_create} instead. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_present_or_create(h_void *a, size_t len)} +@item @emph{Prototype}: @tab @code{void *acc_pcreate(h_void *a, size_t len)} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_present_or_create(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_present_or_create(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_pcreate(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_pcreate(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.21. +@end table + + + +@node acc_copyout +@section @code{acc_copyout} -- Copy device memory to host memory. +@table @asis +@item @emph{Description} +This function copies mapped device memory to host memory which is specified +by host address @var{a} for a length @var{len} bytes in C/C++. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_copyout(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_copyout_async(h_void *a, size_t len, int async);} +@item @emph{Prototype}: @tab @code{acc_copyout_finalize(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_copyout_finalize_async(h_void *a, size_t len, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_copyout(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_copyout(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_finalize(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_finalize(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_finalize_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_copyout_finalize_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.22. +@end table + + + +@node acc_delete +@section @code{acc_delete} -- Free device memory. +@table @asis +@item @emph{Description} +This function frees previously allocated device memory specified by +the device address @var{a} and the length of @var{len} bytes. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_delete(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_delete_async(h_void *a, size_t len, int async);} +@item @emph{Prototype}: @tab @code{acc_delete_finalize(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_delete_finalize_async(h_void *a, size_t len, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_delete(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_delete(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_delete_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_delete_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_delete_finalize(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_delete_finalize(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_delete_async_finalize(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_delete_async_finalize(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.23. +@end table + + + +@node acc_update_device +@section @code{acc_update_device} -- Update device memory from mapped host memory. +@table @asis +@item @emph{Description} +This function updates the device copy from the previously mapped host memory. +The host memory is specified with the host address @var{a} and a length of +@var{len} bytes. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_update_device(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_update_device(h_void *a, size_t len, async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_update_device(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_update_device(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_update_device_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_update_device_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.24. +@end table + + + +@node acc_update_self +@section @code{acc_update_self} -- Update host memory from mapped device memory. +@table @asis +@item @emph{Description} +This function updates the host copy from the previously mapped device memory. +The host memory is specified with the host address @var{a} and a length of +@var{len} bytes. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_update_self(h_void *a, size_t len);} +@item @emph{Prototype}: @tab @code{acc_update_self_async(h_void *a, size_t len, int async);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{subroutine acc_update_self(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @emph{Interface}: @tab @code{subroutine acc_update_self(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @emph{Interface}: @tab @code{subroutine acc_update_self_async(a, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer(acc_handle_kind) :: async} +@item @emph{Interface}: @tab @code{subroutine acc_update_self_async(a, len, async)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{integer(acc_handle_kind) :: async} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.25. +@end table + + + +@node acc_map_data +@section @code{acc_map_data} -- Map previously allocated device memory to host memory. +@table @asis +@item @emph{Description} +This function maps previously allocated device and host memory. The device +memory is specified with the device address @var{d}. The host memory is +specified with the host address @var{h} and a length of @var{len}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_map_data(h_void *h, d_void *d, size_t len);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.26. +@end table + + + +@node acc_unmap_data +@section @code{acc_unmap_data} -- Unmap device memory from host memory. +@table @asis +@item @emph{Description} +This function unmaps previously mapped device and host memory. The latter +specified by @var{h}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_unmap_data(h_void *h);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.27. +@end table + + + +@node acc_deviceptr +@section @code{acc_deviceptr} -- Get device pointer associated with specific host address. +@table @asis +@item @emph{Description} +This function returns the device address that has been mapped to the +host address specified by @var{h}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_deviceptr(h_void *h);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.28. +@end table + + + +@node acc_hostptr +@section @code{acc_hostptr} -- Get host pointer associated with specific device address. +@table @asis +@item @emph{Description} +This function returns the host address that has been mapped to the +device address specified by @var{d}. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_hostptr(d_void *d);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.29. +@end table + + + +@node acc_is_present +@section @code{acc_is_present} -- Indicate whether host variable / array is present on device. +@table @asis +@item @emph{Description} +This function indicates whether the specified host address in @var{a} and a +length of @var{len} bytes is present on the device. In C/C++, a non-zero +value is returned to indicate the presence of the mapped memory on the +device. A zero is returned to indicate the memory is not mapped on the +device. + +In Fortran, two (2) forms are supported. In the first form, @var{a} specifies +a contiguous array section. The second form @var{a} specifies a variable or +array element and @var{len} specifies the length in bytes. If the host +memory is mapped to device memory, then a @code{true} is returned. Otherwise, +a @code{false} is return to indicate the mapped memory is not present. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_is_present(h_void *a, size_t len);} +@end multitable + +@item @emph{Fortran}: +@multitable @columnfractions .20 .80 +@item @emph{Interface}: @tab @code{function acc_is_present(a)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{logical acc_is_present} +@item @emph{Interface}: @tab @code{function acc_is_present(a, len)} +@item @tab @code{type, dimension(:[,:]...) :: a} +@item @tab @code{integer len} +@item @tab @code{logical acc_is_present} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.30. +@end table + + + +@node acc_memcpy_to_device +@section @code{acc_memcpy_to_device} -- Copy host memory to device memory. +@table @asis +@item @emph{Description} +This function copies host memory specified by host address of @var{src} to +device memory specified by the device address @var{dest} for a length of +@var{bytes} bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_memcpy_to_device(d_void *dest, h_void *src, size_t bytes);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.31. +@end table + + + +@node acc_memcpy_from_device +@section @code{acc_memcpy_from_device} -- Copy device memory to host memory. +@table @asis +@item @emph{Description} +This function copies host memory specified by host address of @var{src} from +device memory specified by the device address @var{dest} for a length of +@var{bytes} bytes. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_memcpy_from_device(d_void *dest, h_void *src, size_t bytes);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.32. +@end table + + + +@node acc_attach +@section @code{acc_attach} -- Let device pointer point to device-pointer target. +@table @asis +@item @emph{Description} +This function updates a pointer on the device from pointing to a host-pointer +address to pointing to the corresponding device data. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_attach(h_void **ptr);} +@item @emph{Prototype}: @tab @code{acc_attach_async(h_void **ptr, int async);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.34. +@end table + + + +@node acc_detach +@section @code{acc_detach} -- Let device pointer point to host-pointer target. +@table @asis +@item @emph{Description} +This function updates a pointer on the device from pointing to a device-pointer +address to pointing to the corresponding host data. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_detach(h_void **ptr);} +@item @emph{Prototype}: @tab @code{acc_detach_async(h_void **ptr, int async);} +@item @emph{Prototype}: @tab @code{acc_detach_finalize(h_void **ptr);} +@item @emph{Prototype}: @tab @code{acc_detach_finalize_async(h_void **ptr, int async);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +3.2.35. +@end table + + + +@node acc_get_current_cuda_device +@section @code{acc_get_current_cuda_device} -- Get CUDA device handle. +@table @asis +@item @emph{Description} +This function returns the CUDA device handle. This handle is the same +as used by the CUDA Runtime or Driver API's. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_get_current_cuda_device(void);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +A.2.1.1. +@end table + + + +@node acc_get_current_cuda_context +@section @code{acc_get_current_cuda_context} -- Get CUDA context handle. +@table @asis +@item @emph{Description} +This function returns the CUDA context handle. This handle is the same +as used by the CUDA Runtime or Driver API's. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_get_current_cuda_context(void);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +A.2.1.2. +@end table + + + +@node acc_get_cuda_stream +@section @code{acc_get_cuda_stream} -- Get CUDA stream handle. +@table @asis +@item @emph{Description} +This function returns the CUDA stream handle for the queue @var{async}. +This handle is the same as used by the CUDA Runtime or Driver API's. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void *acc_get_cuda_stream(int async);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +A.2.1.3. +@end table + + + +@node acc_set_cuda_stream +@section @code{acc_set_cuda_stream} -- Set CUDA stream handle. +@table @asis +@item @emph{Description} +This function associates the stream handle specified by @var{stream} with +the queue @var{async}. + +This cannot be used to change the stream handle associated with +@code{acc_async_sync}. + +The return value is not specified. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{int acc_set_cuda_stream(int async, void *stream);} +@end multitable + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +A.2.1.4. +@end table + + + +@node acc_prof_register +@section @code{acc_prof_register} -- Register callbacks. +@table @asis +@item @emph{Description}: +This function registers callbacks. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void acc_prof_register (acc_event_t, acc_prof_callback, acc_register_t);} +@end multitable + +@item @emph{See also}: +@ref{OpenACC Profiling Interface} + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +5.3. +@end table + + + +@node acc_prof_unregister +@section @code{acc_prof_unregister} -- Unregister callbacks. +@table @asis +@item @emph{Description}: +This function unregisters callbacks. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void acc_prof_unregister (acc_event_t, acc_prof_callback, acc_register_t);} +@end multitable + +@item @emph{See also}: +@ref{OpenACC Profiling Interface} + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +5.3. +@end table + + + +@node acc_prof_lookup +@section @code{acc_prof_lookup} -- Obtain inquiry functions. +@table @asis +@item @emph{Description}: +Function to obtain inquiry functions. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{acc_query_fn acc_prof_lookup (const char *);} +@end multitable + +@item @emph{See also}: +@ref{OpenACC Profiling Interface} + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +5.3. +@end table + + + +@node acc_register_library +@section @code{acc_register_library} -- Library registration. +@table @asis +@item @emph{Description}: +Function for library registration. + +@item @emph{C/C++}: +@multitable @columnfractions .20 .80 +@item @emph{Prototype}: @tab @code{void acc_register_library (acc_prof_reg, acc_prof_reg, acc_prof_lookup_func);} +@end multitable + +@item @emph{See also}: +@ref{OpenACC Profiling Interface}, @ref{ACC_PROFLIB} + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +5.3. +@end table + + + +@c --------------------------------------------------------------------- +@c OpenACC Environment Variables +@c --------------------------------------------------------------------- + +@node OpenACC Environment Variables +@chapter OpenACC Environment Variables + +The variables @env{ACC_DEVICE_TYPE} and @env{ACC_DEVICE_NUM} +are defined by section 4 of the OpenACC specification in version 2.0. +The variable @env{ACC_PROFLIB} +is defined by section 4 of the OpenACC specification in version 2.6. +The variable @env{GCC_ACC_NOTIFY} is used for diagnostic purposes. + +@menu +* ACC_DEVICE_TYPE:: +* ACC_DEVICE_NUM:: +* ACC_PROFLIB:: +* GCC_ACC_NOTIFY:: +@end menu + + + +@node ACC_DEVICE_TYPE +@section @code{ACC_DEVICE_TYPE} +@table @asis +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +4.1. +@end table + + + +@node ACC_DEVICE_NUM +@section @code{ACC_DEVICE_NUM} +@table @asis +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +4.2. +@end table + + + +@node ACC_PROFLIB +@section @code{ACC_PROFLIB} +@table @asis +@item @emph{See also}: +@ref{acc_register_library}, @ref{OpenACC Profiling Interface} + +@item @emph{Reference}: +@uref{https://www.openacc.org, OpenACC specification v2.6}, section +4.3. +@end table + + + +@node GCC_ACC_NOTIFY +@section @code{GCC_ACC_NOTIFY} +@table @asis +@item @emph{Description}: +Print debug information pertaining to the accelerator. +@end table + + + +@c --------------------------------------------------------------------- +@c CUDA Streams Usage +@c --------------------------------------------------------------------- + +@node CUDA Streams Usage +@chapter CUDA Streams Usage + +This applies to the @code{nvptx} plugin only. + +The library provides elements that perform asynchronous movement of +data and asynchronous operation of computing constructs. This +asynchronous functionality is implemented by making use of CUDA +streams@footnote{See "Stream Management" in "CUDA Driver API", +TRM-06703-001, Version 5.5, for additional information}. + +The primary means by that the asynchronous functionality is accessed +is through the use of those OpenACC directives which make use of the +@code{async} and @code{wait} clauses. When the @code{async} clause is +first used with a directive, it creates a CUDA stream. If an +@code{async-argument} is used with the @code{async} clause, then the +stream is associated with the specified @code{async-argument}. + +Following the creation of an association between a CUDA stream and the +@code{async-argument} of an @code{async} clause, both the @code{wait} +clause and the @code{wait} directive can be used. When either the +clause or directive is used after stream creation, it creates a +rendezvous point whereby execution waits until all operations +associated with the @code{async-argument}, that is, stream, have +completed. + +Normally, the management of the streams that are created as a result of +using the @code{async} clause, is done without any intervention by the +caller. This implies the association between the @code{async-argument} +and the CUDA stream will be maintained for the lifetime of the program. +However, this association can be changed through the use of the library +function @code{acc_set_cuda_stream}. When the function +@code{acc_set_cuda_stream} is called, the CUDA stream that was +originally associated with the @code{async} clause will be destroyed. +Caution should be taken when changing the association as subsequent +references to the @code{async-argument} refer to a different +CUDA stream. + + + +@c --------------------------------------------------------------------- +@c OpenACC Library Interoperability +@c --------------------------------------------------------------------- + +@node OpenACC Library Interoperability +@chapter OpenACC Library Interoperability + +@section Introduction + +The OpenACC library uses the CUDA Driver API, and may interact with +programs that use the Runtime library directly, or another library +based on the Runtime library, e.g., CUBLAS@footnote{See section 2.26, +"Interactions with the CUDA Driver API" in +"CUDA Runtime API", Version 5.5, and section 2.27, "VDPAU +Interoperability", in "CUDA Driver API", TRM-06703-001, Version 5.5, +for additional information on library interoperability.}. +This chapter describes the use cases and what changes are +required in order to use both the OpenACC library and the CUBLAS and Runtime +libraries within a program. + +@section First invocation: NVIDIA CUBLAS library API + +In this first use case (see below), a function in the CUBLAS library is called +prior to any of the functions in the OpenACC library. More specifically, the +function @code{cublasCreate()}. + +When invoked, the function initializes the library and allocates the +hardware resources on the host and the device on behalf of the caller. Once +the initialization and allocation has completed, a handle is returned to the +caller. The OpenACC library also requires initialization and allocation of +hardware resources. Since the CUBLAS library has already allocated the +hardware resources for the device, all that is left to do is to initialize +the OpenACC library and acquire the hardware resources on the host. + +Prior to calling the OpenACC function that initializes the library and +allocate the host hardware resources, you need to acquire the device number +that was allocated during the call to @code{cublasCreate()}. The invoking of the +runtime library function @code{cudaGetDevice()} accomplishes this. Once +acquired, the device number is passed along with the device type as +parameters to the OpenACC library function @code{acc_set_device_num()}. + +Once the call to @code{acc_set_device_num()} has completed, the OpenACC +library uses the context that was created during the call to +@code{cublasCreate()}. In other words, both libraries will be sharing the +same context. + +@smallexample + /* Create the handle */ + s = cublasCreate(&h); + if (s != CUBLAS_STATUS_SUCCESS) + @{ + fprintf(stderr, "cublasCreate failed %d\n", s); + exit(EXIT_FAILURE); + @} + + /* Get the device number */ + e = cudaGetDevice(&dev); + if (e != cudaSuccess) + @{ + fprintf(stderr, "cudaGetDevice failed %d\n", e); + exit(EXIT_FAILURE); + @} + + /* Initialize OpenACC library and use device 'dev' */ + acc_set_device_num(dev, acc_device_nvidia); + +@end smallexample +@center Use Case 1 + +@section First invocation: OpenACC library API + +In this second use case (see below), a function in the OpenACC library is +called prior to any of the functions in the CUBLAS library. More specificially, +the function @code{acc_set_device_num()}. + +In the use case presented here, the function @code{acc_set_device_num()} +is used to both initialize the OpenACC library and allocate the hardware +resources on the host and the device. In the call to the function, the +call parameters specify which device to use and what device +type to use, i.e., @code{acc_device_nvidia}. It should be noted that this +is but one method to initialize the OpenACC library and allocate the +appropriate hardware resources. Other methods are available through the +use of environment variables and these will be discussed in the next section. + +Once the call to @code{acc_set_device_num()} has completed, other OpenACC +functions can be called as seen with multiple calls being made to +@code{acc_copyin()}. In addition, calls can be made to functions in the +CUBLAS library. In the use case a call to @code{cublasCreate()} is made +subsequent to the calls to @code{acc_copyin()}. +As seen in the previous use case, a call to @code{cublasCreate()} +initializes the CUBLAS library and allocates the hardware resources on the +host and the device. However, since the device has already been allocated, +@code{cublasCreate()} will only initialize the CUBLAS library and allocate +the appropriate hardware resources on the host. The context that was created +as part of the OpenACC initialization is shared with the CUBLAS library, +similarly to the first use case. + +@smallexample + dev = 0; + + acc_set_device_num(dev, acc_device_nvidia); + + /* Copy the first set to the device */ + d_X = acc_copyin(&h_X[0], N * sizeof (float)); + if (d_X == NULL) + @{ + fprintf(stderr, "copyin error h_X\n"); + exit(EXIT_FAILURE); + @} + + /* Copy the second set to the device */ + d_Y = acc_copyin(&h_Y1[0], N * sizeof (float)); + if (d_Y == NULL) + @{ + fprintf(stderr, "copyin error h_Y1\n"); + exit(EXIT_FAILURE); + @} + + /* Create the handle */ + s = cublasCreate(&h); + if (s != CUBLAS_STATUS_SUCCESS) + @{ + fprintf(stderr, "cublasCreate failed %d\n", s); + exit(EXIT_FAILURE); + @} + + /* Perform saxpy using CUBLAS library function */ + s = cublasSaxpy(h, N, &alpha, d_X, 1, d_Y, 1); + if (s != CUBLAS_STATUS_SUCCESS) + @{ + fprintf(stderr, "cublasSaxpy failed %d\n", s); + exit(EXIT_FAILURE); + @} + + /* Copy the results from the device */ + acc_memcpy_from_device(&h_Y1[0], d_Y, N * sizeof (float)); + +@end smallexample +@center Use Case 2 + +@section OpenACC library and environment variables + +There are two environment variables associated with the OpenACC library +that may be used to control the device type and device number: +@env{ACC_DEVICE_TYPE} and @env{ACC_DEVICE_NUM}, respectively. These two +environment variables can be used as an alternative to calling +@code{acc_set_device_num()}. As seen in the second use case, the device +type and device number were specified using @code{acc_set_device_num()}. +If however, the aforementioned environment variables were set, then the +call to @code{acc_set_device_num()} would not be required. + + +The use of the environment variables is only relevant when an OpenACC function +is called prior to a call to @code{cudaCreate()}. If @code{cudaCreate()} +is called prior to a call to an OpenACC function, then you must call +@code{acc_set_device_num()}@footnote{More complete information +about @env{ACC_DEVICE_TYPE} and @env{ACC_DEVICE_NUM} can be found in +sections 4.1 and 4.2 of the @uref{https://www.openacc.org, OpenACC} +Application Programming Interfaceā€¯, Version 2.6.} + + + +@c --------------------------------------------------------------------- +@c OpenACC Profiling Interface +@c --------------------------------------------------------------------- + +@node OpenACC Profiling Interface +@chapter OpenACC Profiling Interface + +@section Implementation Status and Implementation-Defined Behavior + +We're implementing the OpenACC Profiling Interface as defined by the +OpenACC 2.6 specification. We're clarifying some aspects here as +@emph{implementation-defined behavior}, while they're still under +discussion within the OpenACC Technical Committee. + +This implementation is tuned to keep the performance impact as low as +possible for the (very common) case that the Profiling Interface is +not enabled. This is relevant, as the Profiling Interface affects all +the @emph{hot} code paths (in the target code, not in the offloaded +code). Users of the OpenACC Profiling Interface can be expected to +understand that performance will be impacted to some degree once the +Profiling Interface has gotten enabled: for example, because of the +@emph{runtime} (libgomp) calling into a third-party @emph{library} for +every event that has been registered. + +We're not yet accounting for the fact that @cite{OpenACC events may +occur during event processing}. +We just handle one case specially, as required by CUDA 9.0 +@command{nvprof}, that @code{acc_get_device_type} +(@ref{acc_get_device_type})) may be called from +@code{acc_ev_device_init_start}, @code{acc_ev_device_init_end} +callbacks. + +We're not yet implementing initialization via a +@code{acc_register_library} function that is either statically linked +in, or dynamically via @env{LD_PRELOAD}. +Initialization via @code{acc_register_library} functions dynamically +loaded via the @env{ACC_PROFLIB} environment variable does work, as +does directly calling @code{acc_prof_register}, +@code{acc_prof_unregister}, @code{acc_prof_lookup}. + +As currently there are no inquiry functions defined, calls to +@code{acc_prof_lookup} will always return @code{NULL}. + +There aren't separate @emph{start}, @emph{stop} events defined for the +event types @code{acc_ev_create}, @code{acc_ev_delete}, +@code{acc_ev_alloc}, @code{acc_ev_free}. It's not clear if these +should be triggered before or after the actual device-specific call is +made. We trigger them after. + +Remarks about data provided to callbacks: + +@table @asis + +@item @code{acc_prof_info.event_type} +It's not clear if for @emph{nested} event callbacks (for example, +@code{acc_ev_enqueue_launch_start} as part of a parent compute +construct), this should be set for the nested event +(@code{acc_ev_enqueue_launch_start}), or if the value of the parent +construct should remain (@code{acc_ev_compute_construct_start}). In +this implementation, the value will generally correspond to the +innermost nested event type. + +@item @code{acc_prof_info.device_type} +@itemize + +@item +For @code{acc_ev_compute_construct_start}, and in presence of an +@code{if} clause with @emph{false} argument, this will still refer to +the offloading device type. +It's not clear if that's the expected behavior. + +@item +Complementary to the item before, for +@code{acc_ev_compute_construct_end}, this is set to +@code{acc_device_host} in presence of an @code{if} clause with +@emph{false} argument. +It's not clear if that's the expected behavior. + +@end itemize + +@item @code{acc_prof_info.thread_id} +Always @code{-1}; not yet implemented. + +@item @code{acc_prof_info.async} +@itemize + +@item +Not yet implemented correctly for +@code{acc_ev_compute_construct_start}. + +@item +In a compute construct, for host-fallback +execution/@code{acc_device_host} it will always be +@code{acc_async_sync}. +It's not clear if that's the expected behavior. + +@item +For @code{acc_ev_device_init_start} and @code{acc_ev_device_init_end}, +it will always be @code{acc_async_sync}. +It's not clear if that's the expected behavior. + +@end itemize + +@item @code{acc_prof_info.async_queue} +There is no @cite{limited number of asynchronous queues} in libgomp. +This will always have the same value as @code{acc_prof_info.async}. + +@item @code{acc_prof_info.src_file} +Always @code{NULL}; not yet implemented. + +@item @code{acc_prof_info.func_name} +Always @code{NULL}; not yet implemented. + +@item @code{acc_prof_info.line_no} +Always @code{-1}; not yet implemented. + +@item @code{acc_prof_info.end_line_no} +Always @code{-1}; not yet implemented. + +@item @code{acc_prof_info.func_line_no} +Always @code{-1}; not yet implemented. + +@item @code{acc_prof_info.func_end_line_no} +Always @code{-1}; not yet implemented. + +@item @code{acc_event_info.event_type}, @code{acc_event_info.*.event_type} +Relating to @code{acc_prof_info.event_type} discussed above, in this +implementation, this will always be the same value as +@code{acc_prof_info.event_type}. + +@item @code{acc_event_info.*.parent_construct} +@itemize + +@item +Will be @code{acc_construct_parallel} for all OpenACC compute +constructs as well as many OpenACC Runtime API calls; should be the +one matching the actual construct, or +@code{acc_construct_runtime_api}, respectively. + +@item +Will be @code{acc_construct_enter_data} or +@code{acc_construct_exit_data} when processing variable mappings +specified in OpenACC @emph{declare} directives; should be +@code{acc_construct_declare}. + +@item +For implicit @code{acc_ev_device_init_start}, +@code{acc_ev_device_init_end}, and explicit as well as implicit +@code{acc_ev_alloc}, @code{acc_ev_free}, +@code{acc_ev_enqueue_upload_start}, @code{acc_ev_enqueue_upload_end}, +@code{acc_ev_enqueue_download_start}, and +@code{acc_ev_enqueue_download_end}, will be +@code{acc_construct_parallel}; should reflect the real parent +construct. + +@end itemize + +@item @code{acc_event_info.*.implicit} +For @code{acc_ev_alloc}, @code{acc_ev_free}, +@code{acc_ev_enqueue_upload_start}, @code{acc_ev_enqueue_upload_end}, +@code{acc_ev_enqueue_download_start}, and +@code{acc_ev_enqueue_download_end}, this currently will be @code{1} +also for explicit usage. + +@item @code{acc_event_info.data_event.var_name} +Always @code{NULL}; not yet implemented. + +@item @code{acc_event_info.data_event.host_ptr} +For @code{acc_ev_alloc}, and @code{acc_ev_free}, this is always +@code{NULL}. + +@item @code{typedef union acc_api_info} +@dots{} as printed in @cite{5.2.3. Third Argument: API-Specific +Information}. This should obviously be @code{typedef @emph{struct} +acc_api_info}. + +@item @code{acc_api_info.device_api} +Possibly not yet implemented correctly for +@code{acc_ev_compute_construct_start}, +@code{acc_ev_device_init_start}, @code{acc_ev_device_init_end}: +will always be @code{acc_device_api_none} for these event types. +For @code{acc_ev_enter_data_start}, it will be +@code{acc_device_api_none} in some cases. + +@item @code{acc_api_info.device_type} +Always the same as @code{acc_prof_info.device_type}. + +@item @code{acc_api_info.vendor} +Always @code{-1}; not yet implemented. + +@item @code{acc_api_info.device_handle} +Always @code{NULL}; not yet implemented. + +@item @code{acc_api_info.context_handle} +Always @code{NULL}; not yet implemented. + +@item @code{acc_api_info.async_handle} +Always @code{NULL}; not yet implemented. + +@end table + +Remarks about certain event types: + +@table @asis + +@item @code{acc_ev_device_init_start}, @code{acc_ev_device_init_end} +@itemize + +@item +@c See 'DEVICE_INIT_INSIDE_COMPUTE_CONSTRUCT' in +@c 'libgomp.oacc-c-c++-common/acc_prof-kernels-1.c', +@c 'libgomp.oacc-c-c++-common/acc_prof-parallel-1.c'. +When a compute construct triggers implicit +@code{acc_ev_device_init_start} and @code{acc_ev_device_init_end} +events, they currently aren't @emph{nested within} the corresponding +@code{acc_ev_compute_construct_start} and +@code{acc_ev_compute_construct_end}, but they're currently observed +@emph{before} @code{acc_ev_compute_construct_start}. +It's not clear what to do: the standard asks us provide a lot of +details to the @code{acc_ev_compute_construct_start} callback, without +(implicitly) initializing a device before? + +@item +Callbacks for these event types will not be invoked for calls to the +@code{acc_set_device_type} and @code{acc_set_device_num} functions. +It's not clear if they should be. + +@end itemize + +@item @code{acc_ev_enter_data_start}, @code{acc_ev_enter_data_end}, @code{acc_ev_exit_data_start}, @code{acc_ev_exit_data_end} +@itemize + +@item +Callbacks for these event types will also be invoked for OpenACC +@emph{host_data} constructs. +It's not clear if they should be. + +@item +Callbacks for these event types will also be invoked when processing +variable mappings specified in OpenACC @emph{declare} directives. +It's not clear if they should be. + +@end itemize + +@end table + +Callbacks for the following event types will be invoked, but dispatch +and information provided therein has not yet been thoroughly reviewed: + +@itemize +@item @code{acc_ev_alloc} +@item @code{acc_ev_free} +@item @code{acc_ev_update_start}, @code{acc_ev_update_end} +@item @code{acc_ev_enqueue_upload_start}, @code{acc_ev_enqueue_upload_end} +@item @code{acc_ev_enqueue_download_start}, @code{acc_ev_enqueue_download_end} +@end itemize + +During device initialization, and finalization, respectively, +callbacks for the following event types will not yet be invoked: + +@itemize +@item @code{acc_ev_alloc} +@item @code{acc_ev_free} +@end itemize + +Callbacks for the following event types have not yet been implemented, +so currently won't be invoked: + +@itemize +@item @code{acc_ev_device_shutdown_start}, @code{acc_ev_device_shutdown_end} +@item @code{acc_ev_runtime_shutdown} +@item @code{acc_ev_create}, @code{acc_ev_delete} +@item @code{acc_ev_wait_start}, @code{acc_ev_wait_end} +@end itemize + +For the following runtime library functions, not all expected +callbacks will be invoked (mostly concerning implicit device +initialization): + +@itemize +@item @code{acc_get_num_devices} +@item @code{acc_set_device_type} +@item @code{acc_get_device_type} +@item @code{acc_set_device_num} +@item @code{acc_get_device_num} +@item @code{acc_init} +@item @code{acc_shutdown} +@end itemize + +Aside from implicit device initialization, for the following runtime +library functions, no callbacks will be invoked for shared-memory +offloading devices (it's not clear if they should be): + +@itemize +@item @code{acc_malloc} +@item @code{acc_free} +@item @code{acc_copyin}, @code{acc_present_or_copyin}, @code{acc_copyin_async} +@item @code{acc_create}, @code{acc_present_or_create}, @code{acc_create_async} +@item @code{acc_copyout}, @code{acc_copyout_async}, @code{acc_copyout_finalize}, @code{acc_copyout_finalize_async} +@item @code{acc_delete}, @code{acc_delete_async}, @code{acc_delete_finalize}, @code{acc_delete_finalize_async} +@item @code{acc_update_device}, @code{acc_update_device_async} +@item @code{acc_update_self}, @code{acc_update_self_async} +@item @code{acc_map_data}, @code{acc_unmap_data} +@item @code{acc_memcpy_to_device}, @code{acc_memcpy_to_device_async} +@item @code{acc_memcpy_from_device}, @code{acc_memcpy_from_device_async} +@end itemize + +@c --------------------------------------------------------------------- +@c OpenMP-Implementation Specifics +@c --------------------------------------------------------------------- + +@node OpenMP-Implementation Specifics +@chapter OpenMP-Implementation Specifics + +@menu +* OpenMP Context Selectors:: +* Memory allocation with libmemkind:: +@end menu + +@node OpenMP Context Selectors +@section OpenMP Context Selectors + +@code{vendor} is always @code{gnu}. References are to the GCC manual. + +@multitable @columnfractions .60 .10 .25 +@headitem @code{arch} @tab @code{kind} @tab @code{isa} +@item @code{x86}, @code{x86_64}, @code{i386}, @code{i486}, + @code{i586}, @code{i686}, @code{ia32} + @tab @code{host} + @tab See @code{-m...} flags in ``x86 Options'' (without @code{-m}) +@item @code{amdgcn}, @code{gcn} + @tab @code{gpu} + @tab See @code{-march=} in ``AMD GCN Options'' +@item @code{nvptx} + @tab @code{gpu} + @tab See @code{-march=} in ``Nvidia PTX Options'' +@end multitable + +@node Memory allocation with libmemkind +@section Memory allocation with libmemkind + +On Linux systems, where the @uref{https://github.com/memkind/memkind, memkind +library} (@code{libmemkind.so.0}) is available at runtime, it is used when +creating memory allocators requesting + +@itemize +@item the memory space @code{omp_high_bw_mem_space} +@item the memory space @code{omp_large_cap_mem_space} +@item the partition trait @code{omp_atv_interleaved} +@end itemize + + +@c --------------------------------------------------------------------- +@c Offload-Target Specifics +@c --------------------------------------------------------------------- + +@node Offload-Target Specifics +@chapter Offload-Target Specifics + +The following sections present notes on the offload-target specifics + +@menu +* AMD Radeon:: +* nvptx:: +@end menu + +@node AMD Radeon +@section AMD Radeon (GCN) + +On the hardware side, there is the hierarchy (fine to coarse): +@itemize +@item work item (thread) +@item wavefront +@item work group +@item compute unite (CU) +@end itemize + +All OpenMP and OpenACC levels are used, i.e. +@itemize +@item OpenMP's simd and OpenACC's vector map to work items (thread) +@item OpenMP's threads (``parallel'') and OpenACC's workers map + to wavefronts +@item OpenMP's teams and OpenACC's gang use a threadpool with the + size of the number of teams or gangs, respectively. +@end itemize + +The used sizes are +@itemize +@item Number of teams is the specified @code{num_teams} (OpenMP) or + @code{num_gangs} (OpenACC) or otherwise the number of CU +@item Number of wavefronts is 4 for gfx900 and 16 otherwise; + @code{num_threads} (OpenMP) and @code{num_workers} (OpenACC) + overrides this if smaller. +@item The wavefront has 102 scalars and 64 vectors +@item Number of workitems is always 64 +@item The hardware permits maximally 40 workgroups/CU and + 16 wavefronts/workgroup up to a limit of 40 wavefronts in total per CU. +@item 80 scalars registers and 24 vector registers in non-kernel functions + (the chosen procedure-calling API). +@item For the kernel itself: as many as register pressure demands (number of + teams and number of threads, scaled down if registers are exhausted) +@end itemize + +The implementation remark: +@itemize +@item I/O within OpenMP target regions and OpenACC parallel/kernels is supported + using the C library @code{printf} functions and the Fortran + @code{print}/@code{write} statements. +@end itemize + + + +@node nvptx +@section nvptx + +On the hardware side, there is the hierarchy (fine to coarse): +@itemize +@item thread +@item warp +@item thread block +@item streaming multiprocessor +@end itemize + +All OpenMP and OpenACC levels are used, i.e. +@itemize +@item OpenMP's simd and OpenACC's vector map to threads +@item OpenMP's threads (``parallel'') and OpenACC's workers map to warps +@item OpenMP's teams and OpenACC's gang use a threadpool with the + size of the number of teams or gangs, respectively. +@end itemize + +The used sizes are +@itemize +@item The @code{warp_size} is always 32 +@item CUDA kernel launched: @code{dim=@{#teams,1,1@}, blocks=@{#threads,warp_size,1@}}. +@end itemize + +Additional information can be obtained by setting the environment variable to +@code{GOMP_DEBUG=1} (very verbose; grep for @code{kernel.*launch} for launch +parameters). + +GCC generates generic PTX ISA code, which is just-in-time compiled by CUDA, +which caches the JIT in the user's directory (see CUDA documentation; can be +tuned by the environment variables @code{CUDA_CACHE_@{DISABLE,MAXSIZE,PATH@}}. + +Note: While PTX ISA is generic, the @code{-mptx=} and @code{-march=} commandline +options still affect the used PTX ISA code and, thus, the requirments on +CUDA version and hardware. + +The implementation remark: +@itemize +@item I/O within OpenMP target regions and OpenACC parallel/kernels is supported + using the C library @code{printf} functions. Note that the Fortran + @code{print}/@code{write} statements are not supported, yet. +@item Compilation OpenMP code that contains @code{requires reverse_offload} + requires at least @code{-march=sm_35}, compiling for @code{-march=sm_30} + is not supported. +@end itemize + + +@c --------------------------------------------------------------------- +@c The libgomp ABI +@c --------------------------------------------------------------------- + +@node The libgomp ABI +@chapter The libgomp ABI + +The following sections present notes on the external ABI as +presented by libgomp. Only maintainers should need them. + +@menu +* Implementing MASTER construct:: +* Implementing CRITICAL construct:: +* Implementing ATOMIC construct:: +* Implementing FLUSH construct:: +* Implementing BARRIER construct:: +* Implementing THREADPRIVATE construct:: +* Implementing PRIVATE clause:: +* Implementing FIRSTPRIVATE LASTPRIVATE COPYIN and COPYPRIVATE clauses:: +* Implementing REDUCTION clause:: +* Implementing PARALLEL construct:: +* Implementing FOR construct:: +* Implementing ORDERED construct:: +* Implementing SECTIONS construct:: +* Implementing SINGLE construct:: +* Implementing OpenACC's PARALLEL construct:: +@end menu + + +@node Implementing MASTER construct +@section Implementing MASTER construct + +@smallexample +if (omp_get_thread_num () == 0) + block +@end smallexample + +Alternately, we generate two copies of the parallel subfunction +and only include this in the version run by the primary thread. +Surely this is not worthwhile though... + + + +@node Implementing CRITICAL construct +@section Implementing CRITICAL construct + +Without a specified name, + +@smallexample + void GOMP_critical_start (void); + void GOMP_critical_end (void); +@end smallexample + +so that we don't get COPY relocations from libgomp to the main +application. + +With a specified name, use omp_set_lock and omp_unset_lock with +name being transformed into a variable declared like + +@smallexample + omp_lock_t gomp_critical_user_<name> __attribute__((common)) +@end smallexample + +Ideally the ABI would specify that all zero is a valid unlocked +state, and so we wouldn't need to initialize this at +startup. + + + +@node Implementing ATOMIC construct +@section Implementing ATOMIC construct + +The target should implement the @code{__sync} builtins. + +Failing that we could add + +@smallexample + void GOMP_atomic_enter (void) + void GOMP_atomic_exit (void) +@end smallexample + +which reuses the regular lock code, but with yet another lock +object private to the library. + + + +@node Implementing FLUSH construct +@section Implementing FLUSH construct + +Expands to the @code{__sync_synchronize} builtin. + + + +@node Implementing BARRIER construct +@section Implementing BARRIER construct + +@smallexample + void GOMP_barrier (void) +@end smallexample + + +@node Implementing THREADPRIVATE construct +@section Implementing THREADPRIVATE construct + +In _most_ cases we can map this directly to @code{__thread}. Except +that OMP allows constructors for C++ objects. We can either +refuse to support this (how often is it used?) or we can +implement something akin to .ctors. + +Even more ideally, this ctor feature is handled by extensions +to the main pthreads library. Failing that, we can have a set +of entry points to register ctor functions to be called. + + + +@node Implementing PRIVATE clause +@section Implementing PRIVATE clause + +In association with a PARALLEL, or within the lexical extent +of a PARALLEL block, the variable becomes a local variable in +the parallel subfunction. + +In association with FOR or SECTIONS blocks, create a new +automatic variable within the current function. This preserves +the semantic of new variable creation. + + + +@node Implementing FIRSTPRIVATE LASTPRIVATE COPYIN and COPYPRIVATE clauses +@section Implementing FIRSTPRIVATE LASTPRIVATE COPYIN and COPYPRIVATE clauses + +This seems simple enough for PARALLEL blocks. Create a private +struct for communicating between the parent and subfunction. +In the parent, copy in values for scalar and "small" structs; +copy in addresses for others TREE_ADDRESSABLE types. In the +subfunction, copy the value into the local variable. + +It is not clear what to do with bare FOR or SECTION blocks. +The only thing I can figure is that we do something like: + +@smallexample +#pragma omp for firstprivate(x) lastprivate(y) +for (int i = 0; i < n; ++i) + body; +@end smallexample + +which becomes + +@smallexample +@{ + int x = x, y; + + // for stuff + + if (i == n) + y = y; +@} +@end smallexample + +where the "x=x" and "y=y" assignments actually have different +uids for the two variables, i.e. not something you could write +directly in C. Presumably this only makes sense if the "outer" +x and y are global variables. + +COPYPRIVATE would work the same way, except the structure +broadcast would have to happen via SINGLE machinery instead. + + + +@node Implementing REDUCTION clause +@section Implementing REDUCTION clause + +The private struct mentioned in the previous section should have +a pointer to an array of the type of the variable, indexed by the +thread's @var{team_id}. The thread stores its final value into the +array, and after the barrier, the primary thread iterates over the +array to collect the values. + + +@node Implementing PARALLEL construct +@section Implementing PARALLEL construct + +@smallexample + #pragma omp parallel + @{ + body; + @} +@end smallexample + +becomes + +@smallexample + void subfunction (void *data) + @{ + use data; + body; + @} + + setup data; + GOMP_parallel_start (subfunction, &data, num_threads); + subfunction (&data); + GOMP_parallel_end (); +@end smallexample + +@smallexample + void GOMP_parallel_start (void (*fn)(void *), void *data, unsigned num_threads) +@end smallexample + +The @var{FN} argument is the subfunction to be run in parallel. + +The @var{DATA} argument is a pointer to a structure used to +communicate data in and out of the subfunction, as discussed +above with respect to FIRSTPRIVATE et al. + +The @var{NUM_THREADS} argument is 1 if an IF clause is present +and false, or the value of the NUM_THREADS clause, if +present, or 0. + +The function needs to create the appropriate number of +threads and/or launch them from the dock. It needs to +create the team structure and assign team ids. + +@smallexample + void GOMP_parallel_end (void) +@end smallexample + +Tears down the team and returns us to the previous @code{omp_in_parallel()} state. + + + +@node Implementing FOR construct +@section Implementing FOR construct + +@smallexample + #pragma omp parallel for + for (i = lb; i <= ub; i++) + body; +@end smallexample + +becomes + +@smallexample + void subfunction (void *data) + @{ + long _s0, _e0; + while (GOMP_loop_static_next (&_s0, &_e0)) + @{ + long _e1 = _e0, i; + for (i = _s0; i < _e1; i++) + body; + @} + GOMP_loop_end_nowait (); + @} + + GOMP_parallel_loop_static (subfunction, NULL, 0, lb, ub+1, 1, 0); + subfunction (NULL); + GOMP_parallel_end (); +@end smallexample + +@smallexample + #pragma omp for schedule(runtime) + for (i = 0; i < n; i++) + body; +@end smallexample + +becomes + +@smallexample + @{ + long i, _s0, _e0; + if (GOMP_loop_runtime_start (0, n, 1, &_s0, &_e0)) + do @{ + long _e1 = _e0; + for (i = _s0, i < _e0; i++) + body; + @} while (GOMP_loop_runtime_next (&_s0, _&e0)); + GOMP_loop_end (); + @} +@end smallexample + +Note that while it looks like there is trickiness to propagating +a non-constant STEP, there isn't really. We're explicitly allowed +to evaluate it as many times as we want, and any variables involved +should automatically be handled as PRIVATE or SHARED like any other +variables. So the expression should remain evaluable in the +subfunction. We can also pull it into a local variable if we like, +but since its supposed to remain unchanged, we can also not if we like. + +If we have SCHEDULE(STATIC), and no ORDERED, then we ought to be +able to get away with no work-sharing context at all, since we can +simply perform the arithmetic directly in each thread to divide up +the iterations. Which would mean that we wouldn't need to call any +of these routines. + +There are separate routines for handling loops with an ORDERED +clause. Bookkeeping for that is non-trivial... + + + +@node Implementing ORDERED construct +@section Implementing ORDERED construct + +@smallexample + void GOMP_ordered_start (void) + void GOMP_ordered_end (void) +@end smallexample + + + +@node Implementing SECTIONS construct +@section Implementing SECTIONS construct + +A block as + +@smallexample + #pragma omp sections + @{ + #pragma omp section + stmt1; + #pragma omp section + stmt2; + #pragma omp section + stmt3; + @} +@end smallexample + +becomes + +@smallexample + for (i = GOMP_sections_start (3); i != 0; i = GOMP_sections_next ()) + switch (i) + @{ + case 1: + stmt1; + break; + case 2: + stmt2; + break; + case 3: + stmt3; + break; + @} + GOMP_barrier (); +@end smallexample + + +@node Implementing SINGLE construct +@section Implementing SINGLE construct + +A block like + +@smallexample + #pragma omp single + @{ + body; + @} +@end smallexample + +becomes + +@smallexample + if (GOMP_single_start ()) + body; + GOMP_barrier (); +@end smallexample + +while + +@smallexample + #pragma omp single copyprivate(x) + body; +@end smallexample + +becomes + +@smallexample + datap = GOMP_single_copy_start (); + if (datap == NULL) + @{ + body; + data.x = x; + GOMP_single_copy_end (&data); + @} + else + x = datap->x; + GOMP_barrier (); +@end smallexample + + + +@node Implementing OpenACC's PARALLEL construct +@section Implementing OpenACC's PARALLEL construct + +@smallexample + void GOACC_parallel () +@end smallexample + + + +@c --------------------------------------------------------------------- +@c Reporting Bugs +@c --------------------------------------------------------------------- + +@node Reporting Bugs +@chapter Reporting Bugs + +Bugs in the GNU Offloading and Multi Processing Runtime Library should +be reported via @uref{https://gcc.gnu.org/bugzilla/, Bugzilla}. Please add +"openacc", or "openmp", or both to the keywords field in the bug +report, as appropriate. + + + +@c --------------------------------------------------------------------- +@c GNU General Public License +@c --------------------------------------------------------------------- + +@include gpl_v3.texi + + + +@c --------------------------------------------------------------------- +@c GNU Free Documentation License +@c --------------------------------------------------------------------- + +@include fdl.texi + + + +@c --------------------------------------------------------------------- +@c Funding Free Software +@c --------------------------------------------------------------------- + +@include funding.texi + +@c --------------------------------------------------------------------- +@c Index +@c --------------------------------------------------------------------- + +@node Library Index +@unnumbered Library Index + +@printindex cp + +@bye |