path: root/newlib/HOWTO

This HOWTO file contains notes for maintainers and contributors to Newlib.
For information on using Newlib (building, installing), see README.  (In
particular, the "Regenerating Configuration Files" section in README is of
interest to both users and contributors.)
 
(This file could probably use some other "sections" in addition to the
initially-provided sections.  Please help by adding as appropriate.)

DOCUMENTATION
=============

     All the documentation for Newlib comes as part of the machine-readable
distribution.  Functions are documented in the source files, although not
every file contains documentation, as many functions share manual page
information.  For example, lround(), lroundf(), llround(), and llroundf()
share a single man page, which is in the source for lround().  The documentation
is written partially in a custom format and partially in Texinfo format.

     The custom part comprises makedoc.c and doc.str, both of which are in the
doc directory.  doc.str is a macro file that can be used to define things to
be done by makedoc, using building blocks that are built into makedoc.c.
The basic function of makedoc is two-fold.  First, it recognizes comments in
the proper format to pull out of source files.  Second, it adds some Texinfo
commands as well as translating certain sequences into the appropriate
Texinfo commands.  Refer to makedoc.c and doc.str for what these are.
(makedoc.c is not particularly-well commented.)  Another way to see how they
work is to simply examine the source files with documentation comments.

(A couple of examples that use some of the fancier options:
	libm/common/s_isnan.c ("o+" variable-"bullet" list),
	libc/stdio/sprintf.c ("O+" bullet list; "." for example code).)

     In addition to the makedoc.c stuff, Texinfo commands can be directly
used.  Texinfo is a documentation system that uses a single source file to
produce both on-line information and a printed manual.  You can use one of the
Info formatting commands to create the on-line version of the documentation
and TeX (or `texi2roff') to typeset the printed version.  While Newlib contains
a copy of the texinfo package (texinfo.tex), the manual for it is not
included.  The latest one may be found at

https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html

(which could be for a newer version of texinfo.tex than is included in Newlib).

     In addition to Texinfo commands, straight TeX commands can also be used,
but these, however, should be carefully limited and be given alternates for
when a non-printed manual is produced--such as when info pages are produced.
For an example of this kind of usage, see libm/common/s_logb.c.

     If writing a new function that requires documentation, the required
sections are FUNCTION, INDEX, SYNOPSIS, DESCRIPTION, RETURNS,
and PORTABILITY.  BUGS, NOTES, SEEALSO and WARNINGS should be added as
appropriate.

     Source files which contain documentation are processed into ".def"
files with the extracted information.  These .def files are noted in the
makefiles as CHEWOUT_FILES.  These .def files need to be included into an
appropriate .tex file for inclusion in the manuals (one each for libc and libm).
Pay special attention under libc, as the manual is arranged by header file name,
but not all header files are represented by directories (e.g.  wcsftime.c is
found under libc/time, but goes under wchar.h in the manual.)

In summary, to add new documentation:

1.  Add properly-formatted comments to source file (e.g. src.c);
2.  add "chewout" file to CHEWOUT_FILES list in Makefile.am (e.g. src.def),
    re-generate Makefile.in;
3.  @include that .def file in the appropriate .tex file, add texinfo menu
    entries that reference the @node(s) in that .def file;
4.  make ChangeLog entry and generate patch.
 
EL/IX (ELIX_LEVEL_n, ELIX_n_SOURCES)
====================================

     Some of the Makefiles contain definitions of ELIX_LEVEL_1 ... ELIX_LEVEL_4,
and corresponding definitions for ELIX_1_SOURCES ... ELIX_4_SOURCES.
These are referring to the EL/IX standards created by Red Hat for the
purpose of Linux-based open standards for embedded development.  In brief,
the intent is to define multiple levels for API functions that allows the
user to size the library appropriately for their application--at least in
terms of the predefined lists.  For full details, refer to the EL/IX home
page at https://sourceware.org/elix/.  The likely best way to tell how to
classify any new functions in terms of needing an ELIX level qualification
is to ask Jeff Johnston.  To see how it works and try classification on your
own, refer to the API specification on the web site,

https://sourceware.org/elix/api/current/api.pdf

(Unfortunately, it is not complete with respect to either the C99 or POSIX 
standards, so particular C and POSIX functions may or may not be found.)

     The following definitions of the levels are from the (draft) standard.

Level 1
     RTOS compatible layer.  Functions available in both Linux and a
typical deeply embedded operating system (eCos, RTEMS, VxWorks, pSOS, VRTX32,
etc.).  Some functions may have reduced or modified semantics.

Level 2
     Linux single process only. Includes level 1 plus any functions from Linux
that are not easily implemented on an RTOS.  Also full implementations of
reduced functions in Level 1.

Level 3
     Linux multiprocess for embedded applications.  This is basically POSIX.1
with some of the functions that are obviously not for embedded applications
(such as job control) removed.

Level 4
     Full POSIX or Linux compliance.  Essentially these are functions that are
present in a standard Linux kernel but are irrelevant to an embedded system.
These functions do not form part of the EL/IX API.

BUILD SYSTEM
============

     Newlib utilizes GNU Autotools (GNU Autoconf & GNU Automake) for its build
system.  It can be broken down into two phases: configure & compile (make).
NB: GNU Libtool is not used.  Newlib only produces static archives, and GNU
Automake is capable of handling that.

     As noted in the README file, you do not need to worry about which tools to
run at any particular point.  Simply invoke `autoreconf` and it will do it all.

     Tool versions are tightly controlled.  GNU Autoconf 2.69 exactly must be
used.  Attempts to use any other version, newer or older, will be rejected.  If
the current system does not include that version, it can be downloaded from the
GNU Autoconf website and installed into a local directory tree (like ~/.local).
GNU Automake 1.15.1 is required as a minimum.  Newer versions are allowed when
running locally, but developers submitting patches must use 1.15.1 exactly.

     The configure phase naturally starts with the configure script.  It merges
multiple inputs to produce the final script.
* aclocal.m4: Generated by aclocal; do not edit.  Copies all the various m4
  macros used by configure from the system install into the local copy.  This
  allows regenerating configure without needing all the same system packages.
* autom4te.cache/: Generated by aclocal to cache system search results.  Safe
  to delete at any time, but never edit anything in this directory.
* iconv.m4: Generated by libc/iconv/ces/mkdeps.pl; do not edit.  Keeps the
  set of dynamic iconv synced between the source files and configure.
* acinclude.m4: Historically, a way of sharing configure logic with subdirs.
  At some point, it should all be merged into configure.ac directly.
* configure.ac: The main configure script for supporting all Newlib options.
* configure.host: An index of every supported Newlib host and the settings to
  use when compiling for it.
* **/acinclude.m4: Subdirectories may include custom configure tests without
  having to clutter the common configure.ac or acinclude.m4 file.  When adding
  a new file, it must be manually included -- there is no mechanism to search
  and automatically include every such file in the tree.  Look for m4_include
  lines in configure.ac or existing acinclude.m4 files as examples.  Care must
  be taken to not run excessive tests for incompatible targets, or to declare
  generic names or symbols that can collide with other targets.  Any such file
  should try to match $host (or similar) settings before running.  Settings
  from configure.host (e.g. machine_dir or sys_dir) are guaranteed to be
  available.  When creating names (Makefile variables, defines, etc...), try to
  include the respective library and machine or sys name.  Do not confuse these
  with the historical top-level acinclude.m4 file!

     The configure script has to take special consideration to the fact that it
usually runs with a toolchain that lacks a C library (since Newlib will provide
it once it's compiled & installed).  Newlib uses the uncommon AC_NO_EXECUTABLES
macro to tell GNU Autoconf to not automatically require a toolchain that can
link executables.  As a result, Newlib itself cannot perform any link tests.
It shouldn't really need to, but configure authors must keep this in mind and
stick to the appropriate source-level test -- AC_PREPROC for testing behavior
of the preprocessor and AC_COMPILE for testing compiler behavior.

     The newlib.h header is created from newlib.hin during the configure phase.
The newlib.hin file is managed by autoheader, so do not edit it directly.  Any
symbol declared in configure via AC_DEFINE is automatically included.  The file
declares all the Newlib-specific configure settings and will be installed as
part of the standard set of headers.  Critically, all symbols are namespaced
with a leading _ prefix to avoid collisions with user code, and any symbols not
namespaced are automatically removed.  This is necessary due to some system m4
macros automatically adding their own defines that we cannot disable.  It also
means that, when using AC_DEFINE, the _ prefix must be manually added.

     A single Makefile is created during the configure phase.  This will build
all of Newlib using non-recursive make (i.e., there are no subdirectories with
their own Makefiles).  The single Makefile.am file includes Makefile.inc files
in its immediate subdirectories, and those further include Makefile.inc files
in their subdirectories.  This means all variables and targets exist in a
single global flat namespace, so care must be taken to not use generic names
like "SOURCES" in any Makefile.inc file.  Instead, use %C%_ on all variables to
get a unique prefix, and GNU Automake will expand it.  In order to use one of
the standard GNU Automake variables (e.g., CLEANFILES), first declare it at the
top of the Makefile.am by assigning it (e.g., CLEANFILES =), and then always
append to it in Makefile.inc (e.g. CLEANFILES += ...).  Again, GNU Automake
will take care of combining all of this logic together to produce a portable
Makefile.

     Developers used to changing to a subdirectory and running make commands
to iterate quickly on a smaller section of Newlib might be surprised that this
does not work -- there is no Makefile in any subdirectory.  Instead, make has
to be run from the top-level, and the full path to the target is specified.
So instead of running `cd subdir && make foo.o`, run `make subdir/foo.o`.  This
applies to any target, not just objects.  Some common targets:
make libm.a
make libc.a

     Conditionals are allowed in Makefile.inc files, but using them to elide
an entire subdir Makefile.inc should be avoided.  Instead, GNU Automake
conditionals should be used only within a single Makefile.inc file to control
settings specific to that directory.  See the documentation section below for
more information.

     Per-directory compiler settings are allowed.  Similar to the syntax used
above, the _%C% suffix can be used.  See the Makefile.am file for the current
set of flags (e.g. CFLAGS, CPPFLAGS, etc...) that are supported.
AM_CFLAGS_%C% = ...
libm_a_CFLAGS_%C% = ...
libc_a_CFLAGS_%C% = ...

     Per-file compiler settings are allowed.  Similar to the syntax used
above, the _%C% suffix can be used.  See the Makefile.am file for the current
set of flags (e.g. CFLAGS, CPPFLAGS, etc...) that are supported.
AM_CFLAGS_%C%_moo.c = ...
libm_a_CFLAGS_%C%_foo.c = ...
libc_a_CFLAGS_%C%_bar.c = ...

     An important aspect of the build system is allowing machine-specific
directories to override common implementations.  For example, Newlib provides
generic C implementations of string & memory functions in libc/string/ that
perform adequately for any target, but many architectures include hand written
assembly files for smaller & faster execution.  Those files live under the
corresponding libc/machine/<ARCHITECTURE>/ directory.  Both versions are
compiled when building Newlib, but when generating the libc.a archive, the
common object is added first, followed by the machine-specific one.  This works
because the object names follow a consistent convention: libm_a-<filename>.o
for the math library and libc_a-<filename>.o for the C library.  If the common
code stores a function foo in foo.c, then it can be overridden by another foo.c
or foo.S or foo.s file since they all produce foo.o.  For this reason, all
machine Makefile.inc files must be included last after all common Makefile.inc
files.

     Note that the Makefiles do not use VPATH to search source files across
multiple source directories.  VPATH is used to match a single build directory
to the corresponding source directory, but that is it.  For more details on
how machine directories override functions in common directories, see the
previous section.

     Keep in mind that machine overrides only work at the file level, not at
the symbol level.  If a common file defines multiple symbols, then the machine
override must define all the same symbols.  This is why most common code stores
only one function per source file.

     The documentation, both the manual and man pages and header references,
should be consistent regardless of what configure options are used (including
the target system).  This makes it easy to get a complete documentation build
at all times.  If inclusion of chapters depended on specific configure options,
it would be difficult to find the set of options necessary to produce the
complete manual, if it were even possible in the first place (as some configure
options are incompatible).  Since documentation chapters & APIs are declared in
the respective directories and not in a centralized location, it is important
to not use any GNU Automake conditional to control whether Makefile.inc files
are included.  Documentation that covers optional APIs should note that it is
not guaranteed to be included in every build of Newlib for every target.  The
level of detail is left up to the author's discretion.

     Newlib automatically builds itself for all multilib configurations that
the active toolchain supports.  This logic is provided by common code from the
combined toolchain source tree, but bolts itself into the GNU Autoconf and
Automake files.  The critical parts to keep in mind:
* ../config/multi.m4: AM_ENABLE_MULTILIB adds the --{en,dis}able-multilib
  options to configure for users, and takes care of calling ../config-ml.in
  for the Makefile in the current directory.  While not applicable to Newlib
  anymore (since Newlib only produces a single Makefile now), this does not
  automatically run for any subdir Makefiles.  Instead, it must be run manually
  in AC_CONFIG_FILES's commands argument.
* ../config-ml.in: Rewrites the specified Makefile to include the various
  multilib variables and the multi-do command.  Notably, this only makes the
  multi-do rule available, it does not cause it to be used anywhere.
* ../multilib.am: Manually included in Newlib's Makefile.am.  This adds calls
  to multi-do for the common important rules: all (i.e. compiling), install,
  clean, etc...
* FLAGS_TO_PASS: Newlib's Makefile.am declares all the variables to pass to
  each multilib configuration via the multi-do rule.  While the multi-do rule
  includes many common flags automatically, Newlib passes along a few extra
  that are unique to it.

    The testsuite is not integrated at all currently.  It is not really well
tested in general.  The README file has more information.  Hopefully someone
someday will step up to integrate it into the existing framework.