Editing.

1 files changed, 2892 insertions, 2892 deletions

diff --git a/manual/charset.texi b/manual/charset.texi
index bb9cc64..b7b2f73 100644
--- a/manual/charset.texi
+++ b/manual/charset.texi
@@ -1,2892 +1,2892 @@
-@node Character Set Handling, Locales, String and Array Utilities, Top
-@c %MENU% Support for extended character sets
-@chapter Character Set Handling
-
-@ifnottex
-@macro cal{text}
-\text\
-@end macro
-@end ifnottex
-
-Character sets used in the early days of computing had only six, seven,
-or eight bits for each character: there was never a case where more than
-eight bits (one byte) were used to represent a single character.  The
-limitations of this approach became more apparent as more people
-grappled with non-Roman character sets, where not all the characters
-that make up a language's character set can be represented by @math{2^8}
-choices.  This chapter shows the functionality which was added to the C
-library to support multiple character sets.
-
-@menu
-* Extended Char Intro::              Introduction to Extended Characters.
-* Charset Function Overview::        Overview about Character Handling
-                                      Functions.
-* Restartable multibyte conversion:: Restartable multibyte conversion
-                                      Functions.
-* Non-reentrant Conversion::         Non-reentrant Conversion Function.
-* Generic Charset Conversion::       Generic Charset Conversion.
-@end menu
-
-
-@node Extended Char Intro
-@section Introduction to Extended Characters
-
-A variety of solutions to overcome the differences between
-character sets with a 1:1 relation between bytes and characters and
-character sets with ratios of 2:1 or 4:1 exist. The remainder of this
-section gives a few examples to help understand the design decisions
-made while developing the functionality of the @w{C library}.
-
-@cindex internal representation
-A distinction we have to make right away is between internal and
-external representation.  @dfn{Internal representation} means the
-representation used by a program while keeping the text in memory.
-External representations are used when text is stored or transmitted
-through whatever communication channel.  Examples of external
-representations include files lying in a directory that are going to be
-read and parsed.
-
-Traditionally there has been no difference between the two representations.
-It was equally comfortable and useful to use the same single-byte
-representation internally and externally.  This changes with more and
-larger character sets.
-
-One of the problems to overcome with the internal representation is
-handling text that is externally encoded using different character
-sets.  Assume a program which reads two texts and compares them using
-some metric.  The comparison can be usefully done only if the texts are
-internally kept in a common format.
-
-@cindex wide character
-For such a common format (@math{=} character set) eight bits are certainly
-no longer enough.  So the smallest entity will have to grow: @dfn{wide
-characters} will now be used.  Instead of one byte, two or four will
-be used instead.  (Three are not good to address in memory and more
-than four bytes seem not to be necessary).
-
-@cindex Unicode
-@cindex ISO 10646
-As shown in some other part of this manual,
-@c !!! Ahem, wide char string functions are not yet covered -- drepper
-there exists a completely new family of functions which can handle texts
-of this kind in memory.  The most commonly used character sets for such
-internal wide character representations are Unicode and @w{ISO 10646}
-(also known as UCS for Universal Character Set). Unicode was originally
-planned as a 16-bit character set, whereas @w{ISO 10646} was designed to
-be a 31-bit large code space. The two standards are practically identical.
-They have the same character repertoire and code table, but Unicode specifies
-added semantics.  At the moment, only characters in the first @code{0x10000}
-code positions (the so-called Basic Multilingual Plane, BMP) have been
-assigned, but the assignment of more specialized characters outside this
-16-bit space is already in progress. A number of encodings have been
-defined for Unicode and @w{ISO 10646} characters:
-@cindex UCS-2
-@cindex UCS-4
-@cindex UTF-8
-@cindex UTF-16
-UCS-2 is a 16-bit word that can only represent characters
-from the BMP, UCS-4 is a 32-bit word than can represent any Unicode
-and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where
-ASCII characters are represented by ASCII bytes and non-ASCII characters
-by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension
-of UCS-2 in which pairs of certain UCS-2 words can be used to encode
-non-BMP characters up to @code{0x10ffff}.
-
-To represent wide characters the @code{char} type is not suitable.  For
-this reason the @w{ISO C} standard introduces a new type which is
-designed to keep one character of a wide character string.  To maintain
-the similarity there is also a type corresponding to @code{int} for
-those functions which take a single wide character.
-
-@comment stddef.h
-@comment ISO
-@deftp {Data type} wchar_t
-This data type is used as the base type for wide character strings.
-I.e., arrays of objects of this type are the equivalent of @code{char[]}
-for multibyte character strings.  The type is defined in @file{stddef.h}.
-
-The @w{ISO C90} standard, where this type was introduced, does not say
-anything specific about the representation.  It only requires that this
-type is capable of storing all elements of the basic character set.
-Therefore it would be legitimate to define @code{wchar_t} as
-@code{char}.  This might make sense for embedded systems.
-
-But for GNU systems this type is always 32 bits wide.  It is therefore
-capable of representing all UCS-4 values and  therefore covering all of
-@w{ISO 10646}.  Some Unix systems define @code{wchar_t} as a 16-bit type and
-thereby follow Unicode very strictly.  This is perfectly fine with the
-standard but it also means that to represent all characters from Unicode
-and @w{ISO 10646} one has to use UTF-16 surrogate characters which is in
-fact a multi-wide-character encoding.  But this contradicts the purpose
-of the @code{wchar_t} type.
-@end deftp
-
-@comment wchar.h
-@comment ISO
-@deftp {Data type} wint_t
-@code{wint_t} is a data type used for parameters and variables which
-contain a single wide character.  As the name already suggests it is the
-equivalent to @code{int} when using the normal @code{char} strings.  The
-types @code{wchar_t} and @code{wint_t} have often the same
-representation if their size if 32 bits wide but if @code{wchar_t} is
-defined as @code{char} the type @code{wint_t} must be defined as
-@code{int} due to the parameter promotion.
-
-@pindex wchar.h
-This type is defined in @file{wchar.h} and got introduced in
-@w{Amendment 1} to @w{ISO C90}.
-@end deftp
-
-As there are for the @code{char} data type there also exist macros
-specifying the minimum and maximum value representable in an object of
-type @code{wchar_t}.
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WCHAR_MIN
-The macro @code{WCHAR_MIN} evaluates to the minimum value representable
-by an object of type @code{wint_t}.
-
-This macro got introduced in @w{Amendment 1} to @w{ISO C90}.
-@end deftypevr
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WCHAR_MAX
-The macro @code{WCHAR_MAX} evaluates to the maximum value representable
-by an object of type @code{wint_t}.
-
-This macro got introduced in @w{Amendment 1} to @w{ISO C90}.
-@end deftypevr
-
-Another special wide character value is the equivalent to @code{EOF}.
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WEOF
-The macro @code{WEOF} evaluates to a constant expression of type
-@code{wint_t} whose value is different from any member of the extended
-character set.
-
-@code{WEOF} need not be the same value as @code{EOF} and unlike
-@code{EOF} it also need @emph{not} be negative.  I.e., sloppy code like
-
-@smallexample
-@{
-  int c;
-  ...
-  while ((c = getc (fp)) < 0)
-    ...
-@}
-@end smallexample
-
-@noindent
-has to be rewritten to explicitly use @code{WEOF} when wide characters
-are used.
-
-@smallexample
-@{
-  wint_t c;
-  ...
-  while ((c = wgetc (fp)) != WEOF)
-    ...
-@}
-@end smallexample
-
-@pindex wchar.h
-This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is
-defined in @file{wchar.h}.
-@end deftypevr
-
-
-These internal representations present problems when it comes to storing
-and transmittal, since a single wide character consists of more
-than one byte they are effected by byte-ordering.  I.e., machines with
-different endianesses would see different value accessing the same data.
-This also applies for communication protocols which are all byte-based
-and therefore the sender has to decide about splitting the wide
-character in bytes.  A last (but not least important) point is that wide
-characters often require more storage space than an customized byte
-oriented character set.
-
-@cindex multibyte character
-@cindex EBCDIC
-   For all the above reasons, an external encoding which is different
-from the internal encoding is often used if the latter is UCS-2 or UCS-4.
-The external encoding is byte-based and can be chosen appropriately for
-the environment and for the texts to be handled.  There exist a variety
-of different character sets which can be used for this external
-encoding. Information which will not be exhaustively presented
-here--instead, a description of the major groups will suffice.  All of
-the ASCII-based character sets fulfill one requirement: they are
-"filesystem safe".  This means that the character @code{'/'} is used in
-the encoding @emph{only} to represent itself.  Things are a bit
-different for character sets like EBCDIC (Extended Binary Coded Decimal
-Interchange Code, a character set family used by IBM) but if the
-operation system does not understand EBCDIC directly the parameters to
-system calls have to be converted first anyhow.
-
-@itemize @bullet
-@item
-The simplest character sets are single-byte character sets.  There can
-be only up to 256 characters (for @w{8 bit} character sets) which is not
-sufficient to cover all languages but might be sufficient to handle a
-specific text.  Handling of @w{8 bit} character sets is simple.  This is
-not true for the other kinds presented later and therefore the
-application one uses might require the use of @w{8 bit} character sets.
-
-@cindex ISO 2022
-@item
-The @w{ISO 2022} standard defines a mechanism for extended character
-sets where one character @emph{can} be represented by more than one
-byte.  This is achieved by associating a state with the text.  Embedded
-in the text can be characters which can be used to change the state.
-Each byte in the text might have a different interpretation in each
-state.  The state might even influence whether a given byte stands for a
-character on its own or whether it has to be combined with some more
-bytes.
-
-@cindex EUC
-@cindex Shift_JIS
-@cindex SJIS
-In most uses of @w{ISO 2022} the defined character sets do not allow
-state changes which cover more than the next character.  This has the
-big advantage that whenever one can identify the beginning of the byte
-sequence of a character one can interpret a text correctly.  Examples of
-character sets using this policy are the various EUC character sets
-(used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN)
-or Shift_JIS (SJIS, a Japanese encoding).
-
-But there are also character sets using a state which is valid for more
-than one character and has to be changed by another byte sequence.
-Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN.
-
-@item
-@cindex ISO 6937
-Early attempts to fix 8 bit character sets for other languages using the
-Roman alphabet lead to character sets like @w{ISO 6937}.  Here bytes
-representing characters like the acute accent do not produce output
-themselves: one has to combine them with other characters to get the
-desired result.  E.g., the byte sequence @code{0xc2 0x61} (non-spacing
-acute accent, following by lower-case `a') to get the ``small a with
-acute'' character.  To get the acute accent character on its own, one has
-to write @code{0xc2 0x20} (the non-spacing acute followed by a space).
-
-This type of character set is used in some embedded systems such as
-teletex.
-
-@item
-@cindex UTF-8
-Instead of converting the Unicode or @w{ISO 10646} text used internally,
-it is often also sufficient to simply use an encoding different than
-UCS-2/UCS-4.  The Unicode and @w{ISO 10646} standards even specify such an
-encoding: UTF-8.  This encoding is able to represent all of @w{ISO
-10464} 31 bits in a byte string of length one to six.
-
-@cindex UTF-7
-There were a few other attempts to encode @w{ISO 10646} such as UTF-7
-but UTF-8 is today the only encoding which should be used.  In fact,
-UTF-8 will hopefully soon be the only external encoding that has to be
-supported.  It proves to be universally usable and the only disadvantage
-is that it favors Roman languages by making the byte string
-representation of other scripts (Cyrillic, Greek, Asian scripts) longer
-than necessary if using a specific character set for these scripts.
-Methods like the Unicode compression scheme can alleviate these
-problems.
-@end itemize
-
-The question remaining is: how to select the character set or encoding
-to use.  The answer: you cannot decide about it yourself, it is decided
-by the developers of the system or the majority of the users.  Since the
-goal is interoperability one has to use whatever the other people one
-works with use.  If there are no constraints the selection is based on
-the requirements the expected circle of users will have.  I.e., if a
-project is expected to only be used in, say, Russia it is fine to use
-KOI8-R or a similar character set.  But if at the same time people from,
-say, Greece are participating one should use a character set which allows
-all people to collaborate.
-
-The most widely useful solution seems to be: go with the most general
-character set, namely @w{ISO 10646}.  Use UTF-8 as the external encoding
-and problems about users not being able to use their own language
-adequately are a thing of the past.
-
-One final comment about the choice of the wide character representation
-is necessary at this point.  We have said above that the natural choice
-is using Unicode or @w{ISO 10646}.  This is not required, but at least
-encouraged, by the @w{ISO C} standard.  The standard defines at least a
-macro @code{__STDC_ISO_10646__} that is only defined on systems where
-the @code{wchar_t} type encodes @w{ISO 10646} characters.  If this
-symbol is not defined one should as much as possible avoid making
-assumption about the wide character representation.  If the programmer
-uses only the functions provided by the C library to handle wide
-character strings there should not be any compatibility problems with
-other systems.
-
-@node Charset Function Overview
-@section Overview about Character Handling Functions
-
-A Unix @w{C library} contains three different sets of functions in two
-families to handle character set conversion.  The one function family
-is specified in the @w{ISO C} standard and therefore is portable even
-beyond the Unix world.
-
-The most commonly known set of functions, coming from the @w{ISO C90}
-standard, is unfortunately the least useful one.  In fact, these
-functions should be avoided whenever possible, especially when
-developing libraries (as opposed to applications).
-
-The second family of functions got introduced in the early Unix standards
-(XPG2) and is still part of the latest and greatest Unix standard:
-@w{Unix 98}.  It is also the most powerful and useful set of functions.
-But we will start with the functions defined in @w{Amendment 1} to
-@w{ISO C90}.
-
-@node Restartable multibyte conversion
-@section Restartable Multibyte Conversion Functions
-
-The @w{ISO C} standard defines functions to convert strings from a
-multibyte representation to wide character strings.  There are a number
-of peculiarities:
-
-@itemize @bullet
-@item
-The character set assumed for the multibyte encoding is not specified
-as an argument to the functions.  Instead the character set specified by
-the @code{LC_CTYPE} category of the current locale is used; see
-@ref{Locale Categories}.
-
-@item
-The functions handling more than one character at a time require NUL
-terminated strings as the argument.  I.e., converting blocks of text
-does not work unless one can add a NUL byte at an appropriate place.
-The GNU C library contains some extensions the standard which allow
-specifying a size but basically they also expect terminated strings.
-@end itemize
-
-Despite these limitations the @w{ISO C} functions can very well be used
-in many contexts.  In graphical user interfaces, for instance, it is not
-uncommon to have functions which require text to be displayed in a wide
-character string if it is not simple ASCII.  The text itself might come
-from a file with translations and the user should decide about the
-current locale which determines the translation and therefore also the
-external encoding used.  In such a situation (and many others) the
-functions described here are perfect.  If more freedom while performing
-the conversion is necessary take a look at the @code{iconv} functions
-(@pxref{Generic Charset Conversion}).
-
-@menu
-* Selecting the Conversion::     Selecting the conversion and its properties.
-* Keeping the state::            Representing the state of the conversion.
-* Converting a Character::       Converting Single Characters.
-* Converting Strings::           Converting Multibyte and Wide Character
-                                  Strings.
-* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
-@end menu
-
-@node Selecting the Conversion
-@subsection Selecting the conversion and its properties
-
-We already said above that the currently selected locale for the
-@code{LC_CTYPE} category decides about the conversion which is performed
-by the functions we are about to describe.  Each locale uses its own
-character set (given as an argument to @code{localedef}) and this is the
-one assumed as the external multibyte encoding.  The wide character
-character set always is UCS-4, at least on GNU systems.
-
-A characteristic of each multibyte character set is the maximum number
-of bytes which can be necessary to represent one character.  This
-information is quite important when writing code which uses the
-conversion functions.  In the examples below we will see some examples.
-The @w{ISO C} standard defines two macros which provide this information.
-
-
-@comment limits.h
-@comment ISO
-@deftypevr Macro int MB_LEN_MAX
-This macro specifies the maximum number of bytes in the multibyte
-sequence for a single character in any of the supported locales.  It is
-a compile-time constant and it is defined in @file{limits.h}.
-@pindex limits.h
-@end deftypevr
-
-@comment stdlib.h
-@comment ISO
-@deftypevr Macro int MB_CUR_MAX
-@code{MB_CUR_MAX} expands into a positive integer expression that is the
-maximum number of bytes in a multibyte character in the current locale.
-The value is never greater than @code{MB_LEN_MAX}.  Unlike
-@code{MB_LEN_MAX} this macro need not be a compile-time constant and in
-fact, in the GNU C library it is not.
-
-@pindex stdlib.h
-@code{MB_CUR_MAX} is defined in @file{stdlib.h}.
-@end deftypevr
-
-Two different macros are necessary since strictly @w{ISO C90} compilers
-do not allow variable length array definitions but still it is desirable
-to avoid dynamic allocation.  This incomplete piece of code shows the
-problem:
-
-@smallexample
-@{
-  char buf[MB_LEN_MAX];
-  ssize_t len = 0;
-
-  while (! feof (fp))
-    @{
-      fread (&buf[len], 1, MB_CUR_MAX - len, fp);
-      /* @r{... process} buf */
-      len -= used;
-    @}
-@}
-@end smallexample
-
-The code in the inner loop is expected to have always enough bytes in
-the array @var{buf} to convert one multibyte character.  The array
-@var{buf} has to be sized statically since many compilers do not allow a
-variable size.  The @code{fread} call makes sure that always
-@code{MB_CUR_MAX} bytes are available in @var{buf}.  Note that it isn't
-a problem if @code{MB_CUR_MAX} is not a compile-time constant.
-
-
-@node Keeping the state
-@subsection Representing the state of the conversion
-
-@cindex stateful
-In the introduction of this chapter it was said that certain character
-sets use a @dfn{stateful} encoding.  I.e., the encoded values depend in
-some way on the previous bytes in the text.
-
-Since the conversion functions allow converting a text in more than one
-step we must have a way to pass this information from one call of the
-functions to another.
-
-@comment wchar.h
-@comment ISO
-@deftp {Data type} mbstate_t
-@cindex shift state
-A variable of type @code{mbstate_t} can contain all the information
-about the @dfn{shift state} needed from one call to a conversion
-function to another.
-
-@pindex wchar.h
-This type is defined in @file{wchar.h}.  It got introduced in
-@w{Amendment 1} to @w{ISO C90}.
-@end deftp
-
-To use objects of this type the programmer has to define such objects
-(normally as local variables on the stack) and pass a pointer to the
-object to the conversion functions.  This way the conversion function
-can update the object if the current multibyte character set is
-stateful.
-
-There is no specific function or initializer to put the state object in
-any specific state.  The rules are that the object should always
-represent the initial state before the first use and this is achieved by
-clearing the whole variable with code such as follows:
-
-@smallexample
-@{
-  mbstate_t state;
-  memset (&state, '\0', sizeof (state));
-  /* @r{from now on @var{state} can be used.}  */
-  ...
-@}
-@end smallexample
-
-When using the conversion functions to generate output it is often
-necessary to test whether the current state corresponds to the initial
-state.  This is necessary, for example, to decide whether or not to emit
-escape sequences to set the state to the initial state at certain
-sequence points.  Communication protocols often require this.
-
-@comment wchar.h
-@comment ISO
-@deftypefun int mbsinit (const mbstate_t *@var{ps})
-This function determines whether the state object pointed to by @var{ps}
-is in the initial state or not.  If @var{ps} is a null pointer or the
-object is in the initial state the return value is nonzero.  Otherwise
-it is zero.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-Code using this function often looks similar to this:
-
-@c Fix the example to explicitly say how to generate the escape sequence
-@c to restore the initial state.
-@smallexample
-@{
-  mbstate_t state;
-  memset (&state, '\0', sizeof (state));
-  /* @r{Use @var{state}.}  */
-  ...
-  if (! mbsinit (&state))
-    @{
-      /* @r{Emit code to return to initial state.}  */
-      const wchar_t empty[] = L"";
-      const wchar_t *srcp = empty;
-      wcsrtombs (outbuf, &srcp, outbuflen, &state);
-    @}
-  ...
-@}
-@end smallexample
-
-The code to emit the escape sequence to get back to the initial state is
-interesting.  The @code{wcsrtombs} function can be used to determine the
-necessary output code (@pxref{Converting Strings}).  Please note that on
-GNU systems it is not necessary to perform this extra action for the
-conversion from multibyte text to wide character text since the wide
-character encoding is not stateful.  But there is nothing mentioned in
-any standard which prohibits making @code{wchar_t} using a stateful
-encoding.
-
-@node Converting a Character
-@subsection Converting Single Characters
-
-The most fundamental of the conversion functions are those dealing with
-single characters.  Please note that this does not always mean single
-bytes.  But since there is very often a subset of the multibyte
-character set which consists of single byte sequences there are
-functions to help with converting bytes.  One very important and often
-applicable scenario is where ASCII is a subpart of the multibyte
-character set.  I.e., all ASCII characters stand for itself and all
-other characters have at least a first byte which is beyond the range
-@math{0} to @math{127}.
-
-@comment wchar.h
-@comment ISO
-@deftypefun wint_t btowc (int @var{c})
-The @code{btowc} function (``byte to wide character'') converts a valid
-single byte character @var{c} in the initial shift state into the wide
-character equivalent using the conversion rules from the currently
-selected locale of the @code{LC_CTYPE} category.
-
-If @code{(unsigned char) @var{c}} is no valid single byte multibyte
-character or if @var{c} is @code{EOF} the function returns @code{WEOF}.
-
-Please note the restriction of @var{c} being tested for validity only in
-the initial shift state.  There is no @code{mbstate_t} object used from
-which the state information is taken and the function also does not use
-any static state.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-Despite the limitation that the single byte value always is interpreted
-in the initial state this function is actually useful most of the time.
-Most characters are either entirely single-byte character sets or they
-are extension to ASCII.  But then it is possible to write code like this
-(not that this specific example is very useful):
-
-@smallexample
-wchar_t *
-itow (unsigned long int val)
-@{
-  static wchar_t buf[30];
-  wchar_t *wcp = &buf[29];
-  *wcp = L'\0';
-  while (val != 0)
-    @{
-      *--wcp = btowc ('0' + val % 10);
-      val /= 10;
-    @}
-  if (wcp == &buf[29])
-    *--wcp = L'0';
-  return wcp;
-@}
-@end smallexample
-
-Why is it necessary to use such a complicated implementation and not
-simply cast @code{'0' + val % 10} to a wide character?  The answer is
-that there is no guarantee that one can perform this kind of arithmetic
-on the character of the character set used for @code{wchar_t}
-representation.  In other situations the bytes are not constant at
-compile time and so the compiler cannot do the work.  In situations like
-this it is necessary @code{btowc}.
-
-@noindent
-There also is a function for the conversion in the other direction.
-
-@comment wchar.h
-@comment ISO
-@deftypefun int wctob (wint_t @var{c})
-The @code{wctob} function (``wide character to byte'') takes as the
-parameter a valid wide character.  If the multibyte representation for
-this character in the initial state is exactly one byte long the return
-value of this function is this character.  Otherwise the return value is
-@code{EOF}.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-There are more general functions to convert single character from
-multibyte representation to wide characters and vice versa.  These
-functions pose no limit on the length of the multibyte representation
-and they also do not require it to be in the initial state.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps})
-@cindex stateful
-The @code{mbrtowc} function (``multibyte restartable to wide
-character'') converts the next multibyte character in the string pointed
-to by @var{s} into a wide character and stores it in the wide character
-string pointed to by @var{pwc}.  The conversion is performed according
-to the locale currently selected for the @code{LC_CTYPE} category.  If
-the conversion for the character set used in the locale requires a state
-the multibyte string is interpreted in the state represented by the
-object pointed to by @var{ps}.  If @var{ps} is a null pointer, a static,
-internal state variable used only by the @code{mbrtowc} function is
-used.
-
-If the next multibyte character corresponds to the NUL wide character
-the return value of the function is @math{0} and the state object is
-afterwards in the initial state.  If the next @var{n} or fewer bytes
-form a correct multibyte character the return value is the number of
-bytes starting from @var{s} which form the multibyte character.  The
-conversion state is updated according to the bytes consumed in the
-conversion.  In both cases the wide character (either the @code{L'\0'}
-or the one found in the conversion) is stored in the string pointer to
-by @var{pwc} iff @var{pwc} is not null.
-
-If the first @var{n} bytes of the multibyte string possibly form a valid
-multibyte character but there are more than @var{n} bytes needed to
-complete it the return value of the function is @code{(size_t) -2} and
-no value is stored.  Please note that this can happen even if @var{n}
-has a value greater or equal to @code{MB_CUR_MAX} since the input might
-contain redundant shift sequences.
-
-If the first @code{n} bytes of the multibyte string cannot possibly form
-a valid multibyte character also no value is stored, the global variable
-@code{errno} is set to the value @code{EILSEQ} and the function returns
-@code{(size_t) -1}.  The conversion state is afterwards undefined.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-Using this function is straight forward.  A function which copies a
-multibyte string into a wide character string while at the same time
-converting all lowercase character into uppercase could look like this
-(this is not the final version, just an example; it has no error
-checking, and leaks sometimes memory):
-
-@smallexample
-wchar_t *
-mbstouwcs (const char *s)
-@{
-  size_t len = strlen (s);
-  wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
-  wchar_t *wcp = result;
-  wchar_t tmp[1];
-  mbstate_t state;
-  size_t nbytes;
-
-  memset (&state, '\0', sizeof (state));
-  while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
-    @{
-      if (nbytes >= (size_t) -2)
-        /* Invalid input string.  */
-        return NULL;
-      *result++ = towupper (tmp[0]);
-      len -= nbytes;
-      s += nbytes;
-    @}
-  return result;
-@}
-@end smallexample
-
-The use of @code{mbrtowc} should be clear.  A single wide character is
-stored in @code{@var{tmp}[0]} and the number of consumed bytes is stored
-in the variable @var{nbytes}.  In case the the conversion was successful
-the uppercase variant of the wide character is stored in the
-@var{result} array and the pointer to the input string and the number of
-available bytes is adjusted.
-
-The only non-obvious thing about the function might be the way memory is
-allocated for the result.  The above code uses the fact that there can
-never be more wide characters in the converted results than there are
-bytes in the multibyte input string.  This method yields to a
-pessimistic guess about the size of the result and if many wide
-character strings have to be constructed this way or the strings are
-long, the extra memory required allocated because the input string
-contains multibyte characters might be significant.  It would be
-possible to resize the allocated memory block to the correct size before
-returning it.  A better solution might be to allocate just the right
-amount of space for the result right away.  Unfortunately there is no
-function to compute the length of the wide character string directly
-from the multibyte string.  But there is a function which does part of
-the work.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps})
-The @code{mbrlen} function (``multibyte restartable length'') computes
-the number of at most @var{n} bytes starting at @var{s} which form the
-next valid and complete multibyte character.
-
-If the next multibyte character corresponds to the NUL wide character
-the return value is @math{0}.  If the next @var{n} bytes form a valid
-multibyte character the number of bytes belonging to this multibyte
-character byte sequence is returned.
-
-If the the first @var{n} bytes possibly form a valid multibyte
-character but it is incomplete the return value is @code{(size_t) -2}.
-Otherwise the multibyte character sequence is invalid and the return
-value is @code{(size_t) -1}.
-
-The multibyte sequence is interpreted in the state represented by the
-object pointed to by @var{ps}.  If @var{ps} is a null pointer, a state
-object local to @code{mbrlen} is used.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-The tentative reader now will of course note that @code{mbrlen} can be
-implemented as
-
-@smallexample
-mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)
-@end smallexample
-
-This is true and in fact is mentioned in the official specification.
-Now, how can this function be used to determine the length of the wide
-character string created from a multibyte character string?  It is not
-directly usable but we can define a function @code{mbslen} using it:
-
-@smallexample
-size_t
-mbslen (const char *s)
-@{
-  mbstate_t state;
-  size_t result = 0;
-  size_t nbytes;
-  memset (&state, '\0', sizeof (state));
-  while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
-    @{
-      if (nbytes >= (size_t) -2)
-        /* @r{Something is wrong.}  */
-        return (size_t) -1;
-      s += nbytes;
-      ++result;
-    @}
-  return result;
-@}
-@end smallexample
-
-This function simply calls @code{mbrlen} for each multibyte character
-in the string and counts the number of function calls.  Please note that
-we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen}
-call.  This is OK since a) this value is larger then the length of the
-longest multibyte character sequence and b) because we know that the
-string @var{s} ends with a NUL byte which cannot be part of any other
-multibyte character sequence but the one representing the NUL wide
-character.  Therefore the @code{mbrlen} function will never read invalid
-memory.
-
-Now that this function is available (just to make this clear, this
-function is @emph{not} part of the GNU C library) we can compute the
-number of wide character required to store the converted multibyte
-character string @var{s} using
-
-@smallexample
-wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);
-@end smallexample
-
-Please note that the @code{mbslen} function is quite inefficient.  The
-implementation of @code{mbstouwcs} implemented using @code{mbslen} would
-have to perform the conversion of the multibyte character input string
-twice and this conversion might be quite expensive.  So it is necessary
-to think about the consequences of using the easier but imprecise method
-before doing the work twice.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps})
-The @code{wcrtomb} function (``wide character restartable to
-multibyte'') converts a single wide character into a multibyte string
-corresponding to that wide character.
-
-If @var{s} is a null pointer the function resets the the state stored in
-the objects pointer to by @var{ps} (or the internal @code{mbstate_t}
-object) to the initial state.  This can also be achieved by a call like
-this:
-
-@smallexample
-wcrtombs (temp_buf, L'\0', ps)
-@end smallexample
-
-@noindent
-since if @var{s} is a null pointer @code{wcrtomb} performs as if it
-writes into an internal buffer which is guaranteed to be large enough.
-
-If @var{wc} is the NUL wide character @code{wcrtomb} emits, if
-necessary, a shift sequence to get the state @var{ps} into the initial
-state followed by a single NUL byte is stored in the string @var{s}.
-
-Otherwise a byte sequence (possibly including shift sequences) is
-written into the string @var{s}.  This of only happens if @var{wc} is a
-valid wide character, i.e., it has a multibyte representation in the
-character set selected by locale of the @code{LC_CTYPE} category.  If
-@var{wc} is no valid wide character nothing is stored in the strings
-@var{s}, @code{errno} is set to @code{EILSEQ}, the conversion state in
-@var{ps} is undefined and the return value is @code{(size_t) -1}.
-
-If no error occurred the function returns the number of bytes stored in
-the string @var{s}.  This includes all byte representing shift
-sequences.
-
-One word about the interface of the function: there is no parameter
-specifying the length of the array @var{s}.  Instead the function
-assumes that there are at least @code{MB_CUR_MAX} bytes available since
-this is the maximum length of any byte sequence representing a single
-character.  So the caller has to make sure that there is enough space
-available, otherwise buffer overruns can occur.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-Using this function is as easy as using @code{mbrtowc}.  The following
-example appends a wide character string to a multibyte character string.
-Again, the code is not really useful (and correct), it is simply here to
-demonstrate the use and some problems.
-
-@smallexample
-char *
-mbscatwcs (char *s, size_t len, const wchar_t *ws)
-@{
-  mbstate_t state;
-  /* @r{Find the end of the existing string.}  */
-  char *wp = strchr (s, '\0');
-  len -= wp - s;
-  memset (&state, '\0', sizeof (state));
-  do
-    @{
-      size_t nbytes;
-      if (len < MB_CUR_LEN)
-        @{
-          /* @r{We cannot guarantee that the next}
-             @r{character fits into the buffer, so}
-             @r{return an error.}  */
-          errno = E2BIG;
-          return NULL;
-        @}
-      nbytes = wcrtomb (wp, *ws, &state);
-      if (nbytes == (size_t) -1)
-        /* @r{Error in the conversion.}  */
-        return NULL;
-      len -= nbytes;
-      wp += nbytes;
-    @}
-  while (*ws++ != L'\0');
-  return s;
-@}
-@end smallexample
-
-First the function has to find the end of the string currently in the
-array @var{s}.  The @code{strchr} call does this very efficiently since a
-requirement for multibyte character representations is that the NUL byte
-never is used except to represent itself (and in this context, the end
-of the string).
-
-After initializing the state object the loop is entered where the first
-task is to make sure there is enough room in the array @var{s}.  We
-abort if there are not at least @code{MB_CUR_LEN} bytes available.  This
-is not always optimal but we have no other choice.  We might have less
-than @code{MB_CUR_LEN} bytes available but the next multibyte character
-might also be only one byte long.  At the time the @code{wcrtomb} call
-returns it is too late to decide whether the buffer was large enough or
-not.  If this solution is really unsuitable there is a very slow but
-more accurate solution.
-
-@smallexample
-  ...
-  if (len < MB_CUR_LEN)
-    @{
-      mbstate_t temp_state;
-      memcpy (&temp_state, &state, sizeof (state));
-      if (wcrtomb (NULL, *ws, &temp_state) > len)
-        @{
-          /* @r{We cannot guarantee that the next}
-             @r{character fits into the buffer, so}
-             @r{return an error.}  */
-          errno = E2BIG;
-          return NULL;
-        @}
-    @}
-  ...
-@end smallexample
-
-Here we do perform the conversion which might overflow the buffer so
-that we are afterwards in the position to make an exact decision about
-the buffer size.  Please note the @code{NULL} argument for the
-destination buffer in the new @code{wcrtomb} call; since we are not
-interested in the converted text at this point this is a nice way to
-express this.  The most unusual thing about this piece of code certainly
-is the duplication of the conversion state object.  But think about
-this: if a change of the state is necessary to emit the next multibyte
-character we want to have the same shift state change performed in the
-real conversion.  Therefore we have to preserve the initial shift state
-information.
-
-There are certainly many more and even better solutions to this problem.
-This example is only meant for educational purposes.
-
-@node Converting Strings
-@subsection Converting Multibyte and Wide Character Strings
-
-The functions described in the previous section only convert a single
-character at a time.  Most operations to be performed in real-world
-programs include strings and therefore the @w{ISO C} standard also
-defines conversions on entire strings.  However, the defined set of
-functions is quite limited, thus the GNU C library contains a few
-extensions which can help in some important situations.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{mbsrtowcs} function (``multibyte string restartable to wide
-character string'') converts an NUL terminated multibyte character
-string at @code{*@var{src}} into an equivalent wide character string,
-including the NUL wide character at the end.  The conversion is started
-using the state information from the object pointed to by @var{ps} or
-from an internal object of @code{mbsrtowcs} if @var{ps} is a null
-pointer.  Before returning the state object to match the state after the
-last converted character.  The state is the initial state if the
-terminating NUL byte is reached and converted.
-
-If @var{dst} is not a null pointer the result is stored in the array
-pointed to by @var{dst}, otherwise the conversion result is not
-available since it is stored in an internal buffer.
-
-If @var{len} wide characters are stored in the array @var{dst} before
-reaching the end of the input string the conversion stops and @var{len}
-is returned.  If @var{dst} is a null pointer @var{len} is never checked.
-
-Another reason for a premature return from the function call is if the
-input string contains an invalid multibyte sequence.  In this case the
-global variable @code{errno} is set to @code{EILSEQ} and the function
-returns @code{(size_t) -1}.
-
-@c XXX The ISO C9x draft seems to have a problem here.  It says that PS
-@c is not updated if DST is NULL.  This is not said straight forward and
-@c none of the other functions is described like this.  It would make sense
-@c to define the function this way but I don't think it is meant like this.
-
-In all other cases the function returns the number of wide characters
-converted during this call.  If @var{dst} is not null @code{mbsrtowcs}
-stores in the pointer pointed to by @var{src} a null pointer (if the NUL
-byte in the input string was reached) or the address of the byte
-following the last converted multibyte character.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-The definition of this function has one limitation which has to be
-understood.  The requirement that @var{dst} has to be a NUL terminated
-string provides problems if one wants to convert buffers with text.  A
-buffer is normally no collection of NUL terminated strings but instead a
-continuous collection of lines, separated by newline characters.  Now
-assume a function to convert one line from a buffer is needed.  Since
-the line is not NUL terminated the source pointer cannot directly point
-into the unmodified text buffer.  This means, either one inserts the NUL
-byte at the appropriate place for the time of the @code{mbsrtowcs}
-function call (which is not doable for a read-only buffer or in a
-multi-threaded application) or one copies the line in an extra buffer
-where it can be terminated by a NUL byte.  Note that it is not in
-general possible to limit the number of characters to convert by setting
-the parameter @var{len} to any specific value.  Since it is not known
-how many bytes each multibyte character sequence is in length one always
-could do only a guess.
-
-@cindex stateful
-There is still a problem with the method of NUL-terminating a line right
-after the newline character which could lead to very strange results.
-As said in the description of the @var{mbsrtowcs} function above the
-conversion state is guaranteed to be in the initial shift state after
-processing the NUL byte at the end of the input string.  But this NUL
-byte is not really part of the text.  I.e., the conversion state after
-the newline in the original text could be something different than the
-initial shift state and therefore the first character of the next line
-is encoded using this state.  But the state in question is never
-accessible to the user since the conversion stops after the NUL byte
-(which resets the state).  Most stateful character sets in use today
-require that the shift state after a newline is the initial state--but
-this is not a strict guarantee.  Therefore simply NUL terminating a
-piece of a running text is not always an adequate solution and therefore
-never should be used in generally used code.
-
-The generic conversion interface (@pxref{Generic Charset Conversion})
-does not have this limitation (it simply works on buffers, not
-strings), and the GNU C library contains a set of functions which take
-additional parameters specifying the maximal number of bytes which are
-consumed from the input string.  This way the problem of
-@code{mbsrtowcs}'s example above could be solved by determining the line
-length and passing this length to the function.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{wcsrtombs} function (``wide character string restartable to
-multibyte string'') converts the NUL terminated wide character string at
-@code{*@var{src}} into an equivalent multibyte character string and
-stores the result in the array pointed to by @var{dst}.  The NUL wide
-character is also converted.  The conversion starts in the state
-described in the object pointed to by @var{ps} or by a state object
-locally to @code{wcsrtombs} in case @var{ps} is a null pointer.  If
-@var{dst} is a null pointer the conversion is performed as usual but the
-result is not available.  If all characters of the input string were
-successfully converted and if @var{dst} is not a null pointer the
-pointer pointed to by @var{src} gets assigned a null pointer.
-
-If one of the wide characters in the input string has no valid multibyte
-character equivalent the conversion stops early, sets the global
-variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}.
-
-Another reason for a premature stop is if @var{dst} is not a null
-pointer and the next converted character would require more than
-@var{len} bytes in total to the array @var{dst}.  In this case (and if
-@var{dest} is not a null pointer) the pointer pointed to by @var{src} is
-assigned a value pointing to the wide character right after the last one
-successfully converted.
-
-Except in the case of an encoding error the return value of the function
-is the number of bytes in all the multibyte character sequences stored
-in @var{dst}.  Before returning the state in the object pointed to by
-@var{ps} (or the internal object in case @var{ps} is a null pointer) is
-updated to reflect the state after the last conversion.  The state is
-the initial shift state in case the terminating NUL wide character was
-converted.
-
-@pindex wchar.h
-This function was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-The restriction mentions above for the @code{mbsrtowcs} function applies
-also here.  There is no possibility to directly control the number of
-input characters.  One has to place the NUL wide character at the
-correct place or control the consumed input indirectly via the available
-output array size (the @var{len} parameter).
-
-@comment wchar.h
-@comment GNU
-@deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs}
-function.  All the parameters are the same except for @var{nmc} which is
-new.  The return value is the same as for @code{mbsrtowcs}.
-
-This new parameter specifies how many bytes at most can be used from the
-multibyte character string.  I.e., the multibyte character string
-@code{*@var{src}} need not be NUL terminated.  But if a NUL byte is
-found within the @var{nmc} first bytes of the string the conversion
-stops here.
-
-This function is a GNU extensions.  It is meant to work around the
-problems mentioned above.  Now it is possible to convert buffer with
-multibyte character text piece for piece without having to care about
-inserting NUL bytes and the effect of NUL bytes on the conversion state.
-@end deftypefun
-
-A function to convert a multibyte string into a wide character string
-and display it could be written like this (this is not a really useful
-example):
-
-@smallexample
-void
-showmbs (const char *src, FILE *fp)
-@{
-  mbstate_t state;
-  int cnt = 0;
-  memset (&state, '\0', sizeof (state));
-  while (1)
-    @{
-      wchar_t linebuf[100];
-      const char *endp = strchr (src, '\n');
-      size_t n;
-
-      /* @r{Exit if there is no more line.}  */
-      if (endp == NULL)
-        break;
-
-      n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
-      linebuf[n] = L'\0';
-      fprintf (fp, "line %d: \"%S\"\n", linebuf);
-    @}
-@}
-@end smallexample
-
-There is no problem with the state after a call to @code{mbsnrtowcs}.
-Since we don't insert characters in the strings which were not in there
-right from the beginning and we use @var{state} only for the conversion
-of the given buffer there is no problem with altering the state.
-
-@comment wchar.h
-@comment GNU
-@deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{wcsnrtombs} function implements the conversion from wide
-character strings to multibyte character strings.  It is similar to
-@code{wcsrtombs} but it takes, just like @code{mbsnrtowcs}, an extra
-parameter which specifies the length of the input string.
-
-No more than @var{nwc} wide characters from the input string
-@code{*@var{src}} are converted.  If the input string contains a NUL
-wide character in the first @var{nwc} character to conversion stops at
-this place.
-
-This function is a GNU extension and just like @code{mbsnrtowcs} is
-helps in situations where no NUL terminated input strings are available.
-@end deftypefun
-
-
-@node Multibyte Conversion Example
-@subsection A Complete Multibyte Conversion Example
-
-The example programs given in the last sections are only brief and do
-not contain all the error checking etc.  Presented here is a complete
-and documented example.  It features the @code{mbrtowc} function but it
-should be easy to derive versions using the other functions.
-
-@smallexample
-int
-file_mbsrtowcs (int input, int output)
-@{
-  /* @r{Note the use of @code{MB_LEN_MAX}.}
-     @r{@code{MB_CUR_MAX} cannot portably be used here.}  */
-  char buffer[BUFSIZ + MB_LEN_MAX];
-  mbstate_t state;
-  int filled = 0;
-  int eof = 0;
-
-  /* @r{Initialize the state.}  */
-  memset (&state, '\0', sizeof (state));
-
-  while (!eof)
-    @{
-      ssize_t nread;
-      ssize_t nwrite;
-      char *inp = buffer;
-      wchar_t outbuf[BUFSIZ];
-      wchar_t *outp = outbuf;
-
-      /* @r{Fill up the buffer from the input file.}  */
-      nread = read (input, buffer + filled, BUFSIZ);
-      if (nread < 0)
-        @{
-          perror ("read");
-          return 0;
-        @}
-      /* @r{If we reach end of file, make a note to read no more.} */
-      if (nread == 0)
-        eof = 1;
-
-      /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */
-      filled += nread;
-
-      /* @r{Convert those bytes to wide characters--as many as we can.} */
-      while (1)
-        @{
-          size_t thislen = mbrtowc (outp, inp, filled, &state);
-          /* @r{Stop converting at invalid character;}
-             @r{this can mean we have read just the first part}
-             @r{of a valid character.}  */
-          if (thislen == (size_t) -1)
-            break;
-          /* @r{We want to handle embedded NUL bytes}
-             @r{but the return value is 0.  Correct this.}  */
-          if (thislen == 0)
-            thislen = 1;
-          /* @r{Advance past this character.} */
-          inp += thislen;
-          filled -= thislen;
-          ++outp;
-        @}
-
-      /* @r{Write the wide characters we just made.}  */
-      nwrite = write (output, outbuf,
-                      (outp - outbuf) * sizeof (wchar_t));
-      if (nwrite < 0)
-        @{
-          perror ("write");
-          return 0;
-        @}
-
-      /* @r{See if we have a @emph{real} invalid character.} */
-      if ((eof && filled > 0) || filled >= MB_CUR_MAX)
-        @{
-          error (0, 0, "invalid multibyte character");
-          return 0;
-        @}
-
-      /* @r{If any characters must be carried forward,}
-         @r{put them at the beginning of @code{buffer}.} */
-      if (filled > 0)
-        memmove (inp, buffer, filled);
-    @}
-
-  return 1;
-@}
-@end smallexample
-
-
-@node Non-reentrant Conversion
-@section Non-reentrant Conversion Function
-
-The functions described in the last chapter are defined in
-@w{Amendment 1} to @w{ISO C90}.  But the original @w{ISO C90} standard also
-contained functions for character set conversion.  The reason that they
-are not described in the first place is that they are almost entirely
-useless.
-
-The problem is that all the functions for conversion defined in @w{ISO
-C90} use a local state.  This implies that multiple conversions at the
-same time (not only when using threads) cannot be done, and that you
-cannot first convert single characters and then strings since you cannot
-tell the conversion functions which state to use.
-
-These functions are therefore usable only in a very limited set of
-situations.  One must complete converting the entire string before
-starting a new one and each string/text must be converted with the same
-function (there is no problem with the library itself; it is guaranteed
-that no library function changes the state of any of these functions).
-@strong{For the above reasons it is highly requested that the functions
-from the last section are used in place of non-reentrant conversion
-functions.}
-
-@menu
-* Non-reentrant Character Conversion::  Non-reentrant Conversion of Single
-                                         Characters.
-* Non-reentrant String Conversion::     Non-reentrant Conversion of Strings.
-* Shift State::                         States in Non-reentrant Functions.
-@end menu
-
-@node Non-reentrant Character Conversion
-@subsection Non-reentrant Conversion of Single Characters
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size})
-The @code{mbtowc} (``multibyte to wide character'') function when called
-with non-null @var{string} converts the first multibyte character
-beginning at @var{string} to its corresponding wide character code.  It
-stores the result in @code{*@var{result}}.
-
-@code{mbtowc} never examines more than @var{size} bytes.  (The idea is
-to supply for @var{size} the number of bytes of data you have in hand.)
-
-@code{mbtowc} with non-null @var{string} distinguishes three
-possibilities: the first @var{size} bytes at @var{string} start with
-valid multibyte character, they start with an invalid byte sequence or
-just part of a character, or @var{string} points to an empty string (a
-null character).
-
-For a valid multibyte character, @code{mbtowc} converts it to a wide
-character and stores that in @code{*@var{result}}, and returns the
-number of bytes in that character (always at least @math{1}, and never
-more than @var{size}).
-
-For an invalid byte sequence, @code{mbtowc} returns @math{-1}.  For an
-empty string, it returns @math{0}, also storing @code{'\0'} in
-@code{*@var{result}}.
-
-If the multibyte character code uses shift characters, then
-@code{mbtowc} maintains and updates a shift state as it scans.  If you
-call @code{mbtowc} with a null pointer for @var{string}, that
-initializes the shift state to its standard initial value.  It also
-returns nonzero if the multibyte character code in use actually has a
-shift state.  @xref{Shift State}.
-@end deftypefun
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar})
-The @code{wctomb} (``wide character to multibyte'') function converts
-the wide character code @var{wchar} to its corresponding multibyte
-character sequence, and stores the result in bytes starting at
-@var{string}.  At most @code{MB_CUR_MAX} characters are stored.
-
-@code{wctomb} with non-null @var{string} distinguishes three
-possibilities for @var{wchar}: a valid wide character code (one that can
-be translated to a multibyte character), an invalid code, and @code{L'\0'}.
-
-Given a valid code, @code{wctomb} converts it to a multibyte character,
-storing the bytes starting at @var{string}.  Then it returns the number
-of bytes in that character (always at least @math{1}, and never more
-than @code{MB_CUR_MAX}).
-
-If @var{wchar} is an invalid wide character code, @code{wctomb} returns
-@math{-1}.  If @var{wchar} is @code{L'\0'}, it returns @code{0}, also
-storing @code{'\0'} in @code{*@var{string}}.
-
-If the multibyte character code uses shift characters, then
-@code{wctomb} maintains and updates a shift state as it scans.  If you
-call @code{wctomb} with a null pointer for @var{string}, that
-initializes the shift state to its standard initial value.  It also
-returns nonzero if the multibyte character code in use actually has a
-shift state.  @xref{Shift State}.
-
-Calling this function with a @var{wchar} argument of zero when
-@var{string} is not null has the side-effect of reinitializing the
-stored shift state @emph{as well as} storing the multibyte character
-@code{'\0'} and returning @math{0}.
-@end deftypefun
-
-Similar to @code{mbrlen} there is also a non-reentrant function which
-computes the length of a multibyte character.  It can be defined in
-terms of @code{mbtowc}.
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int mblen (const char *@var{string}, size_t @var{size})
-The @code{mblen} function with a non-null @var{string} argument returns
-the number of bytes that make up the multibyte character beginning at
-@var{string}, never examining more than @var{size} bytes.  (The idea is
-to supply for @var{size} the number of bytes of data you have in hand.)
-
-The return value of @code{mblen} distinguishes three possibilities: the
-first @var{size} bytes at @var{string} start with valid multibyte
-character, they start with an invalid byte sequence or just part of a
-character, or @var{string} points to an empty string (a null character).
-
-For a valid multibyte character, @code{mblen} returns the number of
-bytes in that character (always at least @code{1}, and never more than
-@var{size}).  For an invalid byte sequence, @code{mblen} returns
-@math{-1}.  For an empty string, it returns @math{0}.
-
-If the multibyte character code uses shift characters, then @code{mblen}
-maintains and updates a shift state as it scans.  If you call
-@code{mblen} with a null pointer for @var{string}, that initializes the
-shift state to its standard initial value.  It also returns a nonzero
-value if the multibyte character code in use actually has a shift state.
-@xref{Shift State}.
-
-@pindex stdlib.h
-The function @code{mblen} is declared in @file{stdlib.h}.
-@end deftypefun
-
-
-@node Non-reentrant String Conversion
-@subsection Non-reentrant Conversion of Strings
-
-For convenience reasons the @w{ISO C90} standard defines also functions
-to convert entire strings instead of single characters.  These functions
-suffer from the same problems as their reentrant counterparts from
-@w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}.
-
-@comment stdlib.h
-@comment ISO
-@deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size})
-The @code{mbstowcs} (``multibyte string to wide character string'')
-function converts the null-terminated string of multibyte characters
-@var{string} to an array of wide character codes, storing not more than
-@var{size} wide characters into the array beginning at @var{wstring}.
-The terminating null character counts towards the size, so if @var{size}
-is less than the actual number of wide characters resulting from
-@var{string}, no terminating null character is stored.
-
-The conversion of characters from @var{string} begins in the initial
-shift state.
-
-If an invalid multibyte character sequence is found, this function
-returns a value of @math{-1}.  Otherwise, it returns the number of wide
-characters stored in the array @var{wstring}.  This number does not
-include the terminating null character, which is present if the number
-is less than @var{size}.
-
-Here is an example showing how to convert a string of multibyte
-characters, allocating enough space for the result.
-
-@smallexample
-wchar_t *
-mbstowcs_alloc (const char *string)
-@{
-  size_t size = strlen (string) + 1;
-  wchar_t *buf = xmalloc (size * sizeof (wchar_t));
-
-  size = mbstowcs (buf, string, size);
-  if (size == (size_t) -1)
-    return NULL;
-  buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
-  return buf;
-@}
-@end smallexample
-
-@end deftypefun
-
-@comment stdlib.h
-@comment ISO
-@deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size})
-The @code{wcstombs} (``wide character string to multibyte string'')
-function converts the null-terminated wide character array @var{wstring}
-into a string containing multibyte characters, storing not more than
-@var{size} bytes starting at @var{string}, followed by a terminating
-null character if there is room.  The conversion of characters begins in
-the initial shift state.
-
-The terminating null character counts towards the size, so if @var{size}
-is less than or equal to the number of bytes needed in @var{wstring}, no
-terminating null character is stored.
-
-If a code that does not correspond to a valid multibyte character is
-found, this function returns a value of @math{-1}.  Otherwise, the
-return value is the number of bytes stored in the array @var{string}.
-This number does not include the terminating null character, which is
-present if the number is less than @var{size}.
-@end deftypefun
-
-@node Shift State
-@subsection States in Non-reentrant Functions
-
-In some multibyte character codes, the @emph{meaning} of any particular
-byte sequence is not fixed; it depends on what other sequences have come
-earlier in the same string.  Typically there are just a few sequences
-that can change the meaning of other sequences; these few are called
-@dfn{shift sequences} and we say that they set the @dfn{shift state} for
-other sequences that follow.
-
-To illustrate shift state and shift sequences, suppose we decide that
-the sequence @code{0200} (just one byte) enters Japanese mode, in which
-pairs of bytes in the range from @code{0240} to @code{0377} are single
-characters, while @code{0201} enters Latin-1 mode, in which single bytes
-in the range from @code{0240} to @code{0377} are characters, and
-interpreted according to the ISO Latin-1 character set.  This is a
-multibyte code which has two alternative shift states (``Japanese mode''
-and ``Latin-1 mode''), and two shift sequences that specify particular
-shift states.
-
-When the multibyte character code in use has shift states, then
-@code{mblen}, @code{mbtowc} and @code{wctomb} must maintain and update
-the current shift state as they scan the string.  To make this work
-properly, you must follow these rules:
-
-@itemize @bullet
-@item
-Before starting to scan a string, call the function with a null pointer
-for the multibyte character address---for example, @code{mblen (NULL,
-0)}.  This initializes the shift state to its standard initial value.
-
-@item
-Scan the string one character at a time, in order.  Do not ``back up''
-and rescan characters already scanned, and do not intersperse the
-processing of different strings.
-@end itemize
-
-Here is an example of using @code{mblen} following these rules:
-
-@smallexample
-void
-scan_string (char *s)
-@{
-  int length = strlen (s);
-
-  /* @r{Initialize shift state.}  */
-  mblen (NULL, 0);
-
-  while (1)
-    @{
-      int thischar = mblen (s, length);
-      /* @r{Deal with end of string and invalid characters.}  */
-      if (thischar == 0)
-        break;
-      if (thischar == -1)
-        @{
-          error ("invalid multibyte character");
-          break;
-        @}
-      /* @r{Advance past this character.}  */
-      s += thischar;
-      length -= thischar;
-    @}
-@}
-@end smallexample
-
-The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not
-reentrant when using a multibyte code that uses a shift state.  However,
-no other library functions call these functions, so you don't have to
-worry that the shift state will be changed mysteriously.
-
-
-@node Generic Charset Conversion
-@section Generic Charset Conversion
-
-The conversion functions mentioned so far in this chapter all had in
-common that they operate on character sets which are not directly
-specified by the functions.  The multibyte encoding used is specified by
-the currently selected locale for the @code{LC_CTYPE} category.  The
-wide character set is fixed by the implementation (in the case of GNU C
-library it always is UCS-4 encoded @w{ISO 10646}.
-
-This has of course several problems when it comes to general character
-conversion:
-
-@itemize @bullet
-@item
-For every conversion where neither the source or destination character
-set is the character set of the locale for the @code{LC_CTYPE} category,
-one has to change the @code{LC_CTYPE} locale using @code{setlocale}.
-
-This introduces major problems for the rest of the programs since
-several more functions (e.g., the character classification functions,
-@pxref{Classification of Characters}) use the @code{LC_CTYPE} category.
-
-@item
-Parallel conversions to and from different character sets are not
-possible since the @code{LC_CTYPE} selection is global and shared by all
-threads.
-
-@item
-If neither the source nor the destination character set is the character
-set used for @code{wchar_t} representation there is at least a two-step
-process necessary to convert a text using the functions above.  One
-would have to select the source character set as the multibyte encoding,
-convert the text into a @code{wchar_t} text, select the destination
-character set as the multibyte encoding and convert the wide character
-text to the multibyte (@math{=} destination) character set.
-
-Even if this is possible (which is not guaranteed) it is a very tiring
-work.  Plus it suffers from the other two raised points even more due to
-the steady changing of the locale.
-@end itemize
-
-
-The XPG2 standard defines a completely new set of functions which has
-none of these limitations.  They are not at all coupled to the selected
-locales and they but no constraints on the character sets selected for
-source and destination.  Only the set of available conversions is
-limiting them.  The standard does not specify that any conversion at all
-must be available.  It is a measure of the quality of the implementation.
-
-In the following text first the interface to @code{iconv}, the
-conversion function, will be described.  Comparisons with other
-implementations will show what pitfalls lie on the way of portable
-applications.  At last, the implementation is described as far as
-interesting to the advanced user who wants to extend the conversion
-capabilities.
-
-@menu
-* Generic Conversion Interface::    Generic Character Set Conversion Interface.
-* iconv Examples::                  A complete @code{iconv} example.
-* Other iconv Implementations::     Some Details about other @code{iconv}
-                                     Implementations.
-* glibc iconv Implementation::      The @code{iconv} Implementation in the GNU C
-                                     library.
-@end menu
-
-@node Generic Conversion Interface
-@subsection Generic Character Set Conversion Interface
-
-This set of functions follows the traditional cycle of using a resource:
-open--use--close.  The interface consists of three functions, each of
-which implement one step.
-
-Before the interfaces are described it is necessary to introduce a
-datatype.  Just like other open--use--close interface the functions
-introduced here work using a handles and the @file{iconv.h} header
-defines a special type for the handles used.
-
-@comment iconv.h
-@comment XPG2
-@deftp {Data Type} iconv_t
-This data type is an abstract type defined in @file{iconv.h}.  The user
-must not assume anything about the definition of this type, it must be
-completely opaque.
-
-Objects of this type can get assigned handles for the conversions using
-the @code{iconv} functions.  The objects themselves need not be freed but
-the conversions for which the handles stand for have to.
-@end deftp
-
-@noindent
-The first step is the function to create a handle.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode})
-The @code{iconv_open} function has to be used before starting a
-conversion.  The two parameters this function takes determine the
-source and destination character set for the conversion and if the
-implementation has the possibility to perform such a conversion the
-function returns a handle.
-
-If the wanted conversion is not available the function returns
-@code{(iconv_t) -1}.  In this case the global variable @code{errno} can
-have the following values:
-
-@table @code
-@item EMFILE
-The process already has @code{OPEN_MAX} file descriptors open.
-@item ENFILE
-The system limit of open file is reached.
-@item ENOMEM
-Not enough memory to carry out the operation.
-@item EINVAL
-The conversion from @var{fromcode} to @var{tocode} is not supported.
-@end table
-
-It is not possible to use the same descriptor in different threads to
-perform independent conversions.  Within the data structures associated
-with the descriptor there is information about the conversion state.
-This must not be messed up by using it in different conversions.
-
-An @code{iconv} descriptor is like a file descriptor as for every use a
-new descriptor must be created.  The descriptor does not stand for all
-of the conversions from @var{fromset} to @var{toset}.
-
-The GNU C library implementation of @code{iconv_open} has one
-significant extension to other implementations.  To ease the extension
-of the set of available conversions the implementation allows storing
-the necessary files with data and code in arbitrarily many directories.
-How this extension has to be written will be explained below
-(@pxref{glibc iconv Implementation}).  Here it is only important to say
-that all directories mentioned in the @code{GCONV_PATH} environment
-variable are considered if they contain a file @file{gconv-modules}.
-These directories need not necessarily be created by the system
-administrator.  In fact, this extension is introduced to help users
-writing and using their own, new conversions.  Of course this does not work
-for security reasons in SUID binaries; in this case only the system
-directory is considered and this normally is
-@file{@var{prefix}/lib/gconv}.  The @code{GCONV_PATH} environment
-variable is examined exactly once at the first call of the
-@code{iconv_open} function.  Later modifications of the variable have no
-effect.
-
-@pindex iconv.h
-This function got introduced early in the X/Open Portability Guide,
-@w{version 2}.  It is supported by all commercial Unices as it is
-required for the Unix branding.  However, the quality and completeness
-of the implementation varies widely.  The function is declared in
-@file{iconv.h}.
-@end deftypefun
-
-The @code{iconv} implementation can associate large data structure with
-the handle returned by @code{iconv_open}.  Therefore it is crucial to
-free all the resources once all conversions are carried out and the
-conversion is not needed anymore.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun int iconv_close (iconv_t @var{cd})
-The @code{iconv_close} function frees all resources associated with the
-handle @var{cd} which must have been returned by a successful call to
-the @code{iconv_open} function.
-
-If the function call was successful the return value is @math{0}.
-Otherwise it is @math{-1} and @code{errno} is set appropriately.
-Defined error are:
-
-@table @code
-@item EBADF
-The conversion descriptor is invalid.
-@end table
-
-@pindex iconv.h
-This function was introduced together with the rest of the @code{iconv}
-functions in XPG2 and it is declared in @file{iconv.h}.
-@end deftypefun
-
-The standard defines only one actual conversion function.  This has
-therefore the most general interface: it allows conversion from one
-buffer to another.  Conversion from a file to a buffer, vice versa, or
-even file to file can be implemented on top of it.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
-@cindex stateful
-The @code{iconv} function converts the text in the input buffer
-according to the rules associated with the descriptor @var{cd} and
-stores the result in the output buffer.  It is possible to call the
-function for the same text several times in a row since for stateful
-character sets the necessary state information is kept in the data
-structures associated with the descriptor.
-
-The input buffer is specified by @code{*@var{inbuf}} and it contains
-@code{*@var{inbytesleft}} bytes.  The extra indirection is necessary for
-communicating the used input back to the caller (see below).  It is
-important to note that the buffer pointer is of type @code{char} and the
-length is measured in bytes even if the input text is encoded in wide
-characters.
-
-The output buffer is specified in a similar way.  @code{*@var{outbuf}}
-points to the beginning of the buffer with at least
-@code{*@var{outbytesleft}} bytes room for the result.  The buffer
-pointer again is of type @code{char} and the length is measured in
-bytes.  If @var{outbuf} or @code{*@var{outbuf}} is a null pointer the
-conversion is performed but no output is available.
-
-If @var{inbuf} is a null pointer the @code{iconv} function performs the
-necessary action to put the state of the conversion into the initial
-state.  This is obviously a no-op for non-stateful encodings, but if the
-encoding has a state such a function call might put some byte sequences
-in the output buffer which perform the necessary state changes.  The
-next call with @var{inbuf} not being a null pointer then simply goes on
-from the initial state.  It is important that the programmer never makes
-any assumption on whether the conversion has to deal with states or not.
-Even if the input and output character sets are not stateful the
-implementation might still have to keep states.  This is due to the
-implementation chosen for the GNU C library as it is described below.
-Therefore an @code{iconv} call to reset the state should always be
-performed if some protocol requires this for the output text.
-
-The conversion stops for three reasons.  The first is that all
-characters from the input buffer are converted.  This actually can mean
-two things: really all bytes from the input buffer are consumed or
-there are some bytes at the end of the buffer which possibly can form a
-complete character but the input is incomplete.  The second reason for a
-stop is when the output buffer is full.  And the third reason is that
-the input contains invalid characters.
-
-In all these cases the buffer pointers after the last successful
-conversion, for input and output buffer, are stored in @var{inbuf} and
-@var{outbuf} and the available room in each buffer is stored in
-@var{inbytesleft} and @var{outbytesleft}.
-
-Since the character sets selected in the @code{iconv_open} call can be
-almost arbitrary there can be situations where the input buffer contains
-valid characters which have no identical representation in the output
-character set.  The behavior in this situation is undefined.  The
-@emph{current} behavior of the GNU C library in this situation is to
-return with an error immediately.  This certainly is not the most
-desirable solution.  Therefore future versions will provide better ones
-but they are not yet finished.
-
-If all input from the input buffer is successfully converted and stored
-in the output buffer the function returns the number of non-reversible
-conversions performed.  In all other cases the return value is
-@code{(size_t) -1} and @code{errno} is set appropriately.  In this case
-the value pointed to by @var{inbytesleft} is nonzero.
-
-@table @code
-@item EILSEQ
-The conversion stopped because of an invalid byte sequence in the input.
-After the call @code{*@var{inbuf}} points at the first byte of the
-invalid byte sequence.
-
-@item E2BIG
-The conversion stopped because it ran out of space in the output buffer.
-
-@item EINVAL
-The conversion stopped because of an incomplete byte sequence at the end
-of the input buffer.
-
-@item EBADF
-The @var{cd} argument is invalid.
-@end table
-
-@pindex iconv.h
-This function was introduced in the XPG2 standard and is declared in the
-@file{iconv.h} header.
-@end deftypefun
-
-The definition of the @code{iconv} function is quite good overall.  It
-provides quite flexible functionality.  The only problems lie in the
-boundary cases which are incomplete byte sequences at the end of the
-input buffer and invalid input.  A third problem, which is not really
-a design problem, is the way conversions are selected.  The standard
-does not say anything about the legitimate names, a minimal set of
-available conversions.  We will see how this negatively impacts other
-implementations, as is demonstrated below.
-
-
-@node iconv Examples
-@subsection A complete @code{iconv} example
-
-The example below features a solution for a common problem.  Given that
-one knows the internal encoding used by the system for @code{wchar_t}
-strings one often is in the position to read text from a file and store
-it in wide character buffers.  One can do this using @code{mbsrtowcs}
-but then we run into the problems discussed above.
-
-@smallexample
-int
-file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
-@{
-  char inbuf[BUFSIZ];
-  size_t insize = 0;
-  char *wrptr = (char *) outbuf;
-  int result = 0;
-  iconv_t cd;
-
-  cd = iconv_open ("WCHAR_T", charset);
-  if (cd == (iconv_t) -1)
-    @{
-      /* @r{Something went wrong.}  */
-      if (errno == EINVAL)
-        error (0, 0, "conversion from '%s' to wchar_t not available",
-               charset);
-      else
-        perror ("iconv_open");
-
-      /* @r{Terminate the output string.}  */
-      *outbuf = L'\0';
-
-      return -1;
-    @}
-
-  while (avail > 0)
-    @{
-      size_t nread;
-      size_t nconv;
-      char *inptr = inbuf;
-
-      /* @r{Read more input.}  */
-      nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
-      if (nread == 0)
-        @{
-          /* @r{When we come here the file is completely read.}
-             @r{This still could mean there are some unused}
-             @r{characters in the @code{inbuf}.  Put them back.}  */
-          if (lseek (fd, -insize, SEEK_CUR) == -1)
-            result = -1;
-
-          /* @r{Now write out the byte sequence to get into the}
-             @r{initial state if this is necessary.}  */
-          iconv (cd, NULL, NULL, &wrptr, &avail);
-
-          break;
-        @}
-      insize += nread;
-
-      /* @r{Do the conversion.}  */
-      nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
-      if (nconv == (size_t) -1)
-        @{
-          /* @r{Not everything went right.  It might only be}
-             @r{an unfinished byte sequence at the end of the}
-             @r{buffer.  Or it is a real problem.}  */
-          if (errno == EINVAL)
-            /* @r{This is harmless.  Simply move the unused}
-               @r{bytes to the beginning of the buffer so that}
-               @r{they can be used in the next round.}  */
-            memmove (inbuf, inptr, insize);
-          else
-            @{
-              /* @r{It is a real problem.  Maybe we ran out of}
-                 @r{space in the output buffer or we have invalid}
-                 @r{input.  In any case back the file pointer to}
-                 @r{the position of the last processed byte.}  */
-              lseek (fd, -insize, SEEK_CUR);
-              result = -1;
-              break;
-            @}
-        @}
-    @}
-
-  /* @r{Terminate the output string.}  */
-  if (avail >= sizeof (wchar_t))
-    *((wchar_t *) wrptr) = L'\0';
-
-  if (iconv_close (cd) != 0)
-    perror ("iconv_close");
-
-  return (wchar_t *) wrptr - outbuf;
-@}
-@end smallexample
-
-@cindex stateful
-This example shows the most important aspects of using the @code{iconv}
-functions.  It shows how successive calls to @code{iconv} can be used to
-convert large amounts of text.  The user does not have to care about
-stateful encodings as the functions take care of everything.
-
-An interesting point is the case where @code{iconv} return an error and
-@code{errno} is set to @code{EINVAL}.  This is not really an error in
-the transformation.  It can happen whenever the input character set
-contains byte sequences of more than one byte for some character and
-texts are not processed in one piece.  In this case there is a chance
-that a multibyte sequence is cut.  The caller than can simply read the
-remainder of the takes and feed the offending bytes together with new
-character from the input to @code{iconv} and continue the work.  The
-internal state kept in the descriptor is @emph{not} unspecified after
-such an event as it is the case with the conversion functions from the
-@w{ISO C} standard.
-
-The example also shows the problem of using wide character strings with
-@code{iconv}.  As explained in the description of the @code{iconv}
-function above the function always takes a pointer to a @code{char}
-array and the available space is measured in bytes.  In the example the
-output buffer is a wide character buffer.  Therefore we use a local
-variable @var{wrptr} of type @code{char *} which is used in the
-@code{iconv} calls.
-
-This looks rather innocent but can lead to problems on platforms which
-have tight restriction on alignment.  Therefore the caller of
-@code{iconv} has to make sure that the pointers passed are suitable for
-access of characters from the appropriate character set.  Since in the
-above case the input parameter to the function is a @code{wchar_t}
-pointer this is the case (unless the user violates alignment when
-computing the parameter).  But in other situations, especially when
-writing generic functions where one does not know what type of character
-set one uses and therefore treats text as a sequence of bytes, it might
-become tricky.
-
-
-@node Other iconv Implementations
-@subsection Some Details about other @code{iconv} Implementations
-
-This is not really the place to discuss the @code{iconv} implementation
-of other systems but it is necessary to know a bit about them to write
-portable programs.  The above mentioned problems with the specification
-of the @code{iconv} functions can lead to portability issues.
-
-The first thing to notice is that due to the large number of character
-sets in use it is certainly not practical to encode the conversions
-directly in the C library.  Therefore the conversion information must
-come from files outside the C library.  This is usually done in one or
-both of the following ways:
-
-@itemize @bullet
-@item
-The C library contains a set of generic conversion functions which can
-read the needed conversion tables and other information from data files.
-These files get loaded when necessary.
-
-This solution is problematic as it requires a great deal of effort to
-apply to all character sets (potentially an infinite set).  The
-differences in the structure of the different character sets is so large
-that many different variants of the table processing functions must be
-developed.  On top of this the generic nature of these functions make
-them slower than specifically implemented functions.
-
-@item
-The C library only contains a framework which can dynamically load
-object files and execute the therein contained conversion functions.
-
-This solution provides much more flexibility.  The C library itself
-contains only very little code and therefore reduces the general memory
-footprint.  Also, with a documented interface between the C library and
-the loadable modules it is possible for third parties to extend the set
-of available conversion modules.  A drawback of this solution is that
-dynamic loading must be available.
-@end itemize
-
-Some implementations in commercial Unices implement a mixture of these
-these possibilities, the majority only the second solution.  Using
-loadable modules moves the code out of the library itself and keeps the
-door open for extensions and improvements.  But this design is also
-limiting on some platforms since not many platforms support dynamic
-loading in statically linked programs.  On platforms without his
-capability it is therefore not possible to use this interface in
-statically linked programs.  The GNU C library has on ELF platforms no
-problems with dynamic loading in in these situations and therefore this
-point is moot.  The danger is that one gets acquainted with this and
-forgets about the restrictions on other systems.
-
-A second thing to know about other @code{iconv} implementations is that
-the number of available conversions is often very limited.  Some
-implementations provide in the standard release (not special
-international or developer releases) at most 100 to 200 conversion
-possibilities.  This does not mean 200 different character sets are
-supported.  E.g., conversions from one character set to a set of, say,
-10 others counts as 10 conversion.  Together with the other direction
-this makes already 20.  One can imagine the thin coverage these platform
-provide.  Some Unix vendors even provide only a handful of conversions
-which renders them useless for almost all uses.
-
-This directly leads to a third and probably the most problematic point.
-The way the @code{iconv} conversion functions are implemented on all
-known Unix system and the availability of the conversion functions from
-character set @math{@cal{A}} to @math{@cal{B}} and the conversion from
-@math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the
-conversion from @math{@cal{A}} to @math{@cal{C}} is available.
-
-This might not seem unreasonable and problematic at first but it is a
-quite big problem as one will notice shortly after hitting it.  To show
-the problem we assume to write a program which has to convert from
-@math{@cal{A}} to @math{@cal{C}}.  A call like
-
-@smallexample
-cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}");
-@end smallexample
-
-@noindent
-does fail according to the assumption above.  But what does the program
-do now?  The conversion is really necessary and therefore simply giving
-up is no possibility.
-
-This is a nuisance.  The @code{iconv} function should take care of this.
-But how should the program proceed from here on?  If it would try to
-convert to character set @math{@cal{B}} first the two @code{iconv_open}
-calls
-
-@smallexample
-cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}");
-@end smallexample
-
-@noindent
-and
-
-@smallexample
-cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}");
-@end smallexample
-
-@noindent
-will succeed but how to find @math{@cal{B}}?
-
-Unfortunately, the answer is: there is no general solution.  On some
-systems guessing might help.  On those systems most character sets can
-convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text.
-Beside this only some very system-specific methods can help.  Since the
-conversion functions come from loadable modules and these modules must
-be stored somewhere in the filesystem, one @emph{could} try to find them
-and determine from the available file which conversions are available
-and whether there is an indirect route from @math{@cal{A}} to
-@math{@cal{C}}.
-
-This shows one of the design errors of @code{iconv} mentioned above.  It
-should at least be possible to determine the list of available
-conversion programmatically so that if @code{iconv_open} says there is
-no such conversion, one could make sure this also is true for indirect
-routes.
-
-
-@node glibc iconv Implementation
-@subsection The @code{iconv} Implementation in the GNU C library
-
-After reading about the problems of @code{iconv} implementations in the
-last section it is certainly good to note that the implementation in
-the GNU C library has none of the problems mentioned above.  What
-follows is a step-by-step analysis of the points raised above.  The
-evaluation is based on the current state of the development (as of
-January 1999).  The development of the @code{iconv} functions is not
-complete, but basic functionality has solidified.
-
-The GNU C library's @code{iconv} implementation uses shared loadable
-modules to implement the conversions.  A very small number of
-conversions are built into the library itself but these are only rather
-trivial conversions.
-
-All the benefits of loadable modules are available in the GNU C library
-implementation.  This is especially appealing since the interface is
-well documented (see below) and it therefore is easy to write new
-conversion modules.  The drawback of using loadable objects is not a
-problem in the GNU C library, at least on ELF systems.  Since the
-library is able to load shared objects even in statically linked
-binaries this means that static linking needs not to be forbidden in
-case one wants to use @code{iconv}.
-
-The second mentioned problem is the number of supported conversions.
-Currently, the GNU C library supports more than 150 character sets.  The
-way the implementation is designed the number of supported conversions
-is greater than 22350 (@math{150} times @math{149}).  If any conversion
-from or to a character set is missing it can easily be added.
-
-Particularly impressive as it may be, this high number is due to the
-fact that the GNU C library implementation of @code{iconv} does not have
-the third problem mentioned above.  I.e., whenever there is a conversion
-from a character set @math{@cal{A}} to @math{@cal{B}} and from
-@math{@cal{B}} to @math{@cal{C}} it is always possible to convert from
-@math{@cal{A}} to @math{@cal{C}} directly.  If the @code{iconv_open}
-returns an error and sets @code{errno} to @code{EINVAL} this really
-means there is no known way, directly or indirectly, to perform the
-wanted conversion.
-
-@cindex triangulation
-This is achieved by providing for each character set a conversion from
-and to UCS-4 encoded @w{ISO 10646}.  Using @w{ISO 10646} as an
-intermediate representation it is possible to @dfn{triangulate}, i.e.,
-converting with an intermediate representation.
-
-There is no inherent requirement to provide a conversion to @w{ISO
-10646} for a new character set and it is also possible to provide other
-conversions where neither source nor destination character set is @w{ISO
-10646}.  The currently existing set of conversions is simply meant to
-cover all conversions which might be of interest.
-
-@cindex ISO-2022-JP
-@cindex EUC-JP
-All currently available conversions use the triangulation method above,
-making conversion run unnecessarily slow.  If, e.g., somebody often
-needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution
-would involve direct conversion between the two character sets, skipping
-the input to @w{ISO 10646} first.  The two character sets of interest
-are much more similar to each other than to @w{ISO 10646}.
-
-In such a situation one can easy write a new conversion and provide it
-as a better alternative.  The GNU C library @code{iconv} implementation
-would automatically use the module implementing the conversion if it is
-specified to be more efficient.
-
-@subsubsection Format of @file{gconv-modules} files
-
-All information about the available conversions comes from a file named
-@file{gconv-modules} which can be found in any of the directories along
-the @code{GCONV_PATH}.  The @file{gconv-modules} files are line-oriented
-text files, where each of the lines has one of the following formats:
-
-@itemize @bullet
-@item
-If the first non-whitespace character is a @kbd{#} the line contains
-only comments and is ignored.
-
-@item
-Lines starting with @code{alias} define an alias name for a character
-set.  There are two more words expected on the line.  The first one
-defines the alias name and the second defines the original name of the
-character set.  The effect is that it is possible to use the alias name
-in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and
-achieve the same result as when using the real character set name.
-
-This is quite important as a character set has often many different
-names.  There is normally always an official name but this need not
-correspond to the most popular name.  Beside this many character sets
-have special names which are somehow constructed.  E.g., all character
-sets specified by the ISO have an alias of the form
-@code{ISO-IR-@var{nnn}} where @var{nnn} is the registration number.
-This allows programs which know about the registration number to
-construct character set names and use them in @code{iconv_open} calls.
-More on the available names and aliases follows below.
-
-@item
-Lines starting with @code{module} introduce an available conversion
-module.  These lines must contain three or four more words.
-
-The first word specifies the source character set, the second word the
-destination character set of conversion implemented in this module.  The
-third word is the name of the loadable module.  The filename is
-constructed by appending the usual shared object suffix (normally
-@file{.so}) and this file is then supposed to be found in the same
-directory the @file{gconv-modules} file is in.  The last word on the
-line, which is optional, is a numeric value representing the cost of the
-conversion.  If this word is missing a cost of @math{1} is assumed.  The
-numeric value itself does not matter that much; what counts are the
-relative values of the sums of costs for all possible conversion paths.
-Below is a more precise description of the use of the cost value.
-@end itemize
-
-Returning to the example above where one has written a module to directly
-convert from ISO-2022-JP to EUC-JP and back.  All what has to be done is
-to put the new module, be its name ISO2022JP-EUCJP.so, in a directory
-and add a file @file{gconv-modules} with the following content in the
-same directory:
-
-@smallexample
-module  ISO-2022-JP//   EUC-JP//        ISO2022JP-EUCJP    1
-module  EUC-JP//        ISO-2022-JP//   ISO2022JP-EUCJP    1
-@end smallexample
-
-To see why this is sufficient, it is necessary to understand how the
-conversion used by @code{iconv} (and described in the descriptor) is
-selected.  The approach to this problem is quite simple.
-
-At the first call of the @code{iconv_open} function the program reads
-all available @file{gconv-modules} files and builds up two tables: one
-containing all the known aliases and another which contains the
-information about the conversions and which shared object implements
-them.
-
-@subsubsection Finding the conversion path in @code{iconv}
-
-The set of available conversions form a directed graph with weighted
-edges.  The weights on the edges are the costs specified in the
-@file{gconv-modules} files.  The @code{iconv_open} function uses an
-algorithm suitable for search for the best path in such a graph and so
-constructs a list of conversions which must be performed in succession
-to get the transformation from the source to the destination character
-set.
-
-Explaining why the above @file{gconv-modules} files allows the
-@code{iconv} implementation to resolve the specific ISO-2022-JP to
-EUC-JP conversion module instead of the conversion coming with the
-library itself is straightforward.  Since the latter conversion takes two
-steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to
-EUC-JP) the cost is @math{1+1 = 2}.  But the above @file{gconv-modules}
-file specifies that the new conversion modules can perform this
-conversion with only the cost of @math{1}.
-
-A mysterious piece about the @file{gconv-modules} file above (and also
-the file coming with the GNU C library) are the names of the character
-sets specified in the @code{module} lines.  Why do almost all the names
-end in @code{//}?  And this is not all: the names can actually be
-regular expressions.  At this point of time this mystery should not be
-revealed, unless you have the relevant spell-casting materials: ashes
-from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix
-blessed by St.@: Emacs, assorted herbal roots from Central America, sand
-from Cebu, etc.  Sorry!  @strong{The part of the implementation where
-this is used is not yet finished.  For now please simply follow the
-existing examples.  It'll become clearer once it is. --drepper}
-
-A last remark about the @file{gconv-modules} is about the names not
-ending with @code{//}.  There often is a character set named
-@code{INTERNAL} mentioned.  From the discussion above and the chosen
-name it should have become clear that this is the name for the
-representation used in the intermediate step of the triangulation.  We
-have said that this is UCS-4 but actually it is not quite right.  The
-UCS-4 specification also includes the specification of the byte ordering
-used.  Since a UCS-4 value consists of four bytes a stored value is
-effected by byte ordering.  The internal representation is @emph{not}
-the same as UCS-4 in case the byte ordering of the processor (or at least
-the running process) is not the same as the one required for UCS-4.  This
-is done for performance reasons as one does not want to perform
-unnecessary byte-swapping operations if one is not interested in actually
-seeing the result in UCS-4.  To avoid trouble with endianess the internal
-representation consistently is named @code{INTERNAL} even on big-endian
-systems where the representations are identical.
-
-@subsubsection @code{iconv} module data structures
-
-So far this section described how modules are located and considered to
-be used.  What remains to be described is the interface of the modules
-so that one can write new ones.  This section describes the interface as
-it is in use in January 1999.  The interface will change in future a bit
-but hopefully only in an upward compatible way.
-
-The definitions necessary to write new modules are publicly available
-in the non-standard header @file{gconv.h}.  The following text will
-therefore describe the definitions from this header file.  But first it
-is necessary to get an overview.
-
-From the perspective of the user of @code{iconv} the interface is quite
-simple: the @code{iconv_open} function returns a handle which can be
-used in calls to @code{iconv} and finally the handle is freed with a call
-to @code{iconv_close}.  The problem is: the handle has to be able to
-represent the possibly long sequences of conversion steps and also the
-state of each conversion since the handle is all which is passed to the
-@code{iconv} function.  Therefore the data structures are really the
-elements to understanding the implementation.
-
-We need two different kinds of data structures.  The first describes the
-conversion and the second describes the state etc.  There are really two
-type definitions like this in @file{gconv.h}.
-@pindex gconv.h
-
-@comment gconv.h
-@comment GNU
-@deftp {Data type} {struct __gconv_step}
-This data structure describes one conversion a module can perform.  For
-each function in a loaded module with conversion functions there is
-exactly one object of this type.  This object is shared by all users of
-the conversion.  I.e., this object does not contain any information
-corresponding to an actual conversion.  It only describes the conversion
-itself.
-
-@table @code
-@item struct __gconv_loaded_object *__shlib_handle
-@itemx const char *__modname
-@itemx int __counter
-All these elements of the structure are used internally in the C library
-to coordinate loading and unloading the shared.  One must not expect any
-of the other elements be available or initialized.
-
-@item const char *__from_name
-@itemx const char *__to_name
-@code{__from_name} and @code{__to_name} contain the names of the source and
-destination character sets.  They can be used to identify the actual
-conversion to be carried out since one module might implement
-conversions for more than one character set and/or direction.
-
-@item gconv_fct __fct
-@itemx gconv_init_fct __init_fct
-@itemx gconv_end_fct __end_fct
-These elements contain pointers to the functions in the loadable module.
-The interface will be explained below.
-
-@item int __min_needed_from
-@itemx int __max_needed_from
-@itemx int __min_needed_to
-@itemx int __max_needed_to;
-These values have to be filled in the init function of the module.  The
-@code{__min_needed_from} value specifies how many bytes a character of
-the source character set at least needs.  The @code{__max_needed_from}
-specifies the maximum value which also includes possible shift
-sequences.
-
-The @code{__min_needed_to} and @code{__max_needed_to} values serve the
-same purpose but this time for the destination character set.
-
-It is crucial that these values are accurate since otherwise the
-conversion functions will have problems or not work at all.
-
-@item int __stateful
-This element must also be initialized by the init function.  It is
-nonzero if the source character set is stateful.  Otherwise it is zero.
-
-@item void *__data
-This element can be used freely by the conversion functions in the
-module.  It can be used to communicate extra information from one call
-to another.  It need not be initialized if not needed at all.  If this
-element gets assigned a pointer to dynamically allocated memory
-(presumably in the init function) it has to be made sure that the end
-function deallocates the memory.  Otherwise the application will leak
-memory.
-
-It is important to be aware that this data structure is shared by all
-users of this specification conversion and therefore the @code{__data}
-element must not contain data specific to one specific use of the
-conversion function.
-@end table
-@end deftp
-
-@comment gconv.h
-@comment GNU
-@deftp {Data type} {struct __gconv_step_data}
-This is the data structure which contains the information specific to
-each use of the conversion functions.
-
-@table @code
-@item char *__outbuf
-@itemx char *__outbufend
-These elements specify the output buffer for the conversion step.  The
-@code{__outbuf} element points to the beginning of the buffer and
-@code{__outbufend} points to the byte following the last byte in the
-buffer.  The conversion function must not assume anything about the size
-of the buffer but it can be safely assumed the there is room for at
-least one complete character in the output buffer.
-
-Once the conversion is finished and the conversion is the last step the
-@code{__outbuf} element must be modified to point after last last byte
-written into the buffer to signal how much output is available.  If this
-conversion step is not the last one the element must not be modified.
-The @code{__outbufend} element must not be modified.
-
-@item int __is_last
-This element is nonzero if this conversion step is the last one.  This
-information is necessary for the recursion.  See the description of the
-conversion function internals below.  This element must never be
-modified.
-
-@item int __invocation_counter
-The conversion function can use this element to see how many calls of
-the conversion function already happened.  Some character sets require
-when generating output a certain prolog and by comparing this value with
-zero one can find out whether it is the first call and therefore the
-prolog should be emitted or not.  This element must never be modified.
-
-@item int __internal_use
-This element is another one rarely used but needed in certain
-situations.  It got assigned a nonzero value in case the conversion
-functions are used to implement @code{mbsrtowcs} et.al.  I.e., the
-function is not used directly through the @code{iconv} interface.
-
-This sometimes makes a difference as it is expected that the
-@code{iconv} functions are used to translate entire texts while the
-@code{mbsrtowcs} functions are normally only used to convert single
-strings and might be used multiple times to convert entire texts.
-
-But in this situation we would have problem complying with some rules of
-the character set specification.  Some character sets require a prolog
-which must appear exactly once for an entire text.  If a number of
-@code{mbsrtowcs} calls are used to convert the text only the first call
-must add the prolog.  But since there is no communication between the
-different calls of @code{mbsrtowcs} the conversion functions have no
-possibility to find this out.  The situation is different for sequences
-of @code{iconv} calls since the handle allows access to the needed
-information.
-
-This element is mostly used together with @code{__invocation_counter} in
-a way like this:
-
-@smallexample
-if (!data->__internal_use
-     && data->__invocation_counter == 0)
-  /* @r{Emit prolog.}  */
-  ...
-@end smallexample
-
-This element must never be modified.
-
-@item mbstate_t *__statep
-The @code{__statep} element points to an object of type @code{mbstate_t}
-(@pxref{Keeping the state}).  The conversion of an stateful character
-set must use the object pointed to by this element to store information
-about the conversion state.  The @code{__statep} element itself must
-never be modified.
-
-@item mbstate_t __state
-This element @emph{never} must be used directly.  It is only part of
-this structure to have the needed space allocated.
-@end table
-@end deftp
-
-@subsubsection @code{iconv} module interfaces
-
-With the knowledge about the data structures we now can describe the
-conversion functions itself.  To understand the interface a bit of
-knowledge about the functionality in the C library which loads the
-objects with the conversions is necessary.
-
-It is often the case that one conversion is used more than once.  I.e.,
-there are several @code{iconv_open} calls for the same set of character
-sets during one program run.  The @code{mbsrtowcs} et.al.@: functions in
-the GNU C library also use the @code{iconv} functionality which
-increases the number of uses of the same functions even more.
-
-For this reason the modules do not get loaded exclusively for one
-conversion.  Instead a module once loaded can be used by arbitrarily many
-@code{iconv} or @code{mbsrtowcs} calls at the same time.  The splitting
-of the information between conversion function specific information and
-conversion data makes this possible.  The last section showed the two
-data structures used to do this.
-
-This is of course also reflected in the interface and semantics of the
-functions the modules must provide.  There are three functions which
-must have the following names:
-
-@table @code
-@item gconv_init
-The @code{gconv_init} function initializes the conversion function
-specific data structure.  This very same object is shared by all
-conversion which use this conversion and therefore no state information
-about the conversion itself must be stored in here.  If a module
-implements more than one conversion the @code{gconv_init} function will be
-called multiple times.
-
-@item gconv_end
-The @code{gconv_end} function is responsible to free all resources
-allocated by the @code{gconv_init} function.  If there is nothing to do
-this function can be missing.  Special care must be taken if the module
-implements more than one conversion and the @code{gconv_init} function
-does not allocate the same resources for all conversions.
-
-@item gconv
-This is the actual conversion function.  It is called to convert one
-block of text.  It gets passed the conversion step information
-initialized by @code{gconv_init} and the conversion data, specific to
-this use of the conversion functions.
-@end table
-
-There are three data types defined for the three module interface
-function and these define the interface.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *)
-This specifies the interface of the initialization function of the
-module.  It is called exactly once for each conversion the module
-implements.
-
-As explained int the description of the @code{struct __gconv_step} data
-structure above the initialization function has to initialize parts of
-it.
-
-@table @code
-@item __min_needed_from
-@itemx __max_needed_from
-@itemx __min_needed_to
-@itemx __max_needed_to
-These elements must be initialized to the exact numbers of the minimum
-and maximum number of bytes used by one character in the source and
-destination character set respectively.  If the characters all have the
-same size the minimum and maximum values are the same.
-
-@item __stateful
-This element must be initialized to an nonzero value if the source
-character set is stateful.  Otherwise it must be zero.
-@end table
-
-If the initialization function needs to communication some information
-to the conversion function this can happen using the @code{__data}
-element of the @code{__gconv_step} structure.  But since this data is
-shared by all the conversion is must not be modified by the conversion
-function.  How this can be used is shown in the example below.
-
-@smallexample
-#define MIN_NEEDED_FROM         1
-#define MAX_NEEDED_FROM         4
-#define MIN_NEEDED_TO           4
-#define MAX_NEEDED_TO           4
-
-int
-gconv_init (struct __gconv_step *step)
-@{
-  /* @r{Determine which direction.}  */
-  struct iso2022jp_data *new_data;
-  enum direction dir = illegal_dir;
-  enum variant var = illegal_var;
-  int result;
-
-  if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)
-    @{
-      dir = from_iso2022jp;
-      var = iso2022jp;
-    @}
-  else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)
-    @{
-      dir = to_iso2022jp;
-      var = iso2022jp;
-    @}
-  else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)
-    @{
-      dir = from_iso2022jp;
-      var = iso2022jp2;
-    @}
-  else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)
-    @{
-      dir = to_iso2022jp;
-      var = iso2022jp2;
-    @}
-
-  result = __GCONV_NOCONV;
-  if (dir != illegal_dir)
-    @{
-      new_data = (struct iso2022jp_data *)
-        malloc (sizeof (struct iso2022jp_data));
-
-      result = __GCONV_NOMEM;
-      if (new_data != NULL)
-        @{
-          new_data->dir = dir;
-          new_data->var = var;
-          step->__data = new_data;
-
-          if (dir == from_iso2022jp)
-	    @{
-              step->__min_needed_from = MIN_NEEDED_FROM;
-              step->__max_needed_from = MAX_NEEDED_FROM;
-              step->__min_needed_to = MIN_NEEDED_TO;
-              step->__max_needed_to = MAX_NEEDED_TO;
-	    @}
-          else
-            @{
-              step->__min_needed_from = MIN_NEEDED_TO;
-              step->__max_needed_from = MAX_NEEDED_TO;
-              step->__min_needed_to = MIN_NEEDED_FROM;
-              step->__max_needed_to = MAX_NEEDED_FROM + 2;
-            @}
-
-          /* @r{Yes, this is a stateful encoding.}  */
-          step->__stateful = 1;
-
-          result = __GCONV_OK;
-        @}
-    @}
-
-  return result;
-@}
-@end smallexample
-
-The function first checks which conversion is wanted.  The module from
-which this function is taken implements four different conversion and
-which one is selected can be determined by comparing the names.  The
-comparison should always be done without paying attention to the case.
-
-Then a data structure is allocated which contains the necessary
-information about which conversion is selected.  The data structure
-@code{struct iso2022jp_data} is locally defined since outside the module
-this data is not used at all.  Please note that if all four conversions
-this modules supports are requested there are four data blocks.
-
-One interesting thing is the initialization of the @code{__min_} and
-@code{__max_} elements of the step data object.  A single ISO-2022-JP
-character can consist of one to four bytes.  Therefore the
-@code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined
-this way.  The output is always the @code{INTERNAL} character set (aka
-UCS-4) and therefore each character consists of exactly four bytes.  For
-the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into
-account that escape sequences might be necessary to switch the character
-sets.  Therefore the @code{__max_needed_to} element for this direction
-gets assigned @code{MAX_NEEDED_FROM + 2}.  This takes into account the
-two bytes needed for the escape sequences to single the switching.  The
-asymmetry in the maximum values for the two directions can be explained
-easily: when reading ISO-2022-JP text escape sequences can be handled
-alone.  I.e., it is not necessary to process a real character since the
-effect of the escape sequence can be recorded in the state information.
-The situation is different for the other direction.  Since it is in
-general not known which character comes next one cannot emit escape
-sequences to change the state in advance.  This means the escape
-sequences which have to be emitted together with the next character.
-Therefore one needs more room then only for the character itself.
-
-The possible return values of the initialization function are:
-
-@table @code
-@item __GCONV_OK
-The initialization succeeded
-@item __GCONV_NOCONV
-The requested conversion is not supported in the module.  This can
-happen if the @file{gconv-modules} file has errors.
-@item __GCONV_NOMEM
-Memory required to store additional information could not be allocated.
-@end table
-@end deftypevr
-
-The functions called before the module is unloaded is significantly
-easier.  It often has nothing at all to do in which case it can be left
-out completely.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *)
-The task of this function is it to free all resources allocated in the
-initialization function.  Therefore only the @code{__data} element of
-the object pointed to by the argument is of interest.  Continuing the
-example from the initialization function, the finalization function
-looks like this:
-
-@smallexample
-void
-gconv_end (struct __gconv_step *data)
-@{
-  free (data->__data);
-@}
-@end smallexample
-@end deftypevr
-
-The most important function is the conversion function itself.  It can
-get quite complicated for complex character sets.  But since this is not
-of interest here we will only describe a possible skeleton for the
-conversion function.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int)
-The conversion function can be called for two basic reason: to convert
-text or to reset the state.  From the description of the @code{iconv}
-function it can be seen why the flushing mode is necessary.  What mode
-is selected is determined by the sixth argument, an integer.  If it is
-nonzero it means that flushing is selected.
-
-Common to both mode is where the output buffer can be found.  The
-information about this buffer is stored in the conversion step data.  A
-pointer to this is passed as the second argument to this function.  The
-description of the @code{struct __gconv_step_data} structure has more
-information on this.
-
-@cindex stateful
-What has to be done for flushing depends on the source character set.
-If it is not stateful nothing has to be done.  Otherwise the function
-has to emit a byte sequence to bring the state object in the initial
-state.  Once this all happened the other conversion modules in the chain
-of conversions have to get the same chance.  Whether another step
-follows can be determined from the @code{__is_last} element of the step
-data structure to which the first parameter points.
-
-The more interesting mode is when actually text has to be converted.
-The first step in this case is to convert as much text as possible from
-the input buffer and store the result in the output buffer.  The start
-of the input buffer is determined by the third argument which is a
-pointer to a pointer variable referencing the beginning of the buffer.
-The fourth argument is a pointer to the byte right after the last byte
-in the buffer.
-
-The conversion has to be performed according to the current state if the
-character set is stateful.  The state is stored in an object pointed to
-by the @code{__statep} element of the step data (second argument).  Once
-either the input buffer is empty or the output buffer is full the
-conversion stops.  At this point the pointer variable referenced by the
-third parameter must point to the byte following the last processed
-byte.  I.e., if all of the input is consumed this pointer and the fourth
-parameter have the same value.
-
-What now happens depends on whether this step is the last one or not.
-If it is the last step the only thing which has to be done is to update
-the @code{__outbuf} element of the step data structure to point after the
-last written byte.  This gives the caller the information on how much
-text is available in the output buffer.  Beside this the variable
-pointed to by the fifth parameter, which is of type @code{size_t}, must
-be incremented by the number of characters (@emph{not bytes}) which were
-converted in a non-reversible way.  Then the function can return.
-
-In case the step is not the last one the later conversion functions have
-to get a chance to do their work.  Therefore the appropriate conversion
-function has to be called.  The information about the functions is
-stored in the conversion data structures, passed as the first parameter.
-This information and the step data are stored in arrays so the next
-element in both cases can be found by simple pointer arithmetic:
-
-@smallexample
-int
-gconv (struct __gconv_step *step, struct __gconv_step_data *data,
-       const char **inbuf, const char *inbufend, size_t *written,
-       int do_flush)
-@{
-  struct __gconv_step *next_step = step + 1;
-  struct __gconv_step_data *next_data = data + 1;
-  ...
-@end smallexample
-
-The @code{next_step} pointer references the next step information and
-@code{next_data} the next data record.  The call of the next function
-therefore will look similar to this:
-
-@smallexample
-  next_step->__fct (next_step, next_data, &outerr, outbuf,
-                    written, 0)
-@end smallexample
-
-But this is not yet all.  Once the function call returns the conversion
-function might have some more to do.  If the return value of the
-function is @code{__GCONV_EMPTY_INPUT} this means there is more room in
-the output buffer.  Unless the input buffer is empty the conversion
-functions start all over again and processes the rest of the input
-buffer.  If the return value is not @code{__GCONV_EMPTY_INPUT} something
-went wrong and we have to recover from this.
-
-A requirement for the conversion function is that the input buffer
-pointer (the third argument) always points to the last character which
-was put in the converted form in the output buffer.  This is trivially
-true after the conversion performed in the current step.  But if the
-conversion functions deeper down the stream stop prematurely not all
-characters from the output buffer are consumed and therefore the input
-buffer pointers must be backed of to the right position.
-
-This is easy to do if the input and output character sets have a fixed
-width for all characters.  In this situation we can compute how many
-characters are left in the output buffer and therefore can correct the
-input buffer pointer appropriate with a similar computation.  Things are
-getting tricky if either character set has character represented with
-variable length byte sequences and it gets even more complicated if the
-conversion has to take care of the state.  In these cases the conversion
-has to be performed once again, from the known state before the initial
-conversion.  I.e., if necessary the state of the conversion has to be
-reset and the conversion loop has to be executed again.  The difference
-now is that it is known how much input must be created and the
-conversion can stop before converting the first unused character.  Once
-this is done the input buffer pointers must be updated again and the
-function can return.
-
-One final thing should be mentioned.  If it is necessary for the
-conversion to know whether it is the first invocation (in case a prolog
-has to be emitted) the conversion function should just before returning
-to the caller increment the @code{__invocation_counter} element of the
-step data structure.  See the description of the @code{struct
-__gconv_step_data} structure above for more information on how this can
-be used.
-
-The return value must be one of the following values:
-
-@table @code
-@item __GCONV_EMPTY_INPUT
-All input was consumed and there is room left in the output buffer.
-@item __GCONV_FULL_OUTPUT
-No more room in the output buffer.  In case this is not the last step
-this value is propagated down from the call of the next conversion
-function in the chain.
-@item __GCONV_INCOMPLETE_INPUT
-The input buffer is not entirely empty since it contains an incomplete
-character sequence.
-@end table
-
-The following example provides a framework for a conversion function.
-In case a new conversion has to be written the holes in this
-implementation have to be filled and that is it.
-
-@smallexample
-int
-gconv (struct __gconv_step *step, struct __gconv_step_data *data,
-       const char **inbuf, const char *inbufend, size_t *written,
-       int do_flush)
-@{
-  struct __gconv_step *next_step = step + 1;
-  struct __gconv_step_data *next_data = data + 1;
-  gconv_fct fct = next_step->__fct;
-  int status;
-
-  /* @r{If the function is called with no input this means we have}
-     @r{to reset to the initial state.  The possibly partly}
-     @r{converted input is dropped.}  */
-  if (do_flush)
-    @{
-      status = __GCONV_OK;
-
-      /* @r{Possible emit a byte sequence which put the state object}
-         @r{into the initial state.}  */
-
-      /* @r{Call the steps down the chain if there are any but only}
-         @r{if we successfully emitted the escape sequence.}  */
-      if (status == __GCONV_OK && ! data->__is_last)
-        status = fct (next_step, next_data, NULL, NULL,
-                      written, 1);
-    @}
-  else
-    @{
-      /* @r{We preserve the initial values of the pointer variables.}  */
-      const char *inptr = *inbuf;
-      char *outbuf = data->__outbuf;
-      char *outend = data->__outbufend;
-      char *outptr;
-
-      do
-        @{
-          /* @r{Remember the start value for this round.}  */
-          inptr = *inbuf;
-          /* @r{The outbuf buffer is empty.}  */
-          outptr = outbuf;
-
-          /* @r{For stateful encodings the state must be safe here.}  */
-
-          /* @r{Run the conversion loop.  @code{status} is set}
-             @r{appropriately afterwards.}  */
-
-          /* @r{If this is the last step leave the loop, there is}
-             @r{nothing we can do.}  */
-          if (data->__is_last)
-            @{
-              /* @r{Store information about how many bytes are}
-                 @r{available.}  */
-              data->__outbuf = outbuf;
-
-             /* @r{If any non-reversible conversions were performed,}
-                @r{add the number to @code{*written}.}  */
-
-             break;
-           @}
-
-          /* @r{Write out all output which was produced.}  */
-          if (outbuf > outptr)
-            @{
-              const char *outerr = data->__outbuf;
-              int result;
-
-              result = fct (next_step, next_data, &outerr,
-                            outbuf, written, 0);
-
-              if (result != __GCONV_EMPTY_INPUT)
-                @{
-                  if (outerr != outbuf)
-                    @{
-                      /* @r{Reset the input buffer pointer.  We}
-                         @r{document here the complex case.}  */
-                      size_t nstatus;
-
-                      /* @r{Reload the pointers.}  */
-                      *inbuf = inptr;
-                      outbuf = outptr;
-
-                      /* @r{Possibly reset the state.}  */
-
-                      /* @r{Redo the conversion, but this time}
-                         @r{the end of the output buffer is at}
-                         @r{@code{outerr}.}  */
-                    @}
-
-                  /* @r{Change the status.}  */
-                  status = result;
-                @}
-              else
-                /* @r{All the output is consumed, we can make}
-                   @r{ another run if everything was ok.}  */
-                if (status == __GCONV_FULL_OUTPUT)
-                  status = __GCONV_OK;
-           @}
-        @}
-      while (status == __GCONV_OK);
-
-      /* @r{We finished one use of this step.}  */
-      ++data->__invocation_counter;
-    @}
-
-  return status;
-@}
-@end smallexample
-@end deftypevr
-
-This information should be sufficient to write new modules.  Anybody
-doing so should also take a look at the available source code in the GNU
-C library sources.  It contains many examples of working and optimized
-modules.
+@node Character Set Handling, Locales, String and Array Utilities, Top

+@c %MENU% Support for extended character sets

+@chapter Character Set Handling

+

+@ifnottex

+@macro cal{text}

+\text\

+@end macro

+@end ifnottex

+

+Character sets used in the early days of computing had only six, seven,

+or eight bits for each character: there was never a case where more than

+eight bits (one byte) were used to represent a single character.  The

+limitations of this approach became more apparent as more people

+grappled with non-Roman character sets, where not all the characters

+that make up a language's character set can be represented by @math{2^8}

+choices.  This chapter shows the functionality that was added to the C

+library to support multiple character sets.

+

+@menu

+* Extended Char Intro::              Introduction to Extended Characters.

+* Charset Function Overview::        Overview about Character Handling

+                                      Functions.

+* Restartable multibyte conversion:: Restartable multibyte conversion

+                                      Functions.

+* Non-reentrant Conversion::         Non-reentrant Conversion Function.

+* Generic Charset Conversion::       Generic Charset Conversion.

+@end menu

+

+

+@node Extended Char Intro

+@section Introduction to Extended Characters

+

+A variety of solutions is available to overcome the differences between

+character sets with a 1:1 relation between bytes and characters and

+character sets with ratios of 2:1 or 4:1. The remainder of this

+section gives a few examples to help understand the design decisions

+made while developing the functionality of the @w{C library}.

+

+@cindex internal representation

+A distinction we have to make right away is between internal and

+external representation.  @dfn{Internal representation} means the

+representation used by a program while keeping the text in memory.

+External representations are used when text is stored or transmitted

+through some communication channel.  Examples of external

+representations include files waiting in a directory to be

+read and parsed.

+

+Traditionally there has been no difference between the two representations.

+It was equally comfortable and useful to use the same single-byte

+representation internally and externally.  This comfort level decreases

+with more and larger character sets.

+

+One of the problems to overcome with the internal representation is

+handling text that is externally encoded using different character

+sets.  Assume a program that reads two texts and compares them using

+some metric.  The comparison can be usefully done only if the texts are

+internally kept in a common format.

+

+@cindex wide character

+For such a common format (@math{=} character set) eight bits are certainly

+no longer enough.  So the smallest entity will have to grow: @dfn{wide

+characters} will now be used.  Instead of one byte per character, two or

+four will be used instead.  (Three are not good to address in memory and

+more than four bytes seem not to be necessary).

+

+@cindex Unicode

+@cindex ISO 10646

+As shown in some other part of this manual,

+@c !!! Ahem, wide char string functions are not yet covered -- drepper

+a completely new family has been created of functions that can handle wide

+character texts in memory. The most commonly used character sets for such

+internal wide character representations are Unicode and @w{ISO 10646}

+(also known as UCS for Universal Character Set). Unicode was originally

+planned as a 16-bit character set; whereas, @w{ISO 10646} was designed to

+be a 31-bit large code space. The two standards are practically identical.

+They have the same character repertoire and code table, but Unicode specifies

+added semantics.  At the moment, only characters in the first @code{0x10000}

+code positions (the so-called Basic Multilingual Plane, BMP) have been

+assigned, but the assignment of more specialized characters outside this

+16-bit space is already in progress. A number of encodings have been

+defined for Unicode and @w{ISO 10646} characters:

+@cindex UCS-2

+@cindex UCS-4

+@cindex UTF-8

+@cindex UTF-16

+UCS-2 is a 16-bit word that can only represent characters

+from the BMP, UCS-4 is a 32-bit word than can represent any Unicode

+and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where

+ASCII characters are represented by ASCII bytes and non-ASCII characters

+by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension

+of UCS-2 in which pairs of certain UCS-2 words can be used to encode

+non-BMP characters up to @code{0x10ffff}.

+

+To represent wide characters the @code{char} type is not suitable.  For

+this reason the @w{ISO C} standard introduces a new type that is

+designed to keep one character of a wide character string.  To maintain

+the similarity there is also a type corresponding to @code{int} for

+those functions that take a single wide character.

+

+@comment stddef.h

+@comment ISO

+@deftp {Data type} wchar_t

+This data type is used as the base type for wide character strings.

+I.e., arrays of objects of this type are the equivalent of @code{char[]}

+for multibyte character strings.  The type is defined in @file{stddef.h}.

+

+The @w{ISO C90} standard, where @code{wchar_t} was introduced, does not

+say anything specific about the representation.  It only requires that

+this type is capable of storing all elements of the basic character set.

+Therefore it would be legitimate to define @code{wchar_t} as @code{char},

+which might make sense for embedded systems.

+

+But for GNU systems @code{wchar_t} is always 32 bits wide and, therefore,

+capable of representing all UCS-4 values and, therefore, covering all of

+@w{ISO 10646}.  Some Unix systems define @code{wchar_t} as a 16-bit type

+and thereby follow Unicode very strictly. This definition is perfectly

+fine with the standard, but it also means that to represent all

+characters from Unicode and @w{ISO 10646} one has to use UTF-16 surrogate

+characters, which is in fact a multi-wide-character encoding. But

+resorting to multi-wide-character encoding contradicts the purpose of the

+@code{wchar_t} type.

+@end deftp

+

+@comment wchar.h

+@comment ISO

+@deftp {Data type} wint_t

+@code{wint_t} is a data type used for parameters and variables that

+contain a single wide character. As the name suggests this type is the

+equivalent of @code{int} when using the normal @code{char} strings.  The

+types @code{wchar_t} and @code{wint_t} often have the same

+representation if their size is 32 bits wide but if @code{wchar_t} is

+defined as @code{char} the type @code{wint_t} must be defined as

+@code{int} due to the parameter promotion.

+

+@pindex wchar.h

+This type is defined in @file{wchar.h} and was introduced in

+@w{Amendment 1} to @w{ISO C90}.

+@end deftp

+

+As there are for the @code{char} data type macros are available for

+specifying the minimum and maximum value representable in an object of

+type @code{wchar_t}.

+

+@comment wchar.h

+@comment ISO

+@deftypevr Macro wint_t WCHAR_MIN

+The macro @code{WCHAR_MIN} evaluates to the minimum value representable

+by an object of type @code{wint_t}.

+

+This macro was introduced in @w{Amendment 1} to @w{ISO C90}.

+@end deftypevr

+

+@comment wchar.h

+@comment ISO

+@deftypevr Macro wint_t WCHAR_MAX

+The macro @code{WCHAR_MAX} evaluates to the maximum value representable

+by an object of type @code{wint_t}.

+

+This macro was introduced in @w{Amendment 1} to @w{ISO C90}.

+@end deftypevr

+

+Another special wide character value is the equivalent to @code{EOF}.

+

+@comment wchar.h

+@comment ISO

+@deftypevr Macro wint_t WEOF

+The macro @code{WEOF} evaluates to a constant expression of type

+@code{wint_t} whose value is different from any member of the extended

+character set.

+

+@code{WEOF} need not be the same value as @code{EOF} and unlike

+@code{EOF} it also need @emph{not} be negative.  I.e., sloppy code like

+

+@smallexample

+@{

+  int c;

+  ...

+  while ((c = getc (fp)) < 0)

+    ...

+@}

+@end smallexample

+

+@noindent

+has to be rewritten to use @code{WEOF} explicitly when wide characters

+are used:

+

+@smallexample

+@{

+  wint_t c;

+  ...

+  while ((c = wgetc (fp)) != WEOF)

+    ...

+@}

+@end smallexample

+

+@pindex wchar.h

+This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is

+defined in @file{wchar.h}.

+@end deftypevr

+

+

+These internal representations present problems when it comes to storing

+and transmittal. Because each single wide character consists of more

+than one byte, they are effected by byte-ordering.  Thus, machines with

+different endianesses would see different values when accessing the same

+data. This byte ordering concern also applies for communication protocols

+that are all byte-based and, thereforet require that the sender has to

+decide about splitting the wide character in bytes. A last (but not least

+important) point is that wide characters often require more storage space

+than a customized byte-oriented character set.

+

+@cindex multibyte character

+@cindex EBCDIC

+   For all the above reasons, an external encoding that is different

+from the internal encoding is often used if the latter is UCS-2 or UCS-4.

+The external encoding is byte-based and can be chosen appropriately for

+the environment and for the texts to be handled. A variety of different

+character sets can be used for this external encoding (information that

+will not be exhaustively presented here--instead, a description of the

+major groups will suffice). All of the ASCII-based character sets

+[_bkoz_: do you mean Roman character sets? If not, what do you mean

+here?] fulfill one requirement: they are "filesystem safe."  This means

+that the character @code{'/'} is used in the encoding @emph{only} to

+represent itself.  Things are a bit different for character sets like

+EBCDIC (Extended Binary Coded Decimal Interchange Code, a character set

+family used by IBM), but if the operation system does not understand

+EBCDIC directly the parameters-to-system calls have to be converted first

+anyhow.

+

+@itemize @bullet

+@item 

+The simplest character sets are single-byte character sets.  There can 

+be only up to 256 characters (for @w{8 bit} character sets), which is 

+not sufficient to cover all languages but might be sufficient to handle 

+a specific text. Handling of a @w{8 bit} character sets is simple. This 

+is not true for other kinds presented later, and therefore, the 

+application one uses might require the use of @w{8 bit} character sets.

+

+@cindex ISO 2022

+@item

+The @w{ISO 2022} standard defines a mechanism for extended character

+sets where one character @emph{can} be represented by more than one

+byte.  This is achieved by associating a state with the text.

+Characters that can be used to change the state can be embedded in the

+text. Each byte in the text might have a different interpretation in each

+state.  The state might even influence whether a given byte stands for a

+character on its own or whether it has to be combined with some more

+bytes.

+

+@cindex EUC

+@cindex Shift_JIS

+@cindex SJIS

+In most uses of @w{ISO 2022} the defined character sets do not allow

+state changes which cover more than the next character.  This has the

+big advantage that whenever one can identify the beginning of the byte

+sequence of a character one can interpret a text correctly.  Examples of

+character sets using this policy are the various EUC character sets

+(used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN)

+or Shift_JIS (SJIS, a Japanese encoding).

+

+But there are also character sets using a state which is valid for more

+than one character and has to be changed by another byte sequence.

+Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN.

+

+@item

+@cindex ISO 6937

+Early attempts to fix 8 bit character sets for other languages using the

+Roman alphabet lead to character sets like @w{ISO 6937}.  Here bytes

+representing characters like the acute accent do not produce output

+themselves: one has to combine them with other characters to get the

+desired result.  For example, the byte sequence @code{0xc2 0x61}

+(non-spacing acute accent, followed by lower-case `a') to get the ``small

+a with  acute'' character.  To get the acute accent character on its own,

+one has to write @code{0xc2 0x20} (the non-spacing acute followed by a

+space).

+

+Character sets like @w[ISO 6937] are used in some embedded systems such

+as teletex.

+

+@item

+@cindex UTF-8

+Instead of converting the Unicode or @w{ISO 10646} text used internally,

+it is often also sufficient to simply use an encoding different than

+UCS-2/UCS-4.  The Unicode and @w{ISO 10646} standards even specify such an

+encoding: UTF-8.  This encoding is able to represent all of @w{ISO

+10646} 31 bits in a byte string of length one to six.

+

+@cindex UTF-7

+There were a few other attempts to encode @w{ISO 10646} such as UTF-7,

+but UTF-8 is today the only encoding which should be used.  In fact, with

+any luck UTF-8 will soon be the only external encoding that has to be

+supported.  It proves to be universally usable and its only disadvantage

+is that it favors Roman languages by making the byte string

+representation of other scripts (Cyrillic, Greek, Asian scripts) longer

+than necessary if using a specific character set for these scripts.

+Methods like the Unicode compression scheme can alleviate these

+problems.

+@end itemize

+

+The question remaining is: how to select the character set or encoding

+to use.  The answer: you cannot decide about it yourself, it is decided

+by the developers of the system or the majority of the users.  Since the

+goal is interoperability one has to use whatever the other people one

+works with use.  If there are no constraints, the selection is based on

+the requirements the expected circle of users will have.  In other words,

+if a project is expected to be used in only, say, Russia it is fine to use

+KOI8-R or a similar character set.  But if at the same time people from,

+say, Greece are participating one should use a character set which allows

+all people to collaborate.

+

+The most widely useful solution seems to be: go with the most general

+character set, namely @w{ISO 10646}.  Use UTF-8 as the external encoding

+and problems about users not being able to use their own language

+adequately are a thing of the past.

+

+One final comment about the choice of the wide character representation

+is necessary at this point.  We have said above that the natural choice

+is using Unicode or @w{ISO 10646}.  This is not required, but at least

+encouraged, by the @w{ISO C} standard.  The standard defines at least a

+macro @code{__STDC_ISO_10646__} that is only defined on systems where

+the @code{wchar_t} type encodes @w{ISO 10646} characters.  If this

+symbol is not defined one should avoid making assumptions about the wide

+character representation. If the programmer uses only the functions

+provided by the C library to handle wide character strings there should

+be no compatibility problems with other systems.

+

+@node Charset Function Overview

+@section Overview about Character Handling Functions

+

+A Unix @w{C library} contains three different sets of functions in two 

+families to handle character set conversion. One of the function families 

+(the most commonly used) is specified in the @w{ISO C90} standard and, 

+therefore, is portable even beyond the Unix world. Unfortunately this 

+family is the least useful one. These functions should be avoided 

+whenever possible, especially when developing libraries (as opposed to 

+applications). 

+

+The second family of functions got introduced in the early Unix standards

+(XPG2) and is still part of the latest and greatest Unix standard:

+@w{Unix 98}.  It is also the most powerful and useful set of functions.

+But we will start with the functions defined in @w{Amendment 1} to

+@w{ISO C90}.

+

+@node Restartable multibyte conversion

+@section Restartable Multibyte Conversion Functions

+

+The @w{ISO C} standard defines functions to convert strings from a

+multibyte representation to wide character strings.  There are a number

+of peculiarities:

+

+@itemize @bullet

+@item

+The character set assumed for the multibyte encoding is not specified

+as an argument to the functions.  Instead the character set specified by

+the @code{LC_CTYPE} category of the current locale is used; see

+@ref{Locale Categories}.

+

+@item

+The functions handling more than one character at a time require NUL

+terminated strings as the argument.  I.e., converting blocks of text

+does not work unless one can add a NUL byte at an appropriate place.

+The GNU C library contains some extensions to the standard that allow

+specifying a size, but basically they also expect terminated strings.

+@end itemize

+

+Despite these limitations the @w{ISO C} functions can be used in many

+contexts.  In graphical user interfaces, for instance, it is not

+uncommon to have functions that require text to be displayed in a wide

+character string if the text is not simple ASCII.  The text itself might come

+from a file with translations and the user should decide about the

+current locale which determines the translation and therefore also the

+external encoding used. In such a situation (and many others) the

+functions described here are perfect.  If more freedom while performing

+the conversion is necessary take a look at the @code{iconv} functions

+(@pxref{Generic Charset Conversion}).

+

+@menu

+* Selecting the Conversion::     Selecting the conversion and its properties.

+* Keeping the state::            Representing the state of the conversion.

+* Converting a Character::       Converting Single Characters.

+* Converting Strings::           Converting Multibyte and Wide Character

+                                  Strings.

+* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.

+@end menu

+

+@node Selecting the Conversion

+@subsection Selecting the conversion and its properties

+

+We already said above that the currently selected locale for the

+@code{LC_CTYPE} category decides about the conversion which is performed

+by the functions we are about to describe.  Each locale uses its own

+character set (given as an argument to @code{localedef}) and this is the

+one assumed as the external multibyte encoding.  The wide character

+character set always is UCS-4, at least on GNU systems.

+

+A characteristic of each multibyte character set is the maximum number

+of bytes that can be necessary to represent one character.  This

+information is quite important when writing code that uses the

+conversion functions (as shown in the examples below).

+The @w{ISO C} standard defines two macros which provide this information.

+

+

+@comment limits.h

+@comment ISO

+@deftypevr Macro int MB_LEN_MAX

+@code{MB_LEN_MAX} specifies the maximum number of bytes in the multibyte

+sequence for a single character in any of the supported locales.  It is

+a compile-time constant and is defined in @file{limits.h}.

+@pindex limits.h

+@end deftypevr

+

+@comment stdlib.h

+@comment ISO

+@deftypevr Macro int MB_CUR_MAX

+@code{MB_CUR_MAX} expands into a positive integer expression that is the

+maximum number of bytes in a multibyte character in the current locale.

+The value is never greater than @code{MB_LEN_MAX}.  Unlike

+@code{MB_LEN_MAX} this macro need not be a compile-time constant, and in 

+the GNU C library it is not.

+

+@pindex stdlib.h

+@code{MB_CUR_MAX} is defined in @file{stdlib.h}.

+@end deftypevr

+

+Two different macros are necessary since strictly @w{ISO C90} compilers

+do not allow variable length array definitions, but still it is desirable

+to avoid dynamic allocation.  This incomplete piece of code shows the

+problem:

+

+@smallexample

+@{

+  char buf[MB_LEN_MAX];

+  ssize_t len = 0;

+

+  while (! feof (fp))

+    @{

+      fread (&buf[len], 1, MB_CUR_MAX - len, fp);

+      /* @r{... process} buf */

+      len -= used;

+    @}

+@}

+@end smallexample

+

+The code in the inner loop is expected to have always enough bytes in

+the array @var{buf} to convert one multibyte character.  The array

+@var{buf} has to be sized statically since many compilers do not allow a

+variable size.  The @code{fread} call makes sure that @code{MB_CUR_MAX} 

+bytes are always available in @var{buf}.  Note that it isn't

+a problem if @code{MB_CUR_MAX} is not a compile-time constant.

+

+

+@node Keeping the state

+@subsection Representing the state of the conversion

+

+@cindex stateful

+In the introduction of this chapter it was said that certain character

+sets use a @dfn{stateful} encoding.  That is, the encoded values depend 

+in some way on the previous bytes in the text.

+

+Since the conversion functions allow converting a text in more than one

+step we must have a way to pass this information from one call of the

+functions to another.

+

+@comment wchar.h

+@comment ISO

+@deftp {Data type} mbstate_t

+@cindex shift state

+A variable of type @code{mbstate_t} can contain all the information

+about the @dfn{shift state} needed from one call to a conversion

+function to another.

+

+@pindex wchar.h

+@code{mbstate_t} is defined in @file{wchar.h}. It was introduced in

+@w{Amendment 1} to @w{ISO C90}.

+@end deftp

+

+To use objects of type @code{mbstate_t} the programmer has to define such 

+objects (normally as local variables on the stack) and pass a pointer to 

+the object to the conversion functions.  This way the conversion function

+can update the object if the current multibyte character set is stateful.

+

+There is no specific function or initializer to put the state object in

+any specific state.  The rules are that the object should always

+represent the initial state before the first use, and this is achieved by

+clearing the whole variable with code such as follows:

+

+@smallexample

+@{

+  mbstate_t state;

+  memset (&state, '\0', sizeof (state));

+  /* @r{from now on @var{state} can be used.}  */

+  ...

+@}

+@end smallexample

+

+When using the conversion functions to generate output it is often

+necessary to test whether the current state corresponds to the initial

+state.  This is necessary, for example, to decide whether to emit

+escape sequences to set the state to the initial state at certain

+sequence points.  Communication protocols often require this.

+

+@comment wchar.h

+@comment ISO

+@deftypefun int mbsinit (const mbstate_t *@var{ps})

+The @code {mbsinit} function determines whether the state object pointed 

+to by @var{ps} is in the initial state. If @var{ps} is a null pointer or 

+the object is in the initial state the return value is nonzero. Otherwise 

+it is zero.

+

+@pindex wchar.h

+@code {mbsinit} was introduced in @w{Amendment 1} to @w{ISO C90} and is 

+declared in @file{wchar.h}.

+@end deftypefun

+

+Code using @code {mbsinit} often looks similar to this:

+

+@c Fix the example to explicitly say how to generate the escape sequence

+@c to restore the initial state.

+@smallexample

+@{

+  mbstate_t state;

+  memset (&state, '\0', sizeof (state));

+  /* @r{Use @var{state}.}  */

+  ...

+  if (! mbsinit (&state))

+    @{

+      /* @r{Emit code to return to initial state.}  */

+      const wchar_t empty[] = L"";

+      const wchar_t *srcp = empty;

+      wcsrtombs (outbuf, &srcp, outbuflen, &state);

+    @}

+  ...

+@}

+@end smallexample

+

+The code to emit the escape sequence to get back to the initial state is

+interesting. The @code{wcsrtombs} function can be used to determine the

+necessary output code (@pxref{Converting Strings}).  Please note that on

+GNU systems it is not necessary to perform this extra action for the

+conversion from multibyte text to wide character text since the wide

+character encoding is not stateful.  But there is nothing mentioned in

+any standard which prohibits making @code{wchar_t} using a stateful

+encoding.

+

+@node Converting a Character

+@subsection Converting Single Characters

+

+The most fundamental of the conversion functions are those dealing with

+single characters.  Please note that this does not always mean single

+bytes.  But since there is very often a subset of the multibyte

+character set which consists of single byte sequences there are

+functions to help with converting bytes.  Frequently, ASCII is a subpart 

+of the multibyte character set.  In such a scenario, each ASCII character 

+stands for itself, and all other characters have at least a first byte 

+that is beyond the range @math{0} to @math{127}.

+

+@comment wchar.h

+@comment ISO

+@deftypefun wint_t btowc (int @var{c})

+The @code{btowc} function (``byte to wide character'') converts a valid

+single byte character @var{c} in the initial shift state into the wide

+character equivalent using the conversion rules from the currently

+selected locale of the @code{LC_CTYPE} category.

+

+If @code{(unsigned char) @var{c}} is no valid single byte multibyte

+character or if @var{c} is @code{EOF}, the function returns @code{WEOF}.

+

+Please note the restriction of @var{c} being tested for validity only in

+the initial shift state.  No @code{mbstate_t} object is used from

+which the state information is taken, and the function also does not use

+any static state.

+

+@pindex wchar.h

+The @code{btowc} function was introduced in @w{Amendment 1} to @w{ISO C90} 

+and is declared in @file{wchar.h}.

+@end deftypefun

+

+Despite the limitation that the single byte value always is interpreted

+in the initial state this function is actually useful most of the time.

+Most characters are either entirely single-byte character sets or they

+are extension to ASCII.  But then it is possible to write code like this

+(not that this specific example is very useful):

+

+@smallexample

+wchar_t *

+itow (unsigned long int val)

+@{

+  static wchar_t buf[30];

+  wchar_t *wcp = &buf[29];

+  *wcp = L'\0';

+  while (val != 0)

+    @{

+      *--wcp = btowc ('0' + val % 10);

+      val /= 10;

+    @}

+  if (wcp == &buf[29])

+    *--wcp = L'0';

+  return wcp;

+@}

+@end smallexample

+

+Why is it necessary to use such a complicated implementation and not

+simply cast @code{'0' + val % 10} to a wide character?  The answer is

+that there is no guarantee that one can perform this kind of arithmetic

+on the character of the character set used for @code{wchar_t}

+representation.  In other situations the bytes are not constant at

+compile time and so the compiler cannot do the work.  In situations like

+this it is necessary @code{btowc}.

+

+@noindent

+There also is a function for the conversion in the other direction.

+

+@comment wchar.h

+@comment ISO

+@deftypefun int wctob (wint_t @var{c})

+The @code{wctob} function (``wide character to byte'') takes as the

+parameter a valid wide character.  If the multibyte representation for

+this character in the initial state is exactly one byte long the return

+value of this function is this character.  Otherwise the return value is

+@code{EOF}.

+

+@pindex wchar.h

+@code{wctob} was introduced in @w{Amendment 1} to @w{ISO C90} and

+is declared in @file{wchar.h}.

+@end deftypefun

+

+There are more general functions to convert single character from

+multibyte representation to wide characters and vice versa.  These

+functions pose no limit on the length of the multibyte representation

+and they also do not require it to be in the initial state.

+

+@comment wchar.h

+@comment ISO

+@deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps})

+@cindex stateful

+The @code{mbrtowc} function (``multibyte restartable to wide

+character'') converts the next multibyte character in the string pointed

+to by @var{s} into a wide character and stores it in the wide character

+string pointed to by @var{pwc}. The conversion is performed according

+to the locale currently selected for the @code{LC_CTYPE} category.  If

+the conversion for the character set used in the locale requires a state,

+the multibyte string is interpreted in the state represented by the

+object pointed to by @var{ps}. If @var{ps} is a null pointer, a static,

+internal state variable used only by the @code{mbrtowc} function is

+used.

+

+If the next multibyte character corresponds to the NUL wide character,

+the return value of the function is @math{0} and the state object is

+afterwards in the initial state. If the next @var{n} or fewer bytes

+form a correct multibyte character, the return value is the number of

+bytes starting from @var{s} that form the multibyte character.  The

+conversion state is updated according to the bytes consumed in the

+conversion. In both cases the wide character (either the @code{L'\0'}

+or the one found in the conversion) is stored in the string pointed to

+by @var{pwc} if @var{pwc} is not null.

+

+If the first @var{n} bytes of the multibyte string possibly form a valid

+multibyte character but there are more than @var{n} bytes needed to

+complete it, the return value of the function is @code{(size_t) -2} and

+no value is stored.  Please note that this can happen even if @var{n}

+has a value greater than or equal to @code{MB_CUR_MAX} since the input 

+might contain redundant shift sequences.

+

+If the first @code{n} bytes of the multibyte string cannot possibly form

+a valid multibyte character, no value is stored, the global variable

+@code{errno} is set to the value @code{EILSEQ}, and the function returns

+@code{(size_t) -1}. The conversion state is afterwards undefined.

+

+@pindex wchar.h

+@code{mbrtowc} was introduced in @w{Amendment 1} to @w{ISO C90} and

+is declared in @file{wchar.h}.

+@end deftypefun

+

+Use of @code{mbrtowc} is straightforward.  A function which copies a

+multibyte string into a wide character string while at the same time

+converting all lowercase characters into uppercase could look like this

+(this is not the final version, just an example; it has no error

+checking, and sometimes leaks memory):

+

+@smallexample

+wchar_t *

+mbstouwcs (const char *s)

+@{

+  size_t len = strlen (s);

+  wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));

+  wchar_t *wcp = result;

+  wchar_t tmp[1];

+  mbstate_t state;

+  size_t nbytes;

+

+  memset (&state, '\0', sizeof (state));

+  while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)

+    @{

+      if (nbytes >= (size_t) -2)

+        /* Invalid input string.  */

+        return NULL;

+      *result++ = towupper (tmp[0]);

+      len -= nbytes;

+      s += nbytes;

+    @}

+  return result;

+@}

+@end smallexample

+

+The use of @code{mbrtowc} should be clear. A single wide character is

+stored in @code{@var{tmp}[0]}, and the number of consumed bytes is stored

+in the variable @var{nbytes}. If the conversion is successful, the 

+uppercase variant of the wide character is stored in the @var{result} 

+array and the pointer to the input string and the number of available 

+bytes is adjusted.

+

+The only non-obvious thing about @code{mbrtowc} might be the way memory 

+is allocated for the result. The above code uses the fact that there 

+can never be more wide characters in the converted results than there are

+bytes in the multibyte input string. This method yields a pessimistic 

+guess about the size of the result, and if many wide character strings 

+have to be constructed this way or if the strings are long, the extra 

+memory required to be allocated because the input string contains 

+multibyte characters might be significant. The allocated memory block can 

+be resized to the correct size before returning it, but a better solution 

+might be to allocate just the right amount of space for the result right 

+away. Unfortunately there is no function to compute the length of the wide 

+character string directly from the multibyte string. There is, however, a 

+function which does part of the work.

+

+@comment wchar.h

+@comment ISO

+@deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps})

+The @code{mbrlen} function (``multibyte restartable length'') computes

+the number of at most @var{n} bytes starting at @var{s} which form the

+next valid and complete multibyte character.

+

+If the next multibyte character corresponds to the NUL wide character,

+the return value is @math{0}.  If the next @var{n} bytes form a valid

+multibyte character, the number of bytes belonging to this multibyte

+character byte sequence is returned.

+

+If the the first @var{n} bytes possibly form a valid multibyte

+character but the character is incomplete, the return value is 

+@code{(size_t) -2}. Otherwise the multibyte character sequence is invalid 

+and the return value is @code{(size_t) -1}.

+

+The multibyte sequence is interpreted in the state represented by the

+object pointed to by @var{ps}.  If @var{ps} is a null pointer, a state

+object local to @code{mbrlen} is used.

+

+@pindex wchar.h

+@code{mbrlen} was introduced in @w{Amendment 1} to @w{ISO C90} and

+is declared in @file{wchar.h}.

+@end deftypefun

+

+The attentive reader now will note that @code{mbrlen} can be implemented 

+as

+

+@smallexample

+mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)

+@end smallexample

+

+This is true and in fact is mentioned in the official specification.

+How can this function be used to determine the length of the wide

+character string created from a multibyte character string?  It is not

+directly usable, but we can define a function @code{mbslen} using it:

+

+@smallexample

+size_t

+mbslen (const char *s)

+@{

+  mbstate_t state;

+  size_t result = 0;

+  size_t nbytes;

+  memset (&state, '\0', sizeof (state));

+  while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)

+    @{

+      if (nbytes >= (size_t) -2)

+        /* @r{Something is wrong.}  */

+        return (size_t) -1;

+      s += nbytes;

+      ++result;

+    @}

+  return result;

+@}

+@end smallexample

+

+This function simply calls @code{mbrlen} for each multibyte character

+in the string and counts the number of function calls.  Please note that

+we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen}

+call. This is acceptable since a) this value is larger then the length of 

+the longest multibyte character sequence and b) we know that the string 

+@var{s} ends with a NUL byte, which cannot be part of any other multibyte 

+character sequence but the one representing the NUL wide character.  

+Therefore, the @code{mbrlen} function will never read invalid memory.

+

+Now that this function is available (just to make this clear, this

+function is @emph{not} part of the GNU C library) we can compute the

+number of wide character required to store the converted multibyte

+character string @var{s} using

+

+@smallexample

+wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);

+@end smallexample

+

+Please note that the @code{mbslen} function is quite inefficient. The

+implementation of @code{mbstouwcs} with @code{mbslen} would have to 

+perform the conversion of the multibyte character input string twice, and 

+this conversion might be quite expensive. So it is necessary to think 

+about the consequences of using the easier but imprecise method before 

+doing the work twice.

+

+@comment wchar.h

+@comment ISO

+@deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps})

+The @code{wcrtomb} function (``wide character restartable to

+multibyte'') converts a single wide character into a multibyte string

+corresponding to that wide character.

+

+If @var{s} is a null pointer, the function resets the state stored in

+the objects pointed to by @var{ps} (or the internal @code{mbstate_t}

+object) to the initial state.  This can also be achieved by a call like

+this:

+

+@smallexample

+wcrtombs (temp_buf, L'\0', ps)

+@end smallexample

+

+@noindent

+since, if @var{s} is a null pointer, @code{wcrtomb} performs as if it

+writes into an internal buffer, which is guaranteed to be large enough.

+

+If @var{wc} is the NUL wide character, @code{wcrtomb} emits, if

+necessary, a shift sequence to get the state @var{ps} into the initial

+state followed by a single NUL byte, which is stored in the string 

+@var{s}.

+

+Otherwise a byte sequence (possibly including shift sequences) is written 

+into the string @var{s}.  This only happens if @var{wc} is a valid wide 

+character (i.e., it has a multibyte representation in the character set 

+selected by locale of the @code{LC_CTYPE} category).  If @var{wc} is no 

+valid wide character, nothing is stored in the strings @var{s}, 

+@code{errno} is set to @code{EILSEQ}, the conversion state in @var{ps} 

+is undefined and the return value is @code{(size_t) -1}.

+

+If no error occurred the function returns the number of bytes stored in

+the string @var{s}.  This includes all bytes representing shift

+sequences.

+

+One word about the interface of the function: there is no parameter

+specifying the length of the array @var{s}.  Instead the function

+assumes that there are at least @code{MB_CUR_MAX} bytes available since

+this is the maximum length of any byte sequence representing a single

+character.  So the caller has to make sure that there is enough space

+available, otherwise buffer overruns can occur.

+

+@pindex wchar.h

+@code{wcrtomb} was introduced in @w{Amendment 1} to @w{ISO C90} and is

+declared in @file{wchar.h}.

+@end deftypefun

+

+Using @code{wcrtomb} is as easy as using @code{mbrtowc}.  The following

+example appends a wide character string to a multibyte character string.

+Again, the code is not really useful (or correct), it is simply here to

+demonstrate the use and some problems.

+

+@smallexample

+char *

+mbscatwcs (char *s, size_t len, const wchar_t *ws)

+@{

+  mbstate_t state;

+  /* @r{Find the end of the existing string.}  */

+  char *wp = strchr (s, '\0');

+  len -= wp - s;

+  memset (&state, '\0', sizeof (state));

+  do

+    @{

+      size_t nbytes;

+      if (len < MB_CUR_LEN)

+        @{

+          /* @r{We cannot guarantee that the next}

+             @r{character fits into the buffer, so}

+             @r{return an error.}  */

+          errno = E2BIG;

+          return NULL;

+        @}

+      nbytes = wcrtomb (wp, *ws, &state);

+      if (nbytes == (size_t) -1)

+        /* @r{Error in the conversion.}  */

+        return NULL;

+      len -= nbytes;

+      wp += nbytes;

+    @}

+  while (*ws++ != L'\0');

+  return s;

+@}

+@end smallexample

+

+First the function has to find the end of the string currently in the

+array @var{s}.  The @code{strchr} call does this very efficiently since a

+requirement for multibyte character representations is that the NUL byte

+is never used except to represent itself (and in this context, the end

+of the string).

+

+After initializing the state object the loop is entered where the first

+task is to make sure there is enough room in the array @var{s}.  We

+abort if there are not at least @code{MB_CUR_LEN} bytes available.  This

+is not always optimal but we have no other choice.  We might have less

+than @code{MB_CUR_LEN} bytes available but the next multibyte character

+might also be only one byte long.  At the time the @code{wcrtomb} call

+returns it is too late to decide whether the buffer was large enough. If 

+this solution is unsuitable, there is a very slow but more accurate 

+solution.

+

+@smallexample

+  ...

+  if (len < MB_CUR_LEN)

+    @{

+      mbstate_t temp_state;

+      memcpy (&temp_state, &state, sizeof (state));

+      if (wcrtomb (NULL, *ws, &temp_state) > len)

+        @{

+          /* @r{We cannot guarantee that the next}

+             @r{character fits into the buffer, so}

+             @r{return an error.}  */

+          errno = E2BIG;

+          return NULL;

+        @}

+    @}

+  ...

+@end smallexample

+

+Here we perform the conversion that might overflow the buffer so that 

+we are afterwards in the position to make an exact decision about the 

+buffer size. Please note the @code{NULL} argument for the destination 

+buffer in the new @code{wcrtomb} call; since we are not interested in the 

+converted text at this point, this is a nice way to express this. The 

+most unusual thing about this piece of code certainly is the duplication 

+of the conversion state object, but if a change of the state is necessary 

+to emit the next multibyte character, we want to have the same shift state 

+change performed in the real conversion. Therefore, we have to preserve 

+the initial shift state information.

+

+There are certainly many more and even better solutions to this problem.

+This example is only provided for educational purposes.

+

+@node Converting Strings

+@subsection Converting Multibyte and Wide Character Strings

+

+The functions described in the previous section only convert a single

+character at a time.  Most operations to be performed in real-world

+programs include strings and therefore the @w{ISO C} standard also

+defines conversions on entire strings.  However, the defined set of

+functions is quite limited; therefore, the GNU C library contains a few

+extensions which can help in some important situations.

+

+@comment wchar.h

+@comment ISO

+@deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})

+The @code{mbsrtowcs} function (``multibyte string restartable to wide

+character string'') converts an NUL-terminated multibyte character

+string at @code{*@var{src}} into an equivalent wide character string,

+including the NUL wide character at the end.  The conversion is started

+using the state information from the object pointed to by @var{ps} or

+from an internal object of @code{mbsrtowcs} if @var{ps} is a null

+pointer. Before returning, the state object is updated to match the state 

+after the last converted character. The state is the initial state if the

+terminating NUL byte is reached and converted.

+

+If @var{dst} is not a null pointer, the result is stored in the array

+pointed to by @var{dst}; otherwise, the conversion result is not

+available since it is stored in an internal buffer.

+

+If @var{len} wide characters are stored in the array @var{dst} before

+reaching the end of the input string, the conversion stops and @var{len}

+is returned. If @var{dst} is a null pointer, @var{len} is never checked.

+

+Another reason for a premature return from the function call is if the

+input string contains an invalid multibyte sequence.  In this case the

+global variable @code{errno} is set to @code{EILSEQ} and the function

+returns @code{(size_t) -1}.

+

+@c XXX The ISO C9x draft seems to have a problem here.  It says that PS

+@c is not updated if DST is NULL.  This is not said straightforward and

+@c none of the other functions is described like this.  It would make sense

+@c to define the function this way but I don't think it is meant like this.

+

+In all other cases the function returns the number of wide characters

+converted during this call. If @var{dst} is not null, @code{mbsrtowcs}

+stores in the pointer pointed to by @var{src} either a null pointer (if 

+the NUL byte in the input string was reached) or the address of the byte

+following the last converted multibyte character.

+

+@pindex wchar.h

+@code{mbsrtowcs} was introduced in @w{Amendment 1} to @w{ISO C90} and is

+declared in @file{wchar.h}.

+@end deftypefun

+

+The definition of the @code{mbsrtowcs} function has one important 

+limitation. The requirement that @var{dst} has to be a NUL-terminated 

+string provides problems if one wants to convert buffers with text. A

+buffer is normally no collection of NUL-terminated strings but instead a

+continuous collection of lines, separated by newline characters.  Now

+assume that a function to convert one line from a buffer is needed. Since

+the line is not NUL-terminated the source pointer cannot directly point

+into the unmodified text buffer. This means, either one inserts the NUL

+byte at the appropriate place for the time of the @code{mbsrtowcs}

+function call (which is not doable for a read-only buffer or in a

+multi-threaded application) or one copies the line in an extra buffer

+where it can be terminated by a NUL byte. Note that it is not in general 

+possible to limit the number of characters to convert by setting the 

+parameter @var{len} to any specific value.  Since it is not known how 

+many bytes each multibyte character sequence is in length, one can only 

+guess.

+

+@cindex stateful

+There is still a problem with the method of NUL-terminating a line right

+after the newline character which could lead to very strange results.

+As said in the description of the @code{mbsrtowcs} function above the

+conversion state is guaranteed to be in the initial shift state after

+processing the NUL byte at the end of the input string.  But this NUL

+byte is not really part of the text.  I.e., the conversion state after

+the newline in the original text could be something different than the

+initial shift state and therefore the first character of the next line

+is encoded using this state.  But the state in question is never

+accessible to the user since the conversion stops after the NUL byte

+(which resets the state).  Most stateful character sets in use today

+require that the shift state after a newline be the initial state--but

+this is not a strict guarantee.  Therefore, simply NUL-terminating a

+piece of a running text is not always an adequate solution and, 

+therefore, should never be used in generally used code.

+

+The generic conversion interface (@pxref{Generic Charset Conversion})

+does not have this limitation (it simply works on buffers, not

+strings), and the GNU C library contains a set of functions which take

+additional parameters specifying the maximal number of bytes which are

+consumed from the input string.  This way the problem of

+@code{mbsrtowcs}'s example above could be solved by determining the line

+length and passing this length to the function.

+

+@comment wchar.h

+@comment ISO

+@deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})

+The @code{wcsrtombs} function (``wide character string restartable to

+multibyte string'') converts the NUL-terminated wide character string at

+@code{*@var{src}} into an equivalent multibyte character string and 

+stores the result in the array pointed to by @var{dst}. The NUL wide

+character is also converted. The conversion starts in the state

+described in the object pointed to by @var{ps} or by a state object

+locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If

+@var{dst} is a null pointer, the conversion is performed as usual but the

+result is not available. If all characters of the input string were

+successfully converted and if @var{dst} is not a null pointer, the 

+pointer pointed to by @var{src} gets assigned a null pointer.

+

+If one of the wide characters in the input string has no valid multibyte

+character equivalent, the conversion stops early, sets the global

+variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}.

+

+Another reason for a premature stop is if @var{dst} is not a null

+pointer and the next converted character would require more than

+@var{len} bytes in total to the array @var{dst}. In this case (and if

+@var{dest} is not a null pointer) the pointer pointed to by @var{src} is

+assigned a value pointing to the wide character right after the last one

+successfully converted.

+

+Except in the case of an encoding error the return value of the 

+@code{wcsrtombs} function is the number of bytes in all the multibyte 

+character sequences stored in @var{dst}. Before returning the state in 

+the object pointed to by @var{ps} (or the internal object in case 

+@var{ps} is a null pointer) is updated to reflect the state after the 

+last conversion. The state is the initial shift state in case the 

+terminating NUL wide character was converted.

+

+@pindex wchar.h

+The @code{wcsrtombs} function was introduced in @w{Amendment 1} to 

+@w{ISO C90} and is declared in @file{wchar.h}.

+@end deftypefun

+

+The restriction mentioned above for the @code{mbsrtowcs} function applies

+here also. There is no possibility of directly controlling the number of

+input characters. One has to place the NUL wide character at the correct 

+place or control the consumed input indirectly via the available output 

+array size (the @var{len} parameter).

+

+@comment wchar.h

+@comment GNU

+@deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps})

+The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs}

+function. All the parameters are the same except for @var{nmc} which is

+new. The return value is the same as for @code{mbsrtowcs}.

+

+This new parameter specifies how many bytes at most can be used from the

+multibyte character string.  In other words, the multibyte character 

+string @code{*@var{src}} need not be NUL-terminated. But if a NUL byte is

+found within the @var{nmc} first bytes of the string, the conversion 

+stops here.

+

+This function is a GNU extension. It is meant to work around the

+problems mentioned above. Now it is possible to convert a buffer with

+multibyte character text piece for piece without having to care about

+inserting NUL bytes and the effect of NUL bytes on the conversion state.

+@end deftypefun

+

+A function to convert a multibyte string into a wide character string

+and display it could be written like this (this is not a really useful

+example):

+

+@smallexample

+void

+showmbs (const char *src, FILE *fp)

+@{

+  mbstate_t state;

+  int cnt = 0;

+  memset (&state, '\0', sizeof (state));

+  while (1)

+    @{

+      wchar_t linebuf[100];

+      const char *endp = strchr (src, '\n');

+      size_t n;

+

+      /* @r{Exit if there is no more line.}  */

+      if (endp == NULL)

+        break;

+

+      n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);

+      linebuf[n] = L'\0';

+      fprintf (fp, "line %d: \"%S\"\n", linebuf);

+    @}

+@}

+@end smallexample

+

+There is no problem with the state after a call to @code{mbsnrtowcs}.

+Since we don't insert characters in the strings which were not in there

+right from the beginning and we use @var{state} only for the conversion

+of the given buffer, there is no problem with altering the state.

+

+@comment wchar.h

+@comment GNU

+@deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps})

+The @code{wcsnrtombs} function implements the conversion from wide

+character strings to multibyte character strings. It is similar to

+@code{wcsrtombs} but, just like @code{mbsnrtowcs}, it takes an extra

+parameter, which specifies the length of the input string.

+

+No more than @var{nwc} wide characters from the input string

+@code{*@var{src}} are converted.  If the input string contains a NUL

+wide character in the first @var{nwc} characters, the conversion stops at

+this place.

+

+The @code{wcsnrtombs} function is a GNU extension and just like 

+@code{mbsnrtowcs} helps in situations where no NUL-terminated input 

+strings are available.

+@end deftypefun

+

+

+@node Multibyte Conversion Example

+@subsection A Complete Multibyte Conversion Example

+

+The example programs given in the last sections are only brief and do

+not contain all the error checking etc.  Presented here is a complete

+and documented example.  It features the @code{mbrtowc} function but it

+should be easy to derive versions using the other functions.

+

+@smallexample

+int

+file_mbsrtowcs (int input, int output)

+@{

+  /* @r{Note the use of @code{MB_LEN_MAX}.}

+     @r{@code{MB_CUR_MAX} cannot portably be used here.}  */

+  char buffer[BUFSIZ + MB_LEN_MAX];

+  mbstate_t state;

+  int filled = 0;

+  int eof = 0;

+

+  /* @r{Initialize the state.}  */

+  memset (&state, '\0', sizeof (state));

+

+  while (!eof)

+    @{

+      ssize_t nread;

+      ssize_t nwrite;

+      char *inp = buffer;

+      wchar_t outbuf[BUFSIZ];

+      wchar_t *outp = outbuf;

+

+      /* @r{Fill up the buffer from the input file.}  */

+      nread = read (input, buffer + filled, BUFSIZ);

+      if (nread < 0)

+        @{

+          perror ("read");

+          return 0;

+        @}

+      /* @r{If we reach end of file, make a note to read no more.} */

+      if (nread == 0)

+        eof = 1;

+

+      /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */

+      filled += nread;

+

+      /* @r{Convert those bytes to wide characters--as many as we can.} */

+      while (1)

+        @{

+          size_t thislen = mbrtowc (outp, inp, filled, &state);

+          /* @r{Stop converting at invalid character;}

+             @r{this can mean we have read just the first part}

+             @r{of a valid character.}  */

+          if (thislen == (size_t) -1)

+            break;

+          /* @r{We want to handle embedded NUL bytes}

+             @r{but the return value is 0.  Correct this.}  */

+          if (thislen == 0)

+            thislen = 1;

+          /* @r{Advance past this character.} */

+          inp += thislen;

+          filled -= thislen;

+          ++outp;

+        @}

+

+      /* @r{Write the wide characters we just made.}  */

+      nwrite = write (output, outbuf,

+                      (outp - outbuf) * sizeof (wchar_t));

+      if (nwrite < 0)

+        @{

+          perror ("write");

+          return 0;

+        @}

+

+      /* @r{See if we have a @emph{real} invalid character.} */

+      if ((eof && filled > 0) || filled >= MB_CUR_MAX)

+        @{

+          error (0, 0, "invalid multibyte character");

+          return 0;

+        @}

+

+      /* @r{If any characters must be carried forward,}

+         @r{put them at the beginning of @code{buffer}.} */

+      if (filled > 0)

+        memmove (inp, buffer, filled);

+    @}

+

+  return 1;

+@}

+@end smallexample

+

+

+@node Non-reentrant Conversion

+@section Non-reentrant Conversion Function

+

+The functions described in the previous chapter are defined in

+@w{Amendment 1} to @w{ISO C90}, but the original @w{ISO C90} standard 

+also contained functions for character set conversion. The reason that 

+these original functions are not described first is that they are almost 

+entirely useless.

+

+The problem is that all the conversion functions described in the 

+original @w{ISO C90} use a local state. Using a local state implies that 

+multiple conversions at the same time (not only when using threads) 

+cannot be done, and that you cannot first convert single characters and 

+then strings since you cannot tell the conversion functions which state 

+to use.

+

+These original functions are therefore usable only in a very limited set 

+of situations. One must complete converting the entire string before

+starting a new one, and each string/text must be converted with the same

+function (there is no problem with the library itself; it is guaranteed

+that no library function changes the state of any of these functions).

+@strong{For the above reasons it is highly requested that the functions

+described in the previous section be used in place of non-reentrant 

+conversion functions.}

+

+@menu

+* Non-reentrant Character Conversion::  Non-reentrant Conversion of Single

+                                         Characters.

+* Non-reentrant String Conversion::     Non-reentrant Conversion of Strings.

+* Shift State::                         States in Non-reentrant Functions.

+@end menu

+

+@node Non-reentrant Character Conversion

+@subsection Non-reentrant Conversion of Single Characters

+

+@comment stdlib.h

+@comment ISO

+@deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size})

+The @code{mbtowc} (``multibyte to wide character'') function when called

+with non-null @var{string} converts the first multibyte character

+beginning at @var{string} to its corresponding wide character code.  It

+stores the result in @code{*@var{result}}.

+

+@code{mbtowc} never examines more than @var{size} bytes.  (The idea is

+to supply for @var{size} the number of bytes of data you have in hand.)

+

+@code{mbtowc} with non-null @var{string} distinguishes three

+possibilities: the first @var{size} bytes at @var{string} start with

+valid multibyte characters, they start with an invalid byte sequence or

+just part of a character, or @var{string} points to an empty string (a

+null character).

+

+For a valid multibyte character, @code{mbtowc} converts it to a wide

+character and stores that in @code{*@var{result}}, and returns the

+number of bytes in that character (always at least @math{1} and never

+more than @var{size}).

+

+For an invalid byte sequence, @code{mbtowc} returns @math{-1}.  For an

+empty string, it returns @math{0}, also storing @code{'\0'} in

+@code{*@var{result}}.

+

+If the multibyte character code uses shift characters, then

+@code{mbtowc} maintains and updates a shift state as it scans.  If you

+call @code{mbtowc} with a null pointer for @var{string}, that

+initializes the shift state to its standard initial value.  It also

+returns nonzero if the multibyte character code in use actually has a

+shift state.  @xref{Shift State}.

+@end deftypefun

+

+@comment stdlib.h

+@comment ISO

+@deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar})

+The @code{wctomb} (``wide character to multibyte'') function converts

+the wide character code @var{wchar} to its corresponding multibyte

+character sequence, and stores the result in bytes starting at

+@var{string}.  At most @code{MB_CUR_MAX} characters are stored.

+

+@code{wctomb} with non-null @var{string} distinguishes three

+possibilities for @var{wchar}: a valid wide character code (one that can

+be translated to a multibyte character), an invalid code, and @code{L'\0'}.

+

+Given a valid code, @code{wctomb} converts it to a multibyte character,

+storing the bytes starting at @var{string}.  Then it returns the number

+of bytes in that character (always at least @math{1} and never more

+than @code{MB_CUR_MAX}).

+

+If @var{wchar} is an invalid wide character code, @code{wctomb} returns

+@math{-1}.  If @var{wchar} is @code{L'\0'}, it returns @code{0}, also

+storing @code{'\0'} in @code{*@var{string}}.

+

+If the multibyte character code uses shift characters, then

+@code{wctomb} maintains and updates a shift state as it scans. If you

+call @code{wctomb} with a null pointer for @var{string}, that

+initializes the shift state to its standard initial value.  It also

+returns nonzero if the multibyte character code in use actually has a

+shift state.  @xref{Shift State}.

+

+Calling this function with a @var{wchar} argument of zero when

+@var{string} is not null has the side-effect of reinitializing the

+stored shift state @emph{as well as} storing the multibyte character

+@code{'\0'} and returning @math{0}.

+@end deftypefun

+

+Similar to @code{mbrlen} there is also a non-reentrant function which

+computes the length of a multibyte character.  It can be defined in

+terms of @code{mbtowc}.

+

+@comment stdlib.h

+@comment ISO

+@deftypefun int mblen (const char *@var{string}, size_t @var{size})

+The @code{mblen} function with a non-null @var{string} argument returns

+the number of bytes that make up the multibyte character beginning at

+@var{string}, never examining more than @var{size} bytes.  (The idea is

+to supply for @var{size} the number of bytes of data you have in hand.)

+

+The return value of @code{mblen} distinguishes three possibilities: the

+first @var{size} bytes at @var{string} start with valid multibyte

+characters, they start with an invalid byte sequence or just part of a

+character, or @var{string} points to an empty string (a null character).

+

+For a valid multibyte character, @code{mblen} returns the number of

+bytes in that character (always at least @code{1} and never more than

+@var{size}). For an invalid byte sequence, @code{mblen} returns 

+@math{-1}. For an empty string, it returns @math{0}.

+

+If the multibyte character code uses shift characters, then @code{mblen}

+maintains and updates a shift state as it scans.  If you call

+@code{mblen} with a null pointer for @var{string}, that initializes the

+shift state to its standard initial value.  It also returns a nonzero

+value if the multibyte character code in use actually has a shift state.

+@xref{Shift State}.

+

+@pindex stdlib.h

+The function @code{mblen} is declared in @file{stdlib.h}.

+@end deftypefun

+

+

+@node Non-reentrant String Conversion

+@subsection Non-reentrant Conversion of Strings

+

+For convenience the @w{ISO C90} standard also defines functions to 

+convert entire strings instead of single characters. These functions

+suffer from the same problems as their reentrant counterparts from

+@w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}.

+

+@comment stdlib.h

+@comment ISO

+@deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size})

+The @code{mbstowcs} (``multibyte string to wide character string'')

+function converts the null-terminated string of multibyte characters

+@var{string} to an array of wide character codes, storing not more than

+@var{size} wide characters into the array beginning at @var{wstring}.

+The terminating null character counts towards the size, so if @var{size}

+is less than the actual number of wide characters resulting from

+@var{string}, no terminating null character is stored.

+

+The conversion of characters from @var{string} begins in the initial

+shift state.

+

+If an invalid multibyte character sequence is found, the @code{mbstowcs} 

+function returns a value of @math{-1}. Otherwise, it returns the number 

+of wide characters stored in the array @var{wstring}. This number does 

+not include the terminating null character, which is present if the 

+number is less than @var{size}.

+

+Here is an example showing how to convert a string of multibyte

+characters, allocating enough space for the result.

+

+@smallexample

+wchar_t *

+mbstowcs_alloc (const char *string)

+@{

+  size_t size = strlen (string) + 1;

+  wchar_t *buf = xmalloc (size * sizeof (wchar_t));

+

+  size = mbstowcs (buf, string, size);

+  if (size == (size_t) -1)

+    return NULL;

+  buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));

+  return buf;

+@}

+@end smallexample

+

+@end deftypefun

+

+@comment stdlib.h

+@comment ISO

+@deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size})

+The @code{wcstombs} (``wide character string to multibyte string'')

+function converts the null-terminated wide character array @var{wstring}

+into a string containing multibyte characters, storing not more than

+@var{size} bytes starting at @var{string}, followed by a terminating

+null character if there is room. The conversion of characters begins in

+the initial shift state.

+

+The terminating null character counts towards the size, so if @var{size}

+is less than or equal to the number of bytes needed in @var{wstring}, no

+terminating null character is stored.

+

+If a code that does not correspond to a valid multibyte character is

+found, the @code{wcstombs} function returns a value of @math{-1}. 

+Otherwise, the return value is the number of bytes stored in the array 

+@var{string}. This number does not include the terminating null character, 

+which is present if the number is less than @var{size}.

+@end deftypefun

+

+@node Shift State

+@subsection States in Non-reentrant Functions

+

+In some multibyte character codes, the @emph{meaning} of any particular

+byte sequence is not fixed; it depends on what other sequences have come

+earlier in the same string. Typically there are just a few sequences that 

+can change the meaning of other sequences; these few are called 

+@dfn{shift sequences} and we say that they set the @dfn{shift state} for

+other sequences that follow.

+

+To illustrate shift state and shift sequences, suppose we decide that

+the sequence @code{0200} (just one byte) enters Japanese mode, in which

+pairs of bytes in the range from @code{0240} to @code{0377} are single

+characters, while @code{0201} enters Latin-1 mode, in which single bytes

+in the range from @code{0240} to @code{0377} are characters, and

+interpreted according to the ISO Latin-1 character set.  This is a

+multibyte code which has two alternative shift states (``Japanese mode''

+and ``Latin-1 mode''), and two shift sequences that specify particular

+shift states.

+

+When the multibyte character code in use has shift states, then

+@code{mblen}, @code{mbtowc}, and @code{wctomb} must maintain and update

+the current shift state as they scan the string. To make this work

+properly, you must follow these rules:

+

+@itemize @bullet

+@item

+Before starting to scan a string, call the function with a null pointer

+for the multibyte character address---for example, @code{mblen (NULL,

+0)}. This initializes the shift state to its standard initial value.

+

+@item

+Scan the string one character at a time, in order. Do not ``back up''

+and rescan characters already scanned, and do not intersperse the

+processing of different strings.

+@end itemize

+

+Here is an example of using @code{mblen} following these rules:

+

+@smallexample

+void

+scan_string (char *s)

+@{

+  int length = strlen (s);

+

+  /* @r{Initialize shift state.}  */

+  mblen (NULL, 0);

+

+  while (1)

+    @{

+      int thischar = mblen (s, length);

+      /* @r{Deal with end of string and invalid characters.}  */

+      if (thischar == 0)

+        break;

+      if (thischar == -1)

+        @{

+          error ("invalid multibyte character");

+          break;

+        @}

+      /* @r{Advance past this character.}  */

+      s += thischar;

+      length -= thischar;

+    @}

+@}

+@end smallexample

+

+The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not

+reentrant when using a multibyte code that uses a shift state.  However,

+no other library functions call these functions, so you don't have to

+worry that the shift state will be changed mysteriously.

+

+

+@node Generic Charset Conversion

+@section Generic Charset Conversion

+

+The conversion functions mentioned so far in this chapter all had in

+common that they operate on character sets that are not directly

+specified by the functions. The multibyte encoding used is specified by

+the currently selected locale for the @code{LC_CTYPE} category. The

+wide character set is fixed by the implementation (in the case of GNU C

+library it is always UCS-4 encoded @w{ISO 10646}.

+

+This has of course several problems when it comes to general character

+conversion:

+

+@itemize @bullet

+@item

+For every conversion where neither the source nor the destination 

+character set is the character set of the locale for the @code{LC_CTYPE} 

+category, one has to change the @code{LC_CTYPE} locale using 

+@code{setlocale}.

+

+Changing the @code{LC_TYPE} locale introduces major problems for the rest 

+of the programs since several more functions (e.g., the character 

+classification functions, @pxref{Classification of Characters}) use the 

+@code{LC_CTYPE} category.

+

+@item

+Parallel conversions to and from different character sets are not

+possible since the @code{LC_CTYPE} selection is global and shared by all

+threads.

+

+@item

+If neither the source nor the destination character set is the character

+set used for @code{wchar_t} representation, there is at least a two-step

+process necessary to convert a text using the functions above. One would 

+have to select the source character set as the multibyte encoding, 

+convert the text into a @code{wchar_t} text, select the destination

+character set as the multibyte encoding, and convert the wide character

+text to the multibyte (@math{=} destination) character set.

+

+Even if this is possible (which is not guaranteed) it is a very tiring

+work.  Plus it suffers from the other two raised points even more due to

+the steady changing of the locale.

+@end itemize

+

+The XPG2 standard defines a completely new set of functions which has

+none of these limitations. They are not at all coupled to the selected

+locales, and they have no constraints on the character sets selected for

+source and destination. Only the set of available conversions limits 

+them. The standard does not specify that any conversion at all must be 

+available. Such availability is a measure of the quality of the 

+implementation.

+

+In the following text first the interface to @code{iconv} and then the

+conversion function, will be described. Comparisons with other

+implementations will show what obstacles stand in the way of portable

+applications. Finally, the implementation is described in so far as might 

+interest the advanced user who wants to extend conversion capabilities.

+

+@menu

+* Generic Conversion Interface::    Generic Character Set Conversion Interface.

+* iconv Examples::                  A complete @code{iconv} example.

+* Other iconv Implementations::     Some Details about other @code{iconv}

+                                     Implementations.

+* glibc iconv Implementation::      The @code{iconv} Implementation in the GNU C

+                                     library.

+@end menu

+

+@node Generic Conversion Interface

+@subsection Generic Character Set Conversion Interface

+

+This set of functions follows the traditional cycle of using a resource:

+open--use--close.  The interface consists of three functions, each of

+which implements one step.

+

+Before the interfaces are described it is necessary to introduce a

+data type.  Just like other open--use--close interfaces the functions

+introduced here work using handles and the @file{iconv.h} header

+defines a special type for the handles used.

+

+@comment iconv.h

+@comment XPG2

+@deftp {Data Type} iconv_t

+This data type is an abstract type defined in @file{iconv.h}.  The user

+must not assume anything about the definition of this type; it must be

+completely opaque.

+

+Objects of this type can get assigned handles for the conversions using

+the @code{iconv} functions. The objects themselves need not be freed, but

+the conversions for which the handles stand for have to.

+@end deftp

+

+@noindent

+The first step is the function to create a handle.

+

+@comment iconv.h

+@comment XPG2

+@deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode})

+The @code{iconv_open} function has to be used before starting a

+conversion.  The two parameters this function takes determine the

+source and destination character set for the conversion, and if the

+implementation has the possibility to perform such a conversion, the

+function returns a handle.

+

+If the wanted conversion is not available, the @code{iconv_open} function 

+returns @code{(iconv_t) -1}. In this case the global variable 

+@code{errno} can have the following values:

+

+@table @code

+@item EMFILE

+The process already has @code{OPEN_MAX} file descriptors open.

+@item ENFILE

+The system limit of open file is reached.

+@item ENOMEM

+Not enough memory to carry out the operation.

+@item EINVAL

+The conversion from @var{fromcode} to @var{tocode} is not supported.

+@end table

+

+It is not possible to use the same descriptor in different threads to

+perform independent conversions. The data structures associated

+with the descriptor include information about the conversion state.

+This must not be messed up by using it in different conversions.

+

+An @code{iconv} descriptor is like a file descriptor as for every use a

+new descriptor must be created. The descriptor does not stand for all

+of the conversions from @var{fromset} to @var{toset}.

+

+The GNU C library implementation of @code{iconv_open} has one

+significant extension to other implementations. To ease the extension

+of the set of available conversions, the implementation allows storing

+the necessary files with data and code in an arbitrary number of 

+directories. How this extension must be written will be explained below

+(@pxref{glibc iconv Implementation}). Here it is only important to say

+that all directories mentioned in the @code{GCONV_PATH} environment

+variable are considered only if they contain a file @file{gconv-modules}.

+These directories need not necessarily be created by the system

+administrator. In fact, this extension is introduced to help users

+writing and using their own, new conversions. Of course, this does not 

+work for security reasons in SUID binaries; in this case only the system

+directory is considered and this normally is 

+@file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment variable 

+is examined exactly once at the first call of the @code{iconv_open} 

+function. Later modifications of the variable have no effect.

+

+@pindex iconv.h

+The @code{iconv_open} function was introduced early in the X/Open 

+Portability Guide, @w{version 2}. It is supported by all commercial 

+Unices as it is required for the Unix branding. However, the quality and 

+completeness of the implementation varies widely. The @code{iconv_open} 

+function is declared in @file{iconv.h}.

+@end deftypefun

+

+The @code{iconv} implementation can associate large data structure with

+the handle returned by @code{iconv_open}. Therefore, it is crucial to 

+free all the resources once all conversions are carried out and the 

+conversion is not needed anymore.

+

+@comment iconv.h

+@comment XPG2

+@deftypefun int iconv_close (iconv_t @var{cd})

+The @code{iconv_close} function frees all resources associated with the

+handle @var{cd}, which must have been returned by a successful call to

+the @code{iconv_open} function.

+

+If the function call was successful the return value is @math{0}.

+Otherwise it is @math{-1} and @code{errno} is set appropriately.

+Defined error are:

+

+@table @code

+@item EBADF

+The conversion descriptor is invalid.

+@end table

+

+@pindex iconv.h

+The @code{iconv_close} function was introduced together with the rest 

+of the @code{iconv} functions in XPG2 and is declared in @file{iconv.h}.

+@end deftypefun

+

+The standard defines only one actual conversion function.  This has,

+therefore, the most general interface: it allows conversion from one

+buffer to another.  Conversion from a file to a buffer, vice versa, or

+even file to file can be implemented on top of it.

+

+@comment iconv.h

+@comment XPG2

+@deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})

+@cindex stateful

+The @code{iconv} function converts the text in the input buffer

+according to the rules associated with the descriptor @var{cd} and

+stores the result in the output buffer. It is possible to call the

+function for the same text several times in a row since for stateful

+character sets the necessary state information is kept in the data

+structures associated with the descriptor.

+

+The input buffer is specified by @code{*@var{inbuf}} and it contains

+@code{*@var{inbytesleft}} bytes.  The extra indirection is necessary for

+communicating the used input back to the caller (see below).  It is

+important to note that the buffer pointer is of type @code{char} and the

+length is measured in bytes even if the input text is encoded in wide

+characters.

+

+The output buffer is specified in a similar way.  @code{*@var{outbuf}}

+points to the beginning of the buffer with at least

+@code{*@var{outbytesleft}} bytes room for the result.  The buffer

+pointer again is of type @code{char} and the length is measured in

+bytes.  If @var{outbuf} or @code{*@var{outbuf}} is a null pointer, the

+conversion is performed but no output is available.

+

+If @var{inbuf} is a null pointer, the @code{iconv} function performs the

+necessary action to put the state of the conversion into the initial

+state. This is obviously a no-op for non-stateful encodings, but if the

+encoding has a state, such a function call might put some byte sequences

+in the output buffer, which perform the necessary state changes. The

+next call with @var{inbuf} not being a null pointer then simply goes on

+from the initial state. It is important that the programmer never makes

+any assumption as to whether the conversion has to deal with states. Even 

+if the input and output character sets are not stateful, the 

+implementation might still have to keep states. This is due to the

+implementation chosen for the GNU C library as it is described below.

+Therefore an @code{iconv} call to reset the state should always be

+performed if some protocol requires this for the output text.

+

+The conversion stops for one of three reasons. The first is that all

+characters from the input buffer are converted. This actually can mean

+two things: either all bytes from the input buffer are consumed or

+there are some bytes at the end of the buffer that possibly can form a

+complete character but the input is incomplete. The second reason for a

+stop is that the output buffer is full. And the third reason is that

+the input contains invalid characters.

+

+In all of these cases the buffer pointers after the last successful

+conversion, for input and output buffer, are stored in @var{inbuf} and

+@var{outbuf}, and the available room in each buffer is stored in

+@var{inbytesleft} and @var{outbytesleft}.

+

+Since the character sets selected in the @code{iconv_open} call can be

+almost arbitrary, there can be situations where the input buffer contains

+valid characters, which have no identical representation in the output

+character set. The behavior in this situation is undefined. The

+@emph{current} behavior of the GNU C library in this situation is to

+return with an error immediately. This certainly is not the most

+desirable solution; therefore, future versions will provide better ones,

+but they are not yet finished.

+

+If all input from the input buffer is successfully converted and stored

+in the output buffer, the function returns the number of non-reversible

+conversions performed. In all other cases the return value is

+@code{(size_t) -1} and @code{errno} is set appropriately. In such cases

+the value pointed to by @var{inbytesleft} is nonzero.

+

+@table @code

+@item EILSEQ

+The conversion stopped because of an invalid byte sequence in the input.

+After the call, @code{*@var{inbuf}} points at the first byte of the

+invalid byte sequence.

+

+@item E2BIG

+The conversion stopped because it ran out of space in the output buffer.

+

+@item EINVAL

+The conversion stopped because of an incomplete byte sequence at the end

+of the input buffer.

+

+@item EBADF

+The @var{cd} argument is invalid.

+@end table

+

+@pindex iconv.h

+The @code{iconv} function was introduced in the XPG2 standard and is 

+declared in the @file{iconv.h} header.

+@end deftypefun

+

+The definition of the @code{iconv} function is quite good overall. It

+provides quite flexible functionality. The only problems lie in the

+boundary cases, which are incomplete byte sequences at the end of the

+input buffer and invalid input. A third problem, which is not really

+a design problem, is the way conversions are selected. The standard

+does not say anything about the legitimate names, a minimal set of

+available conversions. We will see how this negatively impacts other

+implementations, as demonstrated below.

+

+@node iconv Examples

+@subsection A complete @code{iconv} example

+

+The example below features a solution for a common problem.  Given that

+one knows the internal encoding used by the system for @code{wchar_t}

+strings, one often is in the position to read text from a file and store

+it in wide character buffers. One can do this using @code{mbsrtowcs},

+but then we run into the problems discussed above.

+

+@smallexample

+int

+file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)

+@{

+  char inbuf[BUFSIZ];

+  size_t insize = 0;

+  char *wrptr = (char *) outbuf;

+  int result = 0;

+  iconv_t cd;

+

+  cd = iconv_open ("WCHAR_T", charset);

+  if (cd == (iconv_t) -1)

+    @{

+      /* @r{Something went wrong.}  */

+      if (errno == EINVAL)

+        error (0, 0, "conversion from '%s' to wchar_t not available",

+               charset);

+      else

+        perror ("iconv_open");

+

+      /* @r{Terminate the output string.}  */

+      *outbuf = L'\0';

+

+      return -1;

+    @}

+

+  while (avail > 0)

+    @{

+      size_t nread;

+      size_t nconv;

+      char *inptr = inbuf;

+

+      /* @r{Read more input.}  */

+      nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);

+      if (nread == 0)

+        @{

+          /* @r{When we come here the file is completely read.}

+             @r{This still could mean there are some unused}

+             @r{characters in the @code{inbuf}.  Put them back.}  */

+          if (lseek (fd, -insize, SEEK_CUR) == -1)

+            result = -1;

+

+          /* @r{Now write out the byte sequence to get into the}

+             @r{initial state if this is necessary.}  */

+          iconv (cd, NULL, NULL, &wrptr, &avail);

+

+          break;

+        @}

+      insize += nread;

+

+      /* @r{Do the conversion.}  */

+      nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);

+      if (nconv == (size_t) -1)

+        @{

+          /* @r{Not everything went right.  It might only be}

+             @r{an unfinished byte sequence at the end of the}

+             @r{buffer.  Or it is a real problem.}  */

+          if (errno == EINVAL)

+            /* @r{This is harmless.  Simply move the unused}

+               @r{bytes to the beginning of the buffer so that}

+               @r{they can be used in the next round.}  */

+            memmove (inbuf, inptr, insize);

+          else

+            @{

+              /* @r{It is a real problem.  Maybe we ran out of}

+                 @r{space in the output buffer or we have invalid}

+                 @r{input.  In any case back the file pointer to}

+                 @r{the position of the last processed byte.}  */

+              lseek (fd, -insize, SEEK_CUR);

+              result = -1;

+              break;

+            @}

+        @}

+    @}

+

+  /* @r{Terminate the output string.}  */

+  if (avail >= sizeof (wchar_t))

+    *((wchar_t *) wrptr) = L'\0';

+

+  if (iconv_close (cd) != 0)

+    perror ("iconv_close");

+

+  return (wchar_t *) wrptr - outbuf;

+@}

+@end smallexample

+

+@cindex stateful

+This example shows the most important aspects of using the @code{iconv}

+functions.  It shows how successive calls to @code{iconv} can be used to

+convert large amounts of text.  The user does not have to care about

+stateful encodings as the functions take care of everything.

+

+An interesting point is the case where @code{iconv} returns an error and

+@code{errno} is set to @code{EINVAL}. This is not really an error in the 

+transformation. It can happen whenever the input character set contains 

+byte sequences of more than one byte for some character and texts are not 

+processed in one piece. In this case there is a chance that a multibyte 

+sequence is cut. The caller can then simply read the remainder of the 

+takes and feed the offending bytes together with new character from the 

+input to @code{iconv} and continue the work. The internal state kept in 

+the descriptor is @emph{not} unspecified after such an event as is the 

+case with the conversion functions from the @w{ISO C} standard.

+

+The example also shows the problem of using wide character strings with

+@code{iconv}. As explained in the description of the @code{iconv}

+function above, the function always takes a pointer to a @code{char}

+array and the available space is measured in bytes. In the example, the

+output buffer is a wide character buffer; therefore, we use a local

+variable @var{wrptr} of type @code{char *}, which is used in the

+@code{iconv} calls.

+

+This looks rather innocent but can lead to problems on platforms that

+have tight restriction on alignment. Therefore the caller of @code{iconv} 

+has to make sure that the pointers passed are suitable for access of 

+characters from the appropriate character set. Since, in the

+above case, the input parameter to the function is a @code{wchar_t}

+pointer, this is the case (unless the user violates alignment when

+computing the parameter). But in other situations, especially when

+writing generic functions where one does not know what type of character

+set one uses and, therefore, treats text as a sequence of bytes, it might

+become tricky.

+

+@node Other iconv Implementations

+@subsection Some Details about other @code{iconv} Implementations

+

+This is not really the place to discuss the @code{iconv} implementation

+of other systems but it is necessary to know a bit about them to write

+portable programs.  The above mentioned problems with the specification

+of the @code{iconv} functions can lead to portability issues.

+

+The first thing to notice is that, due to the large number of character

+sets in use, it is certainly not practical to encode the conversions

+directly in the C library. Therefore, the conversion information must

+come from files outside the C library. This is usually done in one or

+both of the following ways:

+

+@itemize @bullet

+@item

+The C library contains a set of generic conversion functions which can

+read the needed conversion tables and other information from data files.

+These files get loaded when necessary.

+

+This solution is problematic as it requires a great deal of effort to

+apply to all character sets (potentially an infinite set). The 

+differences in the structure of the different character sets is so large

+that many different variants of the table-processing functions must be

+developed. In addition, the generic nature of these functions make them 

+slower than specifically implemented functions.

+

+@item

+The C library only contains a framework which can dynamically load

+object files and execute the conversion functions contained therein.

+

+This solution provides much more flexibility. The C library itself

+contains only very little code and therefore reduces the general memory

+footprint. Also, with a documented interface between the C library and

+the loadable modules it is possible for third parties to extend the set

+of available conversion modules. A drawback of this solution is that

+dynamic loading must be available.

+@end itemize

+

+Some implementations in commercial Unices implement a mixture of these 

+possibilities; the majority implement only the second solution. Using 

+loadable modules moves the code out of the library itself and keeps 

+the door open for extensions and improvements, but this design is also

+limiting on some platforms since not many platforms support dynamic

+loading in statically linked programs. On platforms without this

+capability it is therefore not possible to use this interface in

+statically linked programs. The GNU C library has, on ELF platforms, no

+problems with dynamic loading in these situations; therefore, this

+point is moot. The danger is that one gets acquainted with this situation 

+and forgets about the restrictions on other systems.

+

+A second thing to know about other @code{iconv} implementations is that

+the number of available conversions is often very limited. Some

+implementations provide, in the standard release (not special

+international or developer releases), at most 100 to 200 conversion

+possibilities. This does not mean 200 different character sets are

+supported; for example, conversions from one character set to a set of 10 

+others might count as 10 conversions. Together with the other direction

+this makes 20 conversion possibilities used up by one character set. One 

+can imagine the thin coverage these platform provide. Some Unix vendors 

+even provide only a handful of conversions which renders them useless for 

+almost all uses.

+

+This directly leads to a third and probably the most problematic point.

+The way the @code{iconv} conversion functions are implemented on all

+known Unix systems and the availability of the conversion functions from

+character set @math{@cal{A}} to @math{@cal{B}} and the conversion from

+@math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the

+conversion from @math{@cal{A}} to @math{@cal{C}} is available.

+

+This might not seem unreasonable and problematic at first, but it is a

+quite big problem as one will notice shortly after hitting it.  To show

+the problem we assume to write a program which has to convert from

+@math{@cal{A}} to @math{@cal{C}}. A call like

+

+@smallexample

+cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}");

+@end smallexample

+

+@noindent

+fails according to the assumption above. But what does the program

+do now?  The conversion is necessary; therefore, simply giving up is not

+an option.

+

+This is a nuisance.  The @code{iconv} function should take care of this.

+But how should the program proceed from here on?  If it tries to convert 

+to character set @math{@cal{B}}, first the two @code{iconv_open}

+calls

+

+@smallexample

+cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}");

+@end smallexample

+

+@noindent

+and

+

+@smallexample

+cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}");

+@end smallexample

+

+@noindent

+will succeed, but how to find @math{@cal{B}}?

+

+Unfortunately, the answer is: there is no general solution.  On some

+systems guessing might help. On those systems most character sets can

+convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text. Beside 

+this only some very system-specific methods can help. Since the 

+conversion functions come from loadable modules and these modules must

+be stored somewhere in the filesystem, one @emph{could} try to find them

+and determine from the available file which conversions are available

+and whether there is an indirect route from @math{@cal{A}} to

+@math{@cal{C}}.

+

+This example shows one of the design errors of @code{iconv} mentioned 

+above. It should at least be possible to determine the list of available

+conversion programmatically so that if @code{iconv_open} says there is no 

+such conversion, one could make sure this also is true for indirect

+routes.

+

+@node glibc iconv Implementation

+@subsection The @code{iconv} Implementation in the GNU C library

+

+After reading about the problems of @code{iconv} implementations in the

+last section it is certainly good to note that the implementation in

+the GNU C library has none of the problems mentioned above.  What

+follows is a step-by-step analysis of the points raised above.  The

+evaluation is based on the current state of the development (as of

+January 1999).  The development of the @code{iconv} functions is not

+complete, but basic functionality has solidified.

+

+The GNU C library's @code{iconv} implementation uses shared loadable

+modules to implement the conversions.  A very small number of

+conversions are built into the library itself but these are only rather

+trivial conversions.

+

+All the benefits of loadable modules are available in the GNU C library

+implementation.  This is especially appealing since the interface is

+well documented (see below), and it, therefore, is easy to write new

+conversion modules.  The drawback of using loadable objects is not a

+problem in the GNU C library, at least on ELF systems.  Since the

+library is able to load shared objects even in statically linked

+binaries, static linking need not be forbidden in case one wants to use 

+@code{iconv}.

+

+The second mentioned problem is the number of supported conversions.

+Currently, the GNU C library supports more than 150 character sets.  The

+way the implementation is designed the number of supported conversions

+is greater than 22350 (@math{150} times @math{149}).  If any conversion

+from or to a character set is missing, it can be added easily.

+

+Particularly impressive as it may be, this high number is due to the

+fact that the GNU C library implementation of @code{iconv} does not have

+the third problem mentioned above (i.e., whenever there is a conversion

+from a character set @math{@cal{A}} to @math{@cal{B}} and from

+@math{@cal{B}} to @math{@cal{C}} it is always possible to convert from

+@math{@cal{A}} to @math{@cal{C}} directly).  If the @code{iconv_open}

+returns an error and sets @code{errno} to @code{EINVAL}, there is no 

+known way, directly or indirectly, to perform the wanted conversion.

+

+@cindex triangulation

+Triangulation is achieved by providing for each character set a 

+conversion from and to UCS-4 encoded @w{ISO 10646}.  Using @w{ISO 10646} 

+as an intermediate representation it is possible to @dfn{triangulate}

+(i.e., convert with an intermediate representation).

+

+There is no inherent requirement to provide a conversion to @w{ISO

+10646} for a new character set, and it is also possible to provide other

+conversions where neither source nor destination character set is @w{ISO

+10646}.  The existing set of conversions is simply meant to cover all 

+conversions that might be of interest.

+

+@cindex ISO-2022-JP

+@cindex EUC-JP

+All currently available conversions use the triangulation method above,

+making conversion run unnecessarily slow. If, for example, somebody 

+often needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution

+would involve direct conversion between the two character sets, skipping

+the input to @w{ISO 10646} first. The two character sets of interest

+are much more similar to each other than to @w{ISO 10646}.

+

+In such a situation one easily can write a new conversion and provide it

+as a better alternative. The GNU C library @code{iconv} implementation

+would automatically use the module implementing the conversion if it is

+specified to be more efficient.

+

+@subsubsection Format of @file{gconv-modules} files

+

+All information about the available conversions comes from a file named

+@file{gconv-modules} which can be found in any of the directories along

+the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented

+text files, where each of the lines has one of the following formats:

+

+@itemize @bullet

+@item

+If the first non-whitespace character is a @kbd{#} the line contains only 

+comments and is ignored.

+

+@item

+Lines starting with @code{alias} define an alias name for a character 

+set. Two more words are expected on the line.  The first word 

+defines the alias name, and the second defines the original name of the

+character set. The effect is that it is possible to use the alias name

+in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and

+achieve the same result as when using the real character set name.

+

+This is quite important as a character set has often many different

+names. There is normally an official name but this need not correspond to 

+the most popular name.  Beside this many character sets have special 

+names that are somehow constructed.  For example, all character sets 

+specified by the ISO have an alias of the form @code{ISO-IR-@var{nnn}} 

+where @var{nnn} is the registration number. This allows programs which 

+know about the registration number to construct character set names and 

+use them in @code{iconv_open} calls. More on the available names and 

+aliases follows below.

+

+@item

+Lines starting with @code{module} introduce an available conversion

+module. These lines must contain three or four more words.

+

+The first word specifies the source character set, the second word the

+destination character set of conversion implemented in this module, and 

+the third word is the name of the loadable module. The filename is

+constructed by appending the usual shared object suffix (normally

+@file{.so}) and this file is then supposed to be found in the same

+directory the @file{gconv-modules} file is in. The last word on the line, 

+which is optional, is a numeric value representing the cost of the

+conversion. If this word is missing, a cost of @math{1} is assumed. The

+numeric value itself does not matter that much; what counts are the

+relative values of the sums of costs for all possible conversion paths.

+Below is a more precise description of the use of the cost value.

+@end itemize

+

+Returning to the example above where one has written a module to directly

+convert from ISO-2022-JP to EUC-JP and back. All that has to be done is

+to put the new module, let its name be ISO2022JP-EUCJP.so, in a directory

+and add a file @file{gconv-modules} with the following content in the

+same directory:

+

+@smallexample

+module  ISO-2022-JP//   EUC-JP//        ISO2022JP-EUCJP    1

+module  EUC-JP//        ISO-2022-JP//   ISO2022JP-EUCJP    1

+@end smallexample

+

+To see why this is sufficient, it is necessary to understand how the

+conversion used by @code{iconv} (and described in the descriptor) is

+selected. The approach to this problem is quite simple.

+

+At the first call of the @code{iconv_open} function the program reads

+all available @file{gconv-modules} files and builds up two tables: one

+containing all the known aliases and another that contains the

+information about the conversions and which shared object implements

+them.

+

+@subsubsection Finding the conversion path in @code{iconv}

+

+The set of available conversions form a directed graph with weighted

+edges. The weights on the edges are the costs specified in the

+@file{gconv-modules} files. The @code{iconv_open} function uses an

+algorithm suitable for search for the best path in such a graph and so

+constructs a list of conversions which must be performed in succession

+to get the transformation from the source to the destination character

+set.

+

+Explaining why the above @file{gconv-modules} files allows the

+@code{iconv} implementation to resolve the specific ISO-2022-JP to

+EUC-JP conversion module instead of the conversion coming with the

+library itself is straightforward. Since the latter conversion takes two

+steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to

+EUC-JP), the cost is @math{1+1 = 2}.  The above @file{gconv-modules}

+file, however, specifies that the new conversion modules can perform this

+conversion with only the cost of @math{1}.

+

+A mysterious item about the @file{gconv-modules} file above (and also

+the file coming with the GNU C library) are the names of the character

+sets specified in the @code{module} lines. Why do almost all the names

+end in @code{//}?  And this is not all: the names can actually be

+regular expressions.  At this point in time this mystery should not be

+revealed, unless you have the relevant spell-casting materials: ashes

+from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix

+blessed by St.@: Emacs, assorted herbal roots from Central America, sand

+from Cebu, etc.  Sorry!  @strong{The part of the implementation where

+this is used is not yet finished.  For now please simply follow the

+existing examples.  It'll become clearer once it is. --drepper}

+

+A last remark about the @file{gconv-modules} is about the names not

+ending with @code{//}. Aa character set named @code{INTERNAL} is often 

+mentioned. From the discussion above and the chosen name it should have 

+become clear that this is the name for the representation used in the 

+intermediate step of the triangulation. We have said that this is UCS-4 

+but actually that is not quite right. The UCS-4 specification also 

+includes the specification of the byte ordering used. Since a UCS-4 value 

+consists of four bytes, a stored value is effected by byte ordering.  The 

+internal representation is @emph{not} the same as UCS-4 in case the byte 

+ordering of the processor (or at least the running process) is not the 

+same as the one required for UCS-4. This is done for performance reasons 

+as one does not want to perform unnecessary byte-swapping operations if 

+one is not interested in actually seeing the result in UCS-4. To avoid 

+trouble with endianess, the internal representation consistently is named 

+@code{INTERNAL} even on big-endian systems where the representations are 

+identical.

+

+@subsubsection @code{iconv} module data structures

+

+So far this section has described how modules are located and considered 

+to be used. What remains to be described is the interface of the modules

+so that one can write new ones. This section describes the interface as

+it is in use in January 1999. The interface will change a bit in the 

+future but, with luck, only in an upwardly compatible way.

+

+The definitions necessary to write new modules are publicly available

+in the non-standard header @file{gconv.h}.  The following text,

+therefore, describes the definitions from this header file.  First, 

+however, it is necessary to get an overview.

+

+From the perspective of the user of @code{iconv} the interface is quite

+simple: the @code{iconv_open} function returns a handle that can be used 

+in calls to @code{iconv}, and finally the handle is freed with a call to 

+@code{iconv_close}. The problem is that the handle has to be able to

+represent the possibly long sequences of conversion steps and also the

+state of each conversion since the handle is all that is passed to the

+@code{iconv} function. Therefore, the data structures are really the

+elements necessary to understanding the implementation.

+

+We need two different kinds of data structures. The first describes the

+conversion and the second describes the state etc. There are really two

+type definitions like this in @file{gconv.h}.

+@pindex gconv.h

+

+@comment gconv.h

+@comment GNU

+@deftp {Data type} {struct __gconv_step}

+This data structure describes one conversion a module can perform.  For

+each function in a loaded module with conversion functions there is

+exactly one object of this type.  This object is shared by all users of

+the conversion (i.e., this object does not contain any information

+corresponding to an actual conversion; it only describes the conversion

+itself).

+

+@table @code

+@item struct __gconv_loaded_object *__shlib_handle

+@itemx const char *__modname

+@itemx int __counter

+All these elements of the structure are used internally in the C library

+to coordinate loading and unloading the shared. One must not expect any

+of the other elements to be available or initialized.

+

+@item const char *__from_name

+@itemx const char *__to_name

+@code{__from_name} and @code{__to_name} contain the names of the source and

+destination character sets. They can be used to identify the actual

+conversion to be carried out since one module might implement conversions 

+for more than one character set and/or direction.

+

+@item gconv_fct __fct

+@itemx gconv_init_fct __init_fct

+@itemx gconv_end_fct __end_fct

+These elements contain pointers to the functions in the loadable module.

+The interface will be explained below.

+

+@item int __min_needed_from

+@itemx int __max_needed_from

+@itemx int __min_needed_to

+@itemx int __max_needed_to;

+These values have to be supplied in the init function of the module. The

+@code{__min_needed_from} value specifies how many bytes a character of

+the source character set at least needs. The @code{__max_needed_from}

+specifies the maximum value that also includes possible shift sequences.

+

+The @code{__min_needed_to} and @code{__max_needed_to} values serve the

+same purpose as @code{__min_needed_from} and @code{__max_needed_from} but 

+this time for the destination character set.

+

+It is crucial that these values be accurate since otherwise the

+conversion functions will have problems or not work at all.

+

+@item int __stateful

+This element must also be initialized by the init function. 

+@code{int __stateful} is nonzero if the source character set is stateful. 

+Otherwise it is zero.

+

+@item void *__data

+This element can be used freely by the conversion functions in the

+module. @code{void *__data} can be used to communicate extra information 

+from one call to another. @code{void *__data} need not be initialized if 

+not needed at all. If @code{void *__data} element is assigned a pointer 

+to dynamically allocated memory (presumably in the init function) it has 

+to be made sure that the end function deallocates the memory. Otherwise 

+the application will leak memory.

+

+It is important to be aware that this data structure is shared by all

+users of this specification conversion and therefore the @code{__data}

+element must not contain data specific to one specific use of the

+conversion function.

+@end table

+@end deftp

+

+@comment gconv.h

+@comment GNU

+@deftp {Data type} {struct __gconv_step_data}

+This is the data structure that contains the information specific to

+each use of the conversion functions.

+

+

+@table @code

+@item char *__outbuf

+@itemx char *__outbufend

+These elements specify the output buffer for the conversion step. The

+@code{__outbuf} element points to the beginning of the buffer, and

+@code{__outbufend} points to the byte following the last byte in the

+buffer. The conversion function must not assume anything about the size

+of the buffer but it can be safely assumed the there is room for at

+least one complete character in the output buffer.

+

+Once the conversion is finished, if the conversion is the last step, the

+@code{__outbuf} element must be modified to point after the last byte

+written into the buffer to signal how much output is available. If this

+conversion step is not the last one, the element must not be modified.

+The @code{__outbufend} element must not be modified.

+

+@item int __is_last

+This element is nonzero if this conversion step is the last one. This

+information is necessary for the recursion.  See the description of the

+conversion function internals below.  This element must never be

+modified.

+

+@item int __invocation_counter

+The conversion function can use this element to see how many calls of 

+the conversion function already happened. Some character sets require a 

+certain prolog when generating output, and by comparing this value with

+zero, one can find out whether it is the first call and whether, 

+therefore, the prolog should be emitted. This element must never be 

+modified.

+

+@item int __internal_use

+This element is another one rarely used but needed in certain

+situations. It is assigned a nonzero value in case the conversion

+functions are used to implement @code{mbsrtowcs} et.al.@: (i.e., the

+function is not used directly through the @code{iconv} interface).

+

+This sometimes makes a difference as it is expected that the

+@code{iconv} functions are used to translate entire texts while the

+@code{mbsrtowcs} functions are normally used only to convert single

+strings and might be used multiple times to convert entire texts.

+

+But in this situation we would have problem complying with some rules of

+the character set specification. Some character sets require a prolog

+which must appear exactly once for an entire text. If a number of

+@code{mbsrtowcs} calls are used to convert the text, only the first call

+must add the prolog.  However, because there is no communication between the

+different calls of @code{mbsrtowcs}, the conversion functions have no

+possibility to find this out. The situation is different for sequences

+of @code{iconv} calls since the handle allows access to the needed

+information.

+

+The @code{int __internal_use} element is mostly used together with 

+@code{__invocation_counter} as follows:

+

+@smallexample

+if (!data->__internal_use

+     && data->__invocation_counter == 0)

+  /* @r{Emit prolog.}  */

+  ...

+@end smallexample

+

+This element must never be modified.

+

+@item mbstate_t *__statep

+The @code{__statep} element points to an object of type @code{mbstate_t}

+(@pxref{Keeping the state}). The conversion of a stateful character

+set must use the object pointed to by @code{__statep} to store 

+information about the conversion state. The @code{__statep} element 

+itself must never be modified.

+

+@item mbstate_t __state

+This element must @emph{never} be used directly.  It is only part of

+this structure to have the needed space allocated.

+@end table

+@end deftp

+

+@subsubsection @code{iconv} module interfaces

+

+With the knowledge about the data structures we now can describe the

+conversion function itself. To understand the interface a bit of

+knowledge is necessary about the functionality in the C library that 

+loads the objects with the conversions.

+

+It is often the case that one conversion is used more than once (i.e.,

+there are several @code{iconv_open} calls for the same set of character

+sets during one program run).  The @code{mbsrtowcs} et.al.@: functions in

+the GNU C library also use the @code{iconv} functionality, which 

+increases the number of uses of the same functions even more.

+

+Because of this multiple use of conversions, the modules do not get 

+loaded exclusively for one conversion. Instead a module once loaded can 

+be used by an arbitrary number of @code{iconv} or @code{mbsrtowcs} calls 

+at the same time. The splitting of the information between conversion-

+function-specific information and conversion data makes this possible. 

+The last section showed the two data structures used to do this.

+

+This is of course also reflected in the interface and semantics of the

+functions that the modules must provide. There are three functions that

+must have the following names:

+

+@table @code

+@item gconv_init

+The @code{gconv_init} function initializes the conversion function

+specific data structure.  This very same object is shared by all

+conversions that use this conversion and, therefore, no state information

+about the conversion itself must be stored in here. If a module 

+implements more than one conversion, the @code{gconv_init} function will 

+be called multiple times.

+

+@item gconv_end

+The @code{gconv_end} function is responsible for freeing all resources

+allocated by the @code{gconv_init} function. If there is nothing to do,

+this function can be missing. Special care must be taken if the module

+implements more than one conversion and the @code{gconv_init} function

+does not allocate the same resources for all conversions.

+

+@item gconv

+This is the actual conversion function. It is called to convert one

+block of text. It gets passed the conversion step information

+initialized by @code{gconv_init} and the conversion data, specific to

+this use of the conversion functions.

+@end table

+

+There are three data types defined for the three module interface

+functions and these define the interface.

+

+@comment gconv.h

+@comment GNU

+@deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *)

+This specifies the interface of the initialization function of the

+module. It is called exactly once for each conversion the module

+implements.

+

+As explained in the description of the @code{struct __gconv_step} data

+structure above the initialization function has to initialize parts of

+it.

+

+@table @code

+@item __min_needed_from

+@itemx __max_needed_from

+@itemx __min_needed_to

+@itemx __max_needed_to

+These elements must be initialized to the exact numbers of the minimum

+and maximum number of bytes used by one character in the source and

+destination character sets, respectively. If the characters all have the

+same size, the minimum and maximum values are the same.

+

+@item __stateful

+This element must be initialized to an nonzero value if the source

+character set is stateful. Otherwise it must be zero.

+@end table

+

+If the initialization function needs to communicate some information

+to the conversion function, this communication can happen using the 

+@code{__data} element of the @code{__gconv_step} structure. But since 

+this data is shared by all the conversions, it must not be modified by 

+the conversion function. The example below shows how this can be used.

+

+@smallexample

+#define MIN_NEEDED_FROM         1

+#define MAX_NEEDED_FROM         4

+#define MIN_NEEDED_TO           4

+#define MAX_NEEDED_TO           4

+

+int

+gconv_init (struct __gconv_step *step)

+@{

+  /* @r{Determine which direction.}  */

+  struct iso2022jp_data *new_data;

+  enum direction dir = illegal_dir;

+  enum variant var = illegal_var;

+  int result;

+

+  if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)

+    @{

+      dir = from_iso2022jp;

+      var = iso2022jp;

+    @}

+  else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)

+    @{

+      dir = to_iso2022jp;

+      var = iso2022jp;

+    @}

+  else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)

+    @{

+      dir = from_iso2022jp;

+      var = iso2022jp2;

+    @}

+  else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)

+    @{

+      dir = to_iso2022jp;

+      var = iso2022jp2;

+    @}

+

+  result = __GCONV_NOCONV;

+  if (dir != illegal_dir)

+    @{

+      new_data = (struct iso2022jp_data *)

+        malloc (sizeof (struct iso2022jp_data));

+

+      result = __GCONV_NOMEM;

+      if (new_data != NULL)

+        @{

+          new_data->dir = dir;

+          new_data->var = var;

+          step->__data = new_data;

+

+          if (dir == from_iso2022jp)

+            @{

+              step->__min_needed_from = MIN_NEEDED_FROM;

+              step->__max_needed_from = MAX_NEEDED_FROM;

+              step->__min_needed_to = MIN_NEEDED_TO;

+              step->__max_needed_to = MAX_NEEDED_TO;

+            @}

+          else

+            @{

+              step->__min_needed_from = MIN_NEEDED_TO;

+              step->__max_needed_from = MAX_NEEDED_TO;

+              step->__min_needed_to = MIN_NEEDED_FROM;

+              step->__max_needed_to = MAX_NEEDED_FROM + 2;

+            @}

+

+          /* @r{Yes, this is a stateful encoding.}  */

+          step->__stateful = 1;

+

+          result = __GCONV_OK;

+        @}

+    @}

+

+  return result;

+@}

+@end smallexample

+

+The function first checks which conversion is wanted. The module from

+which this function is taken implements four different conversions; 

+which one is selected can be determined by comparing the names. The

+comparison should always be done without paying attention to the case.

+

+Next, a data structure, which contains the necessary information about 

+which conversion is selected, is allocated. The data structure

+@code{struct iso2022jp_data} is locally defined since, outside the 

+module, this data is not used at all. Please note that if all four 

+conversions this modules supports are requested there are four data 

+blocks.

+

+One interesting thing is the initialization of the @code{__min_} and

+@code{__max_} elements of the step data object. A single ISO-2022-JP

+character can consist of one to four bytes. Therefore the

+@code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined

+this way. The output is always the @code{INTERNAL} character set (aka

+UCS-4) and therefore each character consists of exactly four bytes. For

+the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into

+account that escape sequences might be necessary to switch the character

+sets.  Therefore the @code{__max_needed_to} element for this direction

+gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the

+two bytes needed for the escape sequences to single the switching. The

+asymmetry in the maximum values for the two directions can be explained

+easily: when reading ISO-2022-JP text, escape sequences can be handled

+alone (i.e., it is not necessary to process a real character since the

+effect of the escape sequence can be recorded in the state information).

+The situation is different for the other direction. Since it is in

+general not known which character comes next, one cannot emit escape

+sequences to change the state in advance. This means the escape

+sequences that have to be emitted together with the next character.

+Therefore one needs more room than only for the character itself.

+

+The possible return values of the initialization function are:

+

+@table @code

+@item __GCONV_OK

+The initialization succeeded

+@item __GCONV_NOCONV

+The requested conversion is not supported in the module.  This can

+happen if the @file{gconv-modules} file has errors.

+@item __GCONV_NOMEM

+Memory required to store additional information could not be allocated.

+@end table

+@end deftypevr

+

+The function called before the module is unloaded is significantly

+easier. It often has nothing at all to do; in which case it can be left

+out completely.

+

+@comment gconv.h

+@comment GNU

+@deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *)

+The task of this function is to free all resources allocated in the

+initialization function. Therefore only the @code{__data} element of

+the object pointed to by the argument is of interest. Continuing the

+example from the initialization function, the finalization function

+looks like this:

+

+@smallexample

+void

+gconv_end (struct __gconv_step *data)

+@{

+  free (data->__data);

+@}

+@end smallexample

+@end deftypevr

+

+The most important function is the conversion function itself, which can

+get quite complicated for complex character sets. But since this is not

+of interest here, we will only describe a possible skeleton for the

+conversion function.

+

+@comment gconv.h

+@comment GNU

+@deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int)

+The conversion function can be called for two basic reason: to convert

+text or to reset the state. From the description of the @code{iconv}

+function it can be seen why the flushing mode is necessary. What mode

+is selected is determined by the sixth argument, an integer.  This 

+argument being nonzero means that flushing is selected.

+

+Common to both modes is where the output buffer can be found. The

+information about this buffer is stored in the conversion step data. A

+pointer to this information is passed as the second argument to this 

+function. The description of the @code{struct __gconv_step_data} 

+structure has more information on the conversion step data.

+

+@cindex stateful

+What has to be done for flushing depends on the source character set.

+If the source character set is not stateful, nothing has to be done. 

+Otherwise the function has to emit a byte sequence to bring the state 

+object into the initial state. Once this all happened the other 

+conversion modules in the chain of conversions have to get the same 

+chance. Whether another step follows can be determined from the 

+@code{__is_last} element of the step data structure to which the first 

+parameter points.

+

+The more interesting mode is when actual text has to be converted. The 

+first step in this case is to convert as much text as possible from the 

+input buffer and store the result in the output buffer. The start of the 

+input buffer is determined by the third argument which is a pointer to a 

+pointer variable referencing the beginning of the buffer. The fourth 

+argument is a pointer to the byte right after the last byte in the buffer.

+

+The conversion has to be performed according to the current state if the

+character set is stateful. The state is stored in an object pointed to

+by the @code{__statep} element of the step data (second argument). Once

+either the input buffer is empty or the output buffer is full the

+conversion stops. At this point, the pointer variable referenced by the

+third parameter must point to the byte following the last processed

+byte (i.e., if all of the input is consumed, this pointer and the fourth

+parameter have the same value).

+

+What now happens depends on whether this step is the last one. If it is 

+the last step, the only thing that has to be done is to update the 

+@code{__outbuf} element of the step data structure to point after the

+last written byte. This update gives the caller the information on how 

+much text is available in the output buffer. In addition, the variable

+pointed to by the fifth parameter, which is of type @code{size_t}, must

+be incremented by the number of characters (@emph{not bytes}) that were

+converted in a non-reversible way. Then, the function can return.

+

+In case the step is not the last one, the later conversion functions have

+to get a chance to do their work. Therefore, the appropriate conversion

+function has to be called. The information about the functions is

+stored in the conversion data structures, passed as the first parameter.

+This information and the step data are stored in arrays, so the next

+element in both cases can be found by simple pointer arithmetic:

+

+@smallexample

+int

+gconv (struct __gconv_step *step, struct __gconv_step_data *data,

+       const char **inbuf, const char *inbufend, size_t *written,

+       int do_flush)

+@{

+  struct __gconv_step *next_step = step + 1;

+  struct __gconv_step_data *next_data = data + 1;

+  ...

+@end smallexample

+

+The @code{next_step} pointer references the next step information and

+@code{next_data} the next data record.  The call of the next function

+therefore will look similar to this:

+

+@smallexample

+  next_step->__fct (next_step, next_data, &outerr, outbuf,

+                    written, 0)

+@end smallexample

+

+But this is not yet all. Once the function call returns the conversion

+function might have some more to do. If the return value of the function 

+is @code{__GCONV_EMPTY_INPUT}, more room is available in the output 

+buffer. Unless the input buffer is empty the conversion, functions start 

+all over again and process the rest of the input buffer. If the return 

+value is not @code{__GCONV_EMPTY_INPUT}, something went wrong and we have 

+to recover from this.

+

+A requirement for the conversion function is that the input buffer

+pointer (the third argument) always point to the last character that

+was put in converted form into the output buffer. This is trivially

+true after the conversion performed in the current step, but if the

+conversion functions deeper downstream stop prematurely, not all

+characters from the output buffer are consumed and, therefore, the input

+buffer pointers must be backed off to the right position.

+

+Correcting the input buffers is easy to do if the input and output 

+character sets have a fixed width for all characters. In this situation 

+we can compute how many characters are left in the output buffer and, 

+therefore, can correct the input buffer pointer appropriately with a 

+similar computation. Things are getting tricky if either character set 

+has characters represented with variable length byte sequences, and it 

+gets even more complicated if the conversion has to take care of the 

+state. In these cases the conversion has to be performed once again, from 

+the known state before the initial conversion (i.e., if necessary the 

+state of the conversion has to be reset and the conversion loop has to be 

+executed again). The difference now is that it is known how much input 

+must be created, and the conversion can stop before converting the first 

+unused character. Once this is done the input buffer pointers must be 

+updated again and the function can return.

+

+One final thing should be mentioned. If it is necessary for the

+conversion to know whether it is the first invocation (in case a prolog

+has to be emitted), the conversion function should increment the 

+@code{__invocation_counter} element of the step data structure just 

+before returning to the caller. See the description of the @code{struct

+__gconv_step_data} structure above for more information on how this can

+be used.

+

+The return value must be one of the following values:

+

+@table @code

+@item __GCONV_EMPTY_INPUT

+All input was consumed and there is room left in the output buffer.

+@item __GCONV_FULL_OUTPUT

+No more room in the output buffer. In case this is not the last step

+this value is propagated down from the call of the next conversion

+function in the chain.

+@item __GCONV_INCOMPLETE_INPUT

+The input buffer is not entirely empty since it contains an incomplete

+character sequence.

+@end table

+

+The following example provides a framework for a conversion function.

+In case a new conversion has to be written the holes in this

+implementation have to be filled and that is it.

+

+@smallexample

+int

+gconv (struct __gconv_step *step, struct __gconv_step_data *data,

+       const char **inbuf, const char *inbufend, size_t *written,

+       int do_flush)

+@{

+  struct __gconv_step *next_step = step + 1;

+  struct __gconv_step_data *next_data = data + 1;

+  gconv_fct fct = next_step->__fct;

+  int status;

+

+  /* @r{If the function is called with no input this means we have}

+     @r{to reset to the initial state.  The possibly partly}

+     @r{converted input is dropped.}  */

+  if (do_flush)

+    @{

+      status = __GCONV_OK;

+

+      /* @r{Possible emit a byte sequence which put the state object}

+         @r{into the initial state.}  */

+

+      /* @r{Call the steps down the chain if there are any but only}

+         @r{if we successfully emitted the escape sequence.}  */

+      if (status == __GCONV_OK && ! data->__is_last)

+        status = fct (next_step, next_data, NULL, NULL,

+                      written, 1);

+    @}

+  else

+    @{

+      /* @r{We preserve the initial values of the pointer variables.}  */

+      const char *inptr = *inbuf;

+      char *outbuf = data->__outbuf;

+      char *outend = data->__outbufend;

+      char *outptr;

+

+      do

+        @{

+          /* @r{Remember the start value for this round.}  */

+          inptr = *inbuf;

+          /* @r{The outbuf buffer is empty.}  */

+          outptr = outbuf;

+

+          /* @r{For stateful encodings the state must be safe here.}  */

+

+          /* @r{Run the conversion loop.  @code{status} is set}

+             @r{appropriately afterwards.}  */

+

+          /* @r{If this is the last step, leave the loop. There is}

+             @r{nothing we can do.}  */

+          if (data->__is_last)

+            @{

+              /* @r{Store information about how many bytes are}

+                 @r{available.}  */

+              data->__outbuf = outbuf;

+

+             /* @r{If any non-reversible conversions were performed,}

+                @r{add the number to @code{*written}.}  */

+

+             break;

+           @}

+

+          /* @r{Write out all output which was produced.}  */

+          if (outbuf > outptr)

+            @{

+              const char *outerr = data->__outbuf;

+              int result;

+

+              result = fct (next_step, next_data, &outerr,

+                            outbuf, written, 0);

+

+              if (result != __GCONV_EMPTY_INPUT)

+                @{

+                  if (outerr != outbuf)

+                    @{

+                      /* @r{Reset the input buffer pointer.  We}

+                         @r{document here the complex case.}  */

+                      size_t nstatus;

+

+                      /* @r{Reload the pointers.}  */

+                      *inbuf = inptr;

+                      outbuf = outptr;

+

+                      /* @r{Possibly reset the state.}  */

+

+                      /* @r{Redo the conversion, but this time}

+                         @r{the end of the output buffer is at}

+                         @r{@code{outerr}.}  */

+                    @}

+

+                  /* @r{Change the status.}  */

+                  status = result;

+                @}

+              else

+                /* @r{All the output is consumed, we can make}

+                   @r{ another run if everything was ok.}  */

+                if (status == __GCONV_FULL_OUTPUT)

+                  status = __GCONV_OK;

+           @}

+        @}

+      while (status == __GCONV_OK);

+

+      /* @r{We finished one use of this step.}  */

+      ++data->__invocation_counter;

+    @}

+

+  return status;

+@}

+@end smallexample

+@end deftypevr

+

+This information should be sufficient to write new modules.  Anybody

+doing so should also take a look at the available source code in the GNU

+C library sources.  It contains many examples of working and optimized

+modules.

+

+@c File charset.texi edited October 2001 by Dennis Grace, IBM Corporation
\ No newline at end of file