1. Mar 20, 2021
  2. Mar 19, 2021
    • Alexei Starovoitov's avatar
      Merge branch 'BPF static linking' · 6d8b2716
      Alexei Starovoitov authored
      
      
      Andrii Nakryiko says:
      
      ====================
      
      This patch set adds new libbpf APIs and their bpftool integration that allows
      to perform static linking of BPF object files. Currently no extern resolution
      across object files is performed. This is going to be the focus of the follow
      up patches. But, given amount of code and logic necessary to perform just
      basic functionality of linking together mostly independent BPF object files,
      it was decided to land basic BPF linker code and logic first and extend it
      afterwards.
      
      The motivation for BPF static linking is to provide the functionality that is
      naturally assumed for user-space development process: ability to structure
      application's code without artificial restrictions of having all the code and
      data (variables and maps) inside a single source code file.
      
      This enables better engineering practices of splitting code into
      well-encapsulated parts. It provides ability to hide internal state from other
      parts of the code base through static variables and maps. It is also a first
      steps towards having generic reusable BPF libraries.
      
      Please see individual patches (mostly #6 and #7) for more details. Patch #10
      passes all test_progs' individual BPF .o files through BPF static linker,
      which is supposed to be a no-op operation, so is essentially validating that
      BPF static linker doesn't produce corrupted ELF object files. Patch #11 adds
      Makefile infra to be able to specify multi-file BPF object files and adds the
      first multi-file test to validate correctness.
      
      v3->v4:
        - fix Makefile copy/paste error of diff'ing invalid object files (Alexei);
        - fix uninitialized obj_name variable that could lead to bogus object names
          being used during skeleton generation (kernel-patches CI);
      v2->v3:
        - added F(F(F(X))) = F(F(X)) test for all linked BPF object files (Alexei);
        - used reallocarray() more consistently in few places (Alexei);
        - improved bash completions for `gen object` (Quentin);
        - dropped .bpfo extension, but had to add optional `name OBJECT_FILE`
          parameter (path #8) to `gen skeleton` command to specify desired object
          name during skeleton generation;
        - fixed bug of merging DATASECS of special "license" and "version" sections.
          Linker currently strictly validates that all versions and licenses matches
          exactly and keeps only ELF symbols and BTF DATASEC from the very first
          object file with license/version. For all other object files, we ignore
          ELF symbols, but weren't ignoring DATASECs, which caused further problems
          of not being able to find a corresponding ELF symbol, if variable name
          differs between two files (which we test deliberately in multi-file
          linking selftest). The fix is to ignore BTF DATASECS;
      v1->v2:
        - extracted `struct strset` to manage unique set of strings both for BTF and
          ELF SYMTAB (patch #4, refactors btf and btf_dedup logic as well) (Alexei);
        - fixed bugs in bpftool gen command; renamed it to `gen object`, added BASH
          completions and extended/updated man page (Quentin).
      ====================
      
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      6d8b2716
    • Andrii Nakryiko's avatar
      selftests/bpf: Add multi-file statically linked BPF object file test · a0964f52
      Andrii Nakryiko authored
      
      
      Add Makefile infra to specify multi-file BPF object files (and derivative
      skeletons). Add first selftest validating BPF static linker can merge together
      successfully two independent BPF object files and resulting object and
      skeleton are correct and usable.
      
      Use the same F(F(F(X))) = F(F(X)) identity test on linked object files as for
      the case of single BPF object files.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-13-andrii@kernel.org
      a0964f52
    • Andrii Nakryiko's avatar
      selftests/bpf: Pass all BPF .o's through BPF static linker · 14137f3c
      Andrii Nakryiko authored
      
      
      Pass all individual BPF object files (generated from progs/*.c) through
      `bpftool gen object` command to validate that BPF static linker doesn't
      corrupt them.
      
      As an additional sanity checks, validate that passing resulting object files
      through linker again results in identical ELF files. Exact same ELF contents
      can be guaranteed only after two passes, as after the first pass ELF sections
      order changes, and thus .BTF.ext data sections order changes. That, in turn,
      means that strings are added into the final BTF string sections in different
      order, so .BTF strings data might not be exactly the same. But doing another
      round of linking afterwards should result in the identical ELF file, which is
      checked with additional `diff` command.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-12-andrii@kernel.org
      14137f3c
    • Andrii Nakryiko's avatar
      selftests/bpf: Re-generate vmlinux.h and BPF skeletons if bpftool changed · cab62c37
      Andrii Nakryiko authored
      Trigger vmlinux.h and BPF skeletons re-generation if detected that bpftool was
      re-compiled. Otherwise full `make clean` is required to get updated skeletons,
      if bpftool is modified.
      
      Fixes: acbd0620
      
       ("selftests/bpf: Add vmlinux.h selftest exercising tracing of syscalls")
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-11-andrii@kernel.org
      cab62c37
    • Andrii Nakryiko's avatar
      bpftool: Add `gen object` command to perform BPF static linking · d80b2fcb
      Andrii Nakryiko authored
      
      
      Add `bpftool gen object <output-file> <input_file>...` command to statically
      link multiple BPF ELF object files into a single output BPF ELF object file.
      
      This patch also updates bash completions and man page. Man page gets a short
      section on `gen object` command, but also updates the skeleton example to show
      off workflow for BPF application with two .bpf.c files, compiled individually
      with Clang, then resulting object files are linked together with `gen object`,
      and then final object file is used to generate usable BPF skeleton. This
      should help new users understand realistic workflow w.r.t. compiling
      mutli-file BPF application.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Reviewed-by: default avatarQuentin Monnet <quentin@isovalent.com>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-10-andrii@kernel.org
      d80b2fcb
    • Andrii Nakryiko's avatar
      bpftool: Add ability to specify custom skeleton object name · c4122665
      Andrii Nakryiko authored
      
      
      Add optional name OBJECT_NAME parameter to `gen skeleton` command to override
      default object name, normally derived from input file name. This allows much
      more flexibility during build time.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-9-andrii@kernel.org
      c4122665
    • Andrii Nakryiko's avatar
      libbpf: Add BPF static linker BTF and BTF.ext support · 8fd27bf6
      Andrii Nakryiko authored
      
      
      Add .BTF and .BTF.ext static linking logic.
      
      When multiple BPF object files are linked together, their respective .BTF and
      .BTF.ext sections are merged together. BTF types are not just concatenated,
      but also deduplicated. .BTF.ext data is grouped by type (func info, line info,
      core_relos) and target section names, and then all the records are
      concatenated together, preserving their relative order. All the BTF type ID
      references and string offsets are updated as necessary, to take into account
      possibly deduplicated strings and types.
      
      BTF DATASEC types are handled specially. Their respective var_secinfos are
      accumulated separately in special per-section data and then final DATASEC
      types are emitted at the very end during bpf_linker__finalize() operation,
      just before emitting final ELF output file.
      
      BTF data can also provide "section annotations" for some extern variables.
      Such concept is missing in ELF, but BTF will have DATASEC types for such
      special extern datasections (e.g., .kconfig, .ksyms). Such sections are called
      "ephemeral" internally. Internally linker will keep metadata for each such
      section, collecting variables information, but those sections won't be emitted
      into the final ELF file.
      
      Also, given LLVM/Clang during compilation emits BTF DATASECS that are
      incomplete, missing section size and variable offsets for static variables,
      BPF static linker will initially fix up such DATASECs, using ELF symbols data.
      The final DATASECs will preserve section sizes and all variable offsets. This
      is handled correctly by libbpf already, so won't cause any new issues. On the
      other hand, it's actually a nice property to have a complete BTF data without
      runtime adjustments done during bpf_object__open() by libbpf. In that sense,
      BPF static linker is also a BTF normalizer.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-8-andrii@kernel.org
      8fd27bf6
    • Andrii Nakryiko's avatar
      libbpf: Add BPF static linker APIs · faf6ed32
      Andrii Nakryiko authored
      
      
      Introduce BPF static linker APIs to libbpf. BPF static linker allows to
      perform static linking of multiple BPF object files into a single combined
      resulting object file, preserving all the BPF programs, maps, global
      variables, etc.
      
      Data sections (.bss, .data, .rodata, .maps, maps, etc) with the same name are
      concatenated together. Similarly, code sections are also concatenated. All the
      symbols and ELF relocations are also concatenated in their respective ELF
      sections and are adjusted accordingly to the new object file layout.
      
      Static variables and functions are handled correctly as well, adjusting BPF
      instructions offsets to reflect new variable/function offset within the
      combined ELF section. Such relocations are referencing STT_SECTION symbols and
      that stays intact.
      
      Data sections in different files can have different alignment requirements, so
      that is taken care of as well, adjusting sizes and offsets as necessary to
      satisfy both old and new alignment requirements.
      
      DWARF data sections are stripped out, currently. As well as LLLVM_ADDRSIG
      section, which is ignored by libbpf in bpf_object__open() anyways. So, in
      a way, BPF static linker is an analogue to `llvm-strip -g`, which is a pretty
      nice property, especially if resulting .o file is then used to generate BPF
      skeleton.
      
      Original string sections are ignored and instead we construct our own set of
      unique strings using libbpf-internal `struct strset` API.
      
      To reduce the size of the patch, all the .BTF and .BTF.ext processing was
      moved into a separate patch.
      
      The high-level API consists of just 4 functions:
        - bpf_linker__new() creates an instance of BPF static linker. It accepts
          output filename and (currently empty) options struct;
        - bpf_linker__add_file() takes input filename and appends it to the already
          processed ELF data; it can be called multiple times, one for each BPF
          ELF object file that needs to be linked in;
        - bpf_linker__finalize() needs to be called to dump final ELF contents into
          the output file, specified when bpf_linker was created; after
          bpf_linker__finalize() is called, no more bpf_linker__add_file() and
          bpf_linker__finalize() calls are allowed, they will return error;
        - regardless of whether bpf_linker__finalize() was called or not,
          bpf_linker__free() will free up all the used resources.
      
      Currently, BPF static linker doesn't resolve cross-object file references
      (extern variables and/or functions). This will be added in the follow up patch
      set.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-7-andrii@kernel.org
      faf6ed32
    • Andrii Nakryiko's avatar
      libbpf: Add generic BTF type shallow copy API · 9af44bc5
      Andrii Nakryiko authored
      
      
      Add btf__add_type() API that performs shallow copy of a given BTF type from
      the source BTF into the destination BTF. All the information and type IDs are
      preserved, but all the strings encountered are added into the destination BTF
      and corresponding offsets are rewritten. BTF type IDs are assumed to be
      correct or such that will be (somehow) modified afterwards.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-6-andrii@kernel.org
      9af44bc5
    • Andrii Nakryiko's avatar
      libbpf: Extract internal set-of-strings datastructure APIs · 90d76d3e
      Andrii Nakryiko authored
      
      
      Extract BTF logic for maintaining a set of strings data structure, used for
      BTF strings section construction in writable mode, into separate re-usable
      API. This data structure is going to be used by bpf_linker to maintains ELF
      STRTAB section, which has the same layout as BTF strings section.
      
      Suggested-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-5-andrii@kernel.org
      90d76d3e
    • Andrii Nakryiko's avatar
      libbpf: Rename internal memory-management helpers · 3b029e06
      Andrii Nakryiko authored
      
      
      Rename btf_add_mem() and btf_ensure_mem() helpers that abstract away details
      of dynamically resizable memory to use libbpf_ prefix, as they are not
      BTF-specific. No functional changes.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-4-andrii@kernel.org
      3b029e06
    • Andrii Nakryiko's avatar
      libbpf: Generalize BTF and BTF.ext type ID and strings iteration · f36e99a4
      Andrii Nakryiko authored
      
      
      Extract and generalize the logic to iterate BTF type ID and string offset
      fields within BTF types and .BTF.ext data. Expose this internally in libbpf
      for re-use by bpf_linker.
      
      Additionally, complete strings deduplication handling for BTF.ext (e.g., CO-RE
      access strings), which was previously missing. There previously was no
      case of deduplicating .BTF.ext data, but bpf_linker is going to use it.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-3-andrii@kernel.org
      f36e99a4
    • Andrii Nakryiko's avatar
      libbpf: Expose btf_type_by_id() internally · e14ef4bf
      Andrii Nakryiko authored
      
      
      btf_type_by_id() is internal-only convenience API returning non-const pointer
      to struct btf_type. Expose it outside of btf.c for re-use.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210318194036.3521577-2-andrii@kernel.org
      e14ef4bf
  3. Mar 18, 2021
    • Lorenzo Bianconi's avatar
      bpf, devmap: Move drop error path to devmap for XDP_REDIRECT · fdc13979
      Lorenzo Bianconi authored
      
      
      We want to change the current ndo_xdp_xmit drop semantics because it will
      allow us to implement better queue overflow handling. This is working
      towards the larger goal of a XDP TX queue-hook. Move XDP_REDIRECT error
      path handling from each XDP ethernet driver to devmap code. According to
      the new APIs, the driver running the ndo_xdp_xmit pointer, will break tx
      loop whenever the hw reports a tx error and it will just return to devmap
      caller the number of successfully transmitted frames. It will be devmap
      responsibility to free dropped frames.
      
      Move each XDP ndo_xdp_xmit capable driver to the new APIs:
      
      - veth
      - virtio-net
      - mvneta
      - mvpp2
      - socionext
      - amazon ena
      - bnxt
      - freescale (dpaa2, dpaa)
      - xen-frontend
      - qede
      - ice
      - igb
      - ixgbe
      - i40e
      - mlx5
      - ti (cpsw, cpsw-new)
      - tun
      - sfc
      
      Signed-off-by: default avatarLorenzo Bianconi <lorenzo@kernel.org>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Reviewed-by: default avatarIoana Ciornei <ioana.ciornei@nxp.com>
      Reviewed-by: default avatarIlias Apalodimas <ilias.apalodimas@linaro.org>
      Reviewed-by: default avatarCamelia Groza <camelia.groza@nxp.com>
      Acked-by: default avatarEdward Cree <ecree.xilinx@gmail.com>
      Acked-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Acked-by: default avatarShay Agroskin <shayagr@amazon.com>
      Link: https://lore.kernel.org/bpf/ed670de24f951cfd77590decf0229a0ad7fd12f6.1615201152.git.lorenzo@kernel.org
      fdc13979
    • Alexei Starovoitov's avatar
      Merge branch 'Provide NULL and KERNEL_VERSION macros in bpf_helpers.h' · 6b282765
      Alexei Starovoitov authored
      
      
      Andrii Nakryiko says:
      
      ====================
      
      Provide NULL and KERNEL_VERSION macros in bpf_helpers.h. Patch #2 removes such
      custom NULL definition from one of the selftests.
      
      v2->v3:
        - instead of vmlinux.h, do this in bpf_helpers.h;
        - added KERNEL_VERSION, which comes up periodically as well;
        - I dropped strict compilation patches for now, because we run into new
          warnings (e.g., not checking read() result) in kernel-patches CI, which
          I can't even reproduce locally. Also -Wdiscarded-qualifiers pragma for
          jit_disasm.c is not supported by Clang, it needs to be
          -Wincompatible-pointer-types-discards-qualifiers for Clang; we don't have
          to deal with that in this patch set;
      v1->v2:
        - fix few typos and wrong copy/paste;
        - fix #pragma push -> pop.
      ====================
      
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      6b282765
    • Andrii Nakryiko's avatar
      selftests/bpf: drop custom NULL #define in skb_pkt_end selftest · c53a3355
      Andrii Nakryiko authored
      
      
      Now that bpftool generates NULL definition as part of vmlinux.h, drop custom
      NULL definition in skb_pkt_end.c.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Link: https://lore.kernel.org/r/20210317200510.1354627-3-andrii@kernel.org
      
      
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c53a3355
    • Andrii Nakryiko's avatar
      libbpf: provide NULL and KERNEL_VERSION macros in bpf_helpers.h · 9ae2c26e
      Andrii Nakryiko authored
      
      
      Given that vmlinux.h is not compatible with headers like stddef.h, NULL poses
      an annoying problem: it is defined as #define, so is not captured in BTF, so
      is not emitted into vmlinux.h. This leads to users either sticking to explicit
      0, or defining their own NULL (as progs/skb_pkt_end.c does).
      
      But it's easy for bpf_helpers.h to provide (conditionally) NULL definition.
      Similarly, KERNEL_VERSION is another commonly missed macro that came up
      multiple times. So this patch adds both of them, along with offsetof(), that
      also is typically defined in stddef.h, just like NULL.
      
      This might cause compilation warning for existing BPF applications defining
      their own NULL and/or KERNEL_VERSION already:
      
        progs/skb_pkt_end.c:7:9: warning: 'NULL' macro redefined [-Wmacro-redefined]
        #define NULL 0
                ^
        /tmp/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:4:9: note: previous definition is here
        #define NULL ((void *)0)
      	  ^
      
      It is trivial to fix, though, so long-term benefits outweight temporary
      inconveniences.
      
      Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
      Link: https://lore.kernel.org/r/20210317200510.1354627-2-andrii@kernel.org
      
      
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      9ae2c26e
    • Yonghong Song's avatar
      bpf: net: Emit anonymous enum with BPF_TCP_CLOSE value explicitly · 97a19caf
      Yonghong Song authored
      
      
      The selftest failed to compile with clang-built bpf-next.
      Adding LLVM=1 to your vmlinux and selftest build will use clang.
      The error message is:
        progs/test_sk_storage_tracing.c:38:18: error: use of undeclared identifier 'BPF_TCP_CLOSE'
                if (newstate == BPF_TCP_CLOSE)
                                ^
        1 error generated.
        make: *** [Makefile:423: /bpf-next/tools/testing/selftests/bpf/test_sk_storage_tracing.o] Error 1
      
      The reason for the failure is that BPF_TCP_CLOSE, a value of
      an anonymous enum defined in uapi bpf.h, is not defined in
      vmlinux.h. gcc does not have this problem. Since vmlinux.h
      is derived from BTF which is derived from vmlinux DWARF,
      that means gcc-produced vmlinux DWARF has BPF_TCP_CLOSE
      while llvm-produced vmlinux DWARF does not have.
      
      BPF_TCP_CLOSE is referenced in net/ipv4/tcp.c as
        BUILD_BUG_ON((int)BPF_TCP_CLOSE != (int)TCP_CLOSE);
      The following test mimics the above BUILD_BUG_ON, preprocessed
      with clang compiler, and shows gcc DWARF contains BPF_TCP_CLOSE while
      llvm DWARF does not.
      
        $ cat t.c
        enum {
          BPF_TCP_ESTABLISHED = 1,
          BPF_TCP_CLOSE = 7,
        };
        enum {
          TCP_ESTABLISHED = 1,
          TCP_CLOSE = 7,
        };
      
        int test() {
          do {
            extern void __compiletime_assert_767(void) ;
            if ((int)BPF_TCP_CLOSE != (int)TCP_CLOSE) __compiletime_assert_767();
          } while (0);
          return 0;
        }
        $ clang t.c -O2 -c -g && llvm-dwarfdump t.o | grep BPF_TCP_CLOSE
        $ gcc t.c -O2 -c -g && llvm-dwarfdump t.o | grep BPF_TCP_CLOSE
                          DW_AT_name    ("BPF_TCP_CLOSE")
      
      Further checking clang code find clang actually tried to
      evaluate condition at compile time. If it is definitely
      true/false, it will perform optimization and the whole if condition
      will be removed before generating IR/debuginfo.
      
      This patch explicited add an expression after the
      above mentioned BUILD_BUG_ON in net/ipv4/tcp.c like
        (void)BPF_TCP_ESTABLISHED
      to enable generation of debuginfo for the anonymous
      enum which also includes BPF_TCP_CLOSE.
      
      Signed-off-by: default avatarYonghong Song <yhs@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Link: https://lore.kernel.org/bpf/20210317174132.589276-1-yhs@fb.com
      97a19caf
  4. Mar 17, 2021
  5. Mar 16, 2021
  6. Mar 11, 2021
  7. Mar 10, 2021
    • David S. Miller's avatar
      Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next · c1acda98
      David S. Miller authored
      
      
      Alexei Starovoitov says:
      
      ====================
      pull-request: bpf-next 2021-03-09
      
      The following pull-request contains BPF updates for your *net-next* tree.
      
      We've added 90 non-merge commits during the last 17 day(s) which contain
      a total of 114 files changed, 5158 insertions(+), 1288 deletions(-).
      
      The main changes are:
      
      1) Faster bpf_redirect_map(), from Björn.
      
      2) skmsg cleanup, from Cong.
      
      3) Support for floating point types in BTF, from Ilya.
      
      4) Documentation for sys_bpf commands, from Joe.
      
      5) Support for sk_lookup in bpf_prog_test_run, form Lorenz.
      
      6) Enable task local storage for tracing programs, from Song.
      
      7) bpf_for_each_map_elem() helper, from Yonghong.
      ====================
      
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      c1acda98