最近在研究ebpf的應用,網上對較低版本的內核和centos操作系統的相關資料較少,這裡記錄一個自己環境配置&編譯運行一個ebpf的helloworld程式的過程。 > 環境是centos7.9,虛擬機安裝記憶體需要分配高一些,後續編譯llvm很吃性能 # 基礎依賴安裝 ## 升級內核版本 ebpf需 ...
最近在研究ebpf的應用,網上對較低版本的內核和centos操作系統的相關資料較少,這裡記錄一個自己環境配置&編譯運行一個ebpf的helloworld程式的過程。
環境是centos7.9,虛擬機安裝記憶體需要分配高一些,後續編譯llvm很吃性能
基礎依賴安裝
升級內核版本
ebpf需要至少內核是4.6+以上的版本,這裡選擇了4.18版本的內核
下載4.18版本內核的安裝包,下載鏈接
# 安裝4.18版本的內核
yum install -y kernel-ml-4.18.0-1.el7.elrepo.x86_64.rpm
# 修改啟動內核順序
yum install -y grub2-pc
grub-set-default 'CentOS Linux (4.18.0-1.el7.elrepo.x86_64) 7 (Core)'
重啟後可以確認下是否切換到4.18版本的內核
uname -sr
# Linux 4.18.0-1.el7.elrepo.x86_64
安裝gcc
編譯llvm 10+需要高版本gcc,這裡使用scl軟體集的方式升級gcc到7.3.1
yum install centos-release-scl
yum install devtoolset-7 -y
scl enable devtoolset-7 bash
echo "source /opt/rh/devtoolset-7/enable" >> ~/.bash_profile
source /opt/rh/devtoolset-7/enable
# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-7/root/usr --mandir=/opt/rh/devtoolset-7/root/usr/share/man --infodir=/opt/rh/devtoolset-7/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --with-default-libstdcxx-abi=gcc4-compatible --with-isl=/builddir/build/BUILD/gcc-7.3.1-20180303/obj-x86_64-redhat-linux/isl-install --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)
安裝llvm10+
libbpf框架的最新版本需要llvm10+的編譯支持,這裡下載了11版本,解壓後編譯安裝 鏈接
這裡make的需要的時間很長,建議掛著編譯去睡覺
unzip llvm-project-release-11.x.zip
cd llvm-project-release-11.x
mkdir build
cd build/
cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_RTTI=ON -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi" -G "Unix Makefiles" ../llvm
make
make install
藉助libbpf-bootstrap腳手架跑一個helloworld程式
搭建腳手架
libbpf-bootstrap項目提供了一個快速構建ebpf程式的框架,包括libbpf和bpftool兩大工具;項目包含一系列示常式序在examples/c
文件夾中,並提供了一個相對通用的Makefile可以供我們瞭解一個ebpf程式是如何編譯起來的
項目地址
克隆libbpf-bootstrap項目,並更新子項目
git clone https://github.com/libbpf/libbpf-bootstrap.git
git submodule update --init --recursive
編寫一個helloworld程式
現在可以在examples/c
文件夾下新建兩個文件,分別命名為hello.bpf.c
和hello.c
// hello.bpf.c
#define BPF_NO_GLOBAL_DATA
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("tracepoint/syscalls/sys_enter_execve")
int bpf_prog(void *ctx) {
char msg[] = "Hello, World!";
__bpf_printk("invoke bpf_prog: %s\n", msg);
return 0;
}
char LICENSE[] SEC("license") = "Dual BSD/GPL";
// hello.c
#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>
#include <bpf/libbpf.h>
#include "hello.skel.h"
static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
return vfprintf(stderr, format, args);
}
int main(int argc, char **argv)
{
struct hello_bpf *skel;
int err;
libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
/* Set up libbpf errors and debug info callback */
libbpf_set_print(libbpf_print_fn);
/* Open BPF application */
skel = hello_bpf__open();
if (!skel) {
fprintf(stderr, "Failed to open BPF skeleton\n");
return 1;
}
/* Load & verify BPF programs */
err = hello_bpf__load(skel);
if (err) {
fprintf(stderr, "Failed to load and verify BPF skeleton\n");
goto cleanup;
}
/* Attach tracepoint handler */
err = hello_bpf__attach(skel);
if (err) {
fprintf(stderr, "Failed to attach BPF skeleton\n");
goto cleanup;
}
printf("Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` "
"to see output of the BPF programs.\n");
for (;;) {
/* trigger our BPF program */
fprintf(stderr, ".");
sleep(1);
}
cleanup:
hello_bpf__destroy(skel);
return -err;
}
helloworld的程式hook了execve
系統調用,在進程創建時會調用execve
系統調用進入內核空間,此時被註入內核空間的ebpf位元組碼列印出一個字元串。
需要註意的是:
- 這個列印行為並不是在用戶空間進行的,需要通過
/sys/kernel/debug/tracing/trace_pipe
文件觀察列印輸出 - 在
hello.bpf.c
文件中在#include <bpf/bpf_helpers.h>
之前定義了一個#define BPF_NO_GLOBAL_DATA
巨集,這裡是關閉ebpf的全局變數功能,因為低版本的內核不支持這個特性,項目的示例minimal
和minimal_legacy
很明確的標註了這個問題
This version of minimal is modified to allow running on even older kernels that do not allow global variables. bpf_printk uses global variables unless BPF_NO_GLOBAL_DATA is defined before including bpf_helpers.h. Additionally, the global variable my_pid has been replaced with an array of one element to hold the process pid.
編譯運行helloworld程式
將自己的構建目標加入Makefile的APPS變數中,運行make $obj
即可編譯自己的ebpf程式
# Makefile
APPS = minimal minimal_legacy bootstrap uprobe kprobe fentry usdt sockfilter tc ksyscall hello
運行make hello
後,列印出整個項目的編譯流程,主要分為三個步驟如下
- 編譯libbpf
- 編譯bpftool
- 編譯hello
當然編譯一個ebpf程式的過程不是這麼簡單的過程,helloworld程式的具體編譯的過程會在後文介紹
# make hello
which: no cargo in (/opt/rh/devtoolset-7/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
MKDIR .output
MKDIR .output/libbpf
LIB libbpf.a
MKDIR /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/bpf.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/btf.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/libbpf.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/libbpf_errno.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/netlink.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/nlattr.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/str_error.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/libbpf_probes.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/bpf_prog_linfo.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/btf_dump.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/hashmap.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/ringbuf.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/strset.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/linker.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/gen_loader.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/relo_core.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/usdt.o
CC /root/libbpf-bootstrap/examples/c/.output//libbpf/staticobjs/zip.o
AR /root/libbpf-bootstrap/examples/c/.output//libbpf/libbpf.a
INSTALL bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h skel_internal.h libbpf_version.h usdt.bpf.h
INSTALL /root/libbpf-bootstrap/examples/c/.output//libbpf/libbpf.pc
INSTALL /root/libbpf-bootstrap/examples/c/.output//libbpf/libbpf.a
MKDIR bpftool
BPFTOOL bpftool/bootstrap/bpftool
... libbfd: [ OFF ]
... clang-bpf-co-re: [ on ]
... llvm: [ on ]
... libcap: [ OFF ]
MKDIR /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/include/bpf
INSTALL /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/include/bpf/hashmap.h
INSTALL /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/include/bpf/relo_core.h
INSTALL /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/include/bpf/libbpf_internal.h
MKDIR /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/
MKDIR /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/
MKDIR /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/bpf.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/btf.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/libbpf.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/libbpf_errno.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/netlink.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/nlattr.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/str_error.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/libbpf_probes.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/btf_dump.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/hashmap.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/ringbuf.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/strset.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/linker.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/gen_loader.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/relo_core.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/staticobjs/usdt.o
AR /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/libbpf/libbpf.a
INSTALL bpf.h libbpf.h btf.h libbpf_common.h libbpf_legacy.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h skel_internal.h libbpf_version.h usdt.bpf.h
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/main.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/common.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/json_writer.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/gen.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/btf.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/xlated_dumper.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/btf_dumper.o
CC /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/disasm.o
LINK /root/libbpf-bootstrap/examples/c/.output/bpftool/bootstrap/bpftool
BPF .output/hello.bpf.o
GEN-SKEL .output/hello.skel.h
CC .output/hello.o
BINARY hello
這裡我們的helloworld程式就編譯好了,嘗試運行一下,需要使用root用戶:
# ./hello
libbpf: loading object 'hello_bpf' from buffer
libbpf: elf: section(2) .symtab, size 120, link 1, flags 0, type=2
libbpf: elf: section(3) tracepoint/syscalls/sys_enter_execve, size 200, link 0, flags 6, type=1
libbpf: sec 'tracepoint/syscalls/sys_enter_execve': found program 'bpf_prog' at insn offset 0 (0 bytes), code size 25 insns (200 bytes)
libbpf: elf: section(4) .rodata.str1.1, size 35, link 0, flags 32, type=1
libbpf: elf: section(5) license, size 13, link 0, flags 3, type=1
libbpf: license of hello_bpf is Dual BSD/GPL
libbpf: elf: section(6) .BTF, size 438, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: looking for externs among 5 symbols...
libbpf: collected 0 externs total
libbpf: map '.rodata.str1.1' (global data): at sec_idx 4, offset 0, flags 80.
libbpf: map 0 is ".rodata.str1.1"
libbpf: map '.rodata.str1.1': skipped auto-creating...
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.
此時打開另一個終端查看/sys/kernel/debug/tracing/trace_pipe
文件可以看到程式列印:
# cat /sys/kernel/debug/tracing/trace_pipe
<...>-79115 [000] .... 67971.749552: 0: invoke bpf_prog: Hello, World!
<...>-79117 [000] .... 67971.753866: 0: invoke bpf_prog: Hello, World!
<...>-79119 [000] .... 67971.757380: 0: invoke bpf_prog: Hello, World!
<...>-79120 [000] .... 67971.761711: 0: invoke bpf_prog: Hello, World!
<...>-79121 [000] .... 67971.764028: 0: invoke bpf_prog: Hello, World!
<...>-79123 [000] .... 67971.769068: 0: invoke bpf_prog: Hello, World!
<...>-79124 [000] .... 67971.771786: 0: invoke bpf_prog: Hello, World!
<...>-79126 [000] .... 67971.776243: 0: invoke bpf_prog: Hello, World!
<...>-79127 [000] .... 67971.779373: 0: invoke bpf_prog: Hello, World!
<...>-79129 [000] .... 67971.784824: 0: invoke bpf_prog: Hello, World!
<...>-79131 [000] .... 67971.791846: 0: invoke bpf_prog: Hello, World!
<...>-79132 [000] .... 67979.733858: 0: invoke bpf_prog: Hello, World!
這裡我們的helloworld程式就運行成功了
to be continued