RbBCC demo

Rucy

• A compiler, which compiles Ruby scripts into BPF bytecode
• Presented at RubyKaigi 2021-takeout
• Uses mruby's bytecode specification and its compiler internally
• Can only run very simple programs

"Ruby" code sample for Rucy

license! "GPL"
section! "dev/cgroup"

class Ctx
  attr :access_type, :u32
  attr :major, :u32
  attr :minor, :u32
end

def prog(ctx)
  if ctx.minor == 9
    return 0
  else
    return 1
  end
end

This is (virtually) equivalent to this C code

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("cgroup/dev")
int bpf_prog1(struct bpf_cgroup_dev_ctx *ctx)
{
    if (ctx->minor == 9) {
        return 0;
    } else {
        return 1;
    }
}

char _license[] SEC("license") = "GPL";

Details of eBPF program displayed above

• To help understand the following slides
• Uses eBPF cgroup device program type
  • BPF_PROG_TYPE_CGROUP_DEVICE
• Filters devices accessible from containers (using cgroup v2)
  • e.g. deny if device is /dev/urandom, pass otherwise

Compile Ruby code into eBPF binary object

$ rucy object ./sample.rb --dest obj.o

Rucy、Rubyのスクリプト辺を渡すとその情報からBPFオブジェクトファイルを「コンパイル」するようになりました。
このオブジェクトをそのまま使って他言語でツールを書くなりできますが、せっかくなのでmrubyのlibbpfバインディングも作る。 pic.twitter.com/8JSwJySnNJ

— Uchio Kondo (@udzura) August 24, 2021

Basic loading code example of CGROUP_DEVICE

• Example using libbpf

// 抜粋
#include <bpf/libbpf.h>
struct bpf_object *obj;
int prog_fd, cgroup_fd;
bpf_prog_load("./obj.o", BPF_PROG_TYPE_CGROUP_DEVICE, &obj, &prog_fd);

cgroup_fd = open("/sys/fs/cgroup/test-device",  O_RDONLY);
bpf_prog_attach(prog_fd, cgroup_fd, BPF_CGROUP_DEVICE, 0);

Basic loading example (2)

• Confirming successfully loaded

$ sudo mkdir /sys/fs/cgroup/test-device
$ sudo ./loader
$ sudo bpftool prog
...
128: cgroup_device  name bpf_prog1  tag 02de78d75c0e331c  gpl
        loaded_at 2024-08-15T22:17:11+0900  uid 0
        xlated 40B  jited 84B  memlock 4096B
        btf_id 64

Verification

• Start a docker container with --pid=host and get the shell PID

$ sudo docker run -ti --pid=host debian:11-slim bash
root@987bbaa4c62c:/# echo $$
51959

• Write PID to the cgroup

$ echo 51959 | sudo tee /sys/fs/cgroup/test-device/cgroup.procs
51959

Verification (2)

• Confirm that it belongs to the specified cgroup

root@987bbaa4c62c:/# cat /proc/self/cgroup
0::/../../test-device

• Confirm that only /dev/urandom cannot be accessed

root@987bbaa4c62c:/# head -c 4 /dev/random | od
0000000 106533 052260
0000004
root@987bbaa4c62c:/# head -c 4 /dev/urandom | od
head: cannot open '/dev/urandom' for reading: Operation not permitted
0000000

c.f. kprobe example also available

• https://github.com/udzura/mruby-rubykaigi-rucy-sample

Technologies Required for Rucy Development

• mruby bytecode spec
• ELF binary & layout
• Rust
• How to create BPF binaries ...

How to create BPF binaries

• How are BPF binaries created?
• Let's follow the smallest sample

C code (again)

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("cgroup/dev")
int bpf_prog1(struct bpf_cgroup_dev_ctx *ctx)
{
    if (ctx->minor == 9) {
        return 0;
    } else {
        return 1;
    }
}

char _license[] SEC("license") = "GPL";

Convert to LLVM-IR

$ clang -g -O1 -c -S -emit-llvm \
    -target bpf \
    -o cgroup1.ll

; Excerpts/omitted debug information
define dso_local i32 @bpf_prog1(...) #0 section "cgroup/dev" {
  %2 = getelementptr inbounds %struct.bpf_cgroup_dev_ctx,
    %struct.bpf_cgroup_dev_ctx* %0, i64 0, i32 2
  %3 = load i32, i32* %2, align 4, !tbaa !3
  %4 = icmp ne i32 %3, 9
  %5 = zext i1 %4 to i32
  ret i32 %5
}

Convert to BPF binary

$ clang -g -O1 -c -target bpf cgroup1.ll -o cgroup1.o
$ llvm-objdump -x cgroup1.o
cgroup1.o:      file format elf64-bpf
architecture: bpfel
start address: 0x0000000000000000
...
Sections:
Idx Name          Size     VMA              Type
  0               00000000 0000000000000000 
  1 .strtab       00000054 0000000000000000 
  2 .text         00000000 0000000000000000 TEXT
  3 cgroup/dev    00000028 0000000000000000 TEXT
  4 license       00000004 0000000000000000 DATA ...

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000 cgroup1.c
0000000000000020 l       cgroup/dev     0000000000000000 LBB0_2
0000000000000000 g     F cgroup/dev     0000000000000028 bpf_prog1
0000000000000000 g     O license        0000000000000004 _license

Check out the generated bytecode

$ llvm-objdump -Sd cgroup1.o
...
Disassembly of section cgroup/dev:

0000000000000000 <bpf_prog1>:
       0:       61 11 08 00 00 00 00 00 r1 = *(u32 *)(r1 + 8)
       1:       b7 00 00 00 01 00 00 00 r0 = 1
       2:       55 01 01 00 09 00 00 00 if r1 != 9 goto +1 <LBB0_2>
       3:       b7 00 00 00 00 00 00 00 r0 = 0

0000000000000020 <LBB0_2>:
       4:       95 00 00 00 00 00 00 00 exit

Interpret the generated bytecode

0000000000000000 <bpf_prog1>:
                                     # Following the offset of ctx->minor
    0:       61 11 08 00 00 00 00 00 r1 = *(u32 *)(r1 + 8)
                                     # Setting the default return value
    1:       b7 00 00 00 01 00 00 00 r0 = 1
                                     # If ctx->minor != 9, then goto exit
    2:       55 01 01 00 09 00 00 00 if r1 != 9 goto +1 <LBB0_2>
                                     # Otherwise, set the return value to 0
    3:       b7 00 00 00 00 00 00 00 r0 = 0

0000000000000020 <LBB0_2>:
                                     # Exit the program
    4:       95 00 00 00 00 00 00 00 exit

FYI: Instruction format

• See: https://datatracker.ietf.org/doc/draft-ietf-bpf-isa/

   A basic instruction is encoded as follows:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    opcode     |     regs      |            offset             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                              imm                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

• e.g. 55 01 01 00 09 00 00 00
  • opcode = 55 reg = 01 offset = 01 00 immidiate = 09 00 00 00
  • little endian

Loading the binary

• Reposted codes

bpf_prog_load("./obj.o", BPF_PROG_TYPE_CGROUP_DEVICE, &obj, &prog_fd);
bpf_prog_attach(prog_fd, cgroup_fd, BPF_CGROUP_DEVICE, 0);

Looking at the implementation of cgroup-v2 device

• I generally avoid delving into kernel code, but I'll give it a try...
• Targeting Ubuntu 22.04 5.15.0-118-generic

I'm not familiar with Linux, but...

• [code link]

int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor,
				      short access, enum cgroup_bpf_attach_type atype)
{
	struct cgroup *cgrp;
	struct bpf_cgroup_dev_ctx ctx = {
		.access_type = (access << 16) | dev_type,
		.major = major, .minor = minor,
	};
	int allow;
	rcu_read_lock();
	cgrp = task_dfl_cgroup(current);
	allow = BPF_PROG_RUN_ARRAY_CG(cgrp->bpf.effective[atype], &ctx,
				      bpf_prog_run);
	rcu_read_unlock();
	return !allow;
}

static __always_inline u32
BPF_PROG_RUN_ARRAY_CG(const struct bpf_prog_array __rcu *array_rcu,
		      const void *ctx, bpf_prog_run_fn run_prog)
{
	const struct bpf_prog_array_item *item;
	const struct bpf_prog *prog;
	const struct bpf_prog_array *array;
	struct bpf_run_ctx *old_run_ctx;
	struct bpf_cg_run_ctx run_ctx;
	u32 ret = 1;

	migrate_disable();
	rcu_read_lock();
	array = rcu_dereference(array_rcu);
	item = &array->items[0];
	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
	while ((prog = READ_ONCE(item->prog))) {
		run_ctx.prog_item = item;
		ret &= run_prog(prog, ctx);
		item++;
	}
	bpf_reset_run_ctx(old_run_ctx);
	rcu_read_unlock();
	migrate_enable();
	return ret;
}

typedef unsigned int (*bpf_dispatcher_fn)(const void *ctx,
					  const struct bpf_insn *insnsi,
					  unsigned int (*bpf_func)(const void *,
								   const struct bpf_insn *));
//...
static __always_inline u32 __bpf_prog_run(const struct bpf_prog *prog,
					  const void *ctx,
					  bpf_dispatcher_fn dfunc)
{
	u32 ret;

	cant_migrate();
	if (static_branch_unlikely(&bpf_stats_enabled_key)) {
		struct bpf_prog_stats *stats;
		u64 start = sched_clock();

		ret = dfunc(ctx, prog->insnsi, prog->bpf_func);
		stats = this_cpu_ptr(prog->stats);
		u64_stats_update_begin(&stats->syncp);
		stats->cnt++;
		stats->nsecs += sched_clock() - start;
		u64_stats_update_end(&stats->syncp);
	} else {
		ret = dfunc(ctx, prog->insnsi, prog->bpf_func);
	}
	return ret;
}

static __always_inline u32 bpf_prog_run(const struct bpf_prog *prog, const void *ctx)
{
	return __bpf_prog_run(prog, ctx, bpf_dispatcher_nop_func);
}

BTW: What did I implement with Rucy?

• Created pass from mruby bytecode -> to BPF bytecode. Slide

Rucy's overall compilation pass

• RubyKaigi 2021 slide

Recap so far

• I've created RbBCC:
  • Ported BCC to Ruby
  • Got a rough idea of the API as a eBPF user
• I've created Rucy:
  • Enhanced understanding of how BPF programs work
  • Small programs can run if we implement the pass carefully
    • Became friends with binaries (?)

Incidentally, these stories...

• Are mostly in "Learning eBPF"
• More details are included in the book!
  (really? including about binary format too?)

To be honest

• Because I wanted to talk at RubyKaigi... lol
• For RbBCC: I wanted to learn more
  • So, why not port it?
  • After a quick search, I realized that libbcc was just one of FFI use cases, so I thought I could port it with less effort than I've expected
    • I was almost new to Python, but finally I became really good at ctypes...
  • It was fun to port it line by line

To be honest (2)

• For Rucy...
  • In the first place, I wanted to avoid to C boilerplate...
  • But I suddenly had an idea, and when I implemented it, it worked surprisingly and interestingly well
    • So I created the PoC
  • I thought it'd be interesting if I could do it, so I completed