Tales, thoughts and random blurbs on software and systems

Opentelemetry

2023-05-27T00:00:00+10:00

OpenTelemetry Report

I recently went on a couple of months long migration to OpenTelemetry from DataDog, and here are some thoughts specific to OTLP that may be of interest to you.

TL;DR

Emitting all three signals (metrics, traces and logging) in concert and expecting to be able to easily correlate them in the telemetry backend is not straight forward path (also it highly depends on which telemetry backend you choose).

Metrics

For OpenTelemetry’s OTLP metrics, you can be assured that for a long-running service, this is pretty stable. You can choose to ingest with OTLP and fan-out to other vendors like DataDog; there are no surprises involved.

For FaaS workloads (AWS Lambda), it’s a different story. The OpenTelemetry Collector-based, metrics in a FaaS environment is not reliable. I hope it will eventually get there, but exactly when that will happen is unknown. It depends on some key features being implemented in Lambda runtime to mitigate issues involved with the ‘freeze’ and ‘thaw’ of containers.

Traces

OTLP-based traces are pretty stable in the long-running service world. Again, you can choose to ingest with OTLP and fan-out to other vendors like DataDog, and there are no surprises involved.

However, for FaaS workloads such as AWS Lambda, due to the aforementioned freeze & thaw interaction within the extensions’ environment, is yet to materialize fully.

Logs

For OTLP-based logs are a complicated story. It varies on the log sink involved, execution environment, and other constraints. In general, your mileage will vary. OTLP-based log pipelines are not yet widely adopted and are not production ready. So, this is a case of YMMV.

Continuous Profiling

OTLP-based continuous profiling is unheard of; it currently exists in the proposals’ basket.

Observations

OpenTelemetry’s list of dirty laundry is long, and it will take a while to get it widely adopted. But here are a few key ones I have observed:

Contributing vendors have many self-driven interests and agendas that do not necessarily benefit the OpenTelemetry’s open ecosystem.
OpenTelemetry working groups often fail to come to consensus thus impacting deliveries caused by countless number of mundane arguments on the technical merits of a given implementation and design, in addition to lack of leadership (sure, I understand large opensource efforts are not trivial, but it does not take 2+ years to deliver a feature).
OpenTelemetry development is actually siloed; I probably need not say more.

All of this brings me to the following conclusion:

Closed vendors like DataDog can charge $65 million dollars to one customer (just like a raging bull) because their o11y stack is fully baked (all four signals - metrics, traces, logs and profiling) and production ready.

However, all is not lost. OpenTelemetry has a lot of good things going for it, and I am excited to see where it goes. Here are some of the good things I love about OpenTelemetry:

Wide number of telemetry integrations with you name it vendor
An opportunity to shape the Observability landscape for the foreseeable future
Wide variety of language support, telemetry receivers, processors and exporters to choose from
A very active and responsive core team that is willing to help and contribute
Extreme momentum in the community relating to releasing features and bug fixes
Many future enhancements are shaping as we speak, such as using eBPF, client-side metrics aggregations, exemplars, and many Rust-based :) telemetry components are in the works, and many more.

Xdp Ebpf Fw

2020-12-26T00:00:00+11:00

XDP/eBPF IP-layer firewall in Rust!

One of the option available today to do sub-millisecond packet filtering at scale is to harness the facilities afforded by XDP, eBPF, Linux kernel and support provided by various NIC manufacturers.

The two (XDP and eBPF) have grown quite rapidly in the 5.x release of the Linux kernel and there is absolutely no reason you would not want to use it for your low-latency, high-performance use case.

In this post, I will be demonstrating a sample XDP code that performs IP-layer firewall function traditionally done by Netfilter/IPtables (thanks for all the fish Rusty Russell!) but ran as a offloaded XDP program by the eBPF VM in the Linux kernel. The walk-through was thoroughly tested on Ubuntu 20.10 with LLVM version 11.0.0 on QEMU/virtio. LLVM is required today because it can produce eBPF targets.

According to XDP developers, on a benchmark they conducted, XDP far out-performed traditional kernel network stack on Linux on all fronts - tx, rx and forward (in the order of 12-25x!) when used with a NIC that that offloads XDP such as the ones from Intel/Mellanox/Netronome (mlx/ixgbe/nfp etc).

Dependencies Installation and Rust setup

Install dependencies:

$ sudo apt-get install          \
    build-essential             \    
    libelf-dev                  \
    ca-certificates             \
    ca-certificates-java        \
    zlib1g-dev                  \
    llvm-11-dev                 \
    libclang-11-dev             \
    linux-headers-$(uname -r)

Install Rust (single user) using rustup

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ . ~/.bashrc # or logout then login.
$ rustup install nightly
$ rustup default nightly
$ rustc --version
 

Install Rust (system-wide) using rustup

$ sudo -i
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs |       \
    env RUSTUP_HOME=/opt/rust/rustup CARGO_HOME=/opt/rust/cargo     \
    sh -s -- --default-toolchain stable --profile default --no-modify-path -y
$ tee -a /root/.bashrc <<HD
# setup Rust environment
export RUSTUP_HOME=/opt/rust/rustup
export PATH=${PATH}:/opt/rust/cargo/bin
HD
$ . /root/.bashrc # or logout then login.
$ rustup install nightly
$ rustup default nightly
$ rustc --version

Install carfgo-bpf:

$ cargo install cargo-bpf

Walk-through

Create the eBPF project:

$ cargo bpf new xdp-ebpf-fw

Add a new eBPF program called fw:

$ cd xdp-ebpf-fw
$ cargo bpf add fw

Source for xdp-ebpf-fw/Cargo.toml:

[package]
name = "xdp-ebpf-fw"
version = "0.1.0"
edition = '2018'

[dependencies]
cty = "0.2"
redbpf-macros = "1.3"
redbpf-probes = "1.3"

[build-dependencies]
cargo-bpf = { version = "1.3", default-features = false }

[features]
default = []
probes = []

[lib]
path = "src/lib.rs"

[[bin]]
name = "fw"
path = "src/fw/main.rs"
required-features = ["probes"]

Source for fw/main.rs:

#![no_std]
#![no_main]

use core::fmt::Error;
use cty::*;
use redbpf_probes::xdp::prelude::*;

program!(0xFFFFFFFE, "GPL");

const TCP_XDP_DROP: XdpAction = XdpAction::Drop;
const UDP_XDP_DROP: XdpAction = XdpAction::Drop;
const XDP_PASS: XdpAction = XdpAction::Pass;

// XDP/eBPF based IP-layer firewall to drop all UDP packets.
// And, also drop all TCP packets destined to port 80.
#[xdp]
pub fn xdp_ip_firewall(ctx: XdpContext) -> XdpResult {
    if let Ok(ip_protocol) = get_ip_protocol(&ctx) {
        match ip_protocol as u32 {
            IPPROTO_UDP => return Ok(UDP_XDP_DROP), // drop it on the floor
            IPPROTO_TCP => {
                if let Ok(transport) = ctx.transport() {
                    if transport.dest() == 80 {
                        return Ok(TCP_XDP_DROP);  // drop it on the floor
                    }
                }
            }
            _ => return Ok(XDP_PASS), // pass it up the protocol stack
        }
    }
    return Ok(XDP_PASS); // pass it up the protocol stack
}

fn get_ip_protocol(ctx: &XdpContext) -> Result<u32, Error> {
    if let Ok(ip) = ctx.ip() {
        // We need to make raw pointer into a u32 so `unsafe` is required.
        unsafe {
            return Ok((*ip).protocol as u32);
        }
    }
    // Anything above `255` is reserved.
    return Ok(0x10000);
}

Build eBPF program:

$ cargo bpf build

Network queues on QEMU/virtio for XDP

On QEMU, we need to add 2N queues (where N is the number of vCPUs). The following <driver> section is required inside <interface> (use virsh edit [vmname]) is required (for 4 vCPUs, allocate 8 queues) as of QEMU version 4.2.1.

<interface ...>
[ ... ]
<driver name='vhost' txmode='iothread' ioeventfd='on' event_idx='off' queues='8' rx_queue_size='256' tx_queue_size='256'>
    <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/>
    <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
</driver>
[ ... ]
</interface>

Load the compiled eBPF program!

Load the compiled eBPF program:

$ cd xdp-ebpf-fw
$ cargo bpf load -i eth0 target/bpf/programs/fw/fw.elf

Profit!

:boom:

External resources

Interesting Links

2020-11-07T00:00:00+11:00

Interesting links

LWN’s documentation on io_uring

How io_uring and eBPF Will Revolutionize Programming in Linux

Make sure you have the latest Linux kernel (something like v5.9.0+) before opening the io_uring goodness.

Got 40GigE stream data to do DPI on x86? Do you have performance issue with the pattern matching library currently in use? No problem. Enter Intel Hyperscan.

Both io_uring and hyperscan has multitude of language bindings, so you are not necessarily stuck in C lalaland.

Until next interesting links, adios!

Capture Kernel Crash

2020-06-01T00:00:00+10:00

Capturing Linux kernel panic message on QEMU

In this small post, I will cover the following:

Capture Linux kernel panic message
Using softdog module to set the watchdog to auto-reboot upon Kernel panic.

Capture the panic

Log into your guest and update /etc/default/grub and add console=tty0 console=ttyS0,9600n8 to GRUB_CMDLINE_LINUX; Then run update-grub.

To capture a kernel panic message, run the virtual machine like so:

$ qemu-system-x86_64 -smp 2,sockets=2,cores=1,threads=1 -m 4096 -nographic -serial mon:stdio ubuntu18.04-1.qcow2

You will be greeted with the usual boot messages.

Upon kernel panic, a full trace message will appear on your attached console.

[  247.009745] BUG: stack guard page was hit at 000000006c1ac1e1 (stack is 00000000487682b5..000000001262eb3a)
[  247.012648] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[  247.012648] Modules linked in: helloworld(POE+) binfmt_misc ppdev kvm_amd kvm irqbypass input_leds serio_raw parport_pc parport sch_fq_codel softy
[  247.012648] CPU: 1 PID: 2102 Comm: insmod Tainted: P           OE    4.15.0-64-generic #73-Ubuntu
[  247.012648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[  247.012648] RIP: 0010:_ZN4core3num23_$LT$impl$u20$usize$GT$11checked_mul17h07ae83026124b5eaE+0x10/0x70 [helloworld]
[  247.012648] RSP: 0018:ffffa84b4169ffd0 EFLAGS: 00000282
[  247.012648] RAX: 00000000000000fe RBX: 0000000000000000 RCX: 0000000000000000
[  247.012648] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[  247.012648] RBP: ffffa84b416a0020 R08: ffffffffc0690d60 R09: 0000000000000003
[  247.012648] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc059edf0
[  247.070679] R13: ffff9c5d78655a00 R14: 0000000000000001 R15: ffff9c5d51aeb240
[  247.070679] FS:  00007f690a2ff540(0000) GS:ffff9c5d7fd00000(0000) knlGS:0000000000000000
[  247.070679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  247.070679] CR2: ffffa84b4169ffc8 CR3: 0000000137068000 CR4: 00000000000006e0
[  247.070679] Call Trace:
[  247.070679]  ? _ZN4core3num23_$LT$impl$u20$usize$GT$14saturating_mul17hc7be383430829098E+0x15/0x50 [helloworld]
[  247.070679]  ? _ZN4core5slice18from_raw_parts_mut17hc0afd704cde83aefE+0x62/0xd0 [helloworld]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$17get_unchecked_mut17h8+0x]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$9index_mut17h59d846e1fc+0x]
[  247.070679]  ? _ZN4core3cmp6min_by17h6a1a357c03d17732E+0x3d/0x90 [helloworld]
[  247.070679]  ? _ZN4core5slice77_$LT$impl$u20$core..ops..index..IndexMut$LT$I$GT$$u20$for$u20$$u5b$T$u5d$$GT$9index_mut17h6d899aa41763834eE+0x33/0]
[  247.070679]  ? _ZN79_$LT$linux_kernel_module..printk..LogLineWriter$u20$as$u20$core..fmt..Write$GT$9write_str17hb6b9f63c86335f4fE+0x127/0x250 [he]
[  247.070679]  ? _ZN4core3fmt9Formatter12pad_integral17hea95901f585e791cE+0x260/0x8b0 [helloworld]
[  247.070679]  ? _ZN4core3num23_$LT$impl$u20$usize$GT$14saturating_mul17hc7be383430829098E+0x15/0x50 [helloworld]
[  247.070679]  ? _ZN4core5slice14from_raw_parts17hc54a1a170e771755E+0x8f/0xd0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp7fmt_u6417hb10d5615ab39d4d8E+0x5e6/0x720 [helloworld]
[  247.070679]  ? _ZN4core10intrinsics19copy_nonoverlapping17h66731a07c4e66efbE+0xc2/0xd0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp51_$LT$impl$u20$core..fmt..Display$u20$for$u20$u8$GT$3fmt17h8a7427ec99f7ed22E+0x80/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num49_$LT$impl$u20$core..fmt..Debug$u20$for$u20$u8$GT$3fmt17h9010dd741ac356b3E+0x64/0xa0 [helloworld]
[  247.070679]  ? _ZN42_$LT$$RF$T$u20$as$u20$core..fmt..Debug$GT$3fmt17hce45b733e98582cdE+0x18/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner9is_pretty17h61db7b5335d258d0E+0x14/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner5entry28_$u7b$$u7b$closure$u7d$$u7d$17h17d35c1353ea2e80E+0x23c/0x250 [helloworld]
[  247.070679]  ? _ZN79_$LT$linux_kernel_module..printk..LogLineWriter$u20$as$u20$core..fmt..Write$GT$9write_str17hb6b9f63c86335f4fE+0x1be/0x250 [he]
[  247.070679]  ? _ZN4core6result19Result$LT$T$C$E$GT$8and_then17hc710b76977389ea0E+0x4e/0x70 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner5entry17h92ae2868f0df466bE+0x43/0x70 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders9DebugList5entry17he09db0b952ed03ccE+0x23/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders9DebugList7entries17h8eb3996229be728fE+0x88/0xa0 [helloworld]
[  247.070679]  ? _ZN48_$LT$$u5b$T$u5d$$u20$as$u20$core..fmt..Debug$GT$3fmt17h9c57c882f7ac9744E+0x51/0x70 [helloworld]
[  247.070679]  ? _ZN50_$LT$$RF$mut$u20$T$u20$as$u20$core..fmt..Debug$GT$3fmt17hb8d4e0e57dc57deaE+0x2a/0x40 [helloworld]
[  247.070679]  ? _ZN4core3fmt5write17hb916ed05c0beea6fE+0x285/0x640 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN11hello_world4foos17he560009d948fdbe9E+0x189/0x1e0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt10ArgumentV13new17hf3e64bcfe2a81583E+0x50/0x50 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt10ArgumentV13new17hf3e64bcfe2a81583E+0x50/0x50 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN83_$LT$hello_world..HelloWorldModule$u20$as$u20$linux_kernel_module..KernelModule$GT$4init17hcdf4c87881ba8bf7E+0x55/0xb0 [hello]
[  247.092990]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __switch_to_asm+0x41/0x70
[  247.093536]  ? init_module+0x14/0xd0 [helloworld]
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? kfree+0x165/0x180
[  247.093536]  ? do_one_initcall+0x52/0x19f
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? _cond_resched+0x19/0x40
[  247.093536]  ? kmem_cache_alloc_trace+0x14e/0x1b0
[  247.093536]  ? do_init_module+0x27/0x209
[  247.093536]  ? do_init_module+0x5f/0x209
[  247.093536]  ? load_module+0x1939/0x1f30
[  247.093536]  ? ima_post_read_file+0x96/0xa0
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SyS_finit_module+0xe/0x10
[  247.093536]  ? do_syscall_64+0x73/0x130
[  247.093536]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[  247.093536] Code: eb 08 48 c7 45 d0 00 00 00 00 48 8b 45 d0 48 8b 55 d8 48 83 c4 50 5d c3 00 00 00 55 48 89 e5 48 83 ec 50 48 89 7d e0 48 89 75 e 
[  247.097494] RIP: _ZN4core3num23_$LT$impl$u20$usize$GT$11checked_mul17h07ae83026124b5eaE+0x10/0x70 [helloworld] RSP: ffffa84b4169ffd0
[  247.097494] ---[ end trace b512caf3cbcd6f50 ]---
[  247.097494] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[  247.097494] 
[  247.097494] Kernel Offset: 0x1b400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  247.097494] Rebooting in 60 seconds..

Just for your information, you can pipe the kernel panic message to c++filt to get the demangled symbols (the kernel module is written in Rust and it does not yet seem to support [no_mangle]).

$ cat panic | c++filt 
[  247.009745] BUG: stack guard page was hit at 000000006c1ac1e1 (stack is 00000000487682b5..000000001262eb3a)
[  247.012648] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[  247.012648] Modules linked in: helloworld(POE+) binfmt_misc ppdev kvm_amd kvm irqbypass input_leds serio_raw parport_pc parport sch_fq_codel softy
[  247.012648] CPU: 1 PID: 2102 Comm: insmod Tainted: P           OE    4.15.0-64-generic #73-Ubuntu
[  247.012648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[  247.012648] RIP: 0010:core::num::<impl usize>::checked_mul::h07ae83026124b5ea+0x10/0x70 [helloworld]
[  247.012648] RSP: 0018:ffffa84b4169ffd0 EFLAGS: 00000282
[  247.012648] RAX: 00000000000000fe RBX: 0000000000000000 RCX: 0000000000000000
[  247.012648] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[  247.012648] RBP: ffffa84b416a0020 R08: ffffffffc0690d60 R09: 0000000000000003
[  247.012648] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc059edf0
[  247.070679] R13: ffff9c5d78655a00 R14: 0000000000000001 R15: ffff9c5d51aeb240
[  247.070679] FS:  00007f690a2ff540(0000) GS:ffff9c5d7fd00000(0000) knlGS:0000000000000000
[  247.070679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  247.070679] CR2: ffffa84b4169ffc8 CR3: 0000000137068000 CR4: 00000000000006e0
[  247.070679] Call Trace:
[  247.070679]  ? core::num::<impl usize>::saturating_mul::hc7be383430829098+0x15/0x50 [helloworld]
[  247.070679]  ? core::slice::from_raw_parts_mut::hc0afd704cde83aef+0x62/0xd0 [helloworld]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$17get_unchecked_mut17h8+0x]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$9index_mut17h59d846e1fc+0x]
[  247.070679]  ? core::cmp::min_by::h6a1a357c03d17732+0x3d/0x90 [helloworld]
[  247.070679]  ? core::slice::<impl core::ops::index::IndexMut<I> for [T]>::index_mut::h6d899aa41763834e+0x33/0]
[  247.070679]  ? <linux_kernel_module::printk::LogLineWriter as core::fmt::Write>::write_str::hb6b9f63c86335f4f+0x127/0x250 [he]
[  247.070679]  ? core::fmt::Formatter::pad_integral::hea95901f585e791c+0x260/0x8b0 [helloworld]
[  247.070679]  ? core::num::<impl usize>::saturating_mul::hc7be383430829098+0x15/0x50 [helloworld]
[  247.070679]  ? core::slice::from_raw_parts::hc54a1a170e771755+0x8f/0xd0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::fmt_u64::hb10d5615ab39d4d8+0x5e6/0x720 [helloworld]
[  247.070679]  ? core::intrinsics::copy_nonoverlapping::h66731a07c4e66efb+0xc2/0xd0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u8>::fmt::h8a7427ec99f7ed22+0x80/0xa0 [helloworld]
[  247.070679]  ? core::fmt::num::<impl core::fmt::Debug for u8>::fmt::h9010dd741ac356b3+0x64/0xa0 [helloworld]
[  247.070679]  ? <&T as core::fmt::Debug>::fmt::hce45b733e98582cd+0x18/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::is_pretty::h61db7b5335d258d0+0x14/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::entry::::h17d35c1353ea2e80+0x23c/0x250 [helloworld]
[  247.070679]  ? <linux_kernel_module::printk::LogLineWriter as core::fmt::Write>::write_str::hb6b9f63c86335f4f+0x1be/0x250 [he]
[  247.070679]  ? core::result::Result<T,E>::and_then::hc710b76977389ea0+0x4e/0x70 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::entry::h92ae2868f0df466b+0x43/0x70 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugList::entry::he09db0b952ed03cc+0x23/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugList::entries::h8eb3996229be728f+0x88/0xa0 [helloworld]
[  247.070679]  ? <[T] as core::fmt::Debug>::fmt::h9c57c882f7ac9744+0x51/0x70 [helloworld]
[  247.070679]  ? <&mut T as core::fmt::Debug>::fmt::hb8d4e0e57dc57dea+0x2a/0x40 [helloworld]
[  247.070679]  ? core::fmt::write::hb916ed05c0beea6f+0x285/0x640 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? hello_world::foos::he560009d948fdbe9+0x189/0x1e0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? core::fmt::ArgumentV1::new::hf3e64bcfe2a81583+0x50/0x50 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? core::fmt::ArgumentV1::new::hf3e64bcfe2a81583+0x50/0x50 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? <hello_world::HelloWorldModule as linux_kernel_module::KernelModule>::init::hcdf4c87881ba8bf7+0x55/0xb0 [hello]
[  247.092990]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __switch_to_asm+0x41/0x70
[  247.093536]  ? init_module+0x14/0xd0 [helloworld]
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? kfree+0x165/0x180
[  247.093536]  ? do_one_initcall+0x52/0x19f
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? _cond_resched+0x19/0x40
[  247.093536]  ? kmem_cache_alloc_trace+0x14e/0x1b0
[  247.093536]  ? do_init_module+0x27/0x209
[  247.093536]  ? do_init_module+0x5f/0x209
[  247.093536]  ? load_module+0x1939/0x1f30
[  247.093536]  ? ima_post_read_file+0x96/0xa0
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SyS_finit_module+0xe/0x10
[  247.093536]  ? do_syscall_64+0x73/0x130
[  247.093536]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[  247.093536] Code: eb 08 48 c7 45 d0 00 00 00 00 48 8b 45 d0 48 8b 55 d8 48 83 c4 50 5d c3 00 00 00 55 48 89 e5 48 83 ec 50 48 89 7d e0 48 89 75 e 
[  247.097494] RIP: core::num::<impl usize>::checked_mul::h07ae83026124b5ea+0x10/0x70 [helloworld] RSP: ffffa84b4169ffd0
[  247.097494] ---[ end trace b512caf3cbcd6f50 ]---
[  247.097494] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[  247.097494] 
[  247.097494] Kernel Offset: 0x1b400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  247.097494] Rebooting in 60 seconds..

I triggered the above Linux kernel panic while I was trying to understand how the Rust LKM worked; In this case, I allocated large kernel stack which was caught by the stack guard which caused an immediate Kernel panic. If you are into Rust and Linux kernel, I definitely recommend checking out linux-kernel-module-rust.

Auto-reboot upon Linux kernel panic

If you often get kernel panic (while developing/debugging kernel module or what not) or want to ensure system uptime in the face of Kernel instabilities, you could consider enabling softdog in-tree linux kernel module. This LKM runs a watchdog timer with a periodic heartbeat. Upon heartbeat failure, the timer associated with the heartbeat expires due to a kernel panic upon which, the system is rebooted. For this to work, a kernel parameter needs to be added to GRUB - GRUB_CMDLINE_LINUX="panic=60" in /etc/default/grub and then update-grub (for Ubuntu-like distro) needs to be run; Then the system needs to be rebooted.

Ubuntu has commented out the automatic loading of softdog kernel module, so it needs to be added back in. Remove softdog module from files that reside in /lib/modprobe.d/*.conf and then add it to auto-load in /etc/modules-load.d. Then run the following:

$ sudo update-initramfs -u
$ sudo depmod -a
$ sudo systemctl reboot

Feel free to drop a comment if anything. As usual, until next time - Adios!

Underrated Socat

2020-05-01T00:00:00+10:00

Socat

Socat is one of the most underrated software in Linux. It is a tool to help in debugging and applying network interops in ways the original author of a software probably had not anticipated. The author of socat describes it as multipurpose relay software (socket cat). People lovingly synonymize it as like cat but for sockets. I first came across socat when I was working as a pentester back in the day (I have worn many hats in the past) whilst trying to work my way through protocol debugging.

TL;DR? Refer to the commands instead.

The socat story

So, why would you need socat? Follow through on the fictionalised story below ¹.

Let’s pretend that we want to have a TCP client sitting locally to be able to connect to a TCP server sitting elsewhere on port 3450; i.e., a TCP network clients connect to a TCP socket on the server at port 3450. This is the trivial part and software libraries have evolved to abstract almost 99% of this game so engineers can have a good sleep.

Now, here’s the catch - say that Bob the sysadmin appeared on the deck of this ship and said that clients can only connect to TCP sockets on the server on port 80 and 443 ²

You can use many tools in Linux and *BSDs (and that’s the beauty of it - each to their own) to achieve the precise outcome (HAProxy, Nginx, iptables DNATs, SSH port forwards, IPsec tunnels, sshuttle, Wireguard etc..) but in this instance I would like to focus only on socat.

So, how could socat be of help here for a quick win? The TCP client will now connect to the server on port 80. That is, the TCP endpoint to the client will look like server.ip.address:80.

On the server, Bob, the sysadmin will spin up socat and run a TCP relay from the listening TCP port 80 to the local loopback address’ TCP port 3450. Thus, Bob, the sysadmin will run this on the server:

$ socat -v tcp-listen:80,reuseaddr,fork tcp-connect:127.0.0.1:3450

This will relay all TCP packets that connect to the server on port 80 to be sent to 127.0.0.1 on port 3450. As socat accepts a TCP client connection on port 80, it will fork a new child process, set up a bi-directional socket pairs on the child process and forward all incoming and outgoing TCP packets via these descriptors. The parent process will continue to service new TCP connections.

Until a permanent solution is in place², Bob can now proceed to block non-standard TCP port to ingress the server.

socat the swiss-army knife

An exhaustive list of socat capabilities is documented in the manual page for socat(1) and it is still the authoritative source. Below are some common ones one might need.

Relay TCP to TCP

$ socat -v tcp-listen:80,reuseaddr,fork tcp-connect:172.16.2.10:8080

Relay TCP to UDP

$ socat -v tcp-listen:80,reuseaddr,fork udp-connect:172.16.2.10:8080

Relay UDP to UDP

$ socat -v udp-listen:80,reuseaddr,fork udp-connect:172.16.2.10:8080

Relay UDP to TCP

$ socat -v udp-listen:80,reuseaddr,fork tcp-connect:172.16.2.10:8080

Relay TCP to Unix Domain Sockets

$ socat -v tcp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to TCP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork tcp-connect:172.16.2.10:80

Relay UDP to Unix Domain Sockets

$ socat -v udp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to UDP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork udp-connect:172.16.2.10:80

Relay SCTP to Unix Domain Sockets

$ socat -v sctp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to SCTP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork sctp-connect:172.16.2.10:80

Tunnel Unix Domain Socket packets inside TLS relay

Please avoid setting verify=0 on production if you can.

$ socat -v                                          \
    unix-listen:/var/run/uds.sock,reuseaddr,fork    \
    openssl:server.domain.name:443,certificate=cert.pem,cafile=cert.pem,verify=1,key=key.pem,commonname=server.domain.name

Relay TLS packets to Unix Domain Socket (aka virtual patching)

$ socat -v                                                                                             \
    openssl-listen:443,reuseaddr,pf=ip4,fork,key=key.pem,cafile=cert.pem,cert=cert.pem,method=TLS1.2   \
    unix-client:/var/run/uds.sock

Capture TLS packets to /dev/stdout for quick debugging

Please avoid setting verify=0 on production if you can.

$ socat -v                                                                                       				\
    openssl-listen:443,reuseaddr,pf=ip4,fork,key=key.pem,cafile=cert.pem,cert=cert.pem,verify=0,commonname=server.domain.name	\
    file:/dev/stdout

Feel free to drop me a comment if anything. Until next time adios!

¹ Everything in this story is fictional and has been trivialised for the purpose of simplicity. I am a big fan of KISS philosophy although I haven’t had the time to listen to the band KISS at all :smile:.

² Let’s pretend that there is a forcing function here.

Dnssec For The Win

2020-04-15T00:00:00+10:00

Fast Intro

DNSSEC uses digital signature to verify the authenticity and integrity of DNS records. In essence, it sets up a non-spoofable chain of trust right from the root zone down to the authoritative nameserver and further to modern caching resolvers (e.g., BIND, Unbound etc.). Of course, there’s more, on it and some more.

Why DNSSEC

There are many reasons to turn on DNSSEC, some of which some of them are outright detrimental to businesses (loss of revenue for instance).

It may surprise you to find a number of household domains do not use DNSSEC.

In general these are some reasons to use DNSSEC:

DNS Protocol Attacks
BGP Hijacking Attack¹
DNS Hijacking (Credential Theft)
Domain Theft
Cache Poisoning

It goes without saying that DNSSEC is not a panacea :pill:

Turn the secure bits on

To have DNSSEC turned on for a domain, you need three parties working in tandem with each other :revolving_hearts:

The Domain Name Registrar (e.g., Namecheap, Godaddy, tucows etc.) of the domain
The Authoritative Name Server (e.g., BYO BIND/NSD/PDNS, Route53, Vultr, NS1 etc.) of the domain
The “abstracted” Root Name Servers (I say “abstracted” as it will appear almost invisible to us :boom:)

For illustrative purpose, I will use my domain ishworgurung.com.

My personal domain ishworgurung.com is registered using Namecheap and the authoritative name server is at Vultr.

So, to setup DNSSEC for my domain, I need to generate a fresh copy of DS Records (Delegation Signer) from Vultr. Of particular use are:

Key Tag / Key Type (A unique key per DS record used for lookups) see here and here
Algorithm (crypto algorithm)
Digest Type (hashing algorithm)
Digest (hexadecimal representation of the digest)

Vultr by default generates three DS records, copy them to Namecheap’s DNSSEC console.

Handy tools

Some handy tools the internets has to offer :100:

https://dnsviz.net
https://dnssec-debugger.verisignlabs.com
Dig from Bind
- dig +short ds ishworgurung.com
- dig +short dnskey ishworgurung.com
- dig +short nsec ishworgurung.com

Conclusion

Someone on the internet said:

DNSSEC is a tool, not a religion. Please try to understand how the tool works before criticizing it.

And I agree. DNSSEC is a good thing - let’s do more of it; not less.

I leave you with Dr. Casey Deccio’s Hello Summer Break :sound:. Dr. Casey is the original author of DNSViz :beers:

¹Is BGP Safe Yet.

Vlan On Openbsd

2019-12-07T00:00:00+11:00

Three minute guide to set up VLAN on OpenBSD

If you live under the rock, go check out OpenBSD. It’s pretty nice. I have been running OpenBSD on my servers for about five years and I absolutely love it. Simplicity, robust, secure and minimal - you name it; It’s all there. If you got an old intel 386 or an RaspberryPI, it will be sufficient to run the humble OpenBSD.

Now onto how I VLAN my OpenBSD guest.

The vio0 NIC is based on a QEMU’ed guest on the usual VIO(4) driver.

Get these files up:

$> cat /etc/hostname.vio0 
up
$> cat /etc/hostname.vlan8
inet 10.2.1.61 255.255.255.0 NONE vlan 8 vlandev vio0
up

Then restart the network (or simply reboot)

$> sh /etc/netstart

Then after the ifconfig should look like this:

$> ifconfig # removed listing of other interfaces for brevity
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	lladdr ab:cd:ef:ab:cd:ef
	index 1 priority 0 llprio 3
	media: Ethernet autoselect
	status: active
vlan8: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	lladdr ab:cd:ef:ab:cd:ef
	index 4 priority 0 llprio 3
	encap: vnetid 8 parent vio0 txprio packet rxprio outer
	groups: vlan egress
	media: Ethernet autoselect
	status: active
	inet 10.2.1.61 netmask 0xffffff00 broadcast 10.2.1.255

Watch out for inet, parent, vnetid and broadcast in the above listing.

NB: Please note that the interfaces that sit between the packet path should all have equal MTUs (e.g. in the above listing, the MTU is 1500). Any switches that sit in between should also support 802.1q protocol so the VLAN’ed packets can be tagged accordingly. Thanks to martian67 for this.

PSA (Public Service Announcement)

If you run a network with IoT (Internet Of shit Things), consider putting them in a filtered VLAN that can only communicate within the LAN and not to the internet. This is because a lot of these IoT (Internet Of shit Things) call home and could potentially ex-filtrate private informations :boom:

Interesting Links 8

2019-12-06T00:00:00+11:00

Interesting links #8

Ftrace

BPF Performance Tools: Linux System and Application Observability

Interesting Links 7

2019-09-24T00:00:00+10:00

Interesting links #7

TL;DR

Watch Can strace make you fail? strace syscall fault injection by Dmitry Levin who is its current maintainer.

Non TL;DR

I find myself needing to use strace(1) to debug stuffs a lot but recently, I found myself re-visiting it. Enter syscall tampering / fault injection - Youtube talk by Dmitry Levin Fault Injection in strace.

The reason fault injection becomes a useful tool to have in the arsenal is because you may need to dynamically inject error path into codes developed using Go / Rust / C (amongst the modern language ecosystem). With strace(1), from the application’s point of view, it is all as if Linux kernel handled the system call as per your application’s need; where in-fact strace(1) dynamically instrumented the target syscall and injected the fault for your application.

The alternative to strace(1) is a little costly - bcc/ebpf that need to be instrumented at compile-time (but bcc/ebpf is extremely powerful!) or perf (extremely heavyweight).

Until next Interesting links, keep testing injecting those faults in your code! :-)

Interesting Links 6

2019-09-20T00:00:00+10:00

Interesting links #6

Pure Bash Bible

Platform Security Summit 2018 Videos