What Insights Can eBPF Provide into Real-Time SSL/TLS Encrypted Traffic and How?
In applications utilizing secure communication like HTTPS or DoT (DNS over TLS) protocol, network traffic is encrypted. This encryption prevents traditional packet sniffing tools, such as Wireshark, from effectively monitoring or interpreting the data, as the captured information appears jiberish.
In today’s newsletter, I’ll demonstrate how eBPF user space probes (uprobes) can help overcome this limitation.
The Problem
SSL inspection tools, also known as legitimate man-in-the-middle (MiTM), intercept and decrypt SSL traffic for inspection by security software like L7 firewalls or anti-malware. Approved sessions are then re-encrypted and sent to their destination.
These tools can operate as client software on individual machines or on a proxy server at the network’s edge. Both approaches usually require TLS/SSL certificates and are computationally intensive due to the repetitive process of decryption and re-encryption.
Although decrypting data may seem unavoidable (since we need to inspect the content anyway), the exchange of SSL certificates poses potential security risks, as they must be safely stored and managed in multiple locations.
To avoid this, some companies just embed the observation logic directly into the application, before the point where data is encrypted by the TLS/SSL library. However, this approach has several flaws:
The responsibility of maintaining these changes within the application
Time required to merge the changes into production
The need for an application restart to apply the changes
So the question remains?
How can we efficiently monitor encrypted traffic without compromising security?
The Solution
Application traffic encryption is handled by user space libraries, and if you are familiar with eBPF, you may have come across user space probes or just uprobes.
uprobes allow you to dynamically attach eBPF programs to specific functions within user space libraries. These programs are then executed when a certain function is entered (or at specific offsets inside it), capturing its input arguments.
Similarly, the concept of uretprobes allows you to attach probes to the exit of functions and capture their return values once execution is complete.
Why am I talking about this?
User space probes are perfect for inspecting encrypted traffic. They allow you to hook into functions right before encryption (and right after decryption) takes place, without altering the application code. These probes:
Operate independently of the application
Can be dynamically attached, so you don’t have to restart your application
Do not require access to the SSL certificate, as encryption and decryption are unnecessary, reducing computational load.
The only real limitation of this approach is that it only works when the eBPF program is deployed on a node that handles both encryption and decryption of network traffic. This means it functions on individual machines but not at the network edge, although it still covers many use cases, such as company-issued computers pre-installed with observability or security tools.
We'll be observing application traffic encrypted with the OpenSSL library. If your applications use a different encryption library, you'll need to hook into that specific library, as each uprobe (and uretprobe) is tailored to the individual user-space function it targets.
To capture application data before encryption occurs, we need to attach probes to two functions:
SSL_write(): function used inside the OpenSSL library to write data to an SSL/TLS connection. We primarily utilize it to intercept the data (not yet encrypted) written by the client.
SSL_read(): function used inside the OpenSSL library to read data from an SSL/TLS connection. uprobe and uretprobe intercept the decrypted data received by the server as well as parse the return code e.g. HTTP status code like 200 (Successful GET request).
Let’s see it in action.
↖️ HTTP/S Server
↙️ eBPF Observability Program
➡️ HTTP Client
You can find the complete code on this link.
Performance Evaluation
A key concern outlined before was also that interception and parsing of each message sent to or from your application can impact application performance like latency or additional load on the host.
To address this, I ran a series of tests, measuring the average latency across 1,000 requests for each HTTP method type.
The perfomance of separate request type were fairly similar so I did not separate the tests for that.
The results indicate that the eBPF program adds a constant eBPF overhead of approximately 0.2µs on average — yes, remarkably low. Additionally, the average CPU load introduced by each hook is as follows:
0.1% for
uprobe/SSL_write*0.007% for
uprobe/SSL_read*0.3% for
uretprobe/SSL_read.
These findings address the trade-off between the added latency and CPU load due to eBPF instrumentation and the benefits of SSL/TLS encrypted traffic observation and analysis.
⏪ Did you miss the previous issues? I'm sure you wouldn't, but JUST in case:
I hope you find this resource as enlightening as I did. Stay tuned for more exciting developments and updates in the world of eBPF in next week's newsletter.
Until then, keep 🐝-ing!
Warm regards, Teodor