Protecting HashiCorp Vault using KubeArmor
Protecting K8s Secrets using Zero Trust Policy
HashiCorp Vault and CyberArk Conjur are widely used to store and manage encryption keys, passwords, and API credentials in on-premises clusters.
While these secrets are encrypted by default, encryption alone doesn't prevent ransomware attackers from deleting or re-encrypting them.
This alone can compromise the availability of your services that depend on these secrets, potentially resulting in significant financial and operational impacts.
In today’s newsletter, we’ll discuss how KubeArmor blocks malicious attacks and why use it instead of solutions like Tetragon or Falco.
⏪ Did you miss the previous issues? I'm sure you wouldn't, but JUST in case:
The Problem
In service deployments, secrets represent one of the most valuable targets for attackers, as they provide access to sensitive data and privileged operations.
Such incidents aren't rare at all—consider the UNC3944 Cyber Threat Group that successfully exploited a victim's HashiCorp Vault.
Not to mention, several other threats that can scrutinize the security of your systems:
Security misconfigurations, like overly permissive Role-Based Access Control (RBAC) policies
Inconsistent patch management, where failure to regularly update software and systems leaves known vulnerabilities open for exploitation
Phishing attacks, where attackers deceive employees into revealing credentials or sensitive information
Shadow IT, where employees use unauthorized applications or tools that bypass security controls
🔗 Here’s a great guide how HashiCorp Vault can be compromised.
With that in mind, regulations like GDPR, PCI-DSS, and HIPAA mandate strict protection of sensitive data, including secure secret management.
In Kubernetes, security is enforced through mechanisms like AppArmor, and SELinux. However, these solutions often complicate rather than resolve security challenges for several reasons:
AppArmor and SELinux require custom static rules set tailored to each service Pod to achieve true least privilege. Keeping these rules updated as services evolve is a significant operational challenge.
If an administrator deploys AppArmor or SELinux settings, and the application fails due to misconfiguration, it’s really-really hard to pinpoint the cause.
GCP and Azure support AppArmor, while EKS (Amazon Linux) and RHEL support only SELinux, making it harder to ensure consistent policy coverage since they operate in exclusive mode i.e., cannot be enabled at the same time
For these reasons, solutions like Falco, Tetragon, and KubeArmor are gaining popularity in the software industry, providing a high-level interface for enforcing granular security policies, simplifying debugging, and offering broad support across nearly all cloud environments.
Although Falco and Tetragon are widely adopted and provide various integrations, they share a critical limitation—they function on the principle of Post Attack Mitigation:
Falco can detect and alert on suspicious events but cannot inherently block the execution of attacks that lead to those events.
Tetragon, while capable of blocking threats, only does so after malicious actions have already occurred.
Why is their approach flawed?
As shown in the image above, Post Attack Mitigation works by terminating suspicious processes in response to alerts indicating malicious intent. By this time, however, the attacker could already gain access to sensitive data or even disable defenses before mitigation can take effect.
In other words, if we allow the attacker to execute malicious code—even for just a few microseconds—the damage is often already done.
For instance, ransomware can encrypt, move, or delete sensitive assets in mere fractions of a second. Consequently, any remedial action taken will likely be too late to prevent such malicious activities.
Check this blog from Grsecurity showcasing what can happen.
UPDATE 2024-11-10: Tetragon also leverages
bpf_override_return(), which allows it to override the return value of a function call, potentially blocking malicious function execution. However, this does not address the bypass techniques described in the blog above.
The Solution
KubeArmor takes a different approach — namely the Preemptive Mitigation.
In other words, only whitelisted actions are allowed to be spawned and everything else is denied before the attack even takes place.
However, it's not just about using the right tools to mitigate attacks; it's also about how easily these tools can be implemented and managed.
In other words, developers shouldn't have to constantly grapple with security policy updates and complex configurations, as misconfigurations can quickly lead to vulnerabilities.
So how does a developer determine which security policy to use for their deployments?
With KubeArmor, there’s a community maintained policy template repository, providing a set of pre-configured policies that are based on industry-leading compliance and attack frameworks such as CIS, MITRE, NIST-800-53, and STIGs.
But you don’t actually have to search through the repository for your policy; instead, you can simply use the kArmor CLI to suggest the policy for you.
These policies are constantly updated as software tools like HashiCorp Vault release new versions, so there’s no need for users to maintain them.
But pre-configured policies in that repositories only secure your workloads against known attacks.
What about unknown attacks? Is it even possible to defend against them?
This is where Zero Trust Policy comes into play.
This is just another way of saying — don't trust anything by default, just the actions you specifically allow.
To achieve that, you first need to identify legitimate traffic and operations within your service, by either using the KubeArmor GitHub Action or audit it using karmor logs.
To demonstrate all this, I decided to secure HashiCorp Vault Secret Manager.
I find code example renders in Substack tedious, so I’ll refer to my GitHub repository with the complete code demonstration.
Here’s the link.
I hope you find this resource as interesting as I do. Stay tuned for more exciting developments and updates in the world of eBPF in next week's newsletter.
Until then, keep 🐝-ing!
Warm regards, Teodor