Stealing the keys from the octopus: Exfiltrate Git Credentials in Argocd

Intro

Hello world, hello hackers! We're Future Sight - your soon-to-be favorite research group. Never heard of us? Don't worry, nobody has. This is literally our first drop.

Today, we will make some red teamers happy with a new technique we have discovered that allows an authenticated user in ArgoCD to steal the powerful GitHub credentials, further compromising Git accounts and more. For those folks who like Kubernetes security, you will enjoy even more!! Let's dive into the world of ArgoCD, Kubernetes, and Git and use the goddamn octopus against itself!

Motivation

It's the year 2025, and exfiltration of information is a buzz theme in cybersecurity, especially in those poor password managers. We want to contribute to that chaos a little bit and raise awareness among everyone in the cybersecurity field by demonstrating a way to exfiltrate git credentials in ArgoCD and Kubernetes.

Argocd is considered by most to be the best tool in GitOps continuous delivery for Kubernetes, and that's why we targeted it. In the CNCF landscape, it's in first place in the field of "Continuous Integration & Delivery".

Argocd might not be alone in this; similar tools with the same features might also allow these types of attacks. So buckle up kids; this is going to be an interesting ride!

WTF is Kubernetes?

Note: If you're already a black-belt at destroying Kubernetes clusters, accidentally nuking namespaces, and spending hours wondering why your pod refuses to receive traffic (spoiler: it was a missing Service all along)... You can safely skip this section. For the rest of you who have a life, enjoy this introduction!

Kubernetes is a container orchestration framework that helps to manage containers by taking care of scaling and failover. It facilitates the deployment and management of services using a declarative configuration and an automation approach. From the features that Kubernetes has, below are the ones that give Kubernetes its popularity:

• Storage Orchestration
• Load Balancing and Service discovering (using its internal DNS)
• Self-Healing
• Secret and Configuration Management
• Scaling
• Resources Management

Basic Concepts of Kubernetes

The largest unit in Kubernetes is a cluster, which comprises two primary components. On one side, you have the control plane that manages everything in the Kubernetes infrastructure, and on the other side, you have the node, which is where pods, the smallest unit in Kubernetes, are going to run. The Nodes are typically referred to as worker nodes, and the control plane is referred to as the master node. A Kubernetes cluster consists of one control plane and one or more worker nodes.

Below is an overview of the infrastructure:

The control plane is divided into 4 main components:

• kube-apiserver: The kube-apiserver is the frontend of the control plane; it exposes a REST API through which all other components interact.
• etcd: etcd is a key-value datastore to store all the cluster data.
• scheduler: The scheduler assigns nodes to newly created pods. It takes into consideration various factors before choosing one node to deploy the pod.
• The controller manager: The controller manager is a component consisting of other controllers. Its main purpose is to regulate the state of the system. The goal: to bring the current state closer to the desired state. The desired state is typically defined in the declared configuration under the 'spec' field.

Each worker node is divided into 3 main components:

• kubelet: Kubelet is an agent that runs on the Node machine, and its function is to take podspecs from the kubernetes api and ensure the containers defined are deployed.
• kube-proxy: It's a component that manages network traffic to and from Kubernetes Services. It monitors the API server for changes to Service objects and then configures the node's network to direct traffic to the corresponding Pods.
• container runtime: It's a fundamental component to deploy and maintain the lifecycle of containers. Kubernetes supports containerd, CRI-O, and any other implementation of the Kubernetes CRI, which is the interface that allows connecting to different container runtimes.

Kubernetes is composed of resources; below are the fundamental ones for this research:

• Pod: The smallest unit in Kubernetes, it encapsulates one or more containers that share the same volume and network.

• Replicaset: Ensures a specified number of pod replicas are running at any given time, replacing failed pods if necessary.

• Deployment: Manages the scaling and rollbacks of pods via replicaset.

• Service: A Resource that provides network and discovery for pods. It allows the exposure of pods both internally and externally.

• Configmap and Secrets: Configmaps store non-sensitive data; Secrets store sensitive data; Data can be environment variables or config files.

• Namespace: Logical partition inside the cluster. Used to organize the workloads and restrict resources.

Overview detailing how the resources interact with each other:

There are several implementations of the Kubernetes framework, the most notable of which are k3s, kubeadm, minikube, and microk8s. In the majority of these frameworks, besides the components we have already presented, there is another service that comes pre-installed, which is CoreDNS.

CoreDNS

Note: Attention kids this part is very important!

CoreDNS is an authoritative DNS server running in the kube-system namespace, used by Kubernetes clusters to provide service discovery and name resolution. This enables resources, such as pods, to discover other pods by name rather than IP address. When we create a service or a pod, Kubernetes automatically creates an A record for that service and pod in the CoreDNS.

The services can be called by other pods:

• Outside the same namespace: <service_name>.<namespace>, example: http[:]//service1.namespace1
• Inside the same namespace: <service_name>, example: http[:]//service1

Pods typically can be called by <pod-ipv4-address>.<service-name>.<my-namespace>.svc.<cluster-domain.example> however they can have a hostname and subdomain fields too, allowing them to be resolved as:

• <my-hostname>.<my-subdomain>.<my-namespace>.svc.cluster.local
• <my-hostname>.<my-service>.<my-namespace>.svc.cluster.local

By default, when a pod resolves a DNS name, it first checks if the Kubernetes DNS has the record; if not, it forwards the request to an external DNS server. Each pod will have a /etc/resolv.conf, which will be set up by the kubelet. If you view the resolv.conf file, you will see that it lists the IP address of kube-dns as the nameserver for resolving DNS names.

If you want more insight about how DNS works in Kubernetes the following is a good article.

And WTF is Argocd?

Now you have and overview of what Kubernetes is, let's introduce our new friend, Argocd. Argocd is a GitOps declarative continuous integration tool for Kubernetes. And you might ask, what the hell is GitOps? GitOps is a set of practices that use Git as a source of truth to manage and deploy applications. Essentially, Argocd will take a Git repo with Kubernetes manifests as input and apply them to the cluster.

Besides the UI and CLI, ArgoCD has three main components:

• Repository server: It's an internal component that holds a copy of the repositories configured. It is responsible for retrieving the manifests from the Git repository and generating them.

• API server: It's the only exposed component; it's a REST API consumed by the UI and CLI to manage everything in Argocd.

• Application Controller: Finally, the most important component, the Application Controller, is responsible for ensuring that the current state(Kubernetes) is equal to the desired state(Git Repo)

Overview architecture:

Features

As a service that connects to both Git servers and Kubernetes clusters, Argocd presents us with multiple features to support that:

• User Accounts: User Accounts provide Argocd with authentication and authorization when using the other features. They can be restricted by using RBAC (Role-Based Access Control) policies. It comes with the default user admin.

• Repositories: Allow us to connect to Git Repositories using multiple types of connections, from SSH keys to GitHub App.

• Clusters: Allow us to connect to Kubernetes clusters. It comes connected by default to the cluster where Argocd is deployed.

• Certificates: Allow developers to add self-signed certificates to connect to internal Git Servers.

• Applications: Applications are the resources that tell Argocd what Git repos to deploy into what Kubernetes clusters.

• Projects: Projects enable developers to specify which namespaces and Kubernetes resources can be deployed. By default, there is a project called Default with no restrictions.

Argocd Known Attacks

The initial focus of this article was to demonstrate the possible attacks that an authenticated attacker can perform in ArgoCD regarding the permissions of the user they compromise. Most known attacks do the following:

• Exploiting the argocd namespace because that's where the ArgoCD configurations exist; If an attacker can deploy in that namespace, they can override the argocd configmaps.

• Exploiting the default configurations within ArgoCD, such as the Default project or the Default admin user. With permissions to deploy anywhere, an attacker can become a cluster admin and take control of the whole cluster.

However, they are kinda lame and well-known, so we don't want them here.

There are talks that demonstrate some of these attacks: The Hidden Dangers of Defaults and Attacking Argo CD with Argo CD.

Without access to those resources, an attacker can do little more than attempt to probe for accessible services and identify potential permissions to exploit. This was the reason we started digging deeper to find another way to leverage a session that an attacker had obtained.

Game Plan

In ArgoCD, to retrieve manifests from a Git repository, you can configure a connection that stores the credentials for that repository. When called by an application, we don't need to re-enter the same credentials.

However, ArgoCD, for security reasons, hides the credentials once the repository is created, not even allowing the person who created the repository to view the password.

When we saw that, we immediately started to think of ways to leak the credentials, and if you are reading this, it's because we found a way!

Our first try was to find a way to set up a proxy in the repository. However, the proxy configuration is only available at the moment of creation of the repository, so we can't use that.

But what if we can create our own proxy and deceive argocd into thinking that it is connecting to a git server when in fact it's connecting to an attacker service...

The next thing that came to mind was Kubernetes DNS. As explained before, by default, Kubernetes creates DNS records for services and pods. And guess what is the first place that a pod is going to resolve a domain? If you guess the Kubernetes DNS, you are right!

What does that mean? Well, for example, if you can create a github.com DNS record pointing to an attacker-controlled service inside Kubernetes, the service receives the connection and does not go to the actual github.com(Well... we will proxy the requests, so in fact the connection goes to the actual github.com domain, but it will go through us first >:D).

To create a DNS record, an attacker has two paths:

• If he has access to the kube-system namespace, he can configure the coredns configmap and set up a malicious configuration, resolving the DNS names to the cluster IP of the service.
• If he does not have access to the kube-system namespace, he needs to create a DNS record using a namespace name, service name, and, if needed, a pod hostname and subdomain.

Note: We will not cover the coredns path because kube-system is a highly critical namespace, and we assume that it would be protected and monitored.

Okay, assuming the repository server is connected to our malicious service... What about our friend TLS? We need a valid certificate from a trusted CA to actually receive HTTPS connections and look at the content of the HTTP request. Using HTTPS typically stops these attacks, but not this time!

For our convenience, ArgoCD has a feature that allows us to add custom certificates for a domain. This functionality is useful for implementing TLS internally, but an attacker can leverage this to put the certificate used by the service in argocd. Now he can view the HTTP content, as Argocd trusts that certificate.

What is actually happening

• An attacker compromises an account with certain permissions

• He looks at the configured repositories and sees that it has a repo setup with credentials, and the domain is github.com for example

• He proceeds to deploy a malicious service with the name github and namespace com

• After the service is up and running, if the git server uses HTTPS, he creates a certificate in ArgoCD; if not, he moves to the next step

• Once the repo is called, the repository server, the component in argocd with the task of getting the manifests from the git repo, communicates with our malicious server because it resolves the DNS name first to the service cluster IP instead of the github.com public IP

• The attacker can now look at the credentials and forward the request to github.com to look at the response and maintain a successful connection from the argocd side

• Attacker lists the available repos with that creds and extracts the source code and secrets if the creds have read-only permission; otherwise, he can perform worse actions, such as injecting a malicious manifest or CI/CD into the repo

Below is an overview of the attack:

Requisites to perform this technique

It seems like all the stars have aligned... but the attacker still needs some prerequisites in the user account they compromise.

Must have policies:

• certificates: create
• applications: create, get(optional)
• clusters: get
• repositories: get
• projects: get
• logs: get(optional)

Note: Optional means that is not required for the technique to be sucessful but is a quality of life for the attacker

Having those policies, an attacker can now take advantage of the little octopus and start exfiltrating git credentials! Let's see what we have built for him(the attacker obviously hehe).

Here is a script to verify if a compromise user has permissions.

Argexfil

Having put our plan together, we just need to create the service that ArgoCD will connect to, instead of the actual Git service. For that service, we combined the words 'Argocd' and 'exfiltration' to create Argexfil.

Types of Connections

In Argocd, there are four types of connection methods to connect to git repos:

• via ssh
• via http/https
• via GitHub app
• via Google Cloud

We will only take into consideration http/https and github app connections.

The SSH connection we can never steal the private key even when we are between the connection, since it only sends the public key and the signature giving proof that argocd has the private key.

The Google Cloud connection is already deprecated, and it's no longer possible to use it for new customers starting on June 17, 2024, which is already in the past. We focus on the future.

For the HTTP and HTTPS connections, we can choose git or helm, but both are pretty easy to get the contents. For HTTP we don't need anything, just the DNS resolving to argexfil, and for HTTPS, we need an extra step, which is to put the certificate used in argexfil in the argocd.

For the GitHub app connections, it is used only in GitHub and GitHub Enterprise, and it works as follows:

• A developer configures the GitHub app with an app id, installation id and a private key.
• It then uses the private key to generate short-lived JWT tokens and exchanges them with api.github.com for an access token
• With that access token, it then gets the repo from github.com

Here we can do two types of attacks, and both simultaneously:

• capture the JWT token generated with the private key, which has a duration of 10 min max. We can then use it to generate an access token with a bigger expiration time, the default is 8 hours
• capture the access token which argocd defines 1 hour of expiration time, this time can be increased by the env variable: ARGOCD_GITHUB_APP_CREDS_EXPIRATION_DURATION

Git Credentials

Each type of connection has its own way of sending the GitHub credentials. For the http/https, both git and helm will send the credentials in the authorization header, encoding the username and password in base64, so it's pretty easy to extract it.

Example: Authorization: Basic dGVzdHVzZXI6dGVzdHBhc3N3b3JkCg== -> testuser:testpassword

Note: In some cases, developers use classic Person Access Tokens instead of passwords; however, these typically begin with ghp_, which distinguishes them.

For the GitHub app:

• jwt token: it typically has the format of a JWT token beginning with ey and it uses the endpoint /app/installations/<id>/access_tokens
• access token: it comes in the same header and same encoding as the git/helm, but with the prefix string x-access-token.

Features

Besides capturing the requests made by argocd and logging the request information to the output, we decided to add some features. Below are some of the features:

• Redirect the request received to the original destination and return the response
• Send the logs to an attacker destination outside of argocd

To add to that, Argocd lets us delete an application without cascading, meaning that the application is deleted from the ui and from Argocd entirely, but every resource created in that application is not deleted. The service will still run in Kubernetes and still receive ArgoCD connections. This can hide the attackers trace from the argocd side and only be visible to kubernetes side.

We can access the argexfil code here.

POC

Conclusion

In this research, we have identified a technique that enables an authenticated attacker to leak Git credentials in ArgoCD using Kubernetes DNS. This technique has never been mentioned before, and therefore, it has a lot of work to be done and possibly a greater impact. It has its limitations, but if we exfiltrate even one GitHub credential, we can potentially take control of a GitHub account and its repositories, as well as the organizations the user is a part of.

Fun Fact: We have approached the ArgoCD team and discussed this technique. They told us that they had never seen this technique before; however, they did not consider it a vulnerability because they believed it was a problem with Kubernetes default configurations, not Argocd. Additionally, they informed us that the TLS would mitigate this issue, but as demonstrated, if an attacker has a certificate "create" policy, they can still perform the attack. (Once we reported this issue, they were quick to discuss it, so kudos to them.)

Mitigations that can be in place to prevent this technique:

• Least privilege to the user deploying the services in argocd

• Create a more strict project inside argocd

• Monitoring in argocd, access to the argocd UI and CLI, and requests from internal pods to other pods, etc

• Create GitHub tokens with least privilege

• SSH can beat this technique since it never leaks the private key

• Restrict the "Certificates" feature to users, only allowing admins to use it

Future work:

• Find a way to inject a malicious YAML before sending the response to argocd. In our research, we discovered that ArgoCD uses Git packs to retrieve the content from the GitHub repository. Maybe find a way to pollute its content.

• Argocd has a feature to send notifications to Slack and other communication services; we can try to use the same method to exfiltrate tokens from them.

• We can try to leak the cluster keys used by argocd using this technique since the default cluster that comes configured communicates with https://kubernetes.default.svc. If we steal those keys, we might gain cluster admin access.

• We can try the same technique in other similar services like Flux or CircleCI if they share the same features.

Well thats it kids, stay safe! Until next time!