When developers need to provision cloud resources—be it a database, a storage bucket, or a virtual machine—they typically require credentials to authenticate with the cloud provider. In **Google Cloud**, this often involves **service account keys**: JSON files that establish identity and authorization. The inherent challenge with service account keys is their physical nature. They are files, prone to being copied, emailed, inadvertently committed to **Git** repositories, left on laptops, or simply forgotten. When team members depart, organizations frequently struggle to track and revoke all associated keys, leading to significant operational overhead and increased security risks. This phenomenon is widely known as **secret sprawl** – cloud credentials scattered across various systems, pipelines, and codebases, making auditing and removal exceptionally difficult. ## How GitOps Aims to Eliminate Credentials The **Kubernetes** community has addressed this challenge through **GitOps operators** (also known as controllers). The process typically involves: 1. A developer committing YAML configuration files, describing required resources, to Git and applying them to a Kubernetes cluster. 2. A controller running within the cluster reading these files and provisioning or updating the cloud resources on the developer's behalf. Crucially, this model ensures developers never directly handle cloud credentials. The controller authenticates using its own credentials, acting as an intermediary. **Google Kubernetes Config Connector (KCC)** exemplifies this system, commonly operating within a **Google Kubernetes Engine (GKE)** cluster. KCC monitors for configuration files detailing Google Cloud resources and interacts with the respective Google Cloud APIs to create or modify them. For instance, a developer seeking to grant an application permission to read objects from a storage bucket might submit an `IAMPolicyMember` resource: ```yaml apiVersion: iam.cnrm.cloud.google.com/v1beta1 kind: IAMPolicyMember metadata: name: my-binding namespace: my-team spec: member: "serviceAccount:[email protected]" role: roles/storage.objectViewer resourceRef: kind: Project external: "my-project" yaml apiVersion: iam.cnrm.cloud.google.com/v1beta1 kind: IAMPolicyMember metadata: name: escalation namespace: my-team spec: member: "serviceAccount:[email protected]" role: roles/owner resourceRef: apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1 kind: Organization external: "123456789" ``` By applying this YAML via Kubernetes, KCC reads the configuration and requests the IAM change from Google Cloud using its own service account. Since the KCC service account has the necessary permissions, the request is accepted. The attacker then gains control over the Google Cloud organization without ever directly interacting with Google Cloud credentials.  ## Two Authorization Systems, One Missing Check The root cause of ConfigConfusion lies in the interplay between two distinct authorization systems: 1. **Kubernetes RBAC**: This system governs user actions within the cluster. It verifies if a user is permitted to create an `IAMPolicyMember` resource in a given namespace. If allowed, it proceeds, unaware of the broader implications in Google Cloud. 2. **Google Cloud IAM**: This system dictates what a service account can do within Google Cloud. When KCC invokes Google Cloud to establish an IAM binding, it checks if KCC's service account has the necessary permissions. If so, the operation is permitted. Google Cloud, however, has no visibility into which Kubernetes user initiated the request and does not verify if that user should have been authorized to trigger it.  Consequently, Kubernetes only sees the resource being created internally, while Google Cloud only sees KCC's service account making the change. KCC can thus receive instructions from a user with limited privileges and execute them using its own, more extensive Google Cloud permissions. This is a classic example of a **confused deputy problem**: KCC, possessing broad authority, acts on instructions from less privileged users without adequately validating their right to issue such commands. The intentional design decision to remove cloud credentials from developers, while beneficial for secret sprawl, inadvertently severs the link between a developer's Kubernetes identity and the Google Cloud permissions used on their behalf. When KCC acts, Google Cloud attributes the action to KCC, not the original requester. ## Google's Stance: "Working as Designed" Google's official response to ConfigConfusion asserts that KCC is functioning as intended. Their position is that administrators consciously chose to grant KCC an organization-level service account and permitted developers to create `IAMPolicyMember` resources in KCC-managed namespaces. From Google's perspective, these are explicit configuration choices, and KCC merely executes the configured requests. While technically accurate, this explanation overlooks a crucial point: the KCC documentation does not clearly highlight the profound security implications of combining these two configuration decisions. Administrators often consider Kubernetes resource creation permissions and KCC's Google Cloud authority as separate concerns. However, in KCC's operational model, granting a team permission to create an `IAMPolicyMember` resource inherently provides a pathway to leverage KCC's elevated Google Cloud authority. Determining the precise, least-privilege permissions for KCC can be complex, often leading administrators to opt for broader, organization-level access for simplicity. This isn't necessarily poor administration but rather a natural consequence of a design that obscures the resulting authority gap. ## The Challenge of a Fix The most straightforward solution would involve KCC verifying if the Kubernetes user submitting a resource also possesses the corresponding Google Cloud permissions before executing the request. However, this approach is problematic because KCC's core design eliminates the need for developers to have direct Google Cloud identities. Implementing such a check would either necessitate assigning a Google Cloud identity to every Kubernetes user, thereby undermining KCC's original model, or introducing an additional authorization step and API calls to every reconciliation cycle, impacting performance. Therefore, Google's recommended mitigations primarily focus on reducing the scope of authority granted to KCC, rather than altering its authorization mechanism per request. ## Broader Implications: AWS and Azure It's important to note that this authorization vulnerability is not exclusive to KCC. Similar "confused deputy" risks can arise in other cloud environments like **AWS** and **Azure** when using similar GitOps or Infrastructure-as-Code (IaC) tools that operate with elevated permissions on behalf of less privileged users. Security professionals must be vigilant in understanding the full scope of permissions granted to such controllers and the potential for privilege escalation across disparate authorization systems.