MeridianMeridian

Where Secrets Really End Up in Docker Images: Testing 8 Common Methods

A detailed analysis of eight different approaches for handling secrets during Docker image builds reveals that most methods leave sensitive data exposed in layers, configuration files, or build history. Experiments conducted on Ubuntu 24.04.1 LTS with Docker Engine 29.1.3 and BuildKit v0.26.2 showed that simply using RUN rm after COPY or overwriting files fails to remove secrets from immutable layers. Environment variables via ENV and build arguments via ARG both persist in image metadata and history. Only BuildKit Secrets with --mount=type=secret and multi-stage builds that avoid copying secrets into the final stage successfully prevent leakage. The study demonstrates that secrets can be extracted from saved images without running any containers. Developers are advised to adopt these two secure patterns to protect credentials in container supply chains.

HabrTranslated from Russian

Read at source

A comprehensive technical investigation examined exactly where secrets end up when building Docker images using eight different techniques. The tests used Ubuntu 24.04.1 LTS, Docker Engine 29.1.3, and BuildKit v0.26.2, with a test secret string SUPER_SECRET_KOTY_HULIGANY_2026 placed in various ways.

The first method, COPY, adds the secret file directly into an image layer. Even after running docker history, the file remains fully recoverable by extracting layers from a saved tar archive. Adding a subsequent RUN rm command only marks the file as deleted in a new layer; the original data stays intact in the previous layer.

Using ENV stores the secret value in the image configuration JSON under config.Env and also appears in build history. The ARG instruction keeps the value in history.created_by fields even though it does not appear in the final container environment.

The dedicated BuildKit Secrets feature, invoked with --mount=type=secret and --secret build flag, mounts the secret only temporarily during the RUN step. After the build completes, the value is absent from all layers, configuration JSON files, and history entries.

Multi-stage builds also proved effective when the secret is created and used only in an intermediate stage and never copied into the final image. No trace of the secret appears in the published artifact.

Combining echo and rm inside a single RUN instruction still leaves the secret string visible in build history and created_by metadata. Overwriting a copied file with a new value similarly fails because the original layer remains unchanged.

The experiments confirm that only two approaches — BuildKit Secrets and properly isolated multi-stage builds — avoid embedding secrets in the final Docker image. All other common patterns leave extractable data that can be recovered without ever starting a container.

Docker EngineBuildKitAlpine
ShareXFacebookLinkedInReddit
Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Supply Chain & Open Source

Passwordless Kubernetes Image Pulling with Artifact Keeper via ServiceAccount Tokens
Supply Chain & Open Source

Passwordless Kubernetes Image Pulling with Artifact Keeper via ServiceAccount Tokens

Kubernetes 1.34 and later enables kubelet to forward a pod's ServiceAccount token to an external image credential provider plugin, allowing the plugin to exchange it for short-lived read-only credentials from a compatible registry. Artifact Keeper, an open-source Rust-based OCI and package registry, implements an OIDC provider that validates these tokens and issues temporary credentials without requiring imagePullSecrets in manifests. The approach follows zero-trust principles by eliminating long-lived secrets that can leak across namespaces or clusters. A custom credential provider binary handles the token exchange, enforces audience restrictions, and redacts sensitive data from logs. The solution requires specific ClusterRole permissions for nodes, kubelet flags, and registry-side OIDC mappings that bind Kubernetes namespaces to read-only groups. Testing on kind 1.34.3 confirmed successful pulls for authorized ServiceAccounts and clear 401 errors for unauthorized ones when imagePullPolicy is set to Always.

HabrOct 26, 2026Translated from Russian

Malicious Code Injected into LiteLLM Open Source AI Package via Supply Chain Attack
Supply Chain & Open Source

Malicious Code Injected into LiteLLM Open Source AI Package via Supply Chain Attack

The open-source generative AI tool LiteLLM had malicious code inserted into its official distribution packages, enabling credential theft and remote code execution. The compromise affected version 1.107.0 released on October 21, 2026, and was delivered through legitimate channels in a classic software supply chain attack. Seven malicious wheel files were uploaded to PyPI, while official container images on Docker Hub and GitHub Container Registry were also tainted. Security firm JFrog discovered that the malware steals a wide range of secrets including cloud credentials, SSH private keys, browser-stored passwords, development tokens, and AI service API keys from LiteLLM's database. The code also supports execution of externally fetched payloads and abuses Python's startup mechanisms to run even when the application itself is not loaded. This marks the second supply chain incident involving the project in 2026, following a separate event in March.

Security NEXTOct 23, 2026Translated from Japanese

GhostAction Campaign Compromises 790 GitHub Repositories via Malicious Workflows
Supply Chain & Open Source

GhostAction Campaign Compromises 790 GitHub Repositories via Malicious Workflows

Researchers from OpenSourceMalware have identified 790 publicly accessible GitHub repositories containing malicious workflows planted by the GhostAction campaign as of October 9. The campaign abuses trusted maintainer accounts to inject workflows that automatically exfiltrate secrets including AWS access keys and tokens for GitHub, GitLab, Google, Slack, SendGrid, OpenAI, Anthropic, and OpenRouter. Earlier counts by GitGuardian recorded 772 repositories between August 31 and September 30, though the datasets differ and are not cumulative. A separate wave on October 8 impacted 346 repositories linked to two maintainer accounts, including a project belonging to Uber. The malicious workflow runs on code pushes or manual triggers, scans repository files and full Git history for credentials, and sends findings in plaintext to attacker infrastructure. Removing the workflow file does not mitigate exposure, as successful executions already leak credentials and old commits may still contain secrets.

BoletimSecOct 21, 2026Translated from Portuguese