DevSecOps pipeline: security that fails the build
Context
Every new company repository started without a standard: different CI in each one, no security checks and Docker images running as root. I built a template repository with CI/CD and security ready to go, and applied the standard to 9 repositories.
The problem
- Security as an optional step doesn't happen. If the scan doesn't fail the build, nobody looks.
- The image shipped to production might not be the one that was tested, if the release rebuilt it.
- Tools downloaded without verification are a risk: if someone swaps the binary along the way, the pipeline runs foreign code.
What I decided
1. Three checks that fail the build
- Gitleaks: a secret in any commit of the history (not just the last one).
- Trivy: high or critical CVEs in dependencies and in the image, and a Dockerfile without
USER. - Semgrep: insecure code patterns.
2. The scanned image is the one that ships
The build is local, Trivy scans that image and only then is it published. On release, the vX.Y.Z tag just points to the same image, without a rebuild. If the tag's commit never went through CI, the release fails with a clear message.
3. Tools with pinned versions and checksums
Gitleaks and Trivy are downloaded from a specific version, and the SHA-256 is verified before use (the value comes from the official checksums file). Actions are pinned to the commit SHA, and Dependabot updates them.
4. --ignore-unfixed on Trivy
I fail on high/critical CVEs that already have a fix. A CVE with no fix available isn't something the team can resolve, and letting it fail the build only teaches everyone to ignore the scan.
A lesson I learned the hard way
In the first version of the public example repository, the image scan failed on a high CVE in urllib3, msgpack and setuptools. None were app dependencies: they came bundled inside pip and setuptools in the base image. The fix was a two-stage build (dependencies are installed in the first stage) and removing pip and setuptools from the final image, where they aren't needed. The scan did its job.
Result
- The standard is applied across 9 repositories in the organization.
- CI time dropped from 240 s to 139 s on the company pipelines.
- I wrote a generic version from scratch, with a sample app, and made it public. CI runs on PRs and on
main, and the release was tested with a real tag.
What I learned
- Security only works when it fails the build. An ignored warning is the same as no warning.
- What you test must be exactly what you publish.
- Failing on everything teaches people to ignore it. Choosing what fails is part of the design.