I have spent more time than I care to admit staring at CI/CD pipelines. Watching a progress bar crawl forward while a build runs for twenty minutes does something to your soul. Over the years, I have seen the same patterns repeat across teams and organizations: pipelines that started lean and fast, then gradually became bloated, slow, and painful. Here is what I have learned about why that happens and what you can do about it.
The hidden cost of slow pipelines
A slow pipeline is not just an inconvenience. It is a tax on every developer, on every commit, every single day.
Consider the math. If your pipeline takes 20 minutes and you have a team of eight engineers each pushing three times a day, that is 480 minutes of pipeline time per day. The developer does not sit idle during all of that, but context switching while waiting for a build to pass is not free either. Studies on developer productivity consistently show that long feedback loops lead to batching larger changes, which leads to more merge conflicts, which leads to longer review cycles, which leads to fewer deployments. It is a vicious cycle.
Teams with fast pipelines deploy more often. Teams that deploy more often ship smaller changes. Smaller changes are easier to review, easier to roll back, and less likely to cause incidents. Pipeline speed is not a vanity metric. It directly affects how quickly your team can deliver value and recover from mistakes.
Anti-pattern: The monolithic sequential pipeline
The most common pipeline I encounter looks something like this: checkout code, install dependencies, lint, build, run unit tests, run integration tests, build a Docker image, push to registry, deploy to staging, run smoke tests. All of it in a single job, one step after another.
This made sense when the project was small and the whole thing took three minutes. But projects grow. The test suite gets larger. The build gets heavier. New linting rules get added. Before long, the pipeline takes fifteen minutes and every step waits for the previous one to finish, even when there is no actual dependency between them.
Linting does not need to wait for the build. Unit tests do not need to wait for linting. The Docker image build does not need to wait for integration tests if it only packages the compiled artifact. Yet in a sequential pipeline, everything waits for everything.
Anti-pattern: No caching
I have seen pipelines that download 800MB of node_modules on every single run. Or pipelines that pull every Go module from scratch. Or Python pipelines that compile C extensions from source because nobody set up a wheel cache.
Dependency installation is often the single biggest time sink in a pipeline, and it is almost entirely redundant. Your dependency tree does not change on most commits. If your lockfile has not changed, there is no reason to re-resolve and re-download everything.
The same applies to build artifacts. If you are building a Docker image with a multi-stage build, and the first stage installs system packages that rarely change, that layer should be cached. If your compiled output has not changed because only a README was modified, rebuilding from scratch is wasted work.
Anti-pattern: Running everything on every commit
Not every commit needs the full treatment. A typo fix in documentation does not need integration tests. A change to a single microservice does not need to rebuild and test every other service in the monorepo.
Yet many pipelines are configured as all-or-nothing. Every push triggers the same comprehensive pipeline regardless of what changed. This is the safe default, and I understand the instinct, but it trades speed for a false sense of thoroughness. You can be selective without being reckless.
Practical fixes
Parallelize aggressively
Most CI systems support parallel jobs or stages. Use them. Here is how I typically structure a pipeline:
The first stage runs in parallel: linting, unit tests, and the build step all kick off simultaneously. They share nothing except the checked-out source code. The second stage depends on the first: integration tests run against the built artifact. The third stage handles packaging and deployment.
This alone can cut pipeline time by 40-50% without changing a single test or build step. The total compute time is the same, but the wall clock time drops significantly because independent work happens concurrently.
Cache everything that makes sense
At a minimum, cache your dependency directories keyed on the lockfile hash. Every major CI platform supports this natively: GitHub Actions has actions/cache, GitLab CI has its cache directive, and CircleCI has save_cache/restore_cache.
For Docker builds, use BuildKit layer caching. Push your cache layers to a registry so they persist across runs. If you are on GitHub Actions, the docker/build-push-action with cache-from and cache-to options works well.
For compiled languages, cache the build output directory. Go's module and build cache, Rust's target directory, and Gradle's build cache can all be preserved between runs.
The key is to be deliberate about cache keys. A cache that never invalidates is just as bad as no cache at all. Key on the files that actually determine whether the cache is valid: lockfiles for dependencies, source hashes for build artifacts.
Test selectively
This requires more investment but pays off enormously in monorepos or large codebases. The idea is simple: figure out what changed, determine what that change could affect, and only run the relevant tests.
In a monorepo, this might mean using a tool like Nx, Turborepo, or Bazel to understand the dependency graph and run tests only for affected packages. In a polyrepo setup, it might mean skipping integration tests when only unit-tested code has changed, or skipping the entire pipeline when only documentation files were modified.
You do not need a sophisticated build graph tool to start. Even a simple check like "did any source file change, or only markdown files?" can save minutes on documentation-only commits. Start simple and add granularity as the need arises.
Use faster runners when it matters
This is the least interesting optimization but sometimes the most effective. If your pipeline is CPU-bound during compilation or test execution, upgrading from a 2-core runner to an 8-core runner can cut build times by more than half. The cost of larger runners is almost always less than the cost of developer time spent waiting.
Self-hosted runners can also help if you need persistent caches, GPU access, or specific hardware. But they come with maintenance overhead, so weigh that tradeoff carefully.
When to invest in pipeline optimization
Not every slow pipeline needs fixing. Here is my rough heuristic:
If your pipeline takes under 5 minutes, it is probably fine. Developers can grab coffee, check Slack, and come back to a green build. The context switch cost is low.
If it takes 5 to 15 minutes, it is worth spending a day or two on low-hanging fruit: caching, parallelization, and removing unnecessary steps. You will likely recover that time investment within a week.
If it takes over 15 minutes, it is actively hurting your team's productivity and deployment frequency. Prioritize fixing it like you would prioritize fixing a production issue, because in a real sense, it is one.
The goal is not to achieve the theoretical minimum pipeline time. The goal is a feedback loop fast enough that developers stay in flow and deployments stay small. For most teams, that means getting under 10 minutes for the common case and keeping the full pipeline under 15.
Pipeline speed is one of those things that degrades slowly enough that teams do not notice until it is painful. Do not let it get there. Treat your pipeline like production code: measure it, set a budget, and fix it when it regresses.