CodeCrush

## glossary

What is Pipeline?

A pipeline is a sequence of chained processing stages, where the output of each stage serves as input to the next. The term comes from the plumbing analogy: material goes in on one side, passes through successive transformations, and comes out processed on the other.

The concept shows up in several areas of computing, always with the same logic: split a complex process into independent, verifiable stages that can — often — run in parallel.

The most common types

  • CI/CD pipeline — build, tests, security analysis and deploy. Detailed in the CI/CD entry.
  • Data pipeline — extracts from diverse sources, transforms and loads into an analytical destination (the ETL or ELT pattern).
  • Machine learning pipeline — collection, cleaning, feature engineering, training, validation and model publishing.
  • Shell pipeline — commands chained with |, where the output of one becomes the input of the next.
  • CPU pipeline — at the processor level, instructions in simultaneous stages of execution.
The simplest pipeline there is: the shell
# each | passes the output along
cat access.log \
  | grep "POST /api" \
  | awk '{print $7}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

# result: the 10 most-called POST endpoints

Why split into stages

The staged structure brings advantages a monolithic script does not have. Each stage can be tested in isolation. A failure points to exactly where the process broke. Independent stages run in parallel, shortening total time. And you can reprocess only the affected part instead of redoing everything.

A CI pipeline declared in stages
stages:
  - lint
  - test
  - build
  - deploy

lint:
  stage: lint
  script: npm run lint

test:
  stage: test
  script: npm test

build:
  stage: build
  script: npm run build
  artifacts:
    paths: [dist/]

deploy:
  stage: deploy
  script: ./deploy.sh
  only: [main]

Traits of a well-built pipeline

It should be idempotent (running it twice produces the same result), observable (you can see where it is and what failed), fast (late feedback goes unused) and reversible (you can undo what was published). For the specific case of data, the article on data pipelines goes deeper.

## faq

Frequently asked questions

What is the difference between a pipeline and a workflow?

Pipeline suggests a linear flow, where each stage feeds the next. Workflow is broader and admits branching, conditionals, approvals and parallel paths that merge back. In practice, tools mix the two terms.

What are ETL and ELT?

They are different orders for assembling a data pipeline. In ETL, data is transformed before entering the destination. In ELT, it is loaded raw and transformed inside the data warehouse itself, leveraging its processing power — the approach that became dominant with modern warehouses.

How do I speed up a slow pipeline?

Cache dependencies between runs, run independent stages in parallel, execute only the tests affected by the change when possible, and move slow checks after the fast ones — so a trivial failure aborts the process before burning the expensive minutes.

## read next

Related articles