Slow, flaky and dangerous pipelines

Key points

• Everything in a Declarative pipeline outside a node or agent block runs on the controller. Heavy Groovy there consumes the memory and CPU that every other job depends on.
• Pipeline state is serialised at every step so a build can survive a restart. That is why large objects in pipeline variables blow up the controller heap - they are written to disk repeatedly.
• Wrap loops and helpers in @NonCPS when they do not call pipeline steps. Without it every iteration goes through the CPS interpreter, which is dramatically slower.
• Flakiness is usually shared mutable state: a fixed workspace path, a container name, or a port that two concurrent builds both use. Make every one of them unique per build.
• The dangerous part is credentials. A credential interpolated into a shell string with double quotes is expanded by Groovy and can land in the console log; single quotes plus an environment variable keeps it out.
• Set a timeout on every stage and disableConcurrentBuilds where a job touches shared state - an un-timed hung stage occupies an executor indefinitely.

Example

pipeline {
    agent none                        // NOTHING on the controller by default

    options {
        timeout(time: 30, unit: 'MINUTES')      // every pipeline needs one
        disableConcurrentBuilds()               // if it touches shared state
        buildDiscarder(logRotator(numToKeepStr: '30'))  // or disk fills up
        timestamps()
    }

    stages {
        stage('Build') {
            agent { docker { image 'eclipse-temurin:21-jdk'
                             args '-v $HOME/.gradle:/root/.gradle' } }
            steps {
                sh './gradlew --no-daemon build'
                stash name: 'jar', includes: 'build/libs/*.jar'
            }
        }

        stage('Publish') {
            agent any
            steps {
                unstash 'jar'
                // SAFE: single quotes, so Groovy does NOT interpolate.
                // The shell reads it from the environment instead.
                withCredentials([string(credentialsId: 'registry-token',
                                        variable: 'TOKEN')]) {
                    sh 'docker login -u ci --password-stdin <<< "$TOKEN"'
                }
            }
        }
    }

    post { always { cleanWs() } }      // or agents fill their disks
}

/*
 * THE CREDENTIAL LEAK:
 *
 *   sh "docker login -p ${TOKEN}"     <- DOUBLE quotes: Groovy expands it
 *                                        into the command line, which is
 *                                        visible in ps and often the log
 *
 *   sh 'docker login -p $TOKEN'       <- single quotes: the SHELL expands
 *                                        it from the environment
 *
 * Jenkins masks known credentials in console output, but masking is a
 * safety net - it cannot mask a value that was transformed first.
 */

// @NonCPS for anything that does not call pipeline steps.
@NonCPS
def parseVersions(String text) {
    // Without @NonCPS every iteration runs through the CPS interpreter,
    // and the whole loop state is serialised at each step.
    return text.split('\n').collect { it.trim() }.findAll { it }
}

// Unique per build - shared names are the usual cause of flakiness.
//   docker run --name app-${BUILD_NUMBER}
//   PORT=$(shuf -i 20000-30000 -n 1)

Keep work off the controller, time out every stage, and use single quotes with withCredentials so secrets never reach the command line.