Artillery Performance Testing: Modern Load Testing with YAML Scenarios is a critical discipline in modern software quality assurance. According to Google research, as page load time increases from 1 to 3 seconds, the probability of bounce increases 32% (Google/SOASTA Research). According to Akamai, a 100ms delay in page load can decrease conversion rates by 7% (Akamai Performance Study). This guide covers practical approaches that QA teams can apply immediately: from core concepts and tooling to real-world implementation patterns. Whether you are building skills in this area or improving an existing process, you will find actionable techniques backed by industry experience. The goal is not just theoretical understanding but a working framework you can adapt to your team’s context, technology stack, and quality objectives.

TL;DR

  • Define SLAs before writing tests — testing without targets produces meaningless data
  • Run performance tests against production-like data volumes for reliable results
  • Integrate lightweight performance regression tests into CI/CD to catch regressions early

Best for: Teams with defined performance SLAs or traffic growth Skip if: Static sites or internal tools with fewer than 100 concurrent users

Introduction to Artillery

Artillery is a modern, powerful load testing toolkit designed for developers. With YAML-based scenario definitions, built-in support for HTTP, WebSocket, and Socket.io, plus a rich plugin ecosystem, Artillery excels at testing modern, real-time applications.

For API testing fundamentals before load testing, review API Testing Mastery. Teams needing to validate API performance under load should also explore API Performance Testing. For CI/CD integration strategies, see CI/CD Pipeline Optimization for QA Teams and Jenkins Pipeline for Test Automation.

“Performance testing reveals system behavior under stress that no functional test ever will. I always say: run your load tests against production-like data volumes — synthetic data gives you false confidence.” — Yuri Kan, Senior QA Lead

Basic YAML Scenario 

# load-test.yml
config:
  target: "https://api.example.com"
  phases:

    - duration: 60
      arrivalRate: 10  # 10 users per second
      name: "Warm up"
    - duration: 120
      arrivalRate: 50  # Ramp to 50 users/sec
      name: "Sustained load"
    - duration: 30
      arrivalRate: 100  # Peak load
      name: "Spike test"

  processor: "./flows.js"  # Custom JavaScript logic

scenarios:

  - name: "Browse and purchase"
    flow:

      - get:
          url: "/api/products"
          capture:

            - json: "$[0].id"
              as: "productId"

      - post:
          url: "/api/cart"
          json:
            productId: "{{ productId }}"
            quantity: 1
          capture:

            - json: "$.cartId"
              as: "cartId"

      - post:
          url: "/api/checkout"
          json:
            cartId: "{{ cartId }}"
            paymentMethod: "credit_card"
          expect:

            - statusCode: 200
            - contentType: json
            - hasProperty: orderId

Advanced Features

Custom JavaScript Processor 

// flows.js
module.exports = {
  setAuthToken,
  generateTestData,
  validateResponse
};

function setAuthToken(requestParams, context, ee, next) {
  // Login to get token
  const request = require('request');

  request.post({
    url: `${context.vars.target}/auth/login`,
    json: {
      username: 'test@example.com',
      password: 'password123'
    }
  }, (err, res, body) => {
    if (!err && res.statusCode === 200) {
      context.vars.authToken = body.token;
    }
    return next();
  });
}

function generateTestData(context, events, done) {
  // Generate random test data
  context.vars.username = `user_${Date.now()}`;
  context.vars.email = `${context.vars.username}@example.com`;
  return done();
}

function validateResponse(requestParams, response, context, ee, next) {
  // Custom validation logic
  const body = JSON.parse(response.body);

  if (body.status !== 'success') {
    ee.emit('error', 'Invalid response status');
  }

  if (response.timings.phases.total > 1000) {
    console.log(`Slow response: ${response.timings.phases.total}ms`);
  }

  return next();
}

WebSocket Testing 

# websocket-test.yml
config:
  target: "ws://localhost:8080"
  phases:

    - duration: 60
      arrivalRate: 10

scenarios:

  - engine: ws
    flow:
      # Connect to WebSocket
      - send: '{"action": "subscribe", "channel": "prices"}'

      # Wait for message
      - think: 1

      # Send message and expect response
      - send: '{"action": "get_price", "symbol": "BTC"}'
      - match:
          - json: "$.symbol"
            value: "BTC"
          - json: "$.price"
            exists: true

      # Loop messages
      - loop:
        - send: '{"action": "ping"}'
        - think: 5
        count: 10

Plugin Ecosystem

Metrics Plugins 

# artillery.yml with plugins
config:
  target: "https://api.example.com"
  plugins:
    # Publish metrics to CloudWatch
    cloudwatch:
      region: "us-east-1"
      namespace: "LoadTests"

    # Publish to Datadog
    datadog:
      apiKey: "{{ $processEnvironment.DD_API_KEY }}"
      tags:

        - "env:production"
        - "team:backend"

    # Publish to Prometheus
    publish-metrics:

      - type: prometheus
        pushgateway: "http://prometheus:9091"

  phases:

    - duration: 300
      arrivalRate: 20

scenarios:

  - flow:
      - get:
          url: "/api/metrics"

Expect Plugin 

config:
  target: "https://api.example.com"
  plugins:
    expect: {}  # Enable expect plugin

scenarios:

  - name: "API validation"
    flow:

      - get:
          url: "/api/users/{{ userId }}"
          expect:

            - statusCode: 200
            - contentType: json
            - hasHeader: "x-request-id"
            - equals:
                - "{{ $. username}}"
                - "john_doe"
            - matchesRegexp:
                - "{{ $.email }}"
                - "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"

CI/CD Integration

GitHub Actions 

# .github/workflows/performance.yml
name: Performance Tests

on:
  schedule:

    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:

jobs:
  artillery-test:
    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install Artillery
        run: npm install -g artillery@latest

      - name: Run Performance Test
        run: |
          artillery run \
            --output report.json \
            load-test.yml

      - name: Generate HTML Report
        run: artillery report report.json --output report.html

      - name: Check SLO Compliance
        run: |
          artillery report report.json \
            --assert \
            "p95 < 500" \
            "p99 < 1000" \
            "errors.rate < 0.01"

      - name: Upload Reports
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: performance-reports
          path: |
            report.json
            report.html

GitLab CI/CD 

# .gitlab-ci.yml
performance-test:
  stage: test
  image: node:18
  script:

    - npm install -g artillery
    - artillery run --output results.json load-test.yml
    - artillery report results.json

  artifacts:
    reports:
      performance: results.json
    paths:

      - results.json

  only:

    - schedules
    - web

Best Practices

1. Realistic Load Profiles 

config:
  phases:
    # Gradual ramp-up
    - duration: 120
      arrivalRate: 1
      rampTo: 50
      name: "Ramp up"

    # Sustained load
    - duration: 600
      arrivalRate: 50
      name: "Sustained"

    # Spike test
    - duration: 60
      arrivalRate: 200
      name: "Spike"

    # Cool down
    - duration: 60
      arrivalRate: 10
      name: "Cool down"

2. Environment-Specific Configs 

# Base config
config:
  target: "{{ $processEnvironment.TARGET_URL }}"
  phases:

    - duration: "{{ $processEnvironment.DURATION }}"
      arrivalRate: "{{ $processEnvironment.ARRIVAL_RATE }}"

# Run with environment variables
# TARGET_URL=https://api.example.com DURATION=300 ARRIVAL_RATE=50 artillery run test.yml

3. Custom Metrics 

// custom-metrics.js
module.exports = { trackBusinessMetrics };

function trackBusinessMetrics(userContext, events, done) {
  events.on('response', (data) => {
    const body = JSON.parse(data.body);

    // Track business-specific metrics
    if (body.orderValue) {
      events.emit('counter', 'orders.total', 1);
      events.emit('histogram', 'orders.value', body.orderValue);
    }

    if (body.discountApplied) {
      events.emit('counter', 'discounts.applied', 1);
    }
  });

  return done();
}

Artillery vs Other Tools

FeatureArtilleryJMeterLocustk6
Config FormatYAMLGUI/XMLPythonJavaScript
Learning CurveLowMedium-HighLow (Python)Low (JS)
WebSocketExcellentLimitedManualGood
Socket.ioNativeNoNoLimited
PluginsExcellentExtensiveLimitedGrowing
CI/CD (as discussed in K6: Modern Load Testing with JavaScript for DevOps Teams)ExcellentGoodExcellentExcellent
Real-time AppsExcellentPoorGoodGood
DistributedCloud (paid)Built-inBuilt-inCloud

Conclusion

Artillery excels at testing modern applications with its YAML-based scenarios, excellent WebSocket/Socket.io support, and rich plugin ecosystem. Its developer-friendly approach and strong CI/CD integration make it ideal for teams practicing continuous performance testing (as discussed in Locust Python Load Testing: Complete Performance Testing Guide).

Choose Artillery when:

  • Testing real-time applications (WebSocket, Socket.io)
  • YAML configuration preferred over code
  • Modern, developer-friendly tooling desired
  • Rich plugin ecosystem valuable
  • CI/CD integration is priority

Choose alternatives when:

  • Distributed testing without cloud service needed (Locust, k6)
  • GUI-based test creation preferred (JMeter)
  • Custom Python logic essential (Locust)
  • Advanced JavaScript scenarios needed (k6)

For teams building modern, real-time applications and practicing DevOps, Artillery provides an excellent balance of simplicity, power, and extensibility for performance testing.

Official Resources

FAQ

What is the difference between load and stress testing? Load testing validates behavior under expected peak traffic. Stress testing pushes beyond capacity to find breaking points and observe failure modes.

How do you choose performance test targets? Define targets based on SLAs, business requirements, and historical baseline data. Common targets: p95 response time < 500ms, error rate < 0.1%, throughput matching peak traffic projections.

What causes performance test results to be unreliable? Common issues: testing against non-production-like data volumes, testing from a single geographic location, not warming up caches, and running tests at different times of day.

How do you integrate performance testing into CI/CD? Add lightweight performance regression tests to your pipeline that complete in under 5 minutes, comparing key metrics against established baselines and failing builds on regressions.

See Also