Multi-Cluster Kubernetes: Argo CD ApplicationSet Patterns

The Wall You Hit Around 10-20 Clusters

Argo CD scales to dozens of Applications cleanly. Then you hit a wall — typically around 10-20 clusters or 50+ services per cluster — where managing GitOps manifests by hand becomes manual toil. Every new cluster needs Application objects copy-pasted with environment-specific tweaks. Every new service needs an Application per cluster. The repo bloats; mistakes happen during copy-paste; the ops team spends more time managing manifests than improving the platform.

Argo CD's ApplicationSet controller solves this by templating Applications across generators — pluggable sources of cluster lists, Git directories, pull requests, or external decision resources. Define one ApplicationSet; it spawns N Applications based on the generator's output. The four generator types each solve a different multi-cluster problem; this article covers when to use each, real production patterns, and the sharp edges that catch teams in the first 6 months of adoption.

For broader Argo CD context vs alternatives see Argo CD vs Flux CD; for Argo Rollouts (the progressive-delivery sibling) see Argo Rollouts.

The Four Generator Types in Plain Terms

1. Cluster List Generator

The simplest. You give ApplicationSet a list of clusters (or a label selector against clusters registered in Argo CD); it generates one Application per cluster. Use case: deploy the same app to many clusters with cluster-specific values.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: production
  template:
    metadata:
      name: 'monitoring-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/example/k8s-monitoring
        targetRevision: HEAD
        path: helm
        helm:
          values: |
            cluster: {{name}}
            region: {{metadata.labels.region}}
      destination:
        server: '{{server}}'
        namespace: monitoring

Result: one Application per matching cluster, named monitoring-prod-east, monitoring-prod-west, etc.

2. Git Directory Generator

Generate Applications from directories in a Git repo. Common pattern: one directory per environment, or one per service. ApplicationSet watches the repo and creates / removes Applications as directories are added or removed.

spec:
  generators:
  - git:
      repoURL: https://github.com/example/services-repo
      revision: HEAD
      directories:
      - path: services/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      source:
        repoURL: https://github.com/example/services-repo
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

Result: every directory under services/ becomes an Application. Add a directory, get an Application; remove it, the Application is pruned.

3. Pull Request Generator

Generate an Application per open PR for preview environments. Powerful and dangerous — get the configuration right or you'll spin up dozens of clusters' worth of resources.

spec:
  generators:
  - pullRequest:
      github:
        owner: example
        repo: my-app
        labels:
        - preview-env
  template:
    metadata:
      name: 'pr-{{number}}'
    spec:
      source:
        repoURL: https://github.com/example/my-app
        targetRevision: '{{head_sha}}'
        path: deploy/preview
      destination:
        namespace: 'pr-{{number}}'

Result: every PR with the preview-env label gets a preview Application that follows the PR's HEAD commit. Closes the PR → Application pruned → namespace cleaned up.

4. Cluster Decision Resource Generator

The most flexible — and most complex. ApplicationSet queries an external Kubernetes resource (often a custom controller's CRD) for the list of clusters and merges that with the template. Use case: dynamic cluster selection based on capacity, region preferences, or external decision logic outside Argo CD.

This is the right answer when your cluster-selection logic is itself complex (e.g., "deploy to the 3 lowest-loaded clusters in the user's region") and you've built a controller to make those decisions. For most teams, the cluster-list generator is sufficient.

Real Production Patterns

Pattern A: Environment Promotion (dev → staging → prod via folder-per-env)

Combine Git Directory generator with environment labels. The repo structure:

infra-repo/
  apps/
    api/
      base/
      overlays/
        dev/
        staging/
        prod/
    web/
      base/
      overlays/
        dev/
        staging/
        prod/

An ApplicationSet per environment (or one ApplicationSet with a matrix generator combining environments × apps) deploys the appropriate overlay to the appropriate cluster. Promotion is a Git PR that updates the dev overlay, then staging, then prod.

Pattern B: Per-Tenant Deployments (one Application per tenant, generated from a CRD)

SaaS scenario: each tenant gets its own namespace and dedicated services. A Cluster Decision Resource generator pulls from a Tenant CRD; ApplicationSet generates one Application per tenant.

spec:
  generators:
  - clusterDecisionResource:
      configMapRef: tenants-config
      name: tenants
  template:
    metadata:
      name: 'tenant-{{name}}'
    spec:
      source:
        helm:
          values: |
            tenant:
              id: {{name}}
              tier: {{tier}}
              region: {{region}}
      destination:
        namespace: 'tenant-{{name}}'

Add a new tenant CRD instance → ApplicationSet generates a new Application → the tenant's services deploy. Powerful for multi-tenant SaaS at scale.

Pattern C: Multi-Region Failover (cluster-list + region label selectors)

Deploy to active and standby clusters across regions. Cluster List generator with region labels; the active region's Application syncs in normal mode, the standby's syncs in standby mode (replicas 0, lower memory).

spec:
  generators:
  - clusters:
      selector:
        matchExpressions:
        - key: region
          operator: In
          values: [us-east, us-west]
  template:
    spec:
      source:
        helm:
          values: |
            replicas: '{{ ternary "3" "0" (eq .metadata.labels.role "active") }}'

Failover is a label flip on cluster registration: change the active label, ApplicationSets re-template, replicas come up in the new active region.

Pattern D: Preview Environments (PR generator)

Combine PR generator with namespace-per-PR. Each open PR gets its own Application running its branch. Closes the PR → namespace pruned. Saves the engineering org from maintaining "the preview environment" as a manually-curated thing.

Sharp edge: cost. 50 open PRs × full preview app stack = real cluster resource consumption. Bound the count explicitly via PR labels or by capping ApplicationSet's preserveResourcesOnDeletion. See FinOps for engineers for cost-control patterns.

Sharp Edges That Catch Teams

1. Template Debugging Is Awful

ApplicationSet templates use Go templating with custom functions ({{name}}, {{path.basename}}, etc.). When a template has a bug, the error often surfaces at kubectl apply time on the generated Application — not at ApplicationSet validation. You see something like "field replicas: invalid type" pointing at the Application, not at the ApplicationSet.

Workarounds: use argocd appset generate (CLI) or kubectl get applicationset -o yaml to dump generated Applications and check them by hand. For complex templates, a CI lint stage that runs argocd appset generate against changes catches bugs before merge.

2. Generator Mistakes Cascade

An ApplicationSet that's wrong about the generator's output may delete dozens of Applications and re-create them with wrong config. The default behavior on generator output changes is "reconcile to match the new desired state" — including pruning Applications that are no longer in the generator's output.

Mitigation: always set preserveResourcesOnDeletion: true for production ApplicationSets. This means deletions of Applications happen by hand, not automatically. Slower but safer.

3. Sync Wave / Sync Phase Conflicts

If your Applications use Argo CD sync waves (deploying things in a specific order), ApplicationSet doesn't synchronize sync waves across the generated Applications. Application A wave 0 might run after Application B wave 1. For ordered cross-cluster deployments, this is a real problem.

Mitigation: don't use sync waves to coordinate cross-Application ordering. Use external orchestration (a CI pipeline, ArgoEvents, or progressive-delivery via Argo Rollouts) for cross-cluster ordering.

4. Dry-Run Doesn't Show Generated Applications

Standard kubectl apply --dry-run on an ApplicationSet shows the ApplicationSet itself but doesn't expand the generators to show what Applications would be created. argocd appset generate does, but you have to remember to run it. Many teams' first ApplicationSet bug is "didn't realize this would generate 47 Applications."

5. RBAC at the Application Level Doesn't Apply to ApplicationSet

Argo CD's per-Application RBAC (which projects each Application can sync, which destinations it can deploy to) doesn't auto-extend to ApplicationSet-generated Applications. The ApplicationSet itself runs as a powerful service account; it generates Applications that may bypass team-level RBAC. Plan ApplicationSet's permissions explicitly.

When ApplicationSet Is Overkill

For under 5 clusters or under 10 Applications per cluster, ApplicationSet's complexity outweighs its benefit. Just write 5 Applications — copy-paste is fine, the template-debugging tax isn't worth it. ApplicationSet pays off when:

• You have over 10 clusters that need similar Applications
• You have a long tail of services where adding a new one means adding it everywhere
• You want preview environments per PR
• You have multi-tenant deployments scaling past hand-curation
• Your environment-promotion flow is painful with hand-written manifests

If none of these apply, pure Applications are fine. ApplicationSet is the answer to a scaling problem; if you don't have the scaling problem, don't take on the complexity.

Operational Practices That Work

Use a Lint Stage

CI pipeline runs argocd appset generate on every PR that modifies an ApplicationSet. The output is the list of Applications that would be created. PR review checks this list — if 47 Applications are created when 5 are expected, the bug surfaces at PR time.

preserveResourcesOnDeletion in Production

Always set preserveResourcesOnDeletion: true on production ApplicationSets. The slight slow-down (manual deletion) is worth the protection against mistakes that would otherwise cascade.

Version-Pin the Generator Source

For Git Directory generator, pin revision to a specific commit or tag rather than HEAD. Otherwise every push to main potentially regenerates Applications. This trades some "automatic" for "predictable."

One ApplicationSet Per Concern

Avoid mega-ApplicationSets that template across many dimensions (clusters × environments × tenants × services). The matrix generator allows it, but the templating gets unmaintainable. Better: one ApplicationSet per concern, even if you end up with 5-10 of them.

Monitor Generator Health

ApplicationSet has metrics for "applications generated, applications synced." Add Prometheus alerts for unexpected drops in count (sign of a generator misconfiguration) or sync failures. See Prometheus and Grafana monitoring.

Decision Matrix

Use case	Generator	Why
Same app to many similar clusters	Cluster List	Simplest, most maintainable
Per-service Applications, services in Git	Git Directory	Auto-discover services from repo structure
Preview environments per PR	Pull Request	Purpose-built for PR-driven flows
Per-tenant deployments at scale	Cluster Decision Resource	Dynamic tenant list from CRD
Environment promotion (dev→staging→prod)	Git Directory + Matrix	One ApplicationSet for app × env
Multi-region active/standby	Cluster List + label selectors	Region-based selection
Under 5 clusters / simple setup	Plain Applications	ApplicationSet is overkill

Pro tip: Start with one ApplicationSet for one specific use case. Watch it work for 4-6 weeks. Add a second only when you've felt the pain of NOT having it. Most teams that adopt ApplicationSet too aggressively burn out on debugging templates and roll back. Incremental adoption beats big-bang ApplicationSet rollout every time.

Frequently Asked Questions

Bottom Line

ApplicationSet is the answer to managing GitOps Applications at scale beyond the hand-written limit. Pick the right generator for your problem (cluster list for similar deploys, Git directory for service inventory, PR generator for preview environments, cluster decision resource for dynamic logic). Always set preserveResourcesOnDeletion in production. Add a CI lint stage that generates Applications and reviews the output. Start with one ApplicationSet for one concern; expand only when you've felt the pain. ApplicationSet is the right tool when the scaling problem is real; for sub-10-cluster, sub-50-service shops, plain Applications remain the simpler answer.