TypeScript Type Performance Testing & Optimization

[jasonkuhrt]

TypeScript Type Performance Testing & Optimization

Goal

Implement type performance infrastructure:

1. Measure type instantiations with @ark/attest
2. Unit benchmarks for DocumentBuilder, InferResult, Variables
3. E2E heavy-load test using GitHub API generation (20MB, 573k LOC)
4. CI regression detection
5. Optimize generators based on data

Problem

• No type performance baselines or tracking
• GitHub API generation: 20MB, 573k lines (selection-sets.ts alone is 13MB)
• No CI checks for type performance regressions
• Potential issues: _$Context threading, union complexity, repeated type patterns

Research

Core Resources

1. Into the Chamber of Secrets: TS Performance Limits
2. Optimizing TypeScript Type Checking Performance - BAM method, --extendedDiagnostics
3. Why Prisma ORM Checks Types Faster Than Drizzle - Structural deduplication (76% size ↓, 45% time ↓)
4. @ark/attest - Type benchmarks + assertions
5. Prisma PR #26894: Type Benchmarks - Real CI implementation
6. @ark/attest GitHub
7. ArkType Docs
8. TypeScript PR #61505 - Real benchmarks example
9. TS Compiler Diagnostics - --extendedDiagnostics, --generateTrace
10. typescript-analyze-trace

Generic Constraint Performance

What We Actually Know (From Official Sources)

Source: TypeScript Performance Wiki

Verified Facts About Caching:

1. ✅ Interfaces are cached: "Type relationships between interfaces are... cached"
2. ❌ Intersection types are NOT cached consistently
3. ✅ Interfaces create flat object types: "a single flat object type that detects property conflicts"
4. ✅ Intersections recursively merge: Properties merged at comparison time, not pre-computed

Verified Facts About Type Checking Cost:

1. ✅ Union comparison is quadratic: "every element of the union" must be compared pairwise
2. ✅ Function calls with unions are expensive: "every time a function with a union type is called, it has to be compared to each element"
3. ✅ Explicit return types save work: "can save the compiler a lot of work"
4. ✅ Declaration file generation cost: Time spent "reading and writing declaration files"

Measurement Tools:

• --extendedDiagnostics: Shows instantiations count (proxy for "amount of work")
• --generateTrace: Detailed analysis of where time is spent
• Metric: Instantiations count = good proxy for type checker workload

Source: GitHub Issue #45405

Real-world evidence of constraint impact:

• Removing generic constraint on complex type (PrismaClient) caused "massive differences in Types, Instantiations, Assignability cache size"
• Generic constraint "prevents TypeScript from using existing type optimizations"
• Performance degradation visible in VS Code responsiveness

What We DON'T Know (Need to Measure)

❓ When exactly constraints are checked (at call site? at definition? both?)
❓ Whether constraint complexity affects caching (official docs don't specify)
❓ How conditional types vs constraints compare (no benchmark data)
❓ Specific LOC thresholds (no official guidance on "large" vs "small")
❓ Whether lifting constraints to conditionals actually helps (hypothesis, not proven)

Actionable Based on Verified Facts

1. Avoid Large Unions in Constraints

// VERIFIED SLOW: Union constraint (quadratic comparison)
type Process<T extends Type1 | Type2 | Type3 | ... | Type50> = ...

// BETTER: Structural constraint or inheritance
type Process<T extends BaseType> = ...  // Single comparison

Why: Official docs state union comparison is quadratic - "every time a function with a union type is called, it has to be compared to each element"

2. Real Evidence: Constraints Block Optimizations

From GitHub #45405:

• Adding generic constraint to complex type (PrismaClient) caused "massive differences in Types, Instantiations, Assignability cache size"
• Constraint "prevents TypeScript from using existing type optimizations"
• Performance degradation visible in editor

Implication: Constraints on large/complex types have measurable overhead, but we don't know the mechanism or how to predict it.

Other Performance Facts (Unrelated to Constraints)

Interfaces vs Intersections (for constraint TYPE itself)

If your constraint is an intersection, make it an interface instead:

// SLOW: Intersection constraint (not cached, recursive merge)
type Constraint = BaseType & { field1: X } & { field2: Y }
type Process<T extends Constraint> = ...

// FAST: Interface constraint (cached, flat)
interface Constraint extends BaseType {
  field1: X
  field2: Y
}
type Process<T extends Constraint> = ...

Why: Official docs: "Type relationships between interfaces are cached" but intersections are not.

Note: This is about the constraint type's structure, not about whether to use constraints at all.

What You Should Benchmark

Since we lack authoritative data on constraint-specific performance, measure everything:

// tests/type-performance/.../constraint-overhead.bench.ts
import { bench } from '@ark/attest'

// HYPOTHESIS: Removing constraint reduces instantiations
bench('WITH constraint', () => {
  type Result = InferFromQuery<
    SelectionSet extends Select.SelectionSet.RootType,  // Constraint
    Schema extends SchemaDrivenDataMap
  >
}).types([/* measure actual result */, 'instantiations'])

bench('WITHOUT constraint (lifted to conditional)', () => {
  type Result = InferFromQueryOptimized<SelectionSet, Schema>
  // where InferFromQueryOptimized checks constraint in conditional
}).types([/* measure actual result */, 'instantiations'])

bench('WITHOUT constraint (any + guard)', () => {
  type Result = InferFromQueryAny<SelectionSet, Schema>
  // where InferFromQueryAny uses any, checks in conditional
}).types([/* measure actual result */, 'instantiations'])

Compare the measurements to see:

1. Does lifting constraint to conditional actually help?
2. How much overhead does the constraint add?
3. Is there a size threshold where it matters?

Questions to Answer Through Benchmarking

1. Constraint size threshold: At what LOC does constraint become expensive?
2. Constraint position: Does T extends X in parameter vs conditional matter?
3. Constraint complexity: Union constraint vs object constraint vs intersection?
4. Caching behavior: Do repeated uses of same constrained type get cached?

Real Example from Graffle (To Benchmark)

// Current (from infer-variables.ts:41-43)
export type InferFromQuery<
  $SS extends Select.SelectionSet.RootType<Select.StaticBuilderContext> | {},
  $ArgsMap extends SchemaDrivenDataMap & { operations: { query: any } }
>

// Alternative 1: Lift to conditional
export type InferFromQueryLifted<$SS, $ArgsMap> =
  $SS extends Select.SelectionSet.RootType<Select.StaticBuilderContext> | {}
    ? $ArgsMap extends SchemaDrivenDataMap & { operations: { query: any } }
      ? /* actual logic */
      : never
    : never

// Alternative 2: No constraint, any
export type InferFromQueryAny<$SS, $ArgsMap> =
  $SS extends object
    ? $ArgsMap extends object
      ? /* actual logic - cast internals to any */
      : never
    : never

Create benchmark comparing all three, then decide based on data.

@ark/attest

Key APIs

Benchmarking (what we need):

import { bench } from '@ark/attest'

bench("operation name", () => {
  type Result = SomeComplexType<Input>
}).types([1000, "instantiations"])  // Threshold - fails if exceeded

Setup (Vitest):

// vitest.config.ts
export default defineConfig({
  test: { globalSetup: ["tests/type-performance/setup-vitest.ts"] }
})

// setup-vitest.ts
import { setup } from "@ark/attest"
export default () => setup({ benchPercentThreshold: 15 })

Add .attest/ to .gitignore

Implementation

Setup

Dependencies:

{
  "devDependencies": { "@ark/attest": "^0.48.2" },
  "scripts": {
    "bench:types": "vitest run --dir . --testNamePattern '.*\\.bench\\.ts$'",
    "measure:instantiations": "tsc --noEmit --extendedDiagnostics 2>&1 | grep 'Instantiations'",
    "trace:types": "tsc --generateTrace trace --incremental false"
  }
}

Vitest config (add to existing):

// vitest.config.ts or similar
export default defineConfig({
  test: {
    include: ['**/*.bench.ts'],
    globalSetup: ['tests/type-performance/setup-vitest.ts'],
  }
})

File structure (co-located with source):

src/extensions/DocumentBuilder/
├── InferResult/
│   ├── __.ts
│   ├── __.test-d.ts
│   └── __.bench.ts          # NEW
├── var/
│   ├── infer-variables.ts
│   ├── infer-variables.test-d.ts
│   └── infer-variables.bench.ts  # NEW
└── ...

tests/e2e/github/
└── type-performance.bench.ts  # Heavy load: 20MB, 573k LOC

tests/type-performance/
└── setup-vitest.ts           # Global attest setup

Unit Benchmarks

src/extensions/DocumentBuilder/InferResult/__.bench.ts:

import { bench } from '@ark/attest'
import type { InferResult } from './__.js'
import type { Schema } from '../__tests__/fixtures/possible/modules/schema.js'

bench('scalar inference', () => {
  type Result = InferResult.OperationQuery<{ id: true }, Schema>
}).types([200, 'instantiations'])

bench('union inference', () => {
  type Result = InferResult.OperationQuery<{
    unionFooBar: { ___on_Foo: { id: true } }
  }, Schema>
}).types([3000, 'instantiations'])

// More: objects, interfaces, lists...

src/extensions/DocumentBuilder/var/infer-variables.bench.ts:

import { bench } from '@ark/attest'
import type { InferVariables } from './infer-variables.js'

bench('simple variable', () => {
  type Vars = InferVariables<{
    query: { field: { $: { arg: $('name', 'String!') } } }
  }>
}).types([500, 'instantiations'])

// More: multiple vars, nested vars...

E2E: GitHub API Heavy Load

tests/e2e/github/type-performance.bench.ts - Real-world stress test (20MB, 573k LOC):

import { bench } from '@ark/attest'
import type { Github } from './graffle/_namespace.js'

// Stats: selection-sets.ts=13MB, schema.ts=4.1MB, total=573,289 lines

bench('simple field', () => {
  type Q = Github.SelectionSets.Query<{
    codeOfConduct: { $: { key: 'foo' }, key: true }
  }>
}).types([5000, 'instantiations'])

bench('complex union query', () => {
  type Q = Github.SelectionSets.Query<{
    search: {
      edges: {
        node: {
          __typename: true
          ___on_Repository: { name: true }
          ___on_Issue: { title: true }
        }
      }
    }
  }>
}).types([20000, 'instantiations'])

bench('result inference - complex', () => {
  type Result = Github.SelectionSets.Query$Infer<{
    repository: {
      issues: { edges: { node: { title: true } } }
    }
  }>
}).types([25000, 'instantiations'])

// More: interface with many implementors, nested traversal...

Why GitHub API? Already generated, real production schema, catches regressions (e.g., issue #1304 depth limits)

CI Integration

.github/workflows/type-performance.yml:

name: Type Performance
on:
  pull_request:
    paths: ['src/**/*.ts', 'tsconfig*.json']

jobs:
  benchmarks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: pnpm build
      - run: pnpm test:e2e:github:gen:graffle  # Generate heavy load
      - run: pnpm bench:types | tee bench.txt
      - run: pnpm measure:instantiations > metrics.txt

      - name: Comment on PR
        uses: actions/github-script@v7
        with:
          script: |
            const bench = require('fs').readFileSync('bench.txt', 'utf8')
            const metrics = require('fs').readFileSync('metrics.txt', 'utf8')
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: `## Type Performance\n\n${metrics}\n\n\`\`\`\n${bench}\n\`\`\``
            })

      - name: Fail on regression
        run: grep -q FAIL bench.txt && exit 1 || exit 0

Add to .github/workflows/pr.yml:

  type-performance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: pnpm build && pnpm bench:types

Generator Optimizations

Based on Prisma's approach - after establishing baselines.

SelectionSets.ts - Structural deduplication:

// BEFORE: Repeated union for every field
interface Query<_$Context> {
  field1?: Query.field1$Expanded<_$Context> | SelectAlias<...>
  field2?: Query.field2$Expanded<_$Context> | SelectAlias<...>
}

// AFTER: Mapped types (expect 30-50% ↓ instantiations)
type FieldSelector<T> = T | SelectAlias<T>
interface Query<_$Context> {
  [K in keyof QueryFields<_$Context>]?: FieldSelector<QueryFields<_$Context>[K]>
}

InferResult/__.ts - Named conditional types:

// BEFORE: Deep conditional chains
type InferResult<T> =
  T extends Scalar ? InferScalar<T> :
  T extends Object ? InferObject<T> :
  // ...

// AFTER: Categorize then dispatch (expect 20-30% ↓ instantiations)
type SelectionKind<T> = T extends Scalar ? 'scalar' : T extends Object ? 'object' : ...
type InferResult<T> = InferByKind<T, SelectionKind<T>>