The City Arcana

Git scaling, sparse checkout and monorepo tooling

Git for large repositories

When a repository reaches GB or tens of GB, standard Git operations become very slow. Git has developed several features to address this.

Partial clone - download only what's needed

A standard git clone downloads all objects from the history - all files, all versions, all trees. For a large repository, this can take hours.

Partial clone lets you download only the necessary objects. Missing objects are fetched on demand, when you need them.

# Clone without blobs (files) - only downloads metadata
# Files are fetched on demand when you access them
git clone --filter=blob:none https://github.com/example/huge-repo.git

# Clone without subdirectory trees
# Even lighter, but slightly slower to use
git clone --filter=tree:0 https://github.com/example/huge-repo.git

# Size-filtered clone - exclude files larger than 1 MB
git clone --filter=blob:limit=1m https://github.com/example/huge-repo.git

Shallow clone - limit history depth

Shallow clone only downloads the last N commits, without the full history.

# Fetch only the last commit
git clone --depth 1 https://github.com/example/huge-repo.git

# Fetch the last 10 commits
git clone --depth 10 https://github.com/example/huge-repo.git

# Fetch more history if needed
git fetch --deepen=50

# Convert a shallow clone to a full clone
git fetch --unshallow

Sparse checkout - fetch only certain folders

In a monorepo, you probably only work on one or two projects. Sparse checkout lets you materialize only the folders you're interested in within your working directory.

# Clone the repo in blobless mode (fast)
git clone --filter=blob:none https://github.com/example/monorepo.git
cd monorepo

# Enable sparse checkout in cone mode
git sparse-checkout init --cone

# Fetch only the folders you need
git sparse-checkout set apps/frontend packages/shared-utils

# Your working directory now only contains:
# apps/frontend/
# packages/shared-utils/
# (plus files at the root)

# Add an additional folder
git sparse-checkout add apps/backend-api

# See included folders
git sparse-checkout list

# Disable sparse checkout (fetch everything)
git sparse-checkout disable

PowerShell (Windows):

# Commands are identical in PowerShell
git clone --filter=blob:none https://github.com/example/monorepo.git
cd monorepo

git sparse-checkout init --cone
git sparse-checkout set apps/frontend packages/shared-utils
git sparse-checkout list
# To fetch everything:
# git sparse-checkout disable

Cone mode

The --cone flag activates "cone" mode which is faster and more intuitive than the classic mode. In cone mode, you specify entire folders to include, not complex patterns.

# Cone mode (recommended) - specify folders
git sparse-checkout set apps/frontend packages/ui

# Classic mode (legacy) - uses gitignore patterns
# More flexible but slower and more complex
git sparse-checkout init  # (without --cone)
echo "apps/frontend/**" >> .git/info/sparse-checkout

Monorepo tools

Git alone isn't enough to manage a monorepo efficiently. You need tools that understand your project's structure and optimize builds and tests.

Nx (JavaScript / TypeScript)

Nx is an intelligent build framework for JavaScript/TypeScript monorepos. It understands the dependencies between your projects and only rebuilds what has changed.

# Create a new Nx workspace
npx create-nx-workspace@latest my-monorepo

# Typical structure of an Nx workspace
my-monorepo/
├── apps/
│   ├── frontend/         # React/Angular/Vue application
│   └── backend/          # NestJS/Express API
├── libs/
│   ├── shared-types/     # Shared TypeScript types
│   ├── ui-components/    # Reusable UI components
│   └── utils/            # Utility functions
├── nx.json               # Nx configuration
├── package.json
└── tsconfig.base.json

# Build only what's affected by changes
npx nx affected:build
# Note: since Nx 17+, the syntax is: nx affected -t build

# Test only what's affected
npx nx affected:test
# Note: since Nx 17+, the syntax is: nx affected -t test

# Visualize the dependency graph
npx nx graph

Key Nx concepts:

• Affected: Nx knows which projects are impacted by a change. If you modify shared-types, Nx knows that frontend and backend need to be retested, but not ui-components.
• Cache: Nx caches build and test results. If nothing has changed, the result is instant.
• Remote cache: the cache can be shared between developers and CI. If a colleague has already built the same code, you get their result instead of rebuilding.

Turborepo

Turborepo (created by Vercel) is a build tool for JavaScript/TypeScript monorepos, simpler than Nx but very effective.

# Create a Turborepo monorepo
npx create-turbo@latest

# Typical structure
my-monorepo/
├── apps/
│   ├── web/              # Next.js application
│   └── docs/             # Documentation site
├── packages/
│   ├── ui/               # Shared components
│   ├── eslint-config/    # Shared ESLint config
│   └── typescript-config/ # Shared TS config
├── turbo.json            # Pipeline configuration
└── package.json

# turbo.json - define pipelines
# Note: since Turborepo 2.x, the "pipeline" key is renamed to "tasks"
{
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**"]
    },
    "test": {
      "dependsOn": ["build"]
    },
    "lint": {}
  }
}

# Build everything (with cache and parallelization)
npx turbo build

# Build only what's affected by changes since main
npx turbo build --filter=...[origin/main]

Bazel (multi-language)

Bazel (created by Google, open source) is the most powerful but also the most complex build tool. It supports any language and works at Google's scale (billions of lines of code).

# Bazel uses BUILD files to describe targets
# Example BUILD file for a Python project
load("@rules_python//python:defs.bzl", "py_binary", "py_test")

py_binary(
    name = "server",
    srcs = ["server.py"],
    deps = [
        "//libs/auth:auth_lib",
        "//libs/database:db_lib",
    ],
)

py_test(
    name = "server_test",
    srcs = ["server_test.py"],
    deps = [":server"],
)

# Build a target
bazel build //apps/server:server

# Test a target
bazel test //apps/server:server_test

# Build everything affected by changes
bazel build //... --keep_going

Key Bazel concepts:

• Hermeticity: every build is isolated and reproducible. The same inputs always produce the same outputs.
• Distributed cache: build results are shared among all developers and CI.
• Multi-language: Bazel works with Java, Python, Go, C++, Rust, JavaScript, and many more.

Managing dependencies in a monorepo

One of the great advantages of a monorepo is code sharing. But you need to do it properly.

npm/pnpm/yarn workspaces

Modern package managers natively support workspaces - a mechanism to link local packages together without going through an npm registry.

# Root package.json with npm workspaces
{
  "name": "my-monorepo",
  "private": true,
  "workspaces": [
    "apps/*",
    "packages/*"
  ]
}

# package.json of an app using a local package
{
  "name": "frontend",
  "dependencies": {
    "shared-utils": "*",
    "ui-components": "*"
  }
}

# npm automatically resolves local dependencies
# No need to publish packages - they are linked directly

# Install all dependencies (root)
npm install

# Run a script in a specific workspace
npm run build --workspace=apps/frontend

With pnpm (recommended for monorepos)

# pnpm-workspace.yaml
packages:
  - 'apps/*'
  - 'packages/*'

# pnpm is faster and uses less disk space than npm
# thanks to its symlink-based storage

# Install all dependencies
pnpm install

# Run a command in all packages
pnpm -r build

# Run in a specific package
pnpm --filter frontend build

# Run in packages affected by a change
pnpm --filter "...[origin/main]" build

CI/CD in a monorepo

The biggest challenge of a monorepo is CI/CD. You don't want to rebuild and retest everything on each commit - that would be too slow. You need to detect what has changed and only process that.

Affected detection with Git

# See which files have changed since the main branch
git diff --name-only origin/main...HEAD

# Example output:
# apps/frontend/src/App.tsx
# packages/shared-utils/src/format.ts

# Script to detect affected packages
# If a file in apps/frontend/ changed -> test frontend
# If a file in packages/shared-utils/ changed -> test everything that depends on it

GitHub Actions with affected detection

# .github/workflows/ci.yml
name: CI
on:
  pull_request:
    branches: [main]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      frontend: ${{ steps.changes.outputs.frontend }}
      backend: ${{ steps.changes.outputs.backend }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: changes
        with:
          filters: |
            frontend:
              - 'apps/frontend/**'
              - 'packages/shared-utils/**'
              - 'packages/ui-components/**'
            backend:
              - 'apps/backend/**'
              - 'packages/shared-utils/**'

  test-frontend:
    needs: detect-changes
    if: ${{ needs.detect-changes.outputs.frontend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run test --workspace=apps/frontend

  test-backend:
    needs: detect-changes
    if: ${{ needs.detect-changes.outputs.backend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run test --workspace=apps/backend

With Nx

# Nx makes affected detection trivial
# In the CI workflow:
- run: npx nx affected:test --base=origin/main
- run: npx nx affected:build --base=origin/main
- run: npx nx affected:lint --base=origin/main
# Note: since Nx 17+, the syntax is:
# nx affected -t test --base=origin/main
# nx affected -t build --base=origin/main
# nx affected -t lint --base=origin/main

# Nx understands the dependency graph and only tests
# the projects actually impacted by the changes

Practical exercise - Mini-monorepo with sparse checkout

Step 1 - Create the monorepo

# Create the monorepo
mkdir world-city-monorepo
cd world-city-monorepo
git init -b main

# Create the structure
mkdir -p apps/frontend/src
mkdir -p apps/backend/src
mkdir -p apps/mobile/src
mkdir -p packages/shared-utils/src
mkdir -p packages/ui-components/src
mkdir -p docs

# Create the monorepo
mkdir world-city-monorepo
cd world-city-monorepo
git init -b main

# Create the structure
New-Item -ItemType Directory -Force -Path apps/frontend/src
New-Item -ItemType Directory -Force -Path apps/backend/src
New-Item -ItemType Directory -Force -Path apps/mobile/src
New-Item -ItemType Directory -Force -Path packages/shared-utils/src
New-Item -ItemType Directory -Force -Path packages/ui-components/src
New-Item -ItemType Directory -Force -Path docs

Step 2 - Add content to each project

# Frontend
cat > apps/frontend/src/index.js << 'EOF'
// World City frontend application
import { formatName } from '../../../packages/shared-utils/src/format.js';
console.log("Frontend loaded!");
console.log(formatName("Adventurer"));
EOF

# Backend
cat > apps/backend/src/server.js << 'EOF'
// World City backend API
const http = require('http');
const server = http.createServer((req, res) => {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('World City API');
});
server.listen(3000);
console.log("Backend server started on port 3000");
EOF

# Mobile
cat > apps/mobile/src/app.js << 'EOF'
// World City mobile application
console.log("Mobile app initialized!");
EOF

# Shared utils
cat > packages/shared-utils/src/format.js << 'EOF'
// Shared utility functions
export function formatName(name) {
  return name.charAt(0).toUpperCase() + name.slice(1).toLowerCase();
}
export function formatDate(date) {
  return new Date(date).toLocaleDateString('en-US');
}
EOF

# UI components
cat > packages/ui-components/src/button.js << 'EOF'
// Shared button component
export function createButton(label, onClick) {
  const btn = document.createElement('button');
  btn.textContent = label;
  btn.addEventListener('click', onClick);
  return btn;
}
EOF

# Documentation
cat > docs/architecture.md << 'EOF'
# World City Architecture

This monorepo contains:
- 3 applications (frontend, backend, mobile)
- 2 shared packages (shared-utils, ui-components)
EOF

# Root README
cat > README.md << 'EOF'
# World City Monorepo
A mini-monorepo exercise for the Versioner's Chronicles.
EOF

# Frontend
@"
// World City frontend application
import { formatName } from '../../../packages/shared-utils/src/format.js';
console.log("Frontend loaded!");
console.log(formatName("Adventurer"));
"@ | Set-Content apps/frontend/src/index.js

# Backend
@"
// World City backend API
const http = require('http');
const server = http.createServer((req, res) => {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('World City API');
});
server.listen(3000);
console.log("Backend server started on port 3000");
"@ | Set-Content apps/backend/src/server.js

# Mobile
@"
// World City mobile application
console.log("Mobile app initialized!");
"@ | Set-Content apps/mobile/src/app.js

# Shared utils
@"
// Shared utility functions
export function formatName(name) {
  return name.charAt(0).toUpperCase() + name.slice(1).toLowerCase();
}
export function formatDate(date) {
  return new Date(date).toLocaleDateString('en-US');
}
"@ | Set-Content packages/shared-utils/src/format.js

# UI components
@"
// Shared button component
export function createButton(label, onClick) {
  const btn = document.createElement('button');
  btn.textContent = label;
  btn.addEventListener('click', onClick);
  return btn;
}
"@ | Set-Content packages/ui-components/src/button.js

# Documentation
@"
# World City Architecture

This monorepo contains:
- 3 applications (frontend, backend, mobile)
- 2 shared packages (shared-utils, ui-components)
"@ | Set-Content docs/architecture.md

# Root README
"# World City Monorepo`nA mini-monorepo exercise for the Versioner's Chronicles." | Set-Content README.md

Step 3 - Commit all content

git add .
git commit -m "Initialize the World City monorepo"

git add .
git commit -m "Initialize the World City monorepo"

Step 4 - Use sparse checkout

Now, let's simulate a frontend developer who only wants to work on their application and the shared packages.

# Enable sparse checkout
git sparse-checkout init --cone

# Fetch only the frontend and shared packages
git sparse-checkout set apps/frontend packages/shared-utils packages/ui-components

# Check what's visible
ls apps/
# -> frontend/  (no backend/ or mobile/)

ls packages/
# -> shared-utils/  ui-components/

# Documentation isn't visible either
ls docs/ 2>/dev/null || echo "docs/ is not in the sparse checkout"

# See active patterns
git sparse-checkout list

# Enable sparse checkout
git sparse-checkout init --cone

# Fetch only the frontend and shared packages
git sparse-checkout set apps/frontend packages/shared-utils packages/ui-components

# Check what's visible
Get-ChildItem apps/
# frontend/  (no backend/ or mobile/)

Get-ChildItem packages/
# shared-utils/  ui-components/

# See active patterns
git sparse-checkout list

Step 5 - Work normally

# Modify a file in the frontend
echo "// New feature" >> apps/frontend/src/index.js

# Git works normally
git add apps/frontend/src/index.js
git commit -m "Add a feature to the frontend"

# The full history is still accessible
git log --oneline

# If you need the backend temporarily:
git sparse-checkout add apps/backend
ls apps/
# -> backend/  frontend/

# To disable and return to normal mode:
git sparse-checkout disable

# Modify a file in the frontend
Add-Content apps/frontend/src/index.js "// New feature"

# Git works normally
git add apps/frontend/src/index.js
git commit -m "Add a feature to the frontend"

# The full history is still accessible
git log --oneline

# If you need the backend temporarily:
git sparse-checkout add apps/backend

# To disable and return to normal mode:
# git sparse-checkout disable

Command summary