This file provides guidance to AI agents when working with code in this repository.
## Overview
Headscale is an open-source implementation of the Tailscale control server written in Go. It provides self-hosted coordination for Tailscale networks (tailnets), managing node registration, IP allocation, policy enforcement, and DERP routing.
go run ./cmd/hi doctor # Check system requirements
go run ./cmd/hi run "TestPattern" # Run specific test
go run ./cmd/hi run "TestPattern" --postgres # With PostgreSQL backend
# Test artifacts are saved to control_logs/ with logs and debug data
```
## Pre-Commit Quality Checks
### **MANDATORY: Automated Pre-Commit Hooks with prek**
**CRITICAL REQUIREMENT**: This repository uses [prek](https://prek.j178.dev/) for automated pre-commit hooks. All commits are automatically validated for code quality, formatting, and common issues.
### Initial Setup
When you first clone the repository or enter the nix shell, install the git hooks:
```bash
# Enter nix development environment
nix develop
# Install prek git hooks (one-time setup)
prek install
```
This installs the pre-commit hook at `.git/hooks/pre-commit` which automatically runs all configured checks before each commit.
### Configured Hooks
The repository uses `.pre-commit-config.yaml` with the following hooks:
**Important**: Always use `--new-from-rev` to only lint changed files. This prevents formatting the entire repository and keeps changes focused on your actual modifications.
- **Database Tests**: Extensive migration and data integrity validation
- **Policy Tests**: ACL rule evaluation and edge cases
- **Performance Tests**: NodeStore and high-frequency operation validation
## Integration Testing System
### Overview
Headscale uses Docker-based integration tests with real Tailscale clients to validate end-to-end functionality. The integration test system is complex and requires specialized knowledge for effective execution and debugging.
### **MANDATORY: Use the headscale-integration-tester Agent**
**CRITICAL REQUIREMENT**: For ANY integration test execution, analysis, troubleshooting, or validation, you MUST use the `headscale-integration-tester` agent. This agent contains specialized knowledge about:
- Test execution strategies and timing requirements
- Infrastructure vs code issue distinction (99% vs 1% failure patterns)
- Security-critical debugging rules and forbidden practices
- Comprehensive artifact analysis workflows
- Real-world failure patterns from HA debugging experiences
All test runs save comprehensive debugging artifacts to `control_logs/TIMESTAMP-ID/` including server logs, client logs, database dumps, MapResponse protocol data, and Prometheus metrics.
**For all integration test work, use the headscale-integration-tester agent - it contains the complete knowledge needed for effective testing and debugging.**
## NodeStore Implementation Details
**Key Insight from Recent Work**: The NodeStore is a critical performance optimization that caches node data in memory while ensuring consistency with the database. When working with route advertisements or node state changes:
1.**Timing Considerations**: Route advertisements need time to propagate from clients to server. Use `require.EventuallyWithT()` patterns in tests instead of immediate assertions.
2.**Synchronization Points**: NodeStore updates happen at specific points like `poll.go:420` after Hostinfo changes. Ensure these are maintained when modifying the polling logic.
3.**Peer Visibility**: The NodeStore's `peersFunc` determines which nodes are visible to each other. Policy-based filtering is separate from monitoring visibility - expired nodes should remain visible for debugging but marked as expired.
## Testing Guidelines
### Integration Test Patterns
#### **CRITICAL: EventuallyWithT Pattern for External Calls**
**All external calls in integration tests MUST be wrapped in EventuallyWithT blocks** to handle eventual consistency in distributed systems. External calls include:
-`client.Status()` - Getting Tailscale client status
-`client.Curl()` - Making HTTP requests through clients
- Use `requirePeerSubnetRoutesWithCollect` instead of `requirePeerSubnetRoutes` inside EventuallyWithT
- Use `requireNodeRouteCountWithCollect` instead of `requireNodeRouteCount` inside EventuallyWithT
- Use `assertTracerouteViaIPWithCollect` instead of `assertTracerouteViaIP` inside EventuallyWithT
```go
// Node route checking by actual node properties, not array position
var routeNode *v1.Node
for _, node := range nodes {
if nodeIDStr := fmt.Sprintf("%d", node.GetId()); expectedRoutes[nodeIDStr] != "" {
routeNode = node
break
}
}
```
### Running Problematic Tests
- Some tests require significant time (e.g., `TestNodeOnlineStatus` runs for 12 minutes)
- Infrastructure issues like disk space can cause test failures unrelated to code changes
- Use `--postgres` flag when testing database-heavy scenarios
## Quality Assurance and Testing Requirements
### **MANDATORY: Always Use Specialized Testing Agents**
**CRITICAL REQUIREMENT**: For ANY task involving testing, quality assurance, review, or validation, you MUST use the appropriate specialized agent at the END of your task list. This ensures comprehensive quality validation and prevents regressions.
**Required Agents for Different Task Types**:
1.**Integration Testing**: Use `headscale-integration-tester` agent for:
- Running integration tests with `cmd/hi`
- Analyzing test failures and artifacts
- Troubleshooting Docker-based test infrastructure
- Validating end-to-end functionality changes
2.**Quality Control**: Use `quality-control-enforcer` agent for:
- Code review and validation
- Ensuring best practices compliance
- Preventing common pitfalls and anti-patterns
- Validating architectural decisions
**Agent Usage Pattern**: Always add the appropriate agent as the FINAL step in any task list to ensure quality validation occurs after all work is complete.
### Integration Test Debugging Reference
Test artifacts are preserved in `control_logs/TIMESTAMP-ID/` including:
- Headscale server logs (stderr/stdout)
- Tailscale client logs and status
- Database dumps and network captures
- MapResponse JSON files for protocol debugging
**For integration test issues, ALWAYS use the headscale-integration-tester agent - do not attempt manual debugging.**
## EventuallyWithT Pattern for Integration Tests
### Overview
EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
### External Calls That Must Be Wrapped
The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
-`headscale.ListNodes()` - Queries server state
-`client.Status()` - Gets client network status
-`client.Curl()` - Makes HTTP requests through the network
Headscale implements a **tags-as-identity** model where tags and user ownership are mutually exclusive ways to identify nodes. This is a fundamental architectural principle that affects node registration, ownership, ACL evaluation, and API behavior.
### Core Principle: Tags XOR User Ownership
Every node in Headscale is **either** tagged **or** user-owned, never both:
- **Tagged Nodes**: Ownership is defined by tags (e.g., `tag:server`, `tag:database`)
- Tags are set during registration via tagged PreAuthKey
- Tags are immutable after registration (cannot be changed via API)
- May have `UserID` set for "created by" tracking, but ownership is via tags
- Identified by: `node.IsTagged()` returns `true`
- **User-Owned Nodes**: Ownership is defined by user assignment
- Registered via OIDC, web auth, or untagged PreAuthKey
2.**Don't assume tagged nodes never have UserID set**
- Tagged nodes MAY have UserID for "created by" tracking
- Always use `IsTagged()` to determine ownership type
3.**Don't allow setting tags on user-owned nodes**
- This violates the tags XOR user principle
- Use API validation to prevent this
4.**Don't forget TaggedDevices in mapper**
- All tagged nodes MUST use `TaggedDevices.ID` in MapResponse
- User ID is only for actual user-owned nodes
### Migration Considerations
When nodes transition between ownership models:
- **No automatic migration**: Tags-as-identity is set at registration and immutable
- **Re-registration required**: To change from user-owned to tagged (or vice versa), node must be deleted and re-registered
- **UserID persistence**: UserID on tagged nodes is informational and not cleared
### Architecture Benefits
The tags-as-identity model provides:
1.**Clear ownership semantics**: No ambiguity about who/what owns a node
2.**ACL simplicity**: Tag-based access control without user conflicts
3.**API safety**: Validation prevents invalid ownership states
4.**Protocol compatibility**: TaggedDevices special user aligns with Tailscale's model
## Logging Patterns
### Incremental Log Event Building
When building log statements with multiple fields, especially with conditional fields, use the **incremental log event pattern** instead of long single-line chains. This improves readability and allows conditional field addition.
**Pattern:**
```go
// GOOD: Incremental building with conditional fields
1.**Assign chained calls back to the variable**: `logEvent = logEvent.Str(...)` - zerolog methods return a new event, so you must capture the return value
2.**Use for conditional fields**: When fields depend on runtime conditions, build incrementally
3.**Use for long log lines**: When a log line exceeds ~100 characters, split it for readability
4.**Call `.Msg()` at the end**: The final `.Msg()` or `.Msgf()` sends the log event
**Anti-pattern to avoid:**
```go
// BAD: Long single-line chains are hard to read and can't have conditional fields
Prefer the incremental log event pattern over creating helper functions that return multiple logging closures. Helper functions like `logPollFunc` create unnecessary indirection and allocate closures.
