feat: initial implementation of @astrojs/discovery integration

This commit introduces a comprehensive Astro integration that automatically
generates discovery files for websites:

Features:
- robots.txt with LLM bot support (Anthropic-AI, GPTBot, etc.)
- llms.txt for AI assistant context and instructions
- humans.txt for team credits and site information
- Automatic sitemap integration via @astrojs/sitemap

Technical Details:
- TypeScript implementation with full type safety
- Configurable HTTP caching headers
- Custom template support for all generated files
- Sensible defaults with extensive customization options
- Date-based versioning (2025.11.03)

Testing:
- 34 unit tests covering all generators
- Test coverage for robots.txt, llms.txt, and humans.txt
- Integration with Vitest

Documentation:
- Comprehensive README with examples
- API reference documentation
- Contributing guidelines
- Example configurations (minimal and full)

CONTRIBUTING.md

@@ -0,0 +1,169 @@
+# Contributing to @astrojs/discovery
+
+Thank you for your interest in contributing to @astrojs/discovery! This guide will help you get started.
+
+## Development Setup
+
+1. **Clone the repository**
+   ```bash
+   git clone https://github.com/withastro/astro-discovery.git
+   cd astro-discovery
+   ```
+
+2. **Install dependencies**
+   ```bash
+   npm install
+   ```
+
+3. **Build the project**
+   ```bash
+   npm run build
+   ```
+
+4. **Run tests**
+   ```bash
+   npm test
+   ```
+
+## Project Structure
+
+```
+@astrojs/discovery/
+├── src/
+│   ├── index.ts              # Main integration entry point
+│   ├── types.ts              # TypeScript type definitions
+│   ├── config-store.ts       # Global config management
+│   ├── generators/
+│   │   ├── robots.ts        # robots.txt generator
+│   │   ├── llms.ts          # llms.txt generator
+│   │   └── humans.ts        # humans.txt generator
+│   ├── routes/
+│   │   ├── robots.ts        # /robots.txt API route
+│   │   ├── llms.ts          # /llms.txt API route
+│   │   └── humans.ts        # /humans.txt API route
+│   └── validators/
+│       └── config.ts         # Configuration validation
+├── dist/                     # Built output (generated)
+├── example/                  # Example configurations
+└── tests/                    # Test files (to be added)
+```
+
+## Making Changes
+
+### Adding New Features
+
+1. Create a new branch for your feature
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. Make your changes following the existing code style
+
+3. Add tests for your changes
+
+4. Update documentation in README.md
+
+5. Build and test
+   ```bash
+   npm run build
+   npm test
+   ```
+
+### Code Style
+
+- Use TypeScript for all code
+- Follow existing naming conventions
+- Add JSDoc comments for public APIs
+- Keep functions focused and small
+- Use meaningful variable names
+
+### Commit Messages
+
+Follow conventional commit format:
+- `feat:` New features
+- `fix:` Bug fixes
+- `docs:` Documentation changes
+- `test:` Test additions/changes
+- `refactor:` Code refactoring
+- `chore:` Maintenance tasks
+
+Example:
+```
+feat: add support for custom LLM bot agents
+
+- Added ability to specify custom LLM bot user agents
+- Updated documentation with examples
+- Added tests for custom agent configuration
+```
+
+## Testing
+
+### Running Tests
+
+```bash
+# Run all tests
+npm test
+
+# Run tests in watch mode
+npm run test:watch
+
+# Run tests with coverage
+npm run test:coverage
+```
+
+### Writing Tests
+
+Place tests in the `tests/` directory. Follow these patterns:
+
+```typescript
+import { describe, it, expect } from 'vitest';
+import { generateRobotsTxt } from '../src/generators/robots';
+
+describe('generateRobotsTxt', () => {
+  it('generates basic robots.txt', () => {
+    const result = generateRobotsTxt({}, new URL('https://example.com'));
+    expect(result).toContain('User-agent: *');
+    expect(result).toContain('Sitemap: https://example.com/sitemap-index.xml');
+  });
+});
+```
+
+## Pull Request Process
+
+1. **Update documentation**: Ensure README.md reflects any changes
+
+2. **Add tests**: All new features should have tests
+
+3. **Update CHANGELOG**: Add your changes to CHANGELOG.md
+
+4. **Create a pull request**:
+   - Use a clear, descriptive title
+   - Reference any related issues
+   - Describe your changes in detail
+   - Include screenshots for UI changes
+
+5. **Address review feedback**: Be responsive to code review comments
+
+## Release Process
+
+(For maintainers)
+
+1. Update version in package.json using date-based versioning (YYYY.MM.DD)
+2. Update CHANGELOG.md
+3. Create a git tag
+4. Push to npm
+
+```bash
+npm version 2025.11.03
+npm publish
+```
+
+## Questions?
+
+- Open an issue for bugs or feature requests
+- Start a discussion for questions or ideas
+- Join our Discord community
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the MIT License.