Delmar/bikinibottom
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added gitea-mirror

Browse Source
This commit is contained in:
Julieñ
2026-01-19 08:34:20 +01:00
parent b956b07d1e
commit e19aff248d
385 changed files with 81357 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

354
Divers/gitea-mirror/docs/DEVELOPMENT_WORKFLOW.md Normal file
View File

@@ -0,0 +1,354 @@
# Development Workflow
This guide covers the development workflow for the open-source Gitea Mirror.
## Getting Started
### Prerequisites
- Bun >= 1.2.9
- Node.js >= 20
- Git
- GitHub account (for API access)
- Gitea instance (for testing)
### Initial Setup
1. **Clone the repository**:
```bash
git clone https://github.com/RayLabsHQ/gitea-mirror.git
cd gitea-mirror
```
2. **Install dependencies and seed the SQLite database**:
```bash
bun run setup
```
3. **Configure environment (optional)**:
```bash
cp .env.example .env
# Edit .env with your settings
```
4. **Start the development server**:
```bash
bun run dev
```
## Development Commands
| Command | Description |
|---------|-------------|
| `bun run dev` | Start the Bun + Astro dev server with hot reload |
| `bun run build` | Build the production bundle |
| `bun run preview` | Preview the production build locally |
| `bun test` | Run the Bun test suite |
| `bun test:watch` | Run tests in watch mode |
| `bun run db:studio` | Launch Drizzle Kit Studio |
## Project Structure
```
gitea-mirror/
├── src/ # Application UI, API routes, and services
│ ├── components/ # React components rendered inside Astro pages
│ ├── pages/ # Astro pages and API routes (e.g., /api/*)
│ ├── lib/ # Core logic: GitHub/Gitea clients, scheduler, recovery, db helpers
│ │ ├── db/ # Drizzle adapter + schema
│ │ ├── modules/ # Module wiring (jobs, integrations)
│ │ └── utils/ # Shared utilities
│ ├── hooks/ # React hooks
│ ├── content/ # In-app documentation and templated content
│ ├── layouts/ # Shared layout components
│ ├── styles/ # Tailwind CSS entrypoints
│ └── types/ # TypeScript types
├── scripts/ # Bun scripts for DB management and maintenance
├── www/ # Marketing site (Astro + MDX use cases)
├── public/ # Static assets served by Vite/Astro
└── tests/ # Dedicated integration/unit test helpers
```
## Feature Development
### Adding a New Feature
1. **Create feature branch**:
```bash
git checkout -b feature/my-feature
```
2. **Plan your changes**:
- UI components live in `src/components/`
- API endpoints live in `src/pages/api/`
- Database logic is under `src/lib/db/` (schema + adapter)
- Shared types are in `src/types/`
3. **Implement the feature**:
**Example: Adding a new API endpoint**
```typescript
// src/pages/api/my-endpoint.ts
import type { APIRoute } from 'astro';
import { getUserFromCookie } from '@/lib/auth-utils';
export const GET: APIRoute = async ({ request }) => {
const user = await getUserFromCookie(request);
if (!user) {
return new Response('Unauthorized', { status: 401 });
}
// Your logic here
return new Response(JSON.stringify({ data: 'success' }), {
headers: { 'Content-Type': 'application/json' }
});
};
```
4. **Write tests**:
```typescript
// src/lib/my-feature.test.ts
import { describe, it, expect } from 'bun:test';
describe('My Feature', () => {
it('should work correctly', () => {
expect(myFunction()).toBe('expected');
});
});
```
5. **Update documentation**:
- Add JSDoc comments
- Update README/docs if needed
- Document API changes
## Database Development
### Schema Changes
1. **Modify schema**:
```typescript
// src/lib/db/schema.ts
export const myTable = sqliteTable('my_table', {
id: text('id').primaryKey(),
name: text('name').notNull(),
createdAt: integer('created_at').notNull(),
});
```
2. **Generate migration**:
```bash
bun run db:generate
```
3. **Apply migration**:
```bash
bun run db:migrate
```
### Writing Queries
```typescript
// src/lib/db/queries/my-queries.ts
import { db } from '../index';
import { myTable } from '../schema';
export async function getMyData(userId: string) {
return db.select()
.from(myTable)
.where(eq(myTable.userId, userId));
}
```
## Testing
### Unit Tests
```bash
# Run all tests
bun test
# Run specific test file
bun test auth
# Watch mode
bun test:watch
# Coverage
bun test:coverage
```
### Manual Testing Checklist
- [ ] Feature works as expected
- [ ] No console errors
- [ ] Responsive on mobile
- [ ] Handles errors gracefully
- [ ] Loading states work
- [ ] Form validation works
- [ ] API returns correct status codes
## Debugging
### VSCode Configuration
Create `.vscode/launch.json`:
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "bun",
"request": "launch",
"name": "Debug Bun",
"program": "${workspaceFolder}/src/index.ts",
"cwd": "${workspaceFolder}",
"env": {
"NODE_ENV": "development"
}
}
]
}
```
### Debug Logging
```typescript
// Development only logging
if (import.meta.env.DEV) {
console.log('[Debug]', data);
}
```
## Code Style
### TypeScript
- Use strict mode
- Define interfaces for all data structures
- Avoid `any` type
- Use proper error handling
### React Components
- Use functional components
- Implement proper loading states
- Handle errors with error boundaries
- Use TypeScript for props
### API Routes
- Always validate input
- Return proper status codes
- Use consistent error format
- Document with JSDoc
## Git Workflow
### Commit Messages
Follow conventional commits:
```
feat: add repository filtering
fix: resolve sync timeout issue
docs: update API documentation
style: format code with prettier
refactor: simplify auth logic
test: add user creation tests
chore: update dependencies
```
### Pull Request Process
1. Create feature branch
2. Make changes
3. Write/update tests
4. Update documentation
5. Create PR with description
6. Address review feedback
7. Squash and merge
## Performance
### Development Tips
- Use React DevTools
- Monitor bundle size
- Profile database queries
- Check memory usage
### Optimization
- Lazy load components
- Optimize images
- Use database indexes
- Cache API responses
## Common Issues
### Port Already in Use
```bash
# Use different port
PORT=3001 bun run dev
```
### Database Locked
```bash
# Reset database
bun run cleanup-db
bun run init-db
```
### Type Errors
```bash
# Check types
bunx tsc --noEmit
```
## Release Process
1. **Update version**:
```bash
npm version patch # or minor/major
```
2. **Update CHANGELOG.md**
3. **Build and test**:
```bash
bun run build
bun test
```
4. **Create release**:
```bash
git tag v2.23.0
git push origin v2.23.0
```
5. **Create GitHub release**
## Contributing
1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to your fork
5. Create a Pull Request
## Resources
- [Astro Documentation](https://docs.astro.build)
- [Bun Documentation](https://bun.sh/docs)
- [Drizzle ORM](https://orm.drizzle.team)
- [React Documentation](https://react.dev)
- [TypeScript Handbook](https://www.typescriptlang.org/docs/)
## Getting Help
- Check existing [issues](https://github.com/yourusername/gitea-mirror/issues)
- Join [discussions](https://github.com/yourusername/gitea-mirror/discussions)
- Read the [FAQ](./FAQ.md)

416
Divers/gitea-mirror/docs/ENVIRONMENT_VARIABLES.md Normal file
View File

@@ -0,0 +1,416 @@
# Environment Variables Documentation
This document provides a comprehensive list of all environment variables supported by Gitea Mirror. These can be used to configure the application via Docker or other deployment methods.
## Environment Variables and UI Interaction
When environment variables are set:
1. They are loaded on application startup
2. Values are stored in the database on first load
3. The UI will display these values and they can be modified
4. UI changes are saved to the database and persist
5. Environment variables provide initial defaults but don't override UI changes
**Note**: Some critical settings like `GITEA_LFS`, `MIRROR_RELEASES`, and `MIRROR_METADATA` will be visible and configurable in the UI even when set via environment variables.
## Table of Contents
- [Core Configuration](#core-configuration)
- [GitHub Configuration](#github-configuration)
- [Gitea Configuration](#gitea-configuration)
- [Mirror Options](#mirror-options)
- [Automation Configuration](#automation-configuration)
- [Database Cleanup Configuration](#database-cleanup-configuration)
- [Authentication Configuration](#authentication-configuration)
- [Docker Configuration](#docker-configuration)
## Core Configuration
Essential application settings required for running Gitea Mirror.
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `NODE_ENV` | Application environment | `production` | No |
| `HOST` | Server host binding | `0.0.0.0` | No |
| `PORT` | Server port | `4321` | No |
| `DATABASE_URL` | Database connection URL | `sqlite://data/gitea-mirror.db` | No |
| `BETTER_AUTH_SECRET` | Secret key for session signing (generate with: `openssl rand -base64 32`) | - | Yes |
| `BETTER_AUTH_URL` | Primary base URL for authentication. This should be the main URL where your application is accessed. | `http://localhost:4321` | No |
| `PUBLIC_BETTER_AUTH_URL` | Client-side auth URL for multi-origin access. Set this to your primary domain when you need to access the app from different origins (e.g., both IP and domain). The client will use this URL for all auth requests instead of the current browser origin. | - | No |
| `BETTER_AUTH_TRUSTED_ORIGINS` | Trusted origins for authentication requests. Comma-separated list of URLs. Use this to specify additional access URLs (e.g., local IP + domain: `http://10.10.20.45:4321,https://gitea-mirror.mydomain.tld`), SSO providers, reverse proxies, etc. | - | No |
| `ENCRYPTION_SECRET` | Optional encryption key for tokens (generate with: `openssl rand -base64 48`) | - | No |
## GitHub Configuration
Settings for connecting to and configuring GitHub repository sources.
### Basic Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITHUB_USERNAME` | Your GitHub username | - | - |
| `GITHUB_TOKEN` | GitHub personal access token (requires repo and admin:org scopes) | - | - |
| `GITHUB_TYPE` | GitHub account type | `personal` | `personal`, `organization` |
### Repository Selection
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `PRIVATE_REPOSITORIES` | Include private repositories | `false` | `true`, `false` |
| `PUBLIC_REPOSITORIES` | Include public repositories | `true` | `true`, `false` |
| `INCLUDE_ARCHIVED` | Include archived repositories | `false` | `true`, `false` |
| `SKIP_FORKS` | Skip forked repositories | `false` | `true`, `false` |
| `MIRROR_STARRED` | Mirror starred repositories | `false` | `true`, `false` |
| `STARRED_REPOS_ORG` | Organization name for starred repos | `starred` | Any string |
### Organization Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `MIRROR_ORGANIZATIONS` | Mirror organization repositories | `false` | `true`, `false` |
| `PRESERVE_ORG_STRUCTURE` | Preserve GitHub organization structure in Gitea | `false` | `true`, `false` |
| `ONLY_MIRROR_ORGS` | Only mirror organization repos (skip personal) | `false` | `true`, `false` |
| `MIRROR_STRATEGY` | Repository organization strategy | `preserve` | `preserve`, `single-org`, `flat-user`, `mixed` |
### Advanced Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SKIP_STARRED_ISSUES` | Enable lightweight mode for starred repos (skip issues) | `false` | `true`, `false` |
## Gitea Configuration
Settings for the destination Gitea instance.
### Connection Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_URL` | Gitea instance URL | - | Valid URL |
| `GITEA_TOKEN` | Gitea access token | - | - |
| `GITEA_USERNAME` | Gitea username | - | - |
| `GITEA_ORGANIZATION` | Default organization for single-org strategy | `github-mirrors` | Any string |
### Repository Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_ORG_VISIBILITY` | Default organization visibility | `public` | `public`, `private`, `limited`, `default` |
| `GITEA_MIRROR_INTERVAL` | Mirror sync interval - **automatically enables scheduled mirroring when set** | `8h` | Duration string (e.g., `30m`, `1h`, `8h`, `24h`, `1d`) or seconds |
| `GITEA_LFS` | Enable LFS support (requires LFS on Gitea server) - Shows in UI | `false` | `true`, `false` |
| `GITEA_CREATE_ORG` | Auto-create organizations | `true` | `true`, `false` |
| `GITEA_PRESERVE_VISIBILITY` | Preserve GitHub repo visibility in Gitea | `false` | `true`, `false` |
### Template Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_TEMPLATE_OWNER` | Template repository owner | - | Any string |
| `GITEA_TEMPLATE_REPO` | Template repository name | - | Any string |
### Topic Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_ADD_TOPICS` | Add topics to repositories | `true` | `true`, `false` |
| `GITEA_TOPIC_PREFIX` | Prefix for repository topics | - | Any string |
### Fork Handling
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_FORK_STRATEGY` | How to handle forked repositories | `reference` | `skip`, `reference`, `full-copy` |
### Additional Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `GITEA_SKIP_TLS_VERIFY` | Skip TLS certificate verification (WARNING: insecure) | `false` | `true`, `false` |
## Mirror Options
Control what content gets mirrored from GitHub to Gitea.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `MIRROR_RELEASES` | Mirror GitHub releases | `false` | `true`, `false` |
| `RELEASE_LIMIT` | Maximum number of releases to mirror per repository | `10` | Number (1-100) |
| `MIRROR_WIKI` | Mirror wiki content | `false` | `true`, `false` |
| `MIRROR_METADATA` | Master toggle for metadata mirroring | `false` | `true`, `false` |
| `MIRROR_ISSUES` | Mirror issues (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_PULL_REQUESTS` | Mirror pull requests (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_LABELS` | Mirror labels (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_MILESTONES` | Mirror milestones (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_ISSUE_CONCURRENCY` | Number of issues processed in parallel. Set above `1` to speed up mirroring at the risk of out-of-order creation. | `3` | Integer ≥ 1 |
| `MIRROR_PULL_REQUEST_CONCURRENCY` | Number of pull requests processed in parallel. Values above `1` may cause ordering differences. | `5` | Integer ≥ 1 |
> **Ordering vs Throughput:** Metadata now mirrors sequentially by default to preserve chronology. Increase the concurrency variables only if you can tolerate minor out-of-order entries.
## Automation Configuration
Configure automatic scheduled mirroring.
### Basic Schedule Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SCHEDULE_ENABLED` | Enable automatic mirroring. **When set to `true`, automatically imports and mirrors all repositories on startup** (v3.5.3+) | `false` | `true`, `false` |
| `SCHEDULE_INTERVAL` | Interval in seconds or cron expression. **Supports cron syntax for scheduled runs** (e.g., `"0 2 * * *"` for 2 AM daily) | `3600` | Number (seconds) or cron string |
| `DELAY` | Legacy: same as SCHEDULE_INTERVAL | `3600` | Number (seconds) |
> **🚀 Auto-Start Feature (v3.5.3+)**
> Setting either `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` triggers auto-start functionality where the service will:
> 1. **Import** all GitHub repositories on startup
> 2. **Mirror** them to Gitea immediately
> 3. **Continue syncing** at the configured interval
> 4. **Auto-discover** new repositories
> 5. **Clean up** deleted repositories (if configured)
>
> This eliminates the need for manual button clicks - perfect for Docker/Kubernetes deployments!
> **⏰ Scheduling with Cron Expressions**
> Use cron expressions in `SCHEDULE_INTERVAL` to run at specific times:
> - `"0 2 * * *"` - Daily at 2 AM
> - `"0 */6 * * *"` - Every 6 hours
> - `"0 0 * * 0"` - Weekly on Sunday at midnight
> - `"0 3 * * 1-5"` - Weekdays at 3 AM (Monday-Friday)
>
> This is useful for optimizing bandwidth usage during low-activity periods.
### Execution Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SCHEDULE_CONCURRENT` | Allow concurrent mirror operations | `false` | `true`, `false` |
| `SCHEDULE_BATCH_SIZE` | Number of repos to process in parallel | `10` | Number |
| `SCHEDULE_PAUSE_BETWEEN_BATCHES` | Pause between batches (milliseconds) | `5000` | Number |
### Retry Configuration
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SCHEDULE_RETRY_ATTEMPTS` | Number of retry attempts | `3` | Number |
| `SCHEDULE_RETRY_DELAY` | Delay between retries (milliseconds) | `60000` | Number |
| `SCHEDULE_TIMEOUT` | Max time for a mirror operation (milliseconds) | `3600000` | Number |
| `SCHEDULE_AUTO_RETRY` | Automatically retry failed operations | `true` | `true`, `false` |
### Update Detection
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `AUTO_IMPORT_REPOS` | Automatically discover and import new GitHub repositories during scheduled syncs | `true` | `true`, `false` |
| `AUTO_MIRROR_REPOS` | Automatically mirror newly imported repositories during scheduled syncs (no manual “Mirror All” required) | `false` | `true`, `false` |
| `SCHEDULE_ONLY_MIRROR_UPDATED` | Only mirror repos with updates | `false` | `true`, `false` |
| `SCHEDULE_UPDATE_INTERVAL` | Check for updates interval (milliseconds) | `86400000` | Number |
| `SCHEDULE_SKIP_RECENTLY_MIRRORED` | Skip recently mirrored repos | `true` | `true`, `false` |
| `SCHEDULE_RECENT_THRESHOLD` | Skip if mirrored within this time (milliseconds) | `3600000` | Number |
### Maintenance & Notifications
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SCHEDULE_CLEANUP_BEFORE_MIRROR` | Run cleanup before mirroring | `false` | `true`, `false` |
| `SCHEDULE_NOTIFY_ON_FAILURE` | Send notifications on failure | `true` | `true`, `false` |
| `SCHEDULE_NOTIFY_ON_SUCCESS` | Send notifications on success | `false` | `true`, `false` |
| `SCHEDULE_LOG_LEVEL` | Logging level | `info` | `error`, `warn`, `info`, `debug` |
| `SCHEDULE_TIMEZONE` | Timezone for scheduling | `UTC` | Valid timezone string |
## Database Cleanup Configuration
Configure automatic cleanup of old events and data.
### Basic Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `CLEANUP_ENABLED` | Enable automatic cleanup | `false` | `true`, `false` |
| `CLEANUP_RETENTION_DAYS` | Days to keep events | `7` | Number |
### Repository Cleanup
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `CLEANUP_DELETE_FROM_GITEA` | Delete repositories from Gitea | `false` | `true`, `false` |
| `CLEANUP_DELETE_IF_NOT_IN_GITHUB` | Delete repos not found in GitHub (automatically enables cleanup) | `true` | `true`, `false` |
| `CLEANUP_ORPHANED_REPO_ACTION` | Action for orphaned repositories. **Note**: `archive` is recommended to preserve backups | `archive` | `skip`, `archive`, `delete` |
| `CLEANUP_DRY_RUN` | Test mode without actual deletion | `false` | `true`, `false` |
| `CLEANUP_PROTECTED_REPOS` | Comma-separated list of protected repository names | - | Comma-separated strings |
**🛡️ Safety Features (Backup Protection)**:
- **GitHub Failures Don't Delete Backups**: Cleanup is automatically skipped if GitHub API returns errors (404, 403, connection issues)
- **Archive Never Deletes**: The `archive` action ALWAYS preserves repository data, it never deletes
- **Graceful Degradation**: If marking as archived fails, the repository remains fully accessible in Gitea
- **The Purpose of Backups**: Your mirrors are preserved even when GitHub sources disappear - that's the whole point!
**Archive Behavior (Aligned with Gitea API)**:
- **Regular repositories**: Uses Gitea's native archive feature (PATCH `/repos/{owner}/{repo}` with `archived: true`)
- Makes repository read-only while preserving all data
- **Mirror repositories**: Uses rename strategy (Gitea API returns 422 for archiving mirrors)
- Renamed with `archived-` prefix for clear identification
- Description updated with preservation notice and timestamp
- Mirror interval set to 8760h (1 year) to minimize sync attempts
- Repository remains fully accessible and cloneable
- **Manual Sync Option**: Archived mirrors are still available on the Repositories page with automatic syncs disabled—use the `Manual Sync` action to refresh them on demand.
### Execution Settings
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `CLEANUP_BATCH_SIZE` | Number of items to process per batch | `10` | Number |
| `CLEANUP_PAUSE_BETWEEN_DELETES` | Pause between deletions (milliseconds) | `2000` | Number |
## Authentication Configuration
Configure authentication methods and SSO.
### Header Authentication (Reverse Proxy SSO)
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `HEADER_AUTH_ENABLED` | Enable header-based authentication | `false` | `true`, `false` |
| `HEADER_AUTH_USER_HEADER` | Header containing username | `X-Authentik-Username` | Header name |
| `HEADER_AUTH_EMAIL_HEADER` | Header containing email | `X-Authentik-Email` | Header name |
| `HEADER_AUTH_NAME_HEADER` | Header containing display name | `X-Authentik-Name` | Header name |
| `HEADER_AUTH_AUTO_PROVISION` | Auto-create users from headers | `false` | `true`, `false` |
| `HEADER_AUTH_ALLOWED_DOMAINS` | Comma-separated list of allowed email domains | - | Comma-separated domains |
## Docker Configuration
Settings specific to Docker deployments.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `DOCKER_REGISTRY` | Docker registry URL | `ghcr.io` | Registry URL |
| `DOCKER_IMAGE` | Docker image name | `raylabshq/gitea-mirror:` | Image name |
| `DOCKER_TAG` | Docker image tag | `latest` | Tag name |
## Example Docker Compose Configuration
Here's an example of how to use these environment variables in a `docker-compose.yml` file:
```yaml
version: '3.8'
services:
gitea-mirror:
image: ghcr.io/raylabshq/gitea-mirror:latest
container_name: gitea-mirror
environment:
# Core Configuration
- NODE_ENV=production
- DATABASE_URL=file:data/gitea-mirror.db
- BETTER_AUTH_SECRET=your-secure-secret-here
# Primary access URL:
- BETTER_AUTH_URL=https://gitea-mirror.mydomain.tld
# Additional access URLs (local network + SSO providers):
# - BETTER_AUTH_TRUSTED_ORIGINS=http://10.10.20.45:4321,http://192.168.1.100:4321,https://auth.provider.com
# GitHub Configuration
- GITHUB_USERNAME=your-username
- GITHUB_TOKEN=ghp_your_token_here
- PRIVATE_REPOSITORIES=true
- MIRROR_STARRED=true
- SKIP_FORKS=false
# Gitea Configuration
- GITEA_URL=http://gitea:3000
- GITEA_USERNAME=admin
- GITEA_TOKEN=your-gitea-token
- GITEA_ORGANIZATION=github-mirrors
- GITEA_ORG_VISIBILITY=public
# Mirror Options
- MIRROR_RELEASES=true
- MIRROR_WIKI=true
- MIRROR_METADATA=true
- MIRROR_ISSUES=true
- MIRROR_PULL_REQUESTS=true
# Automation
- SCHEDULE_ENABLED=true
- SCHEDULE_INTERVAL=3600
# Cleanup
- CLEANUP_ENABLED=true
- CLEANUP_RETENTION_DAYS=30
volumes:
- ./data:/app/data
ports:
- "4321:4321"
```
## Authentication URL Configuration
### Multiple Access URLs
To allow access to Gitea Mirror through multiple URLs (e.g., local IP and public domain), you need to configure both server and client settings:
**Example Configuration:**
```bash
# Primary URL (required) - where the auth server is hosted
BETTER_AUTH_URL=https://gitea-mirror.mydomain.tld
# Client-side URL (optional) - tells the browser where to send auth requests
# Set this to your primary domain when accessing from different origins
PUBLIC_BETTER_AUTH_URL=https://gitea-mirror.mydomain.tld
# Additional trusted origins (optional) - origins allowed to make auth requests
BETTER_AUTH_TRUSTED_ORIGINS=http://10.10.20.45:4321,http://192.168.1.100:4321
```
This setup allows you to:
- Access via local network IP: `http://10.10.20.45:4321`
- Access via public domain: `https://gitea-mirror.mydomain.tld`
- Auth requests from the IP will be sent to the domain (via `PUBLIC_BETTER_AUTH_URL`)
- Each origin requires separate login due to browser cookie isolation
**Important:** When accessing from different origins (IP vs domain), you'll need to log in separately on each origin as cookies cannot be shared across different origins for security reasons.
### Trusted Origins
The `BETTER_AUTH_TRUSTED_ORIGINS` variable serves multiple purposes:
1. **SSO/OIDC Providers**: When using external authentication providers (Google, Authentik, Okta)
2. **Reverse Proxies**: When running behind nginx, Traefik, or other proxies
3. **Cross-Origin Requests**: When the frontend and backend are on different domains
4. **Development**: When testing from different URLs
**Example Scenarios:**
```bash
# For Authentik SSO integration
BETTER_AUTH_TRUSTED_ORIGINS=https://authentik.company.com,https://auth.company.com
# For reverse proxy setup
BETTER_AUTH_TRUSTED_ORIGINS=https://proxy.internal,https://public.domain.com
# For development with multiple environments
BETTER_AUTH_TRUSTED_ORIGINS=http://localhost:3000,http://192.168.1.100:3000
```
**Important Notes:**
- All URLs from `BETTER_AUTH_URL` are automatically trusted
- URLs must be complete with protocol (http/https)
- Multiple origins are separated by commas
- No trailing slashes needed
## Notes
1. **First Run**: Environment variables are loaded when the container starts. The configuration is applied after the first user account is created.
2. **UI Priority**: Manual changes made through the web UI will be preserved. Environment variables only set values for empty fields.
3. **Token Security**: All tokens are encrypted before being stored in the database.
4. **Auto-Enabling Features**: Certain environment variables automatically enable features when set:
- `GITEA_MIRROR_INTERVAL` - Automatically enables scheduled mirroring
- `CLEANUP_DELETE_IF_NOT_IN_GITHUB=true` - Automatically enables repository cleanup
- `SCHEDULE_INTERVAL` or `DELAY` - Automatically enables the scheduler
5. **Backward Compatibility**: The `DELAY` variable is maintained for backward compatibility but `SCHEDULE_INTERVAL` is preferred.
6. **Required Scopes**: The GitHub token requires the following scopes:
- `repo` (full control of private repositories)
- `admin:org` (read organization data)
- Additional scopes may be required for specific features
For more examples and detailed configuration, see the `.env.example` file in the repository.

249
Divers/gitea-mirror/docs/GRACEFUL_SHUTDOWN.md Normal file
View File

@@ -0,0 +1,249 @@
# Graceful Shutdown and Enhanced Job Recovery
This document describes the graceful shutdown and enhanced job recovery capabilities implemented in gitea-mirror v2.8.0+.
## Overview
The gitea-mirror application now includes comprehensive graceful shutdown handling and enhanced job recovery mechanisms designed specifically for containerized environments. These features ensure:
- **No data loss** during container restarts or shutdowns
- **Automatic job resumption** after application restarts
- **Clean termination** of all active processes and connections
- **Container-aware design** optimized for Docker/LXC deployments
## Features
### 1. Graceful Shutdown Manager
The shutdown manager (`src/lib/shutdown-manager.ts`) provides centralized coordination of application termination:
#### Key Capabilities:
- **Active Job Tracking**: Monitors all running mirroring/sync jobs
- **State Persistence**: Saves job progress to database before shutdown
- **Callback System**: Allows services to register cleanup functions
- **Timeout Protection**: Prevents hanging shutdowns with configurable timeouts
- **Signal Coordination**: Works with signal handlers for proper container lifecycle
#### Configuration:
- **Shutdown Timeout**: 30 seconds maximum (configurable)
- **Job Save Timeout**: 10 seconds per job (configurable)
### 2. Signal Handlers
The signal handler system (`src/lib/signal-handlers.ts`) ensures proper response to container lifecycle events:
#### Supported Signals:
- **SIGTERM**: Docker stop, Kubernetes pod termination
- **SIGINT**: Ctrl+C, manual interruption
- **SIGHUP**: Terminal hangup, service reload
- **Uncaught Exceptions**: Emergency shutdown on critical errors
- **Unhandled Rejections**: Graceful handling of promise failures
### 3. Enhanced Job Recovery
Building on the existing recovery system, new enhancements include:
#### Shutdown-Aware Processing:
- Jobs check for shutdown signals during execution
- Automatic state saving when shutdown is detected
- Proper job status management (interrupted vs failed)
#### Container Integration:
- Docker entrypoint script forwards signals correctly
- Startup recovery runs before main application
- Recovery timeouts prevent startup delays
## Usage
### Basic Operation
The graceful shutdown system is automatically initialized when the application starts. No manual configuration is required for basic operation.
### Testing
Test the graceful shutdown functionality:
```bash
# Run the integration test
bun run test-shutdown
# Clean up test data
bun run test-shutdown-cleanup
# Run unit tests
bun test src/lib/shutdown-manager.test.ts
bun test src/lib/signal-handlers.test.ts
```
### Manual Testing
1. **Start the application**:
```bash
bun run dev
# or in production
bun run start
```
2. **Start a mirroring job** through the web interface
3. **Send shutdown signal**:
```bash
# Send SIGTERM (recommended)
kill -TERM <process_id>
# Or use Ctrl+C for SIGINT
```
4. **Verify job state** is saved and can be resumed on restart
### Container Testing
Test with Docker:
```bash
# Build and run container
docker build -t gitea-mirror .
docker run -d --name test-shutdown gitea-mirror
# Start a job, then stop container
docker stop test-shutdown
# Restart and verify recovery
docker start test-shutdown
docker logs test-shutdown
```
## Implementation Details
### Shutdown Flow
1. **Signal Reception**: Signal handlers detect termination request
2. **Shutdown Initiation**: Shutdown manager begins graceful termination
3. **Job State Saving**: All active jobs save current progress to database
4. **Service Cleanup**: Registered callbacks stop background services
5. **Connection Cleanup**: Database connections and resources are released
6. **Process Termination**: Application exits with appropriate code
### Job State Management
During shutdown, active jobs are updated with:
- `inProgress: false` - Mark as not currently running
- `lastCheckpoint: <timestamp>` - Record shutdown time
- `message: "Job interrupted by application shutdown - will resume on restart"`
- Status remains as `"imported"` (not `"failed"`) to enable recovery
### Recovery Integration
The existing recovery system automatically detects and resumes interrupted jobs:
- Jobs with `inProgress: false` and incomplete status are candidates for recovery
- Recovery runs during application startup (before serving requests)
- Jobs resume from their last checkpoint with remaining items
## Configuration
### Environment Variables
```bash
# Optional: Adjust shutdown timeout (default: 30000ms)
SHUTDOWN_TIMEOUT=30000
# Optional: Adjust job save timeout (default: 10000ms)
JOB_SAVE_TIMEOUT=10000
```
### Docker Configuration
The Docker entrypoint script includes proper signal handling:
```dockerfile
# Signals are forwarded to the application process
# SIGTERM is handled gracefully with 30-second timeout
# Container stops cleanly without force-killing processes
```
### Kubernetes Configuration
For Kubernetes deployments, configure appropriate termination grace period:
```yaml
apiVersion: v1
kind: Pod
spec:
terminationGracePeriodSeconds: 45 # Allow time for graceful shutdown
containers:
- name: gitea-mirror
# ... other configuration
```
## Monitoring and Debugging
### Logs
The application provides detailed logging during shutdown:
```
🛑 Graceful shutdown initiated by signal: SIGTERM
📊 Shutdown status: 2 active jobs, 1 callbacks
📝 Step 1: Saving active job states...
Saving state for job abc-123...
✅ Saved state for job abc-123
🔧 Step 2: Executing shutdown callbacks...
✅ Shutdown callback 1 completed
💾 Step 3: Closing database connections...
✅ Graceful shutdown completed successfully
```
### Status Endpoints
Check shutdown manager status via API:
```bash
# Get current status (if application is running)
curl http://localhost:4321/api/health
```
### Troubleshooting
**Problem**: Jobs not resuming after restart
- **Check**: Startup recovery logs for errors
- **Verify**: Database contains interrupted jobs with correct status
- **Test**: Run `bun run startup-recovery` manually
**Problem**: Shutdown timeout reached
- **Check**: Job complexity and database performance
- **Adjust**: Increase `SHUTDOWN_TIMEOUT` environment variable
- **Monitor**: Database connection and disk I/O during shutdown
**Problem**: Container force-killed
- **Check**: Container orchestrator termination grace period
- **Adjust**: Increase grace period to allow shutdown completion
- **Monitor**: Application shutdown logs for timing issues
## Best Practices
### Development
- Always test graceful shutdown during development
- Use the provided test scripts to verify functionality
- Monitor logs for shutdown timing and job state persistence
### Production
- Set appropriate container termination grace periods
- Monitor shutdown logs for performance issues
- Use health checks to verify application readiness after restart
- Consider job complexity when planning maintenance windows
### Monitoring
- Track job recovery success rates
- Monitor shutdown duration metrics
- Alert on forced terminations or recovery failures
- Log analysis for shutdown pattern optimization
## Future Enhancements
Planned improvements for future versions:
1. **Configurable Timeouts**: Environment variable configuration for all timeouts
2. **Shutdown Metrics**: Prometheus metrics for shutdown performance
3. **Progressive Shutdown**: Graceful degradation of service capabilities
4. **Job Prioritization**: Priority-based job saving during shutdown
5. **Health Check Integration**: Readiness probes during shutdown process

486
Divers/gitea-mirror/docs/NIX_DEPLOYMENT.md Normal file
View File

@@ -0,0 +1,486 @@
# Nix Deployment Guide
This guide covers deploying Gitea Mirror using Nix flakes. The Nix deployment follows the same minimal configuration philosophy as `docker-compose.alt.yml` - secrets are auto-generated, and everything else can be configured via the web UI.
## Prerequisites
- Nix 2.4+ installed
- For NixOS module: NixOS 23.05+
### Enable Flakes (Recommended)
To enable flakes permanently and avoid typing flags, add to `/etc/nix/nix.conf` or `~/.config/nix/nix.conf`:
```
experimental-features = nix-command flakes
```
**Note:** If you don't enable flakes globally, add `--extra-experimental-features 'nix-command flakes'` to all nix commands shown below.
## Quick Start (Zero Configuration!)
### Run Immediately - No Setup Required
```bash
# Run directly from the flake (local)
nix run --extra-experimental-features 'nix-command flakes' .#gitea-mirror
# Or from GitHub (once published)
nix run --extra-experimental-features 'nix-command flakes' github:RayLabsHQ/gitea-mirror
# If you have flakes enabled globally, simply:
nix run .#gitea-mirror
```
That's it! On first run:
- Secrets (`BETTER_AUTH_SECRET` and `ENCRYPTION_SECRET`) are auto-generated
- Database is automatically created and initialized
- Startup recovery and repair scripts run automatically
- Access the web UI at http://localhost:4321
Everything else (GitHub credentials, Gitea settings, mirror options) is configured through the web interface after signup.
### Development Environment
```bash
# Enter development shell with all dependencies
nix develop --extra-experimental-features 'nix-command flakes'
# Or use direnv for automatic environment loading (handles flags automatically)
echo "use flake" > .envrc
direnv allow
```
### Build and Install
```bash
# Build the package
nix build --extra-experimental-features 'nix-command flakes'
# Run the built package
./result/bin/gitea-mirror
# Install to your profile
nix profile install --extra-experimental-features 'nix-command flakes' .#gitea-mirror
```
## What Happens on First Run?
Following the same pattern as the Docker deployment, the Nix package automatically:
1. **Creates data directory**: `~/.local/share/gitea-mirror` (or `$DATA_DIR`)
2. **Generates secrets** (stored securely in data directory):
- `BETTER_AUTH_SECRET` - Session authentication (32-char hex)
- `ENCRYPTION_SECRET` - Token encryption (48-char base64)
3. **Initializes database**: SQLite database with Drizzle migrations
4. **Runs startup scripts**:
- Environment configuration loader
- Crash recovery for interrupted jobs
- Repository status repair
5. **Starts the application** with graceful shutdown handling
## NixOS Module - Minimal Deployment
### Simplest Possible Configuration
Add to your NixOS configuration (`/etc/nixos/configuration.nix`):
```nix
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
gitea-mirror.url = "github:RayLabsHQ/gitea-mirror";
};
outputs = { nixpkgs, gitea-mirror, ... }: {
nixosConfigurations.your-hostname = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
gitea-mirror.nixosModules.default
{
# That's it! Just enable the service
services.gitea-mirror.enable = true;
}
];
};
};
}
```
Apply with:
```bash
sudo nixos-rebuild switch
```
Access at http://localhost:4321, sign up (first user is admin), and configure everything via the web UI.
### Production Configuration
For production with custom domain and firewall:
```nix
{
services.gitea-mirror = {
enable = true;
host = "0.0.0.0";
port = 4321;
betterAuthUrl = "https://mirror.example.com";
betterAuthTrustedOrigins = "https://mirror.example.com";
openFirewall = true;
};
# Optional: Use with nginx reverse proxy
services.nginx = {
enable = true;
virtualHosts."mirror.example.com" = {
locations."/" = {
proxyPass = "http://127.0.0.1:4321";
proxyWebsockets = true;
};
enableACME = true;
forceSSL = true;
};
};
}
```
### Advanced: Manual Secret Management
If you prefer to manage secrets manually (e.g., with sops-nix or agenix):
1. Create a secrets file:
```bash
# /var/lib/gitea-mirror/secrets.env
BETTER_AUTH_SECRET=your-32-character-minimum-secret-key-here
ENCRYPTION_SECRET=your-encryption-secret-here
```
2. Reference it in your configuration:
```nix
{
services.gitea-mirror = {
enable = true;
environmentFile = "/var/lib/gitea-mirror/secrets.env";
};
}
```
### Full Configuration Options
```nix
{
services.gitea-mirror = {
enable = true;
package = gitea-mirror.packages.x86_64-linux.default; # Override package
dataDir = "/var/lib/gitea-mirror";
user = "gitea-mirror";
group = "gitea-mirror";
host = "0.0.0.0";
port = 4321;
betterAuthUrl = "https://mirror.example.com";
betterAuthTrustedOrigins = "https://mirror.example.com";
# Concurrency controls (match docker-compose.alt.yml)
mirrorIssueConcurrency = 3; # Set to 1 for perfect chronological order
mirrorPullRequestConcurrency = 5; # Set to 1 for perfect chronological order
environmentFile = null; # Optional secrets file
openFirewall = true;
};
}
```
## Service Management (NixOS)
```bash
# Start the service
sudo systemctl start gitea-mirror
# Stop the service
sudo systemctl stop gitea-mirror
# Restart the service
sudo systemctl restart gitea-mirror
# Check status
sudo systemctl status gitea-mirror
# View logs
sudo journalctl -u gitea-mirror -f
# Health check
curl http://localhost:4321/api/health
```
## Environment Variables
All variables from `docker-compose.alt.yml` are supported:
```bash
# === AUTO-GENERATED (Don't set unless you want specific values) ===
BETTER_AUTH_SECRET # Auto-generated, stored in data dir
ENCRYPTION_SECRET # Auto-generated, stored in data dir
# === CORE SETTINGS (Have good defaults) ===
DATA_DIR="$HOME/.local/share/gitea-mirror"
DATABASE_URL="file:$DATA_DIR/gitea-mirror.db"
HOST="0.0.0.0"
PORT="4321"
NODE_ENV="production"
# === BETTER AUTH (Override for custom domains) ===
BETTER_AUTH_URL="http://localhost:4321"
BETTER_AUTH_TRUSTED_ORIGINS="http://localhost:4321"
PUBLIC_BETTER_AUTH_URL="http://localhost:4321"
# === CONCURRENCY CONTROLS ===
MIRROR_ISSUE_CONCURRENCY=3 # Default: 3 (set to 1 for perfect order)
MIRROR_PULL_REQUEST_CONCURRENCY=5 # Default: 5 (set to 1 for perfect order)
# === CONFIGURE VIA WEB UI (Not needed at startup) ===
# GitHub credentials, Gitea settings, mirror options, scheduling, etc.
# All configured after signup through the web interface
```
## Database Management
The Nix package includes a database management helper:
```bash
# Initialize database (done automatically on first run)
gitea-mirror-db init
# Check database health
gitea-mirror-db check
# Fix database issues
gitea-mirror-db fix
# Reset users
gitea-mirror-db reset-users
```
## Home Manager Integration
For single-user deployments:
```nix
{ config, pkgs, ... }:
let
gitea-mirror = (import (fetchTarball "https://github.com/RayLabsHQ/gitea-mirror/archive/main.tar.gz")).packages.${pkgs.system}.default;
in {
home.packages = [ gitea-mirror ];
# Optional: Run as user service
systemd.user.services.gitea-mirror = {
Unit = {
Description = "Gitea Mirror Service";
After = [ "network.target" ];
};
Service = {
Type = "simple";
ExecStart = "${gitea-mirror}/bin/gitea-mirror";
Restart = "always";
Environment = [
"DATA_DIR=%h/.local/share/gitea-mirror"
"HOST=127.0.0.1"
"PORT=4321"
];
};
Install = {
WantedBy = [ "default.target" ];
};
};
}
```
## Docker Image from Nix (Optional)
You can also use Nix to create a Docker image:
```nix
# Add to flake.nix packages section
dockerImage = pkgs.dockerTools.buildLayeredImage {
name = "gitea-mirror";
tag = "latest";
contents = [ self.packages.${system}.default pkgs.cacert pkgs.openssl ];
config = {
Cmd = [ "${self.packages.${system}.default}/bin/gitea-mirror" ];
ExposedPorts = { "4321/tcp" = {}; };
Env = [
"DATA_DIR=/data"
"DATABASE_URL=file:/data/gitea-mirror.db"
];
Volumes = { "/data" = {}; };
};
};
```
Build and load:
```bash
nix build --extra-experimental-features 'nix-command flakes' .#dockerImage
docker load < result
docker run -p 4321:4321 -v gitea-mirror-data:/data gitea-mirror:latest
```
## Comparison: Docker vs Nix
Both deployment methods follow the same philosophy:
| Feature | Docker Compose | Nix |
|---------|---------------|-----|
| **Configuration** | Minimal (only BETTER_AUTH_SECRET) | Zero config (auto-generated) |
| **Secret Generation** | Auto-generated & persisted | Auto-generated & persisted |
| **Database Init** | Automatic on first run | Automatic on first run |
| **Startup Scripts** | Runs recovery/repair/env-config | Runs recovery/repair/env-config |
| **Graceful Shutdown** | Signal handling in entrypoint | Signal handling in wrapper |
| **Health Check** | Docker healthcheck | systemd timer (optional) |
| **Updates** | `docker pull` | `nix flake update && nixos-rebuild` |
## Troubleshooting
### Check Auto-Generated Secrets
```bash
# For standalone
cat ~/.local/share/gitea-mirror/.better_auth_secret
cat ~/.local/share/gitea-mirror/.encryption_secret
# For NixOS service
sudo cat /var/lib/gitea-mirror/.better_auth_secret
sudo cat /var/lib/gitea-mirror/.encryption_secret
```
### Database Issues
```bash
# Check if database exists
ls -la ~/.local/share/gitea-mirror/gitea-mirror.db
# Reinitialize (deletes all data!)
rm ~/.local/share/gitea-mirror/gitea-mirror.db
gitea-mirror-db init
```
### Permission Issues (NixOS)
```bash
sudo chown -R gitea-mirror:gitea-mirror /var/lib/gitea-mirror
sudo chmod 700 /var/lib/gitea-mirror
```
### Port Already in Use
```bash
# Change port
export PORT=8080
gitea-mirror
# Or in NixOS config
services.gitea-mirror.port = 8080;
```
### View Startup Logs
```bash
# Standalone (verbose output on console)
gitea-mirror
# NixOS service
sudo journalctl -u gitea-mirror -f --since "5 minutes ago"
```
## Updating
### Standalone Installation
```bash
# Update flake lock
nix flake update --extra-experimental-features 'nix-command flakes'
# Rebuild
nix build --extra-experimental-features 'nix-command flakes'
# Or update profile
nix profile upgrade --extra-experimental-features 'nix-command flakes' gitea-mirror
```
### NixOS
```bash
# Update input
sudo nix flake lock --update-input gitea-mirror --extra-experimental-features 'nix-command flakes'
# Rebuild system
sudo nixos-rebuild switch --flake .#your-hostname
```
## Migration from Docker
To migrate from Docker to Nix while keeping your data:
1. **Stop Docker container:**
```bash
docker-compose -f docker-compose.alt.yml down
```
2. **Copy data directory:**
```bash
# For standalone
cp -r ./data ~/.local/share/gitea-mirror
# For NixOS
sudo cp -r ./data /var/lib/gitea-mirror
sudo chown -R gitea-mirror:gitea-mirror /var/lib/gitea-mirror
```
3. **Copy secrets (if you want to keep them):**
```bash
# Extract from Docker volume
docker run --rm -v gitea-mirror_data:/data alpine \
cat /data/.better_auth_secret > better_auth_secret
docker run --rm -v gitea-mirror_data:/data alpine \
cat /data/.encryption_secret > encryption_secret
# Copy to new location
cp better_auth_secret ~/.local/share/gitea-mirror/.better_auth_secret
cp encryption_secret ~/.local/share/gitea-mirror/.encryption_secret
chmod 600 ~/.local/share/gitea-mirror/.*_secret
```
4. **Start Nix version:**
```bash
gitea-mirror
```
## CI/CD Integration
Example GitHub Actions workflow (see `.github/workflows/nix-build.yml`):
```yaml
name: Nix Build
on: [push, pull_request]
permissions:
contents: read
jobs:
build:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: DeterminateSystems/nix-installer-action@main
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: nix flake check
- run: nix build --print-build-logs
```
This uses:
- **Determinate Nix Installer** - Fast, reliable Nix installation with flakes enabled by default
- **Magic Nix Cache** - Free caching using GitHub Actions cache (no account needed)
## Resources
- [Nix Manual](https://nixos.org/manual/nix/stable/)
- [NixOS Options Search](https://search.nixos.org/options)
- [Nix Pills Tutorial](https://nixos.org/guides/nix-pills/)
- [Project Documentation](../README.md)
- [Docker Deployment](../docker-compose.alt.yml) - Equivalent minimal config

322
Divers/gitea-mirror/docs/NIX_DISTRIBUTION.md Normal file
View File

@@ -0,0 +1,322 @@
# Nix Package Distribution Guide
This guide explains how Gitea Mirror is distributed via Nix and how users can consume it.
## Distribution Methods
### Method 1: Direct GitHub Usage (Zero Infrastructure)
**No CI, releases, or setup needed!** Users can consume directly from GitHub:
```bash
# Latest from main branch
nix run --extra-experimental-features 'nix-command flakes' github:RayLabsHQ/gitea-mirror
# Pin to specific commit
nix run github:RayLabsHQ/gitea-mirror/abc123def
# Pin to git tag
nix run github:RayLabsHQ/gitea-mirror/v3.8.11
```
**How it works:**
1. Nix fetches the repository from GitHub
2. Nix reads `flake.nix` and `flake.lock`
3. Nix builds the package locally on the user's machine
4. Package is cached in `/nix/store` for reuse
**Pros:**
- Zero infrastructure needed
- Works immediately after pushing code
- Users always get reproducible builds
**Cons:**
- Users must build from source (slower first time)
- Requires build dependencies (Bun, etc.)
---
### Method 2: CI Build Caching
The GitHub Actions workflow uses **Magic Nix Cache** (by Determinate Systems) to cache builds:
- **Zero configuration required** - no accounts or tokens needed
- **Automatic** - CI workflow handles everything
- **Uses GitHub Actions cache** - fast, reliable, free
#### How It Works:
1. GitHub Actions builds the package on each push/PR
2. Build artifacts are cached in GitHub Actions cache
3. Subsequent builds reuse cached dependencies (faster CI)
Note: This caches CI builds. Users still build locally, but the flake.lock ensures reproducibility.
---
### Method 3: nixpkgs Submission (Official Distribution)
Submit to the official Nix package repository for maximum visibility.
#### Process:
1. **Prepare package** (already done with `flake.nix`)
2. **Test thoroughly**
3. **Submit PR to nixpkgs:** https://github.com/NixOS/nixpkgs
#### User Experience:
```bash
# After acceptance into nixpkgs
nix run nixpkgs#gitea-mirror
# NixOS configuration
environment.systemPackages = [ pkgs.gitea-mirror ];
```
**Pros:**
- Maximum discoverability (official repo)
- Trusted by Nix community
- Included in NixOS search
- Binary caching by cache.nixos.org
**Cons:**
- Submission/review process
- Must follow nixpkgs guidelines
- Updates require PRs
---
## Current Distribution Strategy
### Phase 1: Direct GitHub (Immediate) ✅
Already working! Users can:
```bash
nix run github:RayLabsHQ/gitea-mirror
```
### Phase 2: CI Build Validation ✅
GitHub Actions workflow validates builds on every push/PR:
- Uses Magic Nix Cache for fast CI builds
- Tests on both Linux and macOS
- No setup required - works automatically
### Phase 3: Version Releases (Optional)
Tag releases for version pinning:
```bash
git tag v3.8.11
git push origin v3.8.11
# Users can then pin:
nix run github:RayLabsHQ/gitea-mirror/v3.8.11
```
### Phase 4: nixpkgs Submission (Long Term)
Once package is stable and well-tested, submit to nixpkgs.
---
## User Documentation
### For Users: How to Install
Add this to your `docs/NIX_DEPLOYMENT.md`:
#### Option 1: Direct Install (No Configuration)
```bash
# Run immediately
nix run --extra-experimental-features 'nix-command flakes' github:RayLabsHQ/gitea-mirror
# Install to profile
nix profile install --extra-experimental-features 'nix-command flakes' github:RayLabsHQ/gitea-mirror
```
#### Option 2: Pin to Specific Version
```bash
# Pin to git tag
nix run github:RayLabsHQ/gitea-mirror/v3.8.11
# Pin to commit
nix run github:RayLabsHQ/gitea-mirror/abc123def
# Lock in flake.nix
inputs.gitea-mirror.url = "github:RayLabsHQ/gitea-mirror/v3.8.11";
```
#### Option 3: NixOS Configuration
```nix
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
gitea-mirror.url = "github:RayLabsHQ/gitea-mirror";
# Or pin to version:
# gitea-mirror.url = "github:RayLabsHQ/gitea-mirror/v3.8.11";
};
outputs = { nixpkgs, gitea-mirror, ... }: {
nixosConfigurations.your-host = nixpkgs.lib.nixosSystem {
modules = [
gitea-mirror.nixosModules.default
{
services.gitea-mirror = {
enable = true;
betterAuthUrl = "https://mirror.example.com";
openFirewall = true;
};
}
];
};
};
}
```
---
## Maintaining the Distribution
### Releasing New Versions
```bash
# 1. Update version in package.json
vim package.json # Update version field
# 2. Update flake.nix version (line 17)
vim flake.nix # Update version = "X.Y.Z";
# 3. Commit changes
git add package.json flake.nix
git commit -m "chore: bump version to vX.Y.Z"
# 4. Create git tag
git tag vX.Y.Z
git push origin main
git push origin vX.Y.Z
# 5. GitHub Actions builds and caches automatically
```
Users can then pin to the new version:
```bash
nix run github:RayLabsHQ/gitea-mirror/vX.Y.Z
```
### Updating Flake Lock
The `flake.lock` file pins all dependencies. Update it periodically:
```bash
# Update all inputs
nix flake update
# Update specific input
nix flake lock --update-input nixpkgs
# Test after update
nix build
nix flake check
# Commit the updated lock file
git add flake.lock
git commit -m "chore: update flake dependencies"
git push
```
---
## Troubleshooting Distribution Issues
### Users Report Build Failures
1. **Check GitHub Actions:** Ensure CI is passing
2. **Test locally:** `nix flake check`
3. **Check flake.lock:** May need update if dependencies changed
### CI Cache Not Working
1. **Check workflow logs:** Review GitHub Actions for errors
2. **Clear cache:** GitHub Actions → Caches → Delete relevant cache
3. **Verify flake.lock:** May need `nix flake update` if dependencies changed
### Version Pinning Not Working
```bash
# Verify tag exists
git tag -l
# Ensure tag is pushed
git ls-remote --tags origin
# Test specific tag
nix run github:RayLabsHQ/gitea-mirror/v3.8.11
```
---
## Advanced: Custom Binary Cache
If you prefer self-hosting instead of Cachix:
### Option 1: S3-Compatible Storage
```nix
# Generate signing key
nix-store --generate-binary-cache-key cache.example.com cache-priv-key.pem cache-pub-key.pem
# Push to S3
nix copy --to s3://my-nix-cache?region=us-east-1 $(nix-build)
```
Users configure:
```nix
substituters = https://my-bucket.s3.amazonaws.com/nix-cache
trusted-public-keys = cache.example.com:BASE64_PUBLIC_KEY
```
### Option 2: Self-Hosted Nix Store
Run `nix-serve` on your server:
```bash
# On server
nix-serve -p 8080
# Behind nginx/caddy
proxy_pass http://localhost:8080;
```
Users configure:
```nix
substituters = https://cache.example.com
trusted-public-keys = YOUR_KEY
```
---
## Comparison: Distribution Methods
| Method | Setup Time | User Speed | Cost | Discoverability |
|--------|-----------|------------|------|-----------------|
| Direct GitHub | 0 min | Slow (build) | Free | Low |
| nixpkgs | Hours/days | Fast (binary) | Free | High |
| Self-hosted cache | 30+ min | Fast (binary) | Server cost | Low |
**Current approach:** Direct GitHub consumption with CI validation using Magic Nix Cache. Users build locally (reproducible via flake.lock). Consider **nixpkgs** submission for maximum reach once the package is mature.
---
## Resources
- [Nix Flakes Documentation](https://nixos.wiki/wiki/Flakes)
- [Magic Nix Cache](https://github.com/DeterminateSystems/magic-nix-cache-action)
- [nixpkgs Contributing Guide](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md)
- [Nix Binary Cache Setup](https://nixos.org/manual/nix/stable/package-management/binary-cache-substituter.html)

39
Divers/gitea-mirror/docs/README.md Normal file
View File

@@ -0,0 +1,39 @@
# Gitea Mirror Documentation
This folder contains engineering and operations references for the open-source Gitea Mirror project. Each guide focuses on the parts of the system that still require bespoke explanation beyond the in-app help and the main `README.md`.
## Available Guides
### Core workflow
- **[DEVELOPMENT_WORKFLOW.md](./DEVELOPMENT_WORKFLOW.md)** – Set up a local environment, run scripts, and understand the repo layout (app + marketing site).
- **[ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md)** – Complete reference for every configuration flag supported by the app and Docker images.
### Reliability & recovery
- **[GRACEFUL_SHUTDOWN.md](./GRACEFUL_SHUTDOWN.md)** – How signal handling, shutdown coordination, and job persistence work in v3.
- **[RECOVERY_IMPROVEMENTS.md](./RECOVERY_IMPROVEMENTS.md)** – Deep dive into the startup recovery workflow and supporting scripts.
### Authentication
- **[SSO-OIDC-SETUP.md](./SSO-OIDC-SETUP.md)** – Configure OIDC/SSO providers through the admin UI.
- **[SSO_TESTING.md](./SSO_TESTING.md)** – Recipes for local and staging SSO testing (Google, Keycloak, mock providers).
If you are looking for customer-facing playbooks, see the MDX use cases under `www/src/pages/use-cases/`.
## Quick start for local development
```bash
git clone https://github.com/RayLabsHQ/gitea-mirror.git
cd gitea-mirror
bun run setup # installs deps and seeds the SQLite DB
bun run dev # starts the Astro/Bun app on http://localhost:4321
```
The first user you create locally becomes the administrator. All other configuration—GitHub owners, Gitea targets, scheduling, cleanup—is done through the **Configuration** screen in the UI.
## Contributing & support
- 🎯 Contribution guide: [../CONTRIBUTING.md](../CONTRIBUTING.md)
- 📘 Code of conduct: [../CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md)
- 🐞 Issues & feature requests: <https://github.com/RayLabsHQ/gitea-mirror/issues>
- 💬 Discussions: <https://github.com/RayLabsHQ/gitea-mirror/discussions>
Security disclosures should follow the process in [../SECURITY.md](../SECURITY.md).

170
Divers/gitea-mirror/docs/RECOVERY_IMPROVEMENTS.md Normal file
View File

@@ -0,0 +1,170 @@
# Job Recovery and Resume Process Improvements
This document outlines the comprehensive improvements made to the job recovery and resume process to make it more robust to application restarts, container restarts, and application crashes.
## Problems Addressed
The original recovery system had several critical issues:
1. **Middleware-based initialization**: Recovery only ran when the first request came in
2. **Database connection issues**: No validation of database connectivity before recovery attempts
3. **Limited error handling**: Insufficient error handling for various failure scenarios
4. **No startup recovery**: No mechanism to handle recovery before serving requests
5. **Incomplete job state management**: Jobs could remain in inconsistent states
6. **No retry mechanisms**: Single-attempt recovery with no fallback strategies
## Improvements Implemented
### 1. Enhanced Recovery System (`src/lib/recovery.ts`)
#### New Features:
- **Database connection validation** before attempting recovery
- **Stale job cleanup** for jobs older than 24 hours
- **Retry mechanisms** with configurable attempts and delays
- **Individual job error handling** to prevent one failed job from stopping recovery
- **Recovery state tracking** to prevent concurrent recovery attempts
- **Enhanced logging** with detailed job information
#### Key Functions:
- `initializeRecovery()` - Main recovery function with enhanced error handling
- `validateDatabaseConnection()` - Ensures database is accessible
- `cleanupStaleJobs()` - Removes jobs that are too old to recover
- `getRecoveryStatus()` - Returns current recovery system status
- `forceRecovery()` - Bypasses recent attempt checks
- `hasJobsNeedingRecovery()` - Checks if recovery is needed
### 2. Startup Recovery Script (`scripts/startup-recovery.ts`)
A dedicated script that runs recovery before the application starts serving requests:
#### Features:
- **Timeout protection** (default: 30 seconds)
- **Force recovery option** to bypass recent attempt checks
- **Graceful signal handling** (SIGINT, SIGTERM)
- **Detailed logging** with progress indicators
- **Exit codes** for different scenarios (success, warnings, errors)
#### Usage:
```bash
bun scripts/startup-recovery.ts [--force] [--timeout=30000]
```
### 3. Improved Middleware (`src/middleware.ts`)
The middleware now serves as a fallback recovery mechanism:
#### Changes:
- **Checks if recovery is needed** before attempting
- **Shorter timeout** (15 seconds) for request-time recovery
- **Better error handling** with status logging
- **Prevents multiple attempts** with proper state tracking
### 4. Enhanced Database Queries (`src/lib/helpers.ts`)
#### Improvements:
- **Proper Drizzle ORM syntax** for all database queries
- **Enhanced interrupted job detection** with multiple criteria:
- Jobs with no recent checkpoint (10+ minutes)
- Jobs running too long (2+ hours)
- **Detailed logging** of found interrupted jobs
- **Better error handling** for database operations
### 5. Docker Integration (`docker-entrypoint.sh`)
#### Changes:
- **Automatic startup recovery** runs before application start
- **Exit code handling** with appropriate logging
- **Fallback mechanisms** if recovery script is not found
- **Non-blocking execution** - application starts even if recovery fails
### 6. Health Check Integration (`src/pages/api/health.ts`)
#### New Features:
- **Recovery system status** in health endpoint
- **Job recovery metrics** (jobs needing recovery, recovery in progress)
- **Overall health status** considers recovery state
- **Detailed recovery information** for monitoring
### 7. Testing Infrastructure (`scripts/test-recovery.ts`)
A comprehensive test script to verify recovery functionality:
#### Features:
- **Creates test interrupted jobs** with realistic scenarios
- **Verifies recovery detection** and execution
- **Checks final job states** after recovery
- **Cleanup functionality** for test data
- **Comprehensive logging** of test progress
## Configuration Options
### Recovery System Options:
- `maxRetries`: Number of recovery attempts (default: 3)
- `retryDelay`: Delay between attempts in ms (default: 5000)
- `skipIfRecentAttempt`: Skip if recent attempt made (default: true)
### Startup Recovery Options:
- `--force`: Force recovery even if recent attempt was made
- `--timeout`: Maximum time to wait for recovery (default: 30000ms)
## Usage Examples
### Manual Recovery:
```bash
# Run startup recovery
bun run startup-recovery
# Force recovery
bun run startup-recovery-force
# Test recovery system
bun run test-recovery
# Clean up test data
bun run test-recovery-cleanup
```
### Programmatic Usage:
```typescript
import { initializeRecovery, hasJobsNeedingRecovery } from '@/lib/recovery';
// Check if recovery is needed
const needsRecovery = await hasJobsNeedingRecovery();
// Run recovery with custom options
const success = await initializeRecovery({
maxRetries: 5,
retryDelay: 3000,
skipIfRecentAttempt: false
});
```
## Monitoring and Observability
### Health Check Endpoint:
- **URL**: `/api/health`
- **Recovery Status**: Included in response
- **Monitoring**: Can be used with external monitoring systems
### Log Messages:
- **Startup**: Clear indicators of recovery attempts and results
- **Progress**: Detailed logging of recovery steps
- **Errors**: Comprehensive error information for debugging
## Benefits
1. **Reliability**: Jobs are automatically recovered after application restarts
2. **Resilience**: Multiple retry mechanisms and fallback strategies
3. **Observability**: Comprehensive logging and health check integration
4. **Performance**: Efficient detection and processing of interrupted jobs
5. **Maintainability**: Clear separation of concerns and modular design
6. **Testing**: Built-in testing infrastructure for verification
## Migration Notes
- **Backward Compatible**: All existing functionality is preserved
- **Automatic**: Recovery runs automatically on startup
- **Configurable**: All timeouts and retry counts can be adjusted
- **Monitoring**: Health checks now include recovery status
This comprehensive improvement ensures that the gitea-mirror application can reliably handle job recovery in all deployment scenarios, from development to production container environments.

226
Divers/gitea-mirror/docs/SSO-OIDC-SETUP.md Normal file
View File

@@ -0,0 +1,226 @@
# SSO and OIDC Setup Guide
This guide explains how to configure Single Sign-On (SSO) and OpenID Connect (OIDC) provider functionality in Gitea Mirror.
## Overview
Gitea Mirror supports three authentication methods:
1. **Email & Password** - Traditional authentication (always enabled)
2. **SSO (Single Sign-On)** - Allow users to authenticate using external OIDC providers
3. **OIDC Provider** - Allow other applications to authenticate users through Gitea Mirror
## Configuration
All SSO and OIDC settings are managed through the web UI in the Configuration page under the "Authentication" tab.
## Setting up SSO (Single Sign-On)
SSO allows your users to sign in using external identity providers like Google, Okta, Azure AD, etc.
### Adding an SSO Provider
1. Navigate to Configuration → Authentication → SSO Providers
2. Click "Add Provider"
3. Fill in the provider details:
#### Required Fields
- **Issuer URL**: The OIDC issuer URL (e.g., `https://accounts.google.com`)
- **Domain**: The email domain for this provider (e.g., `example.com`)
- **Provider ID**: A unique identifier for this provider (e.g., `google-sso`)
- **Client ID**: The OAuth client ID from your provider
- **Client Secret**: The OAuth client secret from your provider
#### Auto-Discovery
If your provider supports OIDC discovery, you can:
1. Enter the Issuer URL
2. Click "Discover"
3. The system will automatically fetch the authorization and token endpoints
#### Manual Configuration
For providers without discovery support, manually enter:
- **Authorization Endpoint**: The OAuth authorization URL
- **Token Endpoint**: The OAuth token exchange URL
- **JWKS Endpoint**: The JSON Web Key Set URL (optional)
- **UserInfo Endpoint**: The user information endpoint (optional)
### Redirect URL
When configuring your SSO provider, use this redirect URL:
```
https://your-domain.com/api/auth/sso/callback/{provider-id}
```
Replace `{provider-id}` with your chosen Provider ID.
### Example: Google SSO Setup
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new OAuth 2.0 Client ID
3. Add authorized redirect URI: `https://your-domain.com/api/auth/sso/callback/google-sso`
4. In Gitea Mirror:
- Issuer URL: `https://accounts.google.com`
- Domain: `your-company.com`
- Provider ID: `google-sso`
- Client ID: [Your Google Client ID]
- Client Secret: [Your Google Client Secret]
- Click "Discover" to auto-fill endpoints
### Example: Okta SSO Setup
1. In Okta Admin Console, create a new OIDC Web Application
2. Set redirect URI: `https://your-domain.com/api/auth/sso/callback/okta-sso`
3. In Gitea Mirror:
- Issuer URL: `https://your-okta-domain.okta.com`
- Domain: `your-company.com`
- Provider ID: `okta-sso`
- Client ID: [Your Okta Client ID]
- Client Secret: [Your Okta Client Secret]
- Click "Discover" to auto-fill endpoints
### Example: Authentik SSO Setup
Working Authentik deployments (see [#134](https://github.com/RayLabsHQ/gitea-mirror/issues/134)) follow these steps:
1. In Authentik, create a new **Application** and OIDC **Provider** (implicit flow works well for testing).
2. Start creating an SSO provider inside Gitea Mirror so you can copy the redirect URL shown (`https://your-domain.com/api/auth/sso/callback/authentik` if you pick `authentik` as your Provider ID).
3. Paste that redirect URL into the Authentik Provider configuration and finish creating the provider.
4. Copy the Authentik issuer URL, client ID, and client secret.
5. Back in Gitea Mirror:
- Issuer URL: the exact value from Authentik (keep any trailing slash Authentik shows).
- Provider ID: match the one you used in step 2.
- Click **Discover** so Gitea Mirror stores the authorization, token, and JWKS endpoints (Authentik publishes them via discovery).
- Domain: enter the email domain you expect to match (e.g. `example.com`).
6. Save the provider and test the login flow.
Notes:
- Make sure `BETTER_AUTH_URL` and (if you serve the UI from multiple origins) `BETTER_AUTH_TRUSTED_ORIGINS` point at the public URL users reach. A mismatch can surface as 500 errors after redirect.
- Authentik must report the user’s email as verified (default behavior) so Gitea Mirror can auto-link accounts.
- If you created an Authentik provider before v3.8.10 you should delete it and re-add it after upgrading; older versions saved incomplete endpoint data which leads to the `url.startsWith` error explained in the Troubleshooting section.
## Setting up OIDC Provider
The OIDC Provider feature allows other applications to use Gitea Mirror as their authentication provider.
### Creating OAuth Applications
1. Navigate to Configuration → Authentication → OAuth Applications
2. Click "Create Application"
3. Fill in the application details:
- **Application Name**: Display name for the application
- **Application Type**: Web, Mobile, or Desktop
- **Redirect URLs**: One or more redirect URLs (one per line)
4. After creation, you'll receive:
- **Client ID**: Share this with the application
- **Client Secret**: Keep this secure and share only once
### OIDC Endpoints
Applications can use these standard OIDC endpoints:
- **Discovery**: `https://your-domain.com/.well-known/openid-configuration`
- **Authorization**: `https://your-domain.com/api/auth/oauth2/authorize`
- **Token**: `https://your-domain.com/api/auth/oauth2/token`
- **UserInfo**: `https://your-domain.com/api/auth/oauth2/userinfo`
- **JWKS**: `https://your-domain.com/api/auth/jwks`
### Supported Scopes
- `openid` - Required, provides user ID
- `profile` - User's name, username, and profile picture
- `email` - User's email address and verification status
### Example: Configuring Another Application
For an application to use Gitea Mirror as its OIDC provider:
```javascript
// Example configuration for another app
const oidcConfig = {
issuer: 'https://gitea-mirror.example.com',
clientId: 'client_xxxxxxxxxxxxx',
clientSecret: 'secret_xxxxxxxxxxxxx',
redirectUri: 'https://myapp.com/auth/callback',
scope: 'openid profile email'
};
```
## User Experience
### Logging In with SSO
When SSO is configured:
1. Users see tabs for "Email" and "SSO" on the login page
2. In the SSO tab, they can:
- Click a specific provider button (if configured)
- Enter their work email to be redirected to the appropriate provider
### OAuth Consent Flow
When an application requests authentication:
1. Users are redirected to Gitea Mirror
2. If not logged in, they authenticate first
3. They see a consent screen showing:
- Application name
- Requested permissions
- Option to approve or deny
## Security Considerations
1. **Client Secrets**: Store OAuth client secrets securely
2. **Redirect URLs**: Only add trusted redirect URLs for applications
3. **Scopes**: Applications only receive the data for approved scopes
4. **Token Security**: Access tokens expire and can be revoked
## Troubleshooting
### SSO Login Issues
1. **"Invalid origin" error**: Check that your Gitea Mirror URL matches the configured redirect URI
2. **"Provider not found" error**: Ensure the provider is properly configured and enabled
3. **Redirect loop**: Verify the redirect URI in both Gitea Mirror and the SSO provider match exactly
4. **`TypeError: undefined is not an object (evaluating 'url.startsWith')`**: This indicates the stored provider configuration is missing OIDC endpoints. Delete the provider from Gitea Mirror and re-register it using the **Discover** button so authorization/token URLs are saved (see [#73](https://github.com/RayLabsHQ/gitea-mirror/issues/73) and [#122](https://github.com/RayLabsHQ/gitea-mirror/issues/122) for examples).
### OIDC Provider Issues
1. **Application not found**: Ensure the client ID is correct
2. **Invalid redirect URI**: The redirect URI must match exactly what's configured
3. **Consent not working**: Check browser cookies are enabled
## Managing Access
### Revoking SSO Access
Currently, SSO sessions are managed through the identity provider. To revoke access:
1. Log out of Gitea Mirror
2. Revoke access in your identity provider's settings
### Disabling OAuth Applications
To disable an application:
1. Go to Configuration → Authentication → OAuth Applications
2. Find the application
3. Click the delete button
This immediately prevents the application from authenticating new users.
## Best Practices
1. **Use HTTPS**: Always use HTTPS in production for security
2. **Regular Audits**: Periodically review configured SSO providers and OAuth applications
3. **Principle of Least Privilege**: Only grant necessary scopes to applications
4. **Monitor Usage**: Keep track of which applications are accessing your OIDC provider
5. **Secure Storage**: Store client secrets in a secure location, never in code
## Migration Notes
If migrating from the previous JWT-based authentication:
- Existing users remain unaffected
- Users can continue using email/password authentication
- SSO can be added as an additional authentication method

193
Divers/gitea-mirror/docs/SSO_TESTING.md Normal file
View File

@@ -0,0 +1,193 @@
# Local SSO Testing Guide
This guide explains how to test SSO authentication locally with Gitea Mirror.
## Option 1: Using Google OAuth (Recommended for Quick Testing)
### Setup Steps:
1. **Create a Google OAuth Application**
- Go to [Google Cloud Console](https://console.cloud.google.com/)
- Create a new project or select existing
- Enable Google+ API
- Go to "Credentials" → "Create Credentials" → "OAuth client ID"
- Choose "Web application"
- Add authorized redirect URIs:
- `http://localhost:3000/api/auth/sso/callback/google-sso`
- `http://localhost:9876/api/auth/sso/callback/google-sso`
2. **Configure in Gitea Mirror**
- Go to Configuration → Authentication tab
- Click "Add Provider"
- Select "OIDC / OAuth2"
- Fill in:
- Provider ID: `google-sso`
- Email Domain: `gmail.com` (or your domain)
- Issuer URL: `https://accounts.google.com`
- Click "Discover" to auto-fill endpoints
- Client ID: (from Google Console)
- Client Secret: (from Google Console)
- Save the provider
## Option 2: Using Keycloak (Local Identity Provider)
### Setup with Docker:
```bash
# Run Keycloak
docker run -d --name keycloak \
-p 8080:8080 \
-e KEYCLOAK_ADMIN=admin \
-e KEYCLOAK_ADMIN_PASSWORD=admin \
quay.io/keycloak/keycloak:latest start-dev
# Access at http://localhost:8080
# Login with admin/admin
```
### Configure Keycloak:
1. Create a new realm (e.g., "gitea-mirror")
2. Create a client:
- Client ID: `gitea-mirror`
- Client Protocol: `openid-connect`
- Access Type: `confidential`
- Valid Redirect URIs: `http://localhost:*/api/auth/sso/callback/keycloak`
3. Get credentials from the "Credentials" tab
4. Create test users in "Users" section
### Configure in Gitea Mirror:
- Provider ID: `keycloak`
- Email Domain: `example.com`
- Issuer URL: `http://localhost:8080/realms/gitea-mirror`
- Client ID: `gitea-mirror`
- Client Secret: (from Keycloak)
- Click "Discover" to auto-fill endpoints
## Option 3: Using Mock SSO Provider (Development)
For testing without external dependencies, you can use a mock OIDC provider.
### Using oidc-provider-example:
```bash
# Clone and run mock provider
git clone https://github.com/panva/node-oidc-provider-example.git
cd node-oidc-provider-example
npm install
npm start
# Runs on http://localhost:3001
```
### Configure in Gitea Mirror:
- Provider ID: `mock-provider`
- Email Domain: `test.com`
- Issuer URL: `http://localhost:3001`
- Client ID: `foo`
- Client Secret: `bar`
- Authorization Endpoint: `http://localhost:3001/auth`
- Token Endpoint: `http://localhost:3001/token`
## Testing the SSO Flow
1. **Logout** from Gitea Mirror if logged in
2. Go to `/login`
3. Click on the **SSO** tab
4. Either:
- Click the provider button (e.g., "Sign in with gmail.com")
- Or enter your email and click "Continue with SSO"
5. You'll be redirected to the identity provider
6. Complete authentication
7. You'll be redirected back and logged in
## Troubleshooting
### Common Issues:
1. **"Invalid origin" error**
- Check that `trustedOrigins` in `/src/lib/auth.ts` includes your dev URL
- Restart the dev server after changes
2. **Provider not showing in login**
- Check browser console for errors
- Verify provider was saved successfully
- Check `/api/sso/providers` returns your providers
3. **Redirect URI mismatch**
- Ensure the redirect URI in your OAuth app matches exactly:
`http://localhost:PORT/api/auth/sso/callback/PROVIDER_ID`
4. **CORS errors**
- Add your identity provider domain to CORS allowed origins if needed
### Debug Mode:
Enable debug logging by setting environment variable:
```bash
DEBUG=better-auth:* bun run dev
```
## Testing Different Scenarios
### 1. New User Registration
- Use an email not in the system
- SSO should create a new user automatically
### 2. Existing User Login
- Create a user with email/password first
- Login with SSO using same email
- Should link to existing account
### 3. Domain-based Routing
- Configure multiple providers with different domains
- Test that entering email routes to correct provider
### 4. Organization Provisioning
- Set organizationId on provider
- Test that users are added to correct organization
## Security Testing
1. **Token Expiration**
- Wait for session to expire
- Test refresh flow
2. **Invalid State**
- Modify state parameter in callback
- Should reject authentication
3. **PKCE Flow**
- Enable/disable PKCE
- Verify code challenge works
## Using with Better Auth CLI
Better Auth provides CLI tools for testing:
```bash
# List registered providers
bun run auth:providers list
# Test provider configuration
bun run auth:providers test google-sso
```
## Environment Variables
For production-like testing:
```env
BETTER_AUTH_URL=http://localhost:3000
BETTER_AUTH_SECRET=your-secret-key
```
## Next Steps
After successful SSO setup:
1. Test user attribute mapping
2. Configure role-based access
3. Set up SAML if needed
4. Test with your organization's actual IdP