crawlapi/
├── crates/
│   ├── api/          # Servidor REST (Axum) + seed script
│   ├── worker/       # Worker distribuido con Redis queue
│   ├── shared/       # Tipos y config compartidos
│   └── db/           # Capa de base de datos + migraciones
├── playwright/       # Script Node.js con Browser Pool + Stealth + CAPTCHA
├── frontend/         # Landing + Playground + Billing + Dashboard + Docs
├── e2e/              # Tests E2E con Playwright
├── load-tests/       # k6 load testing scripts
├── k8s/              # Kubernetes manifests + cert-manager
├── legal/            # Terms, Privacy, DPA
├── .github/          # GitHub Actions workflows
├── docker-compose.yml
└── prometheus.yml

Endpoints

Crawl/Scrape/AI

Endpoint	Descripción
`POST /api/crawl`	Full JS-rendered page crawl
`POST /api/content`	Raw HTML
`POST /api/screenshot`	PNG screenshot (subido a S3)
`POST /api/pdf`	PDF export (subido a S3)
`POST /api/markdown`	Markdown extraction
`POST /api/snapshot`	HTML + screenshot combined
`POST /api/scrape`	CSS selector extraction
`POST /api/json`	Structured JSON
`POST /api/links`	Extract all links
`POST /api/extract`	AI-powered extraction con OpenAI

Auth

Endpoint	Descripción
`POST /api/auth/register`	Crear cuenta
`POST /api/auth/login`	Login (devuelve JWT)
`GET /api/auth/google`	URL de OAuth Google
`GET /api/auth/google/callback`	Callback de OAuth (real con token exchange)
`POST /api/auth/api-keys`	Crear API key (requiere JWT)
`GET /api/auth/api-keys`	Listar API keys (requiere JWT)
`DELETE /api/auth/api-keys/{id}`	Eliminar API key (requiere JWT)

Billing

Endpoint	Descripción
`POST /api/stripe/checkout`	Crear checkout session funcional
`POST /api/stripe/webhook`	Webhook de Stripe (procesa eventos reales)

Teams

Endpoint	Descripción
`POST /api/teams`	Crear equipo
`GET /api/teams/{slug}`	Ver equipo y miembros
`POST /api/teams/{slug}/members`	Agregar miembro

Observabilidad

Endpoint	Descripción
`GET /metrics`	Métricas Prometheus
`GET /ws/logs`	WebSocket live logs

Quick Start (Docker)

# 1. Iniciar toda la stack
cd crawlapi
docker-compose up --build

# 2. Crear seed data (en otra terminal)
export DATABASE_URL="postgres://crawlapi:crawlapi@localhost:5432/crawlapi"
source "$HOME/.cargo/env"
cargo run -p api --bin seed

# 3. Servicios disponibles:
# API:        http://localhost:3000
# Frontend:   http://localhost
# MinIO:      http://localhost:9001 (minioadmin/minioadmin)
# Prometheus: http://localhost:9090
# Grafana:    http://localhost:3001 (admin/admin)

CI/CD (GitHub Actions)

# En cada push a main:
# 1. cargo fmt --check
# 2. cargo clippy -- -D warnings
# 3. cargo test --workspace
# 4. cargo audit
# 5. Docker build + push a registry
# 6. Deploy a staging
# 7. Smoke tests
# 8. Deploy a production (solo en tags v*)

Workflows:

.github/workflows/ci.yml — Test, build, push images
.github/workflows/deploy.yml — Deploy a staging y production

Kubernetes

# Instalar cert-manager primero
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# Deploy todo
cd k8s
kubectl apply -f namespace.yaml
kubectl apply -f cert-manager.yaml
kubectl apply -f secrets.yaml
kubectl apply -f postgres.yaml
kubectl apply -f redis.yaml
kubectl apply -f minio.yaml
kubectl apply -f api.yaml
kubectl apply -f worker.yaml
kubectl apply -f frontend.yaml

# Workers auto-scale con HPA (3-20 réplicas)
kubectl get hpa -n crawlapi

Load Testing (k6)

cd load-tests

# Smoke test (1 VU, 1 min)
k6 run smoke.js

# Load test (ramp up a 20 VUs, 14 min)
k6 run load.js

# Stress test (hasta 200 VUs)
k6 run stress.js

# Screenshot test (5 VUs concurrentes)
k6 run screenshot.js

Tests

# Unit tests
cargo test

# E2E tests
cd e2e && npm install && npx playwright test

Features implementadas

Core

✅ 10 endpoints REST (9 crawl + 1 AI)
✅ Browser Pool — 5 navegadores Chromium, 10 páginas cada uno
✅ Session/Cookie Persistence — Guarda cookies por session_id
✅ Mobile Emulation — iPhone 14 viewport
✅ Infinite Scroll — Auto-scroll hasta el final
✅ Custom Headers — Headers arbitrarios por request

Workers

✅ Distributed Queue — Redis LPUSH/BLPOP
✅ Retry con Backoff — 3 retries con espera exponencial (2s, 4s, 8s)
✅ Dead Letter Queue — Jobs fallidos guardados por 24h
✅ Caching — Resultados en Redis con TTL 5 min

Scraping Avanzado

✅ Stealth Mode — Evade detección de bots (webdriver, plugins, canvas)
✅ Proxy Rotation — Múltiples proxies vía PROXY_URL
✅ CAPTCHA Solving — Integración con CapSolver/2captcha

Auth & Billing

✅ Email/Password — Bcrypt + JWT
✅ Google OAuth — Exchange real de code → token → user info
✅ Stripe — Checkout funcional + webhooks reales
✅ Teams — Owner/member roles

Observabilidad

✅ Prometheus — /metrics con counters y histograms
✅ Grafana — Dashboard incluido
✅ Sentry — Error tracking en API y Worker
✅ Structured Logging — JSON logs con correlation IDs
✅ WebSocket Logs — /ws/logs

Seguridad

✅ Input Validation — URLs, webhooks, tamaños (SSRF protection)
✅ Rate Limiting — Por API key (60/min) + por IP (100/min)
✅ IP Blocking — Auto-bloqueo por 1 hora

Infraestructura

✅ Docker Compose — Todo en un comando
✅ Kubernetes — Full manifests con ingress TLS + cert-manager
✅ HPA — Auto-scaling 3-20 workers
✅ Health Checks — Liveness, readiness, startup probes
✅ SSL/TLS — Let's Encrypt automático via cert-manager

Secrets Management

✅ Multi-provider — Env vars → Vault → AWS Secrets Manager
✅ Fallback chain — Intenta cada provider en orden

Frontend

✅ Landing Page
✅ API Documentation
✅ Interactive Playground — Probar endpoints con code snippets
✅ Billing Page — Plans + usage bar
✅ Dashboard — Login, API keys, tester

CI/CD

✅ GitHub Actions — CI con test, clippy, audit
✅ Docker Build & Push — Multi-stage builds
✅ Deploy Staging — Auto-deploy en push a main
✅ Deploy Production — Solo en tags v*
✅ Smoke Tests — Verificación post-deploy

Legal

✅ Terms of Service
✅ Privacy Policy
✅ Data Processing Agreement

Load Testing

✅ k6 Smoke Test — 1 VU
✅ k6 Load Test — Ramp up a 20 VUs
✅ k6 Stress Test — Hasta 200 VUs
✅ k6 Screenshot Test — 5 VUs concurrentes

Variables de entorno

# Core
DATABASE_URL="postgres://..."
REDIS_URL="redis://..."
JWT_SECRET="..."

# Storage
S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY

# Auth
GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET

# Billing
STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET

# AI
OPENAI_API_KEY

# Scraping
PROXY_URL="http://proxy1:8080,http://proxy2:8080"
CAPTCHA_API_KEY="..."

# Error Tracking
SENTRY_DSN="https://..."

# Logging
JSON_LOGGING="true"  # Enable structured JSON logs

# Secrets Management
VAULT_ADDR="https://vault.example.com"
VAULT_TOKEN="..."

# Browser Pool
BROWSER_POOL_SIZE=5
MAX_PAGES_PER_BROWSER=10

Uso de la API

AI Extraction

curl -X POST http://localhost:3000/api/extract \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/products",
    "schema": {"products": [{"name": "string", "price": "number"}]}
  }'

Screenshot con stealth + proxy + CAPTCHA

curl -X POST http://localhost:3000/api/screenshot \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "url": "https://protected-site.com",
    "options": {
      "stealth": true,
      "use_proxy": true,
      "solve_captcha": true,
      "session_id": "user_123"
    }
  }'

Mobile emulation

curl -X POST http://localhost:3000/api/screenshot \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{"url": "https://example.com", "options": {"mobile": true}}'

Licencia

MIT