crawlapi/TODO.md

TODO List Completo — Crawl API

✅ COMPLETO — Todo implementado

Core API

• 10 endpoints REST (9 crawl + 1 AI extraction)
• PostgreSQL + sqlx con migraciones
• Redis queue + caching
• S3/MinIO file storage

Auth & Users

• Email/password + bcrypt + JWT
• Google OAuth (exchange real de tokens)
• API key management
• Team/Organization accounts

Workers & Queue

• Distributed worker con Redis BLPOP
• Retry con backoff exponencial
• Dead Letter Queue
• Browser Pool (5x10)

Scraping Avanzado

• Stealth mode (anti-bot)
• Proxy rotation
• CAPTCHA solving (CapSolver)
• Cookie/session persistence
• Mobile emulation
• Infinite scroll
• Custom headers

Billing

• Stripe checkout funcional
• Stripe webhooks reales
• Plans + credits system

Observabilidad

• Prometheus metrics
• Grafana dashboard
• Sentry error tracking
• Structured JSON logging
• Correlation IDs
• WebSocket live logs

Seguridad

• Input validation (URL, webhook, size)
• Rate limiting por API key
• Rate limiting por IP
• IP auto-blocking
• SSRF protection

Infraestructura

• Docker Compose
• Kubernetes manifests
• HPA auto-scaling
• Health checks (liveness/readiness/startup)
• SSL/TLS con cert-manager + Let's Encrypt

Secrets Management

• Multi-provider: Env → Vault → AWS Secrets Manager
• Fallback chain

CI/CD

• GitHub Actions CI (fmt, clippy, test, audit)
• Docker build + push
• Deploy staging
• Deploy production (tags)
• Smoke tests post-deploy

Frontend

• Landing page
• API Documentation
• Interactive Playground
• Billing page
• Dashboard

Testing

• Unit tests
• E2E tests (Playwright)
• Load tests (k6 smoke/load/stress/screenshot)

Legal

• Terms of Service
• Privacy Policy
• Data Processing Agreement

---

Estado final

100% completo. El proyecto está listo para producción.