nas-tools/docs/TESTING.md

# Testing Guide

## Overview

This guide covers the multi-layered testing approach for the NAS Dashboard to ensure features work correctly before and after deployment.

## Testing Layers

```
       /\
      /E2E\         <- Full user workflows (optional)
     /------\
    /Smoke  \       <- Post-deployment verification
   /----------\
  /Integration\     <- API and component integration
 /--------------\
/   Unit Tests   \  <- Fast, isolated tests
------------------
```

## Running Tests Locally

### Backend Tests

```bash
cd dashboard/backend
python -m venv venv
source venv/bin/activate
pip install -r requirements-dev.txt
pytest tests/ --cov=. --cov-report=term
```

**Run specific test file:**
```bash
pytest tests/integration/test_conversation_tracker.py -v
```

**Run with coverage:**
```bash
pytest tests/ --cov=. --cov-report=html
open htmlcov/index.html
```

### Frontend Tests

```bash
cd dashboard/frontend
npm install
npm run test
```

**Run with coverage:**
```bash
npm run test:coverage
```

**Run in watch mode:**
```bash
npm run test:ui
```

### Smoke Tests

Test dev deployment:
```bash
./scripts/smoke-test-dashboard.sh dev
```

Test production deployment:
```bash
./scripts/smoke-test-dashboard.sh prod
```

## Pre-Deployment Checklist

Before pushing code to dev branch:

- [ ] Run backend tests locally: `cd dashboard/backend && pytest tests/`
- [ ] Run frontend tests locally: `cd dashboard/frontend && npm run test`
- [ ] Verify API paths don't have double `/api/` prefixes
- [ ] Check router registration in `main.py` has correct prefix
- [ ] Test feature manually in local environment if possible
- [ ] Review code changes for obvious issues

## Post-Deployment Verification

### After Deployment to Dev

1. Wait for CI/CD to complete (check Gitea Actions)
2. Run smoke tests:
   ```bash
   ./scripts/smoke-test-dashboard.sh dev
   ```
3. Manually test the feature in browser at `http://nas.jimmygan.com:4001`
4. Check container logs for errors:
   ```bash
   ssh nas "docker logs nas-dashboard-dev --tail 50"
   ```

### After Deployment to Production

1. Run smoke tests:
   ```bash
   ./scripts/smoke-test-dashboard.sh prod
   ```
2. Verify health monitoring is active:
   ```bash
   ssh nas "tail -20 /volume1/repos/nas-tools/logs/health-monitor.log"
   ```
3. Test critical user workflows manually

## Adding Tests for New Features

### 1. Backend API Tests

Create integration test in `dashboard/backend/tests/integration/`:

```python
import pytest

class TestMyNewFeature:
    def test_endpoint_requires_auth(self, test_app):
        response = test_app.get("/api/my-feature/endpoint")
        assert response.status_code == 401
    
    def test_endpoint_with_auth(self, test_app, admin_headers):
        response = test_app.get("/api/my-feature/endpoint", headers=admin_headers)
        assert response.status_code == 200
```

### 2. Frontend API Path Tests

Create test in `dashboard/frontend/tests/integration/`:

```javascript
import { describe, it, expect, vi } from 'vitest';
import { get } from '../../src/lib/api.js';

describe('My Feature API Paths', () => {
  it('should call correct API path', async () => {
    global.fetch = vi.fn().mockResolvedValue({
      ok: true,
      json: async () => ({}),
      headers: new Headers(),
    });

    await get('/my-feature/endpoint');

    expect(global.fetch).toHaveBeenCalledWith(
      expect.stringContaining('/api/my-feature/endpoint'),
      expect.any(Object)
    );
  });
});
```

### 3. Update Smoke Tests

Add checks to `scripts/smoke-test-dashboard.sh`:

```bash
# Test new feature endpoint
echo "Testing my new feature..."
RESPONSE=$(curl -s "$BASE_URL/api/my-feature/health")
if echo "$RESPONSE" | grep -q "ok"; then
    echo "✅ My feature is working"
else
    echo "❌ My feature failed"
    exit 1
fi
```

## CI/CD Pipeline

Tests run automatically on every push via `.gitea/workflows/test.yml`:

### Backend Tests
- Runs pytest with 49% minimum coverage
- Timeout: 30 seconds per test
- Generates JUnit XML and coverage reports

### Frontend Tests
- Runs vitest with coverage
- Tests all components and API integrations

### Deployment Tests
- Smoke tests run after dev deployment
- Health checks verify container is healthy
- Logs checked for critical errors

## Monitoring

### Health Check Monitoring

Runs every 5 minutes via cron on NAS:

```bash
*/5 * * * * /volume1/repos/nas-tools/scripts/monitor-dashboard-health.sh >> /volume1/repos/nas-tools/logs/health-monitor.log 2>&1
```

**What it checks:**
- Production dashboard `/api/health` endpoint
- Dev dashboard `/api/health` endpoint (non-critical)
- Sends Telegram alert on production failure
- Sends recovery notification when fixed

**View logs:**
```bash
ssh nas "tail -f /volume1/repos/nas-tools/logs/health-monitor.log"
```

### Manual Health Check

```bash
# Check production
curl http://nas.jimmygan.com:4000/api/health

# Check dev
curl http://nas.jimmygan.com:4001/api/health

# Check container health
ssh nas "docker ps | grep nas-dashboard"
```

## Troubleshooting

### Tests Failing Locally

**Backend:**
```bash
# Check Python version (should be 3.12+)
python --version

# Reinstall dependencies
cd dashboard/backend
rm -rf venv
python -m venv venv
source venv/bin/activate
pip install -r requirements-dev.txt
```

**Frontend:**
```bash
# Clear node_modules and reinstall
cd dashboard/frontend
rm -rf node_modules package-lock.json
npm install
```

### Smoke Tests Failing

1. Check if container is running:
   ```bash
   ssh nas "docker ps | grep nas-dashboard"
   ```

2. Check container logs:
   ```bash
   ssh nas "docker logs nas-dashboard-dev --tail 100"
   ```

3. Verify network connectivity:
   ```bash
   curl -v http://nas.jimmygan.com:4001/api/health
   ```

4. Check if database is accessible (for conversation tracker):
   ```bash
   ssh nas "ls -la /volume1/docker/claude-code-tracker/data/conversations.db"
   ```

### CI/CD Tests Failing

1. Check Gitea Actions logs in web UI
2. Look for specific test failures in output
3. Run the same tests locally to reproduce
4. Fix issues and push again

## Best Practices

### Writing Tests

1. **Test behavior, not implementation** - Focus on what the API returns, not how it works internally
2. **Use descriptive test names** - `test_endpoint_requires_auth` is better than `test_1`
3. **One assertion per test** - Makes failures easier to diagnose
4. **Mock external dependencies** - Don't rely on real databases, APIs, or services
5. **Test error cases** - Not just the happy path

### Test Coverage

- **Aim for 50%+ coverage** - Current backend is at 49%
- **Focus on critical paths** - Auth, data access, user-facing features
- **Don't chase 100%** - Some code (like error handlers) is hard to test

### Deployment Safety

1. **Always test in dev first** - Never push directly to production
2. **Run smoke tests after deployment** - Automated verification catches issues
3. **Monitor logs after deployment** - Watch for errors in first 5-10 minutes
4. **Have rollback plan** - Know how to revert to previous version

## Common Issues and Solutions

### Issue: Double `/api/` prefix in API calls

**Symptom:** Frontend gets 404 errors, logs show `/api/api/endpoint`

**Solution:**
- Check router definition: should NOT have `prefix="/api/..."` in `APIRouter()`
- Check router registration: SHOULD have `prefix="/api/..."` in `app.include_router()`
- Frontend should call `get("/endpoint")` not `get("/api/endpoint")`

**Test:** Run `dashboard/frontend/tests/integration/conversations.test.js`

### Issue: Tests pass locally but fail in CI

**Possible causes:**
- Different Python/Node versions
- Missing environment variables
- Timing issues (increase timeouts)
- File path differences

**Solution:** Check CI logs for specific error, replicate environment locally

### Issue: Smoke tests timeout

**Possible causes:**
- Container not fully started
- Network issues
- Service crashed

**Solution:**
```bash
# Check container status
ssh nas "docker ps -a | grep nas-dashboard"

# Check container logs
ssh nas "docker logs nas-dashboard-dev --tail 50"

# Restart container
ssh nas "cd /volume1/docker/nas-dashboard && docker compose -f docker-compose.dev.yml up -d"
```

## Resources

- **Backend Test Fixtures:** `dashboard/backend/tests/conftest.py`
- **Frontend Test Setup:** `dashboard/frontend/tests/setup.js`
- **CI/CD Workflows:** `.gitea/workflows/`
- **Smoke Tests:** `scripts/smoke-test-dashboard.sh`
- **Health Monitor:** `scripts/monitor-dashboard-health.sh`

## Getting Help

If tests are failing and you're stuck:

1. Check this documentation first
2. Look at existing tests for examples
3. Check CI/CD logs for specific errors
4. Review recent code changes that might have broken tests
5. Ask for help with specific error messages and context