nas-tools/docs/TESTING.md

Testing Guide

Overview

This guide covers the multi-layered testing approach for the NAS Dashboard to ensure features work correctly before and after deployment.

Testing Layers

       /\
      /E2E\         <- Full user workflows (optional)
     /------\
    /Smoke  \       <- Post-deployment verification
   /----------\
  /Integration\     <- API and component integration
 /--------------\
/   Unit Tests   \  <- Fast, isolated tests
------------------

Running Tests Locally

Backend Tests

cd dashboard/backend
python -m venv venv
source venv/bin/activate
pip install -r requirements-dev.txt
pytest tests/ --cov=. --cov-report=term

Run specific test file:

pytest tests/integration/test_conversation_tracker.py -v

Run with coverage:

pytest tests/ --cov=. --cov-report=html
open htmlcov/index.html

Frontend Tests

cd dashboard/frontend
npm install
npm run test

Run with coverage:

npm run test:coverage

Run in watch mode:

npm run test:ui

Smoke Tests

Test dev deployment:

./scripts/smoke-test-dashboard.sh dev

Test production deployment:

./scripts/smoke-test-dashboard.sh prod

Pre-Deployment Checklist

Before pushing code to dev branch:

• Run backend tests locally: cd dashboard/backend && pytest tests/
• Run frontend tests locally: cd dashboard/frontend && npm run test
• Verify API paths don't have double /api/ prefixes
• Check router registration in main.py has correct prefix
• Test feature manually in local environment if possible
• Review code changes for obvious issues

Post-Deployment Verification

After Deployment to Dev

1. Wait for CI/CD to complete (check Gitea Actions)
2. Run smoke tests:

   ./scripts/smoke-test-dashboard.sh dev
   

3. Manually test the feature in browser at http://nas.jimmygan.com:4001
4. Check container logs for errors:

   ssh nas "docker logs nas-dashboard-dev --tail 50"
   

After Deployment to Production

1. Run smoke tests:

   ./scripts/smoke-test-dashboard.sh prod
   

2. Verify health monitoring is active:

   ssh nas "tail -20 /volume1/repos/nas-tools/logs/health-monitor.log"
   

3. Test critical user workflows manually

Adding Tests for New Features

1. Backend API Tests

Create integration test in dashboard/backend/tests/integration/:

import pytest

class TestMyNewFeature:
    def test_endpoint_requires_auth(self, test_app):
        response = test_app.get("/api/my-feature/endpoint")
        assert response.status_code == 401
    
    def test_endpoint_with_auth(self, test_app, admin_headers):
        response = test_app.get("/api/my-feature/endpoint", headers=admin_headers)
        assert response.status_code == 200

2. Frontend API Path Tests

Create test in dashboard/frontend/tests/integration/:

import { describe, it, expect, vi } from 'vitest';
import { get } from '../../src/lib/api.js';

describe('My Feature API Paths', () => {
  it('should call correct API path', async () => {
    global.fetch = vi.fn().mockResolvedValue({
      ok: true,
      json: async () => ({}),
      headers: new Headers(),
    });

    await get('/my-feature/endpoint');

    expect(global.fetch).toHaveBeenCalledWith(
      expect.stringContaining('/api/my-feature/endpoint'),
      expect.any(Object)
    );
  });
});

3. Update Smoke Tests

Add checks to scripts/smoke-test-dashboard.sh:

# Test new feature endpoint
echo "Testing my new feature..."
RESPONSE=$(curl -s "$BASE_URL/api/my-feature/health")
if echo "$RESPONSE" | grep -q "ok"; then
    echo "✅ My feature is working"
else
    echo "❌ My feature failed"
    exit 1
fi

CI/CD Pipeline

Tests run automatically on every push via .gitea/workflows/test.yml:

Backend Tests

• Runs pytest with 49% minimum coverage
• Timeout: 30 seconds per test
• Generates JUnit XML and coverage reports

Frontend Tests

• Runs vitest with coverage
• Tests all components and API integrations

Deployment Tests

• Smoke tests run after dev deployment
• Health checks verify container is healthy
• Logs checked for critical errors

Monitoring

Health Check Monitoring

Runs every 5 minutes via cron on NAS:

*/5 * * * * /volume1/repos/nas-tools/scripts/monitor-dashboard-health.sh >> /volume1/repos/nas-tools/logs/health-monitor.log 2>&1

What it checks:

• Production dashboard /api/health endpoint
• Dev dashboard /api/health endpoint (non-critical)
• Sends Telegram alert on production failure
• Sends recovery notification when fixed

View logs:

ssh nas "tail -f /volume1/repos/nas-tools/logs/health-monitor.log"

Manual Health Check

# Check production
curl http://nas.jimmygan.com:4000/api/health

# Check dev
curl http://nas.jimmygan.com:4001/api/health

# Check container health
ssh nas "docker ps | grep nas-dashboard"

Troubleshooting

Tests Failing Locally

Backend:

# Check Python version (should be 3.12+)
python --version

# Reinstall dependencies
cd dashboard/backend
rm -rf venv
python -m venv venv
source venv/bin/activate
pip install -r requirements-dev.txt

Frontend:

# Clear node_modules and reinstall
cd dashboard/frontend
rm -rf node_modules package-lock.json
npm install

Smoke Tests Failing

1. Check if container is running:

   ssh nas "docker ps | grep nas-dashboard"
   

2. Check container logs:

   ssh nas "docker logs nas-dashboard-dev --tail 100"
   

3. Verify network connectivity:

   curl -v http://nas.jimmygan.com:4001/api/health
   

4. Check if database is accessible (for conversation tracker):

   ssh nas "ls -la /volume1/docker/claude-code-tracker/data/conversations.db"
   

CI/CD Tests Failing

1. Check Gitea Actions logs in web UI
2. Look for specific test failures in output
3. Run the same tests locally to reproduce
4. Fix issues and push again

Best Practices

Writing Tests

1. Test behavior, not implementation - Focus on what the API returns, not how it works internally
2. Use descriptive test names - test_endpoint_requires_auth is better than test_1
3. One assertion per test - Makes failures easier to diagnose
4. Mock external dependencies - Don't rely on real databases, APIs, or services
5. Test error cases - Not just the happy path

Test Coverage

• Aim for 50%+ coverage - Current backend is at 49%
• Focus on critical paths - Auth, data access, user-facing features
• Don't chase 100% - Some code (like error handlers) is hard to test

Deployment Safety

1. Always test in dev first - Never push directly to production
2. Run smoke tests after deployment - Automated verification catches issues
3. Monitor logs after deployment - Watch for errors in first 5-10 minutes
4. Have rollback plan - Know how to revert to previous version

Common Issues and Solutions

Issue: Double /api/ prefix in API calls

Symptom: Frontend gets 404 errors, logs show /api/api/endpoint

Solution:

• Check router definition: should NOT have prefix="/api/..." in APIRouter()
• Check router registration: SHOULD have prefix="/api/..." in app.include_router()
• Frontend should call get("/endpoint") not get("/api/endpoint")

Test: Run dashboard/frontend/tests/integration/conversations.test.js

Issue: Tests pass locally but fail in CI

Possible causes:

• Different Python/Node versions
• Missing environment variables
• Timing issues (increase timeouts)
• File path differences

Solution: Check CI logs for specific error, replicate environment locally

Issue: Smoke tests timeout

Possible causes:

• Container not fully started
• Network issues
• Service crashed

Solution:

# Check container status
ssh nas "docker ps -a | grep nas-dashboard"

# Check container logs
ssh nas "docker logs nas-dashboard-dev --tail 50"

# Restart container
ssh nas "cd /volume1/docker/nas-dashboard && docker compose -f docker-compose.dev.yml up -d"

Resources

• Backend Test Fixtures: dashboard/backend/tests/conftest.py
• Frontend Test Setup: dashboard/frontend/tests/setup.js
• CI/CD Workflows: .gitea/workflows/
• Smoke Tests: scripts/smoke-test-dashboard.sh
• Health Monitor: scripts/monitor-dashboard-health.sh

Getting Help

If tests are failing and you're stuck:

1. Check this documentation first
2. Look at existing tests for examples
3. Check CI/CD logs for specific errors
4. Review recent code changes that might have broken tests
5. Ask for help with specific error messages and context