nas-tools/TESTING_IMPLEMENTATION_PLAN.md

Testing Infrastructure Implementation Plan

Overview

Add comprehensive unit and integration testing for the NAS Dashboard backend (Python/FastAPI) and frontend (Svelte 5). Currently, the codebase has zero test coverage and relies only on smoke tests and manual verification.

Backend Testing Strategy

1. Testing Framework Setup

• Framework: pytest 8.x with pytest-asyncio for async tests
• Coverage: pytest-cov for coverage reporting
• Mocking: pytest-mock + unittest.mock
• HTTP Testing: httpx for TestClient (FastAPI built-in)
• Fixtures: Centralized fixtures for common test data

2. Test Structure

dashboard/backend/
├── tests/
│   ├── __init__.py
│   ├── conftest.py              # Shared fixtures
│   ├── unit/
│   │   ├── test_auth.py         # JWT, TOTP, password hashing
│   │   ├── test_rbac.py         # Permission checks, role resolution
│   │   ├── test_config.py       # IP parsing, env validation
│   │   └── test_utils.py        # Path validation, encryption
│   └── integration/
│       ├── test_auth_flow.py    # Login, refresh, logout flows
│       ├── test_docker.py       # Container operations (mocked)
│       ├── test_files.py        # File operations with permissions
│       ├── test_terminal.py     # WebSocket terminal (mocked SSH)
│       ├── test_passkey.py      # WebAuthn flows
│       └── test_security.py     # Rate limiting, audit logging

3. Key Test Areas

Unit Tests (High Priority)

1. auth.py (~300 lines to test)

   • JWT token generation/validation
   • Token expiry and refresh logic
   • TOTP generation/verification
   • Password hashing/verification
   • Token version management
   • Encryption/decryption (Fernet + legacy fallback)

2. rbac.py (~200 lines to test)

   • Role resolution (admin/member/viewer)
   • Page access checks
   • Sidebar link filtering
   • User override logic
   • Atomic JSON persistence

3. config.py (~150 lines to test)

   • IP range parsing (LAN, Tailscale, trusted proxies)
   • Environment variable validation
   • Default value handling

4. Path validation utilities

   • Traversal protection
   • Symlink resolution
   • Blocked directory checks

Integration Tests (High Priority)

1. Authentication Flow (routers/auth.py - 316 lines)

   • POST /api/auth/login → TOTP required → token issued
   • POST /api/auth/refresh → new tokens
   • POST /api/auth/logout → token revoked
   • Proxy auth with Remote-User header
   • Rate limiting enforcement

2. Docker Operations (routers/docker_router.py - 37 lines)

   • GET /api/docker/containers → list with health status
   • POST /api/docker/containers/{id}/start → admin only
   • GET /api/docker/containers/{id}/logs → tail limit

3. File Operations (routers/files.py - 128 lines)

   • GET /api/files/browse → directory listing
   • POST /api/files/upload → 100MB limit, admin only
   • DELETE /api/files/delete → permission checks

4. Terminal WebSocket (routers/terminal.py - 220 lines)

   • WebSocket /ws/terminal?host=nas → auth validation
   • Session reservation (max 5 global, 5 per user)
   • SSH connection lifecycle (mocked)
   • Keepalive ping/pong

5. Passkey/WebAuthn (routers/passkey.py - 188 lines)

   • Registration challenge generation
   • Credential verification
   • Credential management

4. Mocking Strategy

Docker SDK (docker.DockerClient):

@pytest.fixture
def mock_docker_client(mocker):
    client = mocker.Mock()
    client.containers.list.return_value = [
        mocker.Mock(id="abc123", name="test-container", status="running")
    ]
    return client

asyncssh (SSH connections):

@pytest.fixture
async def mock_ssh_connection(mocker):
    conn = mocker.AsyncMock()
    conn.create_process.return_value = mocker.AsyncMock()
    mocker.patch('asyncssh.connect', return_value=conn)
    return conn

httpx (External API calls):

@pytest.fixture
def mock_httpx_client(mocker):
    client = mocker.AsyncMock()
    mocker.patch('httpx.AsyncClient', return_value=client)
    return client

5. Test Fixtures (conftest.py)

@pytest.fixture
def test_app():
    """FastAPI test client with overridden dependencies"""
    from main import app
    return TestClient(app)

@pytest.fixture
def valid_jwt_token():
    """Generate valid JWT for testing"""
    return create_access_token({"sub": "testuser", "role": "admin"})

@pytest.fixture
def mock_rbac_data():
    """Sample RBAC configuration"""
    return {
        "role_defaults": {"admin": {"pages": "*", "sidebar_links": "*"}},
        "user_overrides": {}
    }

@pytest.fixture
def temp_auth_file(tmp_path):
    """Temporary auth.json for testing"""
    auth_file = tmp_path / "auth.json"
    auth_file.write_text('{"totp_secrets": {}, "passkey_credentials": {}}')
    return auth_file

6. Coverage Goals

• Target: 80%+ overall coverage
• Critical paths: 95%+ (auth, RBAC, path validation)
• Routers: 70%+ (focus on happy path + error cases)

7. CI Integration

Add to .gitea/workflows/test.yml:

name: Backend Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: nas-runner
    steps:
      - uses: actions/checkout@v3
      - name: Run pytest
        run: |
          cd dashboard/backend
          pip install -r requirements.txt -r requirements-dev.txt
          pytest --cov=. --cov-report=term --cov-report=html
      - name: Upload coverage
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: coverage-report
          path: dashboard/backend/htmlcov/

---

Frontend Testing Strategy

1. Testing Framework Setup

• Framework: Vitest 2.x (Vite-native, fast)
• Component Testing: @testing-library/svelte for Svelte 5
• Mocking: vi.mock() for modules, vi.fn() for functions
• Coverage: @vitest/coverage-v8

2. Test Structure

dashboard/frontend/
├── tests/
│   ├── setup.js                 # Global test setup
│   ├── unit/
│   │   ├── api.test.js          # API client, token management
│   │   └── voice.test.js        # VoiceSession class
│   ├── components/
│   │   ├── Sidebar.test.js      # Navigation, drag-drop, RBAC
│   │   └── App.test.js          # Routing, auth flow
│   └── integration/
│       ├── Login.test.js        # Passkey + password flows
│       ├── Terminal.test.js     # WebSocket, reconnection
│       ├── Docker.test.js       # Container management
│       └── Files.test.js        # File browser operations

3. Key Test Areas

Unit Tests

1. api.js (~200 lines)

   • Token storage/retrieval
   • Auto-refresh on 401
   • Error handling
   • Request methods (get, post, put, del)

2. voice.js (~150 lines)

   • VoiceSession state management
   • STT/TTS integration (mocked)
   • Language switching
   • Talk mode handling

Component Tests

1. App.svelte (~400 lines)

   • Initial auth check
   • Page routing via query params
   • Theme toggle persistence
   • User info loading

2. Sidebar.svelte (~800 lines)

   • Category rendering
   • Drag-drop reordering
   • Role-based link filtering
   • Collapse state persistence

3. Login.svelte (~230 lines)

   • Passkey authentication flow
   • Password + TOTP flow
   • Error message display
   • Form validation

4. Terminal.svelte (~652 lines)

   • WebSocket connection lifecycle
   • Tab management
   • Reconnection with exponential backoff
   • Auth failure recovery
   • Image upload (drag-drop/paste)

5. Docker.svelte (~133 lines)

   • Container list rendering
   • Start/stop/restart actions
   • Log modal display

6. Files.svelte (~151 lines)

   • Directory navigation
   • File upload
   • Download/delete actions

4. Mocking Strategy

Fetch API:

beforeEach(() => {
  global.fetch = vi.fn();
});

test('login success', async () => {
  fetch.mockResolvedValueOnce({
    ok: true,
    json: async () => ({ access_token: 'test-token' })
  });
  // test logic
});

WebSocket:

class MockWebSocket {
  constructor(url) {
    this.url = url;
    this.readyState = WebSocket.CONNECTING;
    setTimeout(() => {
      this.readyState = WebSocket.OPEN;
      this.onopen?.();
    }, 0);
  }
  send(data) { /* mock */ }
  close() { /* mock */ }
}
global.WebSocket = MockWebSocket;

Web Speech API:

global.SpeechRecognition = vi.fn(() => ({
  start: vi.fn(),
  stop: vi.fn(),
  addEventListener: vi.fn()
}));

global.speechSynthesis = {
  speak: vi.fn(),
  cancel: vi.fn()
};

xterm.js:

vi.mock('@xterm/xterm', () => ({
  Terminal: vi.fn(() => ({
    open: vi.fn(),
    write: vi.fn(),
    onData: vi.fn(),
    dispose: vi.fn()
  }))
}));

5. Test Configuration (vitest.config.js)

import { defineConfig } from 'vitest/config';
import { svelte } from '@sveltejs/vite-plugin-svelte';

export default defineConfig({
  plugins: [svelte({ hot: !process.env.VITEST })],
  test: {
    globals: true,
    environment: 'jsdom',
    setupFiles: ['./tests/setup.js'],
    coverage: {
      provider: 'v8',
      reporter: ['text', 'html', 'lcov'],
      exclude: ['node_modules/', 'tests/', 'dist/']
    }
  }
});

6. Coverage Goals

• Target: 70%+ overall coverage
• Critical components: 80%+ (Terminal, Login, App)
• API client: 90%+ (core infrastructure)

7. CI Integration

Add to .gitea/workflows/test.yml:

frontend-tests:
  runs-on: nas-runner
  steps:
    - uses: actions/checkout@v3
    - name: Install dependencies
      run: |
        cd dashboard/frontend
        npm ci
    - name: Run tests
      run: |
        cd dashboard/frontend
        npm run test -- --coverage
    - name: Upload coverage
      if: always()
      uses: actions/upload-artifact@v3
      with:
        name: frontend-coverage
        path: dashboard/frontend/coverage/

---

Implementation Phases

Phase 1: Backend Foundation (Priority: High)

1. Install pytest, pytest-asyncio, pytest-cov, pytest-mock
2. Create test directory structure
3. Write conftest.py with shared fixtures
4. Implement unit tests for auth.py (JWT, TOTP, encryption)
5. Implement unit tests for rbac.py (permissions, role resolution)
6. Implement unit tests for config.py (IP parsing)

Estimated effort: 4-6 hours Files created: 6-8 test files, conftest.py, requirements-dev.txt

Phase 2: Backend Integration Tests (Priority: High)

1. Mock Docker SDK for container tests
2. Mock asyncssh for terminal tests
3. Mock httpx for external API tests
4. Implement integration tests for auth flow (login, refresh, logout)
5. Implement integration tests for Docker operations
6. Implement integration tests for file operations
7. Implement integration tests for terminal WebSocket

Estimated effort: 6-8 hours Files created: 7-10 integration test files

Phase 3: Frontend Foundation (Priority: Medium)

1. Install vitest, @testing-library/svelte, @vitest/coverage-v8, jsdom
2. Create vitest.config.js
3. Create test directory structure and setup.js
4. Implement unit tests for api.js (token management, auto-refresh)
5. Implement unit tests for voice.js (VoiceSession)

Estimated effort: 3-4 hours Files created: vitest.config.js, setup.js, 2-3 test files

Phase 4: Frontend Component Tests (Priority: Medium)

1. Mock fetch, WebSocket, Web Speech API, xterm.js
2. Implement component tests for App.svelte (routing, auth)
3. Implement component tests for Login.svelte (passkey, password flows)
4. Implement component tests for Sidebar.svelte (drag-drop, RBAC)
5. Implement component tests for Terminal.svelte (WebSocket, reconnection)
6. Implement component tests for Docker.svelte (actions, logs)
7. Implement component tests for Files.svelte (browse, upload)

Estimated effort: 8-10 hours Files created: 6-8 component test files

Phase 5: CI Integration (Priority: Medium)

1. Create .gitea/workflows/test.yml
2. Configure pytest in CI with coverage reporting
3. Configure vitest in CI with coverage reporting
4. Add coverage badges to README (optional)
5. Configure coverage thresholds (fail if below target)

Estimated effort: 2-3 hours Files created: 1 workflow file

Phase 6: Documentation & Maintenance (Priority: Low)

1. Add TESTING.md with instructions for running tests
2. Document mocking patterns and fixtures
3. Add pre-commit hook for running tests (optional)
4. Set up coverage monitoring

Estimated effort: 1-2 hours Files created: TESTING.md

---

Dependencies to Add

Backend (requirements-dev.txt)

pytest==8.3.4
pytest-asyncio==0.24.0
pytest-cov==6.0.0
pytest-mock==3.14.0

Frontend (package.json devDependencies)

{
  "vitest": "^2.1.8",
  "@testing-library/svelte": "^5.2.3",
  "@vitest/coverage-v8": "^2.1.8",
  "jsdom": "^25.0.1"
}

---

Success Criteria

1. Backend: 80%+ test coverage with all critical paths tested
2. Frontend: 70%+ test coverage with key components tested
3. CI: Tests run automatically on push/PR with coverage reports
4. Documentation: Clear instructions for running and writing tests
5. Maintainability: Fixtures and mocks are reusable and well-documented

---

Risks & Mitigations

Risk: Tests may be slow due to Docker/SSH mocking complexity Mitigation: Use lightweight mocks, avoid actual Docker/SSH connections

Risk: Frontend tests may be brittle due to Svelte 5 runes Mitigation: Use @testing-library/svelte best practices, test behavior not implementation

Risk: CI runner may lack resources for parallel test execution Mitigation: Run tests sequentially if needed, optimize test fixtures

Risk: Existing code may need refactoring for testability Mitigation: Minimal refactoring, focus on testing current behavior first

---

Open Questions

1. Should we add E2E tests with Playwright in addition to unit/integration tests?

   • Recommendation: Start with unit/integration, add E2E later if needed

2. Should we enforce coverage thresholds in CI (fail build if below target)?

   • Recommendation: Yes, but start with lenient thresholds (60%) and increase gradually

3. Should we test all 13 routers or focus on high-risk areas first?

   • Recommendation: Focus on auth, docker, terminal, files first (80% of risk)

4. Should we mock the file system or use temporary directories?

   • Recommendation: Use pytest's tmp_path fixture for real file operations

5. Should we test WebSocket reconnection logic in detail?

   • Recommendation: Yes, it's complex and critical (Terminal.svelte has 652 lines)