rspade_system/app/RSpade/man/model_normalization.txt

DATABASE NORMALIZATION & MIGRATION SYSTEM
==========================================

RSpade Framework Database Normalization Documentation

OVERVIEW
--------
RSpade enforces an opinionated database schema through a two-layer system:

1. **SqlQueryTransformer** - Rewrites DDL statements during migrations to enforce
   consistent column types and character sets at CREATE/ALTER time

2. **migrate:normalize_schema** - Adds framework-required columns and fixes drift
   in existing tables

This eliminates foreign key type mismatches and ensures consistent schema across
all environments without manual ALTER statements.

ENFORCED COLUMN TYPE STANDARDS
-------------------------------

### Integer Types
  ✓ BIGINT           - All integer columns (IDs, counts, foreign keys)
  ✓ TINYINT(1)       - Boolean values only (true/false)
  ✗ INT, INTEGER     - Automatically converted to BIGINT
  ✗ MEDIUMINT        - Automatically converted to BIGINT
  ✗ SMALLINT         - Automatically converted to BIGINT
  ✗ UNSIGNED         - Removed (framework uses signed integers)

### Floating Point Types
  ✓ DOUBLE           - All floating point numbers
  ✓ DECIMAL(p,s)     - Exact precision (money, percentages)
  ✗ FLOAT, REAL      - Automatically converted to DOUBLE

### String Types
  ✓ VARCHAR(n)       - Variable length strings (with utf8mb4)
  ✓ LONGTEXT         - Large text content (with utf8mb4)
  ✓ JSON             - Structured data (when appropriate)
  ✗ CHAR(n)          - Automatically converted to VARCHAR(n)
  ✗ TEXT, MEDIUMTEXT - Automatically converted to LONGTEXT

### Temporal Types
  ✓ DATE             - Date only (YYYY-MM-DD)
  ✓ DATETIME(3)      - Date and time with millisecond precision
  ✓ TIMESTAMP(3)     - Unix timestamp with millisecond precision
  ✗ YEAR             - FORBIDDEN (use INT or DATE)
  ✗ TIME             - FORBIDDEN (use DATETIME)

### Binary Types
  ✓ BLOB, LONGBLOB   - Binary data (images, files)

### FORBIDDEN Types (Throw Exception)
  ✗ ENUM             - Use VARCHAR with validation instead
  ✗ SET              - Use JSON or separate table
  ✗ YEAR             - Use INT or DATE
  ✗ TIME             - Use DATETIME

RATIONALE: ENUM and SET cannot be modified without table locks, causing deployment
issues in production. YEAR and TIME types are too limited and cause conversion issues.

CHARACTER SET ENFORCEMENT
-------------------------
All string columns (VARCHAR, TEXT) are automatically set to:
  • CHARACTER SET: utf8mb4
  • COLLATION: utf8mb4_unicode_ci

This ensures full Unicode support including emojis and prevents character set
mismatches in foreign keys.

MIGRATION FLOW
--------------
When you run `php artisan migrate`:

1. **Pre-Migration Normalization**
   - Fixes existing tables created before transformer implementation
   - Converts tables to utf8mb4 character set
   - Ensures consistent schema before new migrations

2. **SqlQueryTransformer Enabled**
   - Intercepts all DB::statement() calls
   - Rewrites CREATE TABLE and ALTER TABLE queries
   - Transforms types (INT→BIGINT, FLOAT→DOUBLE, etc.)
   - Adds CHARACTER SET utf8mb4 to all string columns
   - Validates forbidden types (throws exception for ENUM, SET, etc.)

3. **Run Migrations**
   - Migrations execute with transformed SQL
   - Tables created with correct types from the start
   - Foreign keys work immediately (no type mismatches)

4. **Post-Migration Normalization**
   - Adds framework-required columns to new tables:
     • created_at, updated_at (with millisecond precision)
     • created_by, updated_by (audit columns)
     • deleted_by (for soft deletes)
     • Trait-specific columns (site_id, version, etc.)
   - Creates indexes on timestamp columns
   - Upgrades DATETIME/TIMESTAMP precision to milliseconds
   - Syncs Query/Join attribute indexes (see AUTOMATIC INDEX MANAGEMENT below)

5. **Regenerate Constants** (Dev Mode Only)
   - Updates model constants from database schema
   - Exports enum values to JavaScript
   - Generates IDE type hints

6. **SqlQueryTransformer Disabled**
   - Transformer only active during migration runs
   - Normal queries unaffected

TRANSFORMATION EXAMPLES
-----------------------

### Example 1: Integer Type Transformation

User writes:
```sql
CREATE TABLE posts (
    id INT AUTO_INCREMENT PRIMARY KEY,
    user_id INT,
    views INT UNSIGNED,
    is_published TINYINT(1),
    FOREIGN KEY (user_id) REFERENCES users(id)
);
```

Transformer rewrites to:
```sql
CREATE TABLE posts (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    user_id BIGINT,
    views BIGINT,
    is_published TINYINT(1),
    FOREIGN KEY (user_id) REFERENCES users(id)
);
```

Result: Foreign key works immediately (both columns BIGINT).

### Example 2: String Type Transformation

User writes:
```sql
CREATE TABLE users (
    name VARCHAR(255),
    bio TEXT,
    code CHAR(10)
);
```

Transformer rewrites to:
```sql
CREATE TABLE users (
    name VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
    bio LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
    code VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
);
```

Result: Consistent character sets, no FK issues on string columns.

### Example 3: Forbidden Type Rejection

User writes:
```sql
CREATE TABLE products (
    status ENUM('draft', 'published', 'archived')
);
```

Result: Migration fails with exception:
```
ENUM column type is forbidden in RSpade. Use VARCHAR with validation instead.
ENUMs cannot be modified without table locks and cause deployment issues.
```

Developer fixes by using VARCHAR:
```sql
CREATE TABLE products (
    status VARCHAR(20)  -- Add validation in model
);
```

FRAMEWORK-REQUIRED COLUMNS
--------------------------
normalize_schema ensures these columns exist on ALL tables (except excluded tables):

### Audit Columns
  • created_at TIMESTAMP(3) DEFAULT CURRENT_TIMESTAMP(3)
  • updated_at TIMESTAMP(3) DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3)
  • created_by BIGINT NULL
  • updated_by BIGINT NULL
  • deleted_by BIGINT NULL (if deleted_at exists)

### Indexes
  • INDEX created_at(created_at)
  • INDEX updated_at(updated_at)

### Trait-Specific Columns
  • site_id BIGINT (for Siteable trait)
  • version BIGINT (for Versionable/Ajaxable traits)

### Order Column (Manual Sorting)
Tables with an `order` column are automatically normalized:
  • Type: BIGINT DEFAULT NULL
  • Index: order_idx covering (order)
  • Triggers: Auto-increment on INSERT/UPDATE when order is NULL

The triggers ensure that when a record is inserted or updated with `order = NULL`,
it automatically gets assigned `MAX(order) + 1` from that table. This provides
automatic "add to end" behavior while still allowing explicit ordering.

Usage pattern:
```php
// Insert at end (auto-assigns order)
$item = new Menu_Item();
$item->name = 'New Item';
$item->order = null;  // Will be MAX(order) + 1
$item->save();

// Insert at specific position
$item->order = 5;     // Explicit position
$item->save();

// Reorder (swap with another item)
$item1->order = $item2->order;
$item2->order = $item1_old_order;
$item1->save();
$item2->save();
```

Trigger naming convention: {table_name}_order_insert, {table_name}_order_update

Excluded Tables: migrations, api_clients, sessions

AUTOMATIC INDEX MANAGEMENT
---------------------------
RSX automatically creates and maintains database indexes based on #[Query] and
#[Join] attributes declared in model methods. This eliminates manual index
management and ensures all queries have appropriate indexes.

### How It Works

During database normalization (and manifest rebuild in dev mode), the framework:

1. **Scans Models** - Finds all methods with #[Query] or #[Join] attributes
2. **Collects Required Indexes** - Extracts index column lists from attributes
3. **Detects Covering Indexes** - Checks if manual indexes already cover requirements
4. **Removes Redundancy** - Filters overlapping index requirements
5. **Creates Missing Indexes** - Adds only indexes not covered by existing ones
6. **Removes Obsolete Indexes** - Drops auto-generated indexes no longer needed

### Index Naming Convention

Auto-generated indexes are prefixed with `__rsx_query_autoindex_` followed by
a sequential number. These indexes are managed automatically by the framework.

Manual indexes (without this prefix) are never modified. If a manual index
covers the columns required by a Query attribute, no auto-index is created.

### Covering Index Detection

The framework understands that index (a,b,c) can efficiently support queries on:
- (a) alone
- (a,b) together
- (a,b,c) together

But NOT efficiently on:
- (b) alone
- (c) alone
- (b,c) together

If you have #[Query(index: ['user_id', 'status'])] and a manual index already
exists on (user_id, status, created_at), the framework recognizes the manual
index covers your requirement and does not create an auto-index.

### Redundant Index Removal

If multiple Query attributes would create overlapping indexes, the framework
keeps only the more specific ones:

- #[Query(index: ['user_id'])]
- #[Query(index: ['user_id', 'status'])]

Result: Only create index on (user_id, status) because it covers both queries.

### Query Attribute Syntax

Declare required indexes directly in model methods:

```php
class User_Model extends Rsx_Model_Abstract
{
    // Simple single-column index
    #[Query(index: ['status'])]
    public static function find_by_status(int $status)
    {
        return static::where('status', $status)->get();
    }

    // Composite index (order matters!)
    #[Query(index: ['category_id', 'status'])]
    public static function find_by_category(int $category_id)
    {
        return static::where('category_id', $category_id)
            ->where('status', static::STATUS_ACTIVE)
            ->get();
    }
}
```

### Join Attribute Syntax

Joins specify the target table and index to create on that table:

```php
class User_Model extends Rsx_Model_Abstract
{
    #[Join(
        target_table: 'user_logins',
        index: ['user_id', 'logged_in_at']
    )]
    public static function join_user_logins_recent($query, array $conditions = [])
    {
        return $query->leftJoin('user_logins', function($join) {
            $join->on('user_logins.user_id', '=', 'users.id')
                 ->where('user_logins.logged_in_at', '>',
                     DB::raw('NOW() - INTERVAL 30 DAY'));
        });
    }
}
```

The index is created on the TARGET table (user_logins), not the source table.

### When Indexes Are Created

**Development Mode**: Indexes sync automatically during manifest rebuild
```bash
php artisan rsx:manifest:build    # Auto-syncs indexes
```

**Production Mode**: Indexes sync during database normalization
```bash
php artisan migrate:normalize_schema
```

### Viewing Required Indexes

```bash
php artisan rsx:db:check_indexes
```

Shows all Query and Join attributes with their required indexes and whether
those indexes exist in the database.

For complete documentation of the Query/Join system, see:
```bash
php artisan rsx:man database_queries
```

BENEFITS OF THIS SYSTEM
------------------------

1. **No Foreign Key Mismatches**
   - Types match from the start (both BIGINT)
   - No need to drop/recreate foreign keys
   - Migrations run faster in production

2. **Write Natural SQL**
   - Developer writes INT, framework uses BIGINT
   - Developer writes TEXT, framework uses LONGTEXT
   - No need to remember framework conventions

3. **Consistent Schema**
   - All environments have identical schema
   - No drift between dev and production
   - Database migrations are deterministic

4. **Fast Production Migrations**
   - No column type modifications (no row rewrites)
   - No foreign key drop/recreate (metadata only)
   - Tables created correctly on first try

5. **Fail Fast on Bad Types**
   - ENUM throws exception immediately
   - Developer gets clear error message
   - No silent failures or runtime issues

COMMANDS
--------

### php artisan migrate
  Main migration command. Runs the full cycle:
  pre-normalize → migrations → post-normalize → regenerate constants (dev)

### php artisan migrate:begin
  Creates MySQL snapshot before migrations (dev mode only).
  Required for safety in development.

### php artisan migrate:commit
  Deletes snapshot and exits migration mode.
  Runs schema quality checks before committing.

### php artisan migrate:rollback
  Restores database to snapshot state.
  Stays in migration mode for retry.

### php artisan migrate:normalize_schema
  Manually run normalization (rarely needed).
  Add --production flag to skip snapshot requirement.

### php artisan migrate:check
  Check database schema for standards compliance.
  Reports violations without making changes.

SNAPSHOT-BASED DEVELOPMENT
--------------------------
Development migrations require snapshots for safety:

1. `php artisan migrate:begin`    - Create snapshot
2. `php artisan migrate`           - Run migrations
3. `php artisan migrate:commit`    - Keep changes, delete snapshot
   OR
   `php artisan migrate:rollback`  - Discard changes, restore snapshot

Production migrations skip snapshots and run directly.

WHY NO FOREIGN KEY DROP/RECREATE?
---------------------------------
Previous system:
  1. User creates table with INT columns
  2. normalize_schema converts INT→BIGINT (slow, locks table)
  3. Must drop ALL FKs first, recreate after (complex, error-prone)

New system:
  1. Transformer rewrites INT→BIGINT before execution
  2. Table created with BIGINT from the start
  3. normalize_schema finds nothing to change (fast)
  4. No FK drop/recreate needed

The only remaining use case for column modifications is converting pre-existing
tables (created before transformer) to utf8mb4 character set. This is rare and
only affects legacy tables.

TROUBLESHOOTING
---------------

### Migration fails with "ENUM column type is forbidden"
  Solution: Replace ENUM with VARCHAR and add validation in your model.

### Migration fails with "SET column type is forbidden"
  Solution: Use JSON column or create a separate table.

### Foreign key fails with type mismatch
  Check: Are you running migrations outside the framework? Use `php artisan migrate`.
  The transformer only runs during framework migration commands.

### Table has INT instead of BIGINT
  This means the table was created before the transformer was implemented or
  created outside the migration system. Run `php artisan migrate:normalize_schema`
  to fix it, or drop and recreate the table.

### Character set mismatch on foreign keys
  Run `php artisan migrate:normalize_schema` to convert tables to utf8mb4.

ADVANCED: HOW THE TRANSFORMER WORKS
------------------------------------
SqlQueryTransformer uses regex-based pattern matching on SQL statements:

1. Whitespace Normalization
   - Collapses whitespace while preserving quoted strings
   - Handles single quotes, double quotes, and backticks

2. DDL Detection
   - Only transforms CREATE TABLE and ALTER TABLE
   - Other queries (SELECT, INSERT, etc.) pass through unchanged

3. Type Validation
   - Checks for forbidden types first (ENUM, SET, YEAR, TIME)
   - Throws exception immediately if found

4. Type Transformation (in order)
   - Integer types: INT/INTEGER/MEDIUMINT/SMALLINT → BIGINT
   - Float types: FLOAT/REAL → DOUBLE
   - Text types: TEXT/MEDIUMTEXT/CHAR → LONGTEXT/VARCHAR
   - Charset enforcement: Add CHARACTER SET utf8mb4 to string columns

5. Query Execution
   - Transformed query executes via DB::statement()
   - Database never sees the original types

The transformer is enabled only during `php artisan migrate` and disabled after
completion. Normal application queries are not affected.

SEE ALSO
--------
- php artisan rsx:man database_queries   - Query/Join attribute system
- php artisan rsx:man migrations         - Migration guidelines
- php artisan rsx:man rsx_architecture   - Framework architecture
- php artisan rsx:man coding_standards   - Code standards
- /app/RSpade/Core/Database/CLAUDE.md    - Database system docs

AUTHOR
------
RSpade Framework - October 2025