SWE-2002

Replication-Unsafe Schema Design

Cantorian Technical Debt Magnitude: ℵ₁ (Systemic)

Description

Schema designs that don't account for distributed database replication challenges, leading to split-brain scenarios, replication lag issues, or conflict resolution problems. This includes using auto-incrementing IDs without proper coordination, schemas that generate conflicts during concurrent writes, or designs that assume single-master consistency in multi-master environments.

Illustrative Cantor Point

The Cantor Point occurs when choosing between single-region simplicity and multi-region complexity. The decision to implement aggressive automated failover without considering schema-level conflicts, or to use database features that don't replicate well, creates divergent paths where network partitions can cause irreconcilable data states.

Categories: CP-Schema, CP-API, CP-Process

Real-World Examples / Observed In

  • GitHub (2018): 43-second network partition triggered automated failover, creating split-brain with divergent data timelines [See: Cases-By-Year/2018 Data Integrity Failures.md#3]
  • Financial Trading Platform: Concurrent updates to order tables in different regions created conflicting sequences
  • E-commerce: Shopping cart updates during failover lost items due to replication conflicts

Common Consequences & Impacts

Technical Impacts

  • - Data divergence between replicas
  • - Lost writes during failover
  • - Complex reconciliation requirements
  • - Replication lag amplification

Human/Ethical Impacts

  • - Lost user data
  • - Incorrect account balances
  • - Failed critical operations
  • - Engineering stress during incidents

Business Impacts

  • - Service outages during recovery
  • - Data inconsistency
  • - Lost transactions
  • - Customer trust erosion

Recovery Difficulty & Escalation

8
7

ADI Principles & Axioms Violated

  • Principle of Fragile Stability: Network partitions reveal hidden schema assumptions
  • Principle of Cascading Catastrophes: Replication issues cascade across regions

Detection / 60-Second Audit

-- Check for auto-increment primary keys (problematic in multi-master)
SELECT 
    table_name,
    column_name,
    data_type,
    column_default
FROM information_schema.columns
WHERE table_schema = 'public'
AND column_default LIKE 'nextval%'
AND ordinal_position = 1;

-- Identify tables without proper conflict resolution columns
SELECT 
    t.table_name,
    CASE 
        WHEN NOT EXISTS (
            SELECT 1 FROM information_schema.columns c
            WHERE c.table_name = t.table_name
            AND c.column_name IN ('updated_at', 'version', 'last_modified')
        ) THEN 'Missing version/timestamp columns for conflict resolution'
    END as issue
FROM information_schema.tables t
WHERE t.table_schema = 'public'
AND t.table_type = 'BASE TABLE';

-- Check for unique constraints that might conflict across regions
SELECT 
    tc.table_name,
    tc.constraint_name,
    string_agg(kcu.column_name, ', ') as columns
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu 
    ON tc.constraint_name = kcu.constraint_name
WHERE tc.constraint_type = 'UNIQUE'
AND tc.table_schema = 'public'
GROUP BY tc.table_name, tc.constraint_name
ORDER BY tc.table_name;
-- Find tables with auto-increment PKs (replication risk)
SELECT 
    TABLE_NAME,
    COLUMN_NAME,
    DATA_TYPE,
    EXTRA
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND EXTRA LIKE '%auto_increment%'
AND ORDINAL_POSITION = 1;

-- Check for missing conflict resolution columns
SELECT 
    t.TABLE_NAME,
    CASE 
        WHEN NOT EXISTS (
            SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS c
            WHERE c.TABLE_NAME = t.TABLE_NAME
            AND c.COLUMN_NAME IN ('updated_at', 'version', 'last_modified')
            AND c.TABLE_SCHEMA = t.TABLE_SCHEMA
        ) THEN 'Missing version/timestamp for conflict resolution'
    END as issue
FROM INFORMATION_SCHEMA.TABLES t
WHERE t.TABLE_SCHEMA = DATABASE()
AND t.TABLE_TYPE = 'BASE TABLE'
HAVING issue IS NOT NULL;

-- Analyze replication hot spots
SELECT 
    object_name,
    count_read,
    count_write,
    count_write / (count_read + count_write + 0.0001) * 100 as write_percentage,
    CASE 
        WHEN count_write > 10000 THEN 'High write volume - lag risk'
        ELSE 'Normal'
    END as replication_risk
FROM performance_schema.table_io_waits_summary_by_table
WHERE object_schema = DATABASE()
ORDER BY count_write DESC
LIMIT 20;
-- Identify IDENTITY columns (problematic for multi-master)
SELECT 
    OBJECT_NAME(object_id) AS TableName,
    name AS ColumnName,
    seed_value,
    increment_value,
    last_value
FROM sys.identity_columns
WHERE OBJECTPROPERTY(object_id, 'IsMSShipped') = 0;

-- Find tables missing version columns
SELECT 
    t.name AS TableName,
    CASE 
        WHEN NOT EXISTS (
            SELECT 1 FROM sys.columns c
            WHERE c.object_id = t.object_id
            AND c.name IN ('updated_at', 'version', 'last_modified', 'rowversion')
        ) THEN 'Missing conflict resolution columns'
    END as Issue
FROM sys.tables t
WHERE t.is_ms_shipped = 0
AND NOT EXISTS (
    SELECT 1 FROM sys.columns c
    WHERE c.object_id = t.object_id
    AND c.name IN ('updated_at', 'version', 'last_modified', 'rowversion')
);

-- Check for high-frequency write tables
SELECT TOP 20
    OBJECT_NAME(s.object_id) AS TableName,
    SUM(s.user_updates) AS total_writes,
    SUM(s.user_seeks + s.user_scans + s.user_lookups) AS total_reads,
    CASE 
        WHEN SUM(s.user_updates) > 10000 THEN 'High write volume - replication risk'
        ELSE 'Normal volume'
    END AS replication_risk
FROM sys.dm_db_index_usage_stats s
INNER JOIN sys.tables t ON s.object_id = t.object_id
WHERE database_id = DB_ID()
GROUP BY s.object_id
ORDER BY total_writes DESC;

Prevention & Mitigation Best Practices

  1. Use UUIDs Instead of Auto-increment:

    -- Bad: Auto-increment can conflict
    CREATE TABLE orders (
        id SERIAL PRIMARY KEY,
        -- ...
    );
    
    -- Good: UUIDs avoid conflicts
    CREATE TABLE orders (
        id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
        -- ...
    );
    
  2. Implement Conflict Resolution Columns:

    CREATE TABLE critical_data (
        id UUID PRIMARY KEY,
        data JSONB,
        version INTEGER DEFAULT 1,
        updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
        region VARCHAR(20) DEFAULT current_region(),
        -- Use for last-write-wins or version comparison
    );
    
  3. Design for Eventual Consistency: Accept that replicas may lag

  4. Implement Idempotent Operations: Same operation produces same result

  5. Use Database-Specific Features: Leverage built-in conflict resolution

  6. Test Failover Scenarios: Regular split-brain drills

  7. Monitor Replication Lag: Alert on dangerous lag levels

Real World Examples

-- Dangerous: Schema that will cause split-brain conflicts

-- Table with auto-increment ID and no conflict resolution
CREATE TABLE orders (
    id SERIAL PRIMARY KEY,  -- Will conflict between regions!
    user_id INTEGER NOT NULL,
    total DECIMAL(10,2),
    status VARCHAR(20),
    created_at TIMESTAMP DEFAULT NOW()
);

-- Application code that doesn't handle conflicts
const createOrder = async (userId, items) => {
  // This will generate different IDs in different regions
  const result = await db.query(
    'INSERT INTO orders (user_id, total, status) VALUES ($1, $2, $3) RETURNING id',
    [userId, calculateTotal(items), 'pending']
  );
  
  // If regions split, same user could have orders with conflicting IDs
  return result.rows[0].id;
};
-- Another dangerous pattern: Unique constraints without region awareness

CREATE TABLE user_sessions (
    id SERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL,
    token VARCHAR(255) UNIQUE,  -- Will conflict if generated in multiple regions!
    created_at TIMESTAMP DEFAULT NOW(),
    -- No version or region tracking
    CONSTRAINT one_session_per_user UNIQUE(user_id)  -- Breaks during split-brain!
);

-- This will fail during network partition
const createSession = async (userId) => {
  const token = generateToken();
  // Both regions might try to create session for same user
  await db.query(
    'INSERT INTO user_sessions (user_id, token) VALUES ($1, $2)',
    [userId, token]
  );
};
-- Safe: Replication-aware schema design

-- Use UUIDs and include conflict resolution metadata
CREATE TABLE orders (
    id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
    user_id UUID NOT NULL,
    total DECIMAL(10,2) NOT NULL,
    status VARCHAR(20) NOT NULL,
    version INTEGER DEFAULT 1,
    region VARCHAR(20) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    
    -- Include region in unique constraints where needed
    INDEX idx_user_orders (user_id, created_at)
);

-- Conflict-aware session management
CREATE TABLE user_sessions (
    id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
    user_id UUID NOT NULL,
    token VARCHAR(255) NOT NULL,
    region VARCHAR(20) NOT NULL,
    version INTEGER DEFAULT 1,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
    
    -- Region-aware unique constraint
    UNIQUE(user_id, region),
    -- Token includes region prefix to avoid conflicts
    UNIQUE(token),
    INDEX idx_user_sessions (user_id, expires_at)
);

-- Application code that handles replication
const createOrder = async (userId, items, region) => {
  const orderId = uuid.v4();
  const now = new Date().toISOString();
  
  const result = await db.query(`
    INSERT INTO orders (id, user_id, total, status, region, created_at, updated_at)
    VALUES ($1, $2, $3, $4, $5, $6, $6)
    ON CONFLICT (id) DO UPDATE SET
      version = orders.version + 1,
      updated_at = $6
    WHERE orders.updated_at < $6  -- Last-write-wins
    RETURNING *
  `, [orderId, userId, calculateTotal(items), 'pending', region, now]);
  
  return result.rows[0];
};

-- Implement reconciliation for split-brain recovery
CREATE OR REPLACE FUNCTION reconcile_orders(
    primary_region TEXT,
    secondary_region TEXT
) RETURNS TABLE(id UUID, action TEXT) AS $
BEGIN
    RETURN QUERY
    WITH conflicts AS (
        SELECT 
            COALESCE(p.id, s.id) as id,
            p.version as primary_version,
            s.version as secondary_version,
            p.updated_at as primary_updated,
            s.updated_at as secondary_updated
        FROM orders p
        FULL OUTER JOIN orders_replica s ON p.id = s.id
        WHERE p.region = primary_region 
        AND s.region = secondary_region
        AND (p.version != s.version OR p.id IS NULL OR s.id IS NULL)
    )
    SELECT 
        id,
        CASE 
            WHEN primary_version IS NULL THEN 'INSERT_PRIMARY'
            WHEN secondary_version IS NULL THEN 'INSERT_SECONDARY'
            WHEN secondary_updated > primary_updated THEN 'UPDATE_PRIMARY'
            ELSE 'UPDATE_SECONDARY'
        END as action
    FROM conflicts;
END;
$ LANGUAGE plpgsql;

AI Coding Guidance/Prompt

Prompt: "When designing schemas for replicated databases:"
Rules:
  - Warn against auto-incrementing IDs in multi-master setups
  - Require timestamp and version columns for conflict detection
  - Flag unique constraints that might conflict across regions
  - Suggest partition-tolerant ID generation strategies
  - Require explicit conflict resolution strategy
  
Example:
  # Bad: Replication-unsafe design
  CREATE TABLE user_sessions (
    id SERIAL PRIMARY KEY,
    user_id INTEGER,
    session_data TEXT,
    created_at TIMESTAMP DEFAULT NOW()
  );
  
  # Good: Replication-aware design
  CREATE TABLE user_sessions (
    id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
    user_id UUID NOT NULL,
    session_data JSONB,
    version INTEGER DEFAULT 1,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    region VARCHAR(20) NOT NULL,
    
    -- Composite unique constraint includes region
    UNIQUE(user_id, region, created_at)
  );
  
  -- Function for conflict resolution
  CREATE OR REPLACE FUNCTION resolve_session_conflict(
    local_row user_sessions,
    remote_row user_sessions
  ) RETURNS user_sessions AS $
  BEGIN
    -- Last-write-wins with version check
    IF remote_row.version > local_row.version OR
       (remote_row.version = local_row.version AND 
        remote_row.updated_at > local_row.updated_at) THEN
      RETURN remote_row;
    ELSE
      RETURN local_row;
    END IF;
  END;
  $ LANGUAGE plpgsql;

Relevant Keywords

replication unsafe schema design Symptoms: slow queries, data inconsistency, constraint violations Preventive: schema validation, constraint enforcement, proper typing Tech stack: PostgreSQL, MySQL, SQL Server, Oracle Industry: all industries, enterprise, SaaS

Related Patterns

The Cantorian Technical Debt Magnitude scale gives developers an intuitive sense of magnitude beyond simple hour counts - some debts aren't just larger in scale, but qualitatively different in their complexity.

Cantor Points are critical decision junctures—or even moments of non-decision—where seemingly small choices can create drastically divergent futures for a system's integrity, security, and evolvability. These are the "forks in the road" where one path might lead to manageable complexity, while another veers towards systemic entanglement or even chaos. They often appear trivial at the time but can set in motion irreversible or costly-to-reverse consequences.

Applied Data Integrity (ADI) is a framework to understanding the far-reaching consequences of schema and data decisions that impact security and reliability, and accumulate into ethical debt that affects real human lives. Built on research from real-world incidents, ADI uncovered 7 Principles to identify when these decisions are being made, and how to make them better, to avoid future technical debt and potentially catastrophic "butterfly effects" of small decisions that ripple into chaotic technical and ethical debt.