Replication-Unsafe Schema Design
Description
Schema designs that don't account for distributed database replication challenges, leading to split-brain scenarios, replication lag issues, or conflict resolution problems. This includes using auto-incrementing IDs without proper coordination, schemas that generate conflicts during concurrent writes, or designs that assume single-master consistency in multi-master environments.
Illustrative Cantor Point
The Cantor Point occurs when choosing between single-region simplicity and multi-region complexity. The decision to implement aggressive automated failover without considering schema-level conflicts, or to use database features that don't replicate well, creates divergent paths where network partitions can cause irreconcilable data states.
Real-World Examples / Observed In
- GitHub (2018): 43-second network partition triggered automated failover, creating split-brain with divergent data timelines [See: Cases-By-Year/2018 Data Integrity Failures.md#3]
- Financial Trading Platform: Concurrent updates to order tables in different regions created conflicting sequences
- E-commerce: Shopping cart updates during failover lost items due to replication conflicts
Common Consequences & Impacts
Technical Impacts
- - Data divergence between replicas
- - Lost writes during failover
- - Complex reconciliation requirements
- - Replication lag amplification
Human/Ethical Impacts
- - Lost user data
- - Incorrect account balances
- - Failed critical operations
- - Engineering stress during incidents
Business Impacts
- - Service outages during recovery
- - Data inconsistency
- - Lost transactions
- - Customer trust erosion
Recovery Difficulty & Escalation
ADI Principles & Axioms Violated
- Principle of Fragile Stability: Network partitions reveal hidden schema assumptions
- Principle of Cascading Catastrophes: Replication issues cascade across regions
Detection / 60-Second Audit
-- Check for auto-increment primary keys (problematic in multi-master)
SELECT
table_name,
column_name,
data_type,
column_default
FROM information_schema.columns
WHERE table_schema = 'public'
AND column_default LIKE 'nextval%'
AND ordinal_position = 1;
-- Identify tables without proper conflict resolution columns
SELECT
t.table_name,
CASE
WHEN NOT EXISTS (
SELECT 1 FROM information_schema.columns c
WHERE c.table_name = t.table_name
AND c.column_name IN ('updated_at', 'version', 'last_modified')
) THEN 'Missing version/timestamp columns for conflict resolution'
END as issue
FROM information_schema.tables t
WHERE t.table_schema = 'public'
AND t.table_type = 'BASE TABLE';
-- Check for unique constraints that might conflict across regions
SELECT
tc.table_name,
tc.constraint_name,
string_agg(kcu.column_name, ', ') as columns
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu
ON tc.constraint_name = kcu.constraint_name
WHERE tc.constraint_type = 'UNIQUE'
AND tc.table_schema = 'public'
GROUP BY tc.table_name, tc.constraint_name
ORDER BY tc.table_name;
-- Find tables with auto-increment PKs (replication risk)
SELECT
TABLE_NAME,
COLUMN_NAME,
DATA_TYPE,
EXTRA
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND EXTRA LIKE '%auto_increment%'
AND ORDINAL_POSITION = 1;
-- Check for missing conflict resolution columns
SELECT
t.TABLE_NAME,
CASE
WHEN NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS c
WHERE c.TABLE_NAME = t.TABLE_NAME
AND c.COLUMN_NAME IN ('updated_at', 'version', 'last_modified')
AND c.TABLE_SCHEMA = t.TABLE_SCHEMA
) THEN 'Missing version/timestamp for conflict resolution'
END as issue
FROM INFORMATION_SCHEMA.TABLES t
WHERE t.TABLE_SCHEMA = DATABASE()
AND t.TABLE_TYPE = 'BASE TABLE'
HAVING issue IS NOT NULL;
-- Analyze replication hot spots
SELECT
object_name,
count_read,
count_write,
count_write / (count_read + count_write + 0.0001) * 100 as write_percentage,
CASE
WHEN count_write > 10000 THEN 'High write volume - lag risk'
ELSE 'Normal'
END as replication_risk
FROM performance_schema.table_io_waits_summary_by_table
WHERE object_schema = DATABASE()
ORDER BY count_write DESC
LIMIT 20;
-- Identify IDENTITY columns (problematic for multi-master)
SELECT
OBJECT_NAME(object_id) AS TableName,
name AS ColumnName,
seed_value,
increment_value,
last_value
FROM sys.identity_columns
WHERE OBJECTPROPERTY(object_id, 'IsMSShipped') = 0;
-- Find tables missing version columns
SELECT
t.name AS TableName,
CASE
WHEN NOT EXISTS (
SELECT 1 FROM sys.columns c
WHERE c.object_id = t.object_id
AND c.name IN ('updated_at', 'version', 'last_modified', 'rowversion')
) THEN 'Missing conflict resolution columns'
END as Issue
FROM sys.tables t
WHERE t.is_ms_shipped = 0
AND NOT EXISTS (
SELECT 1 FROM sys.columns c
WHERE c.object_id = t.object_id
AND c.name IN ('updated_at', 'version', 'last_modified', 'rowversion')
);
-- Check for high-frequency write tables
SELECT TOP 20
OBJECT_NAME(s.object_id) AS TableName,
SUM(s.user_updates) AS total_writes,
SUM(s.user_seeks + s.user_scans + s.user_lookups) AS total_reads,
CASE
WHEN SUM(s.user_updates) > 10000 THEN 'High write volume - replication risk'
ELSE 'Normal volume'
END AS replication_risk
FROM sys.dm_db_index_usage_stats s
INNER JOIN sys.tables t ON s.object_id = t.object_id
WHERE database_id = DB_ID()
GROUP BY s.object_id
ORDER BY total_writes DESC;
Prevention & Mitigation Best Practices
Use UUIDs Instead of Auto-increment:
-- Bad: Auto-increment can conflict CREATE TABLE orders ( id SERIAL PRIMARY KEY, -- ... ); -- Good: UUIDs avoid conflicts CREATE TABLE orders ( id UUID DEFAULT gen_random_uuid() PRIMARY KEY, -- ... );Implement Conflict Resolution Columns:
CREATE TABLE critical_data ( id UUID PRIMARY KEY, data JSONB, version INTEGER DEFAULT 1, updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), region VARCHAR(20) DEFAULT current_region(), -- Use for last-write-wins or version comparison );Design for Eventual Consistency: Accept that replicas may lag
Implement Idempotent Operations: Same operation produces same result
Use Database-Specific Features: Leverage built-in conflict resolution
Test Failover Scenarios: Regular split-brain drills
Monitor Replication Lag: Alert on dangerous lag levels
Real World Examples
-- Dangerous: Schema that will cause split-brain conflicts
-- Table with auto-increment ID and no conflict resolution
CREATE TABLE orders (
id SERIAL PRIMARY KEY, -- Will conflict between regions!
user_id INTEGER NOT NULL,
total DECIMAL(10,2),
status VARCHAR(20),
created_at TIMESTAMP DEFAULT NOW()
);
-- Application code that doesn't handle conflicts
const createOrder = async (userId, items) => {
// This will generate different IDs in different regions
const result = await db.query(
'INSERT INTO orders (user_id, total, status) VALUES ($1, $2, $3) RETURNING id',
[userId, calculateTotal(items), 'pending']
);
// If regions split, same user could have orders with conflicting IDs
return result.rows[0].id;
};
-- Another dangerous pattern: Unique constraints without region awareness
CREATE TABLE user_sessions (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL,
token VARCHAR(255) UNIQUE, -- Will conflict if generated in multiple regions!
created_at TIMESTAMP DEFAULT NOW(),
-- No version or region tracking
CONSTRAINT one_session_per_user UNIQUE(user_id) -- Breaks during split-brain!
);
-- This will fail during network partition
const createSession = async (userId) => {
const token = generateToken();
// Both regions might try to create session for same user
await db.query(
'INSERT INTO user_sessions (user_id, token) VALUES ($1, $2)',
[userId, token]
);
};
-- Safe: Replication-aware schema design
-- Use UUIDs and include conflict resolution metadata
CREATE TABLE orders (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
user_id UUID NOT NULL,
total DECIMAL(10,2) NOT NULL,
status VARCHAR(20) NOT NULL,
version INTEGER DEFAULT 1,
region VARCHAR(20) NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
-- Include region in unique constraints where needed
INDEX idx_user_orders (user_id, created_at)
);
-- Conflict-aware session management
CREATE TABLE user_sessions (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
user_id UUID NOT NULL,
token VARCHAR(255) NOT NULL,
region VARCHAR(20) NOT NULL,
version INTEGER DEFAULT 1,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
-- Region-aware unique constraint
UNIQUE(user_id, region),
-- Token includes region prefix to avoid conflicts
UNIQUE(token),
INDEX idx_user_sessions (user_id, expires_at)
);
-- Application code that handles replication
const createOrder = async (userId, items, region) => {
const orderId = uuid.v4();
const now = new Date().toISOString();
const result = await db.query(`
INSERT INTO orders (id, user_id, total, status, region, created_at, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $6)
ON CONFLICT (id) DO UPDATE SET
version = orders.version + 1,
updated_at = $6
WHERE orders.updated_at < $6 -- Last-write-wins
RETURNING *
`, [orderId, userId, calculateTotal(items), 'pending', region, now]);
return result.rows[0];
};
-- Implement reconciliation for split-brain recovery
CREATE OR REPLACE FUNCTION reconcile_orders(
primary_region TEXT,
secondary_region TEXT
) RETURNS TABLE(id UUID, action TEXT) AS $
BEGIN
RETURN QUERY
WITH conflicts AS (
SELECT
COALESCE(p.id, s.id) as id,
p.version as primary_version,
s.version as secondary_version,
p.updated_at as primary_updated,
s.updated_at as secondary_updated
FROM orders p
FULL OUTER JOIN orders_replica s ON p.id = s.id
WHERE p.region = primary_region
AND s.region = secondary_region
AND (p.version != s.version OR p.id IS NULL OR s.id IS NULL)
)
SELECT
id,
CASE
WHEN primary_version IS NULL THEN 'INSERT_PRIMARY'
WHEN secondary_version IS NULL THEN 'INSERT_SECONDARY'
WHEN secondary_updated > primary_updated THEN 'UPDATE_PRIMARY'
ELSE 'UPDATE_SECONDARY'
END as action
FROM conflicts;
END;
$ LANGUAGE plpgsql;
AI Coding Guidance/Prompt
Prompt: "When designing schemas for replicated databases:"
Rules:
- Warn against auto-incrementing IDs in multi-master setups
- Require timestamp and version columns for conflict detection
- Flag unique constraints that might conflict across regions
- Suggest partition-tolerant ID generation strategies
- Require explicit conflict resolution strategy
Example:
# Bad: Replication-unsafe design
CREATE TABLE user_sessions (
id SERIAL PRIMARY KEY,
user_id INTEGER,
session_data TEXT,
created_at TIMESTAMP DEFAULT NOW()
);
# Good: Replication-aware design
CREATE TABLE user_sessions (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
user_id UUID NOT NULL,
session_data JSONB,
version INTEGER DEFAULT 1,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
region VARCHAR(20) NOT NULL,
-- Composite unique constraint includes region
UNIQUE(user_id, region, created_at)
);
-- Function for conflict resolution
CREATE OR REPLACE FUNCTION resolve_session_conflict(
local_row user_sessions,
remote_row user_sessions
) RETURNS user_sessions AS $
BEGIN
-- Last-write-wins with version check
IF remote_row.version > local_row.version OR
(remote_row.version = local_row.version AND
remote_row.updated_at > local_row.updated_at) THEN
RETURN remote_row;
ELSE
RETURN local_row;
END IF;
END;
$ LANGUAGE plpgsql;
Relevant Keywords
replication unsafe schema design Symptoms: slow queries, data inconsistency, constraint violations Preventive: schema validation, constraint enforcement, proper typing Tech stack: PostgreSQL, MySQL, SQL Server, Oracle Industry: all industries, enterprise, SaaS