mars/MARS_Packages/REL02_POST/MARS-1409/README.md

MARS-1409: Add A_WORKFLOW_HISTORY_KEY to A_SOURCE_FILE_RECEIVED

Overview

Package for adding A_WORKFLOW_HISTORY_KEY column to A_SOURCE_FILE_RECEIVED table and updating FILE_MANAGER package to populate this value during file registration.

Purpose

Direct tracking of workflow history keys in file registration for improved diagnostics and simplified archival queries.

Structure

MARS-1409/
├── .gitignore
├── install_mars1409.sql                    # Master installation script (8 steps)
├── rollback_mars1409.sql                   # Master rollback script (5 steps)
├── verify_packages_version.sql              # Version verification
├── track_package_versions.sql               # Version tracking
├── 01_MARS_1409_add_workflow_history_key_column.sql
├── 01A_MARS_1409_update_existing_workflow_keys.sql  # Update existing records
├── 01B_MARS_1409_install_CT_MRDS_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.3.0 spec
├── 01C_MARS_1409_install_CT_MRDS_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.3.0 body
├── 02_MARS_1409_install_CT_MRDS_FILE_MANAGER_SPEC.sql
├── 03_MARS_1409_install_CT_MRDS_FILE_MANAGER_BODY.sql
├── 04_MARS_1409_verify_installation.sql
├── 90_MARS_1409_verify_rollback.sql
├── 91_MARS_1409_rollback_workflow_history_key_column.sql
├── 91A_MARS_1409_rollback_existing_workflow_keys.sql  # Clear existing values
├── 92_MARS_1409_rollback_FILE_MANAGER_SPEC.sql
├── 92A_MARS_1409_rollback_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.2.0 spec
├── 92B_MARS_1409_rollback_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.2.0 body
├── 93_MARS_1409_rollback_FILE_MANAGER_BODY.sql
├── new_version/                             # Updated packages
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Updated table definition
│   ├── ENV_MANAGER.pkg                      # v3.3.0
│   ├── ENV_MANAGER.pkb                      # v3.3.0
│   ├── FILE_MANAGER.pkg                     # v3.6.0
│   ├── FILE_MANAGER.pkb                     # v3.6.0
│   └── FILE_ARCHIVER.pkb                    # Current version
├── rollback_version/                        # Previous versions
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Original table definition
│   ├── ENV_MANAGER.pkg                      # v3.2.0
│   ├── ENV_MANAGER.pkb                      # v3.2.0
│   ├── FILE_MANAGER.pkg                     # v3.5.1
│   ├── FILE_MANAGER.pkb                     # v3.5.1
│   └── FILE_ARCHIVER.pkb                    # Previous version
└── log/                                     # Installation logs

Status

TESTED & VERIFIED - Installation and rollback validated in DEV environment (2026-02-27)

• ✅ Installation: SUCCESS (8 steps)
• ✅ Rollback: SUCCESS (5 steps)
• ✅ Package compilation: ALL VALID
• ✅ Version tracking: Working correctly
• ⚠️ Prerequisite: MARS-828 column rename must be applied first

Implementation Details

Database Changes

• Added A_WORKFLOW_HISTORY_KEY NUMBER column to CT_MRDS.A_SOURCE_FILE_RECEIVED
• No FK constraint (workflow history record created later in processing)
• Column populated during VALIDATE_SOURCE_FILE_RECEIVED procedure
• Migration script (01A): Updates A_WORKFLOW_HISTORY_KEY for existing records by extracting values from ODS tables

Package Changes

• ENV_MANAGER v3.3.0: Added error codes CODE_WORKFLOW_KEY_NULL (-20035) and CODE_MULTIPLE_WORKFLOW_KEYS (-20036)
• FILE_MANAGER v3.6.0: Enhanced VALIDATE_SOURCE_FILE_RECEIVED to extract and validate A_WORKFLOW_HISTORY_KEY from external tables

Validation Rules

• NULL values: Fatal error - file must contain A_WORKFLOW_HISTORY_KEY
• Multiple values: Fatal error - each file must have exactly one workflow execution key
• Single value: Value extracted and stored in A_SOURCE_FILE_RECEIVED

Migration of Existing Data (01A Script)

The installation includes automatic migration of A_WORKFLOW_HISTORY_KEY for existing records:

• Scope: Updates records with status VALIDATED, READY_FOR_INGESTION, INGESTED, ARCHIVED*
• Method: Extracts A_WORKFLOW_HISTORY_KEY from ODS tables by matching file$name with SOURCE_FILE_NAME
• Safety: Uses TRY-CATCH for each configuration - continues if ODS table doesn't exist
• Logging: Detailed output showing success/failure for each configuration
• Expected behavior: Some records may remain NULL if:
  • Files not yet ingested into ODS tables
  • Files with status RECEIVED or VALIDATION_FAILED
  • ODS tables don't exist or have different structure
• These NULL records will be populated when files are reprocessed

Next Steps

1. Test installation in DEV environment:

   @install_mars1409.sql
   

2. Review migration results: Check how many existing records were updated
3. Validate new files: Test with sample files containing A_WORKFLOW_HISTORY_KEY
4. Test rollback procedure to ensure clean restoration
5. Deploy to higher environments after successful DEV validation

Installation Flow

install_mars1409.sql (MASTER - 8 steps)
  │
  ├─ 01: Add A_WORKFLOW_HISTORY_KEY column (DDL)
  ├─ 01A: Update existing records (Migration)
  ├─ 01B: Install ENV_MANAGER.pkg (v3.3.0)
  ├─ 01C: Install ENV_MANAGER.pkb (v3.3.0)
  ├─ 02: Install FILE_MANAGER.pkg (v3.6.0)
  ├─ 03: Install FILE_MANAGER.pkb (v3.6.0)
  ├─ 04: Verify installation
  └─ 05: Track package versions

Rollback Flow

rollback_mars1409.sql (MASTER - 5 steps)
  │
  ├─ 01: Restore FILE_MANAGER.pkb (v3.5.1)
  ├─ 02: Restore FILE_MANAGER.pkg (v3.5.1)
  ├─ 02A: Restore ENV_MANAGER.pkb (v3.2.0)
  ├─ 02B: Restore ENV_MANAGER.pkg (v3.2.0)
  ├─ 03: Clear A_WORKFLOW_HISTORY_KEY values
  ├─ 04: Drop A_WORKFLOW_HISTORY_KEY column
  └─ 05: Verify rollback

Post-Installation Verification

Check migration results:

-- Count updated records
SELECT 
    CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END as STATUS,
    COUNT(*) as RECORD_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
GROUP BY CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END;

-- Check by configuration
SELECT 
    sfc.A_SOURCE_KEY,
    sfc.SOURCE_FILE_ID,
    sfc.TABLE_ID,
    COUNT(*) as TOTAL_FILES,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 1 ELSE 0 END) as POPULATED,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NULL THEN 1 ELSE 0 END) as NULL_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
GROUP BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID, sfc.TABLE_ID
ORDER BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID;

Test new file processing:

-- Process a test file (will populate A_WORKFLOW_HISTORY_KEY automatically)
EXEC FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/TEST_FILE/TEST_TABLE/test_file.csv');

-- Verify A_WORKFLOW_HISTORY_KEY was populated
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, A_WORKFLOW_HISTORY_KEY, PROCESSING_STATUS
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE SOURCE_FILE_NAME LIKE '%test_file.csv%';

Related Tickets

• Based on MARS-828 package structure
• Supports FILE_ARCHIVER workflow tracking improvements

Test Results (2026-02-27)

✅ Installation Test

Environment: DEV (ggmichalski_high)
Status: SUCCESS

Installation Steps:

1. Step 1: Add A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
2. Step 1A: Migrate existing records - ✅ SUCCESS (0 updated, 47 NULL expected)
3. Step 1B: Install ENV_MANAGER v3.3.0 specification - ✅ SUCCESS
4. Step 1C: Install ENV_MANAGER v3.3.0 body - ✅ SUCCESS
5. Step 2: Install FILE_MANAGER v3.6.0 specification - ✅ SUCCESS
6. Step 3: Install FILE_MANAGER v3.6.0 body - ✅ SUCCESS
7. Step 4: Verification - ✅ SUCCESS
8. Step 5: Version tracking - ✅ SUCCESS

Post-Installation State:

• FILE_MANAGER version: 3.6.0
• ENV_MANAGER version: 3.3.0
• A_WORKFLOW_HISTORY_KEY column: EXISTS (NUMBER, NULLABLE)
• Package compilation: ALL VALID
• Migration results: 0 records updated (expected - no data in ODS tables)

✅ Rollback Test

Status: SUCCESS

Rollback Steps:

1. Step 1: Restore FILE_MANAGER body v3.5.1 - ✅ SUCCESS
2. Step 2: Restore FILE_MANAGER spec v3.5.1 - ✅ SUCCESS
3. Step 2A: Restore ENV_MANAGER body v3.2.0 - ✅ SUCCESS
4. Step 2B: Restore ENV_MANAGER spec v3.2.0 - ✅ SUCCESS
5. Step 3: Clear A_WORKFLOW_HISTORY_KEY values - ✅ SUCCESS (0 cleared, 47 already NULL)
6. Step 4: Drop A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
7. Step 5: Verification - ✅ SUCCESS

Post-Rollback State:

• FILE_MANAGER version: 3.5.1 (restored)
• ENV_MANAGER version: 3.2.0 (restored)
• A_WORKFLOW_HISTORY_KEY column: REMOVED
• Package compilation: ALL VALID

Critical Findings:

• ⚠️ Prerequisite: MARS-828 column rename must be applied first (ARCHIVE_THRESHOLD_DAYS)
• ⚠️ Database State: rollback_version packages require MARS-828 naming conventions
• ✅ Solution: Applied MARS-828 01a script before testing - now works correctly

Package Ready for:

• ✅ DEV deployment (tested successfully)
• ✅ QA deployment (after DEV validation)
• ⏳ PROD deployment (pending higher environment validation)