Package for adding A_WORKFLOW_HISTORY_KEY column to A_SOURCE_FILE_RECEIVED table and updating FILE_MANAGER package to populate this value during file registration.

Purpose

Direct tracking of workflow history keys in file registration for improved diagnostics and simplified archival queries.

Structure

MARS-1409/
├── .gitignore
├── install_mars1409.sql                    # Master installation script (8 steps)
├── rollback_mars1409.sql                   # Master rollback script (5 steps)
├── verify_packages_version.sql              # Version verification
├── track_package_versions.sql               # Version tracking
├── 01_MARS_1409_add_workflow_history_key_column.sql
├── 01A_MARS_1409_update_existing_workflow_keys.sql  # Update existing records
├── 01B_MARS_1409_install_CT_MRDS_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.3.0 spec
├── 01C_MARS_1409_install_CT_MRDS_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.3.0 body
├── 02_MARS_1409_install_CT_MRDS_FILE_MANAGER_SPEC.sql
├── 03_MARS_1409_install_CT_MRDS_FILE_MANAGER_BODY.sql
├── 04_MARS_1409_verify_installation.sql
├── 90_MARS_1409_verify_rollback.sql
├── 91_MARS_1409_rollback_workflow_history_key_column.sql
├── 91A_MARS_1409_rollback_existing_workflow_keys.sql  # Clear existing values
├── 92_MARS_1409_rollback_FILE_MANAGER_SPEC.sql
├── 92A_MARS_1409_rollback_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.2.0 spec
├── 92B_MARS_1409_rollback_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.2.0 body
├── 93_MARS_1409_rollback_FILE_MANAGER_BODY.sql
├── new_version/                             # Updated packages
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Updated table definition
│   ├── ENV_MANAGER.pkg                      # v3.3.0
│   ├── ENV_MANAGER.pkb                      # v3.3.0
│   ├── FILE_MANAGER.pkg                     # v3.6.0
│   ├── FILE_MANAGER.pkb                     # v3.6.0
│   └── FILE_ARCHIVER.pkb                    # Current version
├── rollback_version/                        # Previous versions
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Original table definition
│   ├── ENV_MANAGER.pkg                      # v3.2.0
│   ├── ENV_MANAGER.pkb                      # v3.2.0
│   ├── FILE_MANAGER.pkg                     # v3.5.1
│   ├── FILE_MANAGER.pkb                     # v3.5.1
│   └── FILE_ARCHIVER.pkb                    # Previous version
└── log/                                     # Installation logs

Status

TESTED & VERIFIED - Installation and rollback validated in DEV environment (2026-02-27)

✅ Installation: SUCCESS (8 steps)
✅ Rollback: SUCCESS (5 steps)
✅ Package compilation: ALL VALID
✅ Version tracking: Working correctly
⚠️ Prerequisite: MARS-828 column rename must be applied first

Implementation Details

Database Changes

Added A_WORKFLOW_HISTORY_KEY NUMBER column to CT_MRDS.A_SOURCE_FILE_RECEIVED
No FK constraint (workflow history record created later in processing)
Column populated during VALIDATE_SOURCE_FILE_RECEIVED procedure
Migration script (01A): Updates A_WORKFLOW_HISTORY_KEY for existing records by extracting values from ODS tables

Package Changes

ENV_MANAGER v3.3.0: Added error codes CODE_WORKFLOW_KEY_NULL (-20035) and CODE_MULTIPLE_WORKFLOW_KEYS (-20036)
FILE_MANAGER v3.6.0: Enhanced VALIDATE_SOURCE_FILE_RECEIVED to extract and validate A_WORKFLOW_HISTORY_KEY from external tables

Validation Rules

NULL values: Fatal error - file must contain A_WORKFLOW_HISTORY_KEY
Multiple values: Fatal error - each file must have exactly one workflow execution key
Single value: Value extracted and stored in A_SOURCE_FILE_RECEIVED

Migration of Existing Data (01A Script)

The installation includes automatic migration of A_WORKFLOW_HISTORY_KEY for existing records:

Scope: Updates records with status VALIDATED, READY_FOR_INGESTION, INGESTED, ARCHIVED*
Method: Extracts A_WORKFLOW_HISTORY_KEY from ODS tables by matching file$name with SOURCE_FILE_NAME
Safety: Uses TRY-CATCH for each configuration - continues if ODS table doesn't exist
Logging: Detailed output showing success/failure for each configuration
Expected behavior: Some records may remain NULL if:
- Files not yet ingested into ODS tables
- Files with status RECEIVED or VALIDATION_FAILED
- ODS tables don't exist or have different structure
These NULL records will be populated when files are reprocessed

Next Steps

Test installation in DEV environment:
```
@install_mars1409.sql
```
Review migration results: Check how many existing records were updated
Validate new files: Test with sample files containing A_WORKFLOW_HISTORY_KEY
Test rollback procedure to ensure clean restoration
Deploy to higher environments after successful DEV validation

Installation Flow

install_mars1409.sql (MASTER - 8 steps)
  │
  ├─ 01: Add A_WORKFLOW_HISTORY_KEY column (DDL)
  ├─ 01A: Update existing records (Migration)
  ├─ 01B: Install ENV_MANAGER.pkg (v3.3.0)
  ├─ 01C: Install ENV_MANAGER.pkb (v3.3.0)
  ├─ 02: Install FILE_MANAGER.pkg (v3.6.0)
  ├─ 03: Install FILE_MANAGER.pkb (v3.6.0)
  ├─ 04: Verify installation
  └─ 05: Track package versions

Rollback Flow

rollback_mars1409.sql (MASTER - 5 steps)
  │
  ├─ 01: Restore FILE_MANAGER.pkb (v3.5.1)
  ├─ 02: Restore FILE_MANAGER.pkg (v3.5.1)
  ├─ 02A: Restore ENV_MANAGER.pkb (v3.2.0)
  ├─ 02B: Restore ENV_MANAGER.pkg (v3.2.0)
  ├─ 03: Clear A_WORKFLOW_HISTORY_KEY values
  ├─ 04: Drop A_WORKFLOW_HISTORY_KEY column
  └─ 05: Verify rollback

Post-Installation Verification

Check migration results:

-- Count updated records
SELECT 
    CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END as STATUS,
    COUNT(*) as RECORD_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
GROUP BY CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END;

-- Check by configuration
SELECT 
    sfc.A_SOURCE_KEY,
    sfc.SOURCE_FILE_ID,
    sfc.TABLE_ID,
    COUNT(*) as TOTAL_FILES,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 1 ELSE 0 END) as POPULATED,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NULL THEN 1 ELSE 0 END) as NULL_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
GROUP BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID, sfc.TABLE_ID
ORDER BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID;

Test new file processing:

-- Process a test file (will populate A_WORKFLOW_HISTORY_KEY automatically)
EXEC FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/TEST_FILE/TEST_TABLE/test_file.csv');

-- Verify A_WORKFLOW_HISTORY_KEY was populated
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, A_WORKFLOW_HISTORY_KEY, PROCESSING_STATUS
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE SOURCE_FILE_NAME LIKE '%test_file.csv%';

Based on MARS-828 package structure
Supports FILE_ARCHIVER workflow tracking improvements

Test Results (2026-02-27)

✅ Installation Test

Environment: DEV (ggmichalski_high)
Status: SUCCESS

Installation Steps:

Step 1: Add A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
Step 1A: Migrate existing records - ✅ SUCCESS (0 updated, 47 NULL expected)
Step 1B: Install ENV_MANAGER v3.3.0 specification - ✅ SUCCESS
Step 1C: Install ENV_MANAGER v3.3.0 body - ✅ SUCCESS
Step 2: Install FILE_MANAGER v3.6.0 specification - ✅ SUCCESS
Step 3: Install FILE_MANAGER v3.6.0 body - ✅ SUCCESS
Step 4: Verification - ✅ SUCCESS
Step 5: Version tracking - ✅ SUCCESS

Post-Installation State:

FILE_MANAGER version: 3.6.0
ENV_MANAGER version: 3.3.0
A_WORKFLOW_HISTORY_KEY column: EXISTS (NUMBER, NULLABLE)
Package compilation: ALL VALID
Migration results: 0 records updated (expected - no data in ODS tables)

✅ Rollback Test

Status: SUCCESS

Rollback Steps:

Step 1: Restore FILE_MANAGER body v3.5.1 - ✅ SUCCESS
Step 2: Restore FILE_MANAGER spec v3.5.1 - ✅ SUCCESS
Step 2A: Restore ENV_MANAGER body v3.2.0 - ✅ SUCCESS
Step 2B: Restore ENV_MANAGER spec v3.2.0 - ✅ SUCCESS
Step 3: Clear A_WORKFLOW_HISTORY_KEY values - ✅ SUCCESS (0 cleared, 47 already NULL)
Step 4: Drop A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
Step 5: Verification - ✅ SUCCESS

Post-Rollback State:

FILE_MANAGER version: 3.5.1 (restored)
ENV_MANAGER version: 3.2.0 (restored)
A_WORKFLOW_HISTORY_KEY column: REMOVED
Package compilation: ALL VALID

Critical Findings:

⚠️ Prerequisite: MARS-828 column rename must be applied first (ARCHIVE_THRESHOLD_DAYS)
⚠️ Database State: rollback_version packages require MARS-828 naming conventions
✅ Solution: Applied MARS-828 01a script before testing - now works correctly

Package Ready for:

✅ DEV deployment (tested successfully)
✅ QA deployment (after DEV validation)
⏳ PROD deployment (pending higher environment validation)

README.md

MARS-1409: Add A_WORKFLOW_HISTORY_KEY to A_SOURCE_FILE_RECEIVED

Overview