# MARS-1409: Add A_WORKFLOW_HISTORY_KEY to A_SOURCE_FILE_RECEIVED

## Overview
Package for adding A_WORKFLOW_HISTORY_KEY column to A_SOURCE_FILE_RECEIVED table and updating FILE_MANAGER package to populate this value during file registration.

## Purpose
Direct tracking of workflow history keys in file registration for improved diagnostics and simplified archival queries.

## Structure
```
MARS-1409/
├── .gitignore
├── install_mars1409.sql                    # Master installation script (8 steps)
├── rollback_mars1409.sql                   # Master rollback script (5 steps)
├── verify_packages_version.sql              # Version verification
├── track_package_versions.sql               # Version tracking
├── 01_MARS_1409_add_workflow_history_key_column.sql
├── 01A_MARS_1409_update_existing_workflow_keys.sql  # Update existing records
├── 01B_MARS_1409_install_CT_MRDS_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.3.0 spec
├── 01C_MARS_1409_install_CT_MRDS_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.3.0 body
├── 02_MARS_1409_install_CT_MRDS_FILE_MANAGER_SPEC.sql
├── 03_MARS_1409_install_CT_MRDS_FILE_MANAGER_BODY.sql
├── 04_MARS_1409_verify_installation.sql
├── 90_MARS_1409_verify_rollback.sql
├── 91_MARS_1409_rollback_workflow_history_key_column.sql
├── 91A_MARS_1409_rollback_existing_workflow_keys.sql  # Clear existing values
├── 92_MARS_1409_rollback_FILE_MANAGER_SPEC.sql
├── 92A_MARS_1409_rollback_ENV_MANAGER_SPEC.sql  # ENV_MANAGER v3.2.0 spec
├── 92B_MARS_1409_rollback_ENV_MANAGER_BODY.sql  # ENV_MANAGER v3.2.0 body
├── 93_MARS_1409_rollback_FILE_MANAGER_BODY.sql
├── new_version/                             # Updated packages
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Updated table definition
│   ├── ENV_MANAGER.pkg                      # v3.3.0
│   ├── ENV_MANAGER.pkb                      # v3.3.0
│   ├── FILE_MANAGER.pkg                     # v3.6.0
│   ├── FILE_MANAGER.pkb                     # v3.6.0
│   └── FILE_ARCHIVER.pkb                    # Current version
├── rollback_version/                        # Previous versions
│   ├── A_SOURCE_FILE_RECEIVED.sql           # Original table definition
│   ├── ENV_MANAGER.pkg                      # v3.2.0
│   ├── ENV_MANAGER.pkb                      # v3.2.0
│   ├── FILE_MANAGER.pkg                     # v3.5.1
│   ├── FILE_MANAGER.pkb                     # v3.5.1
│   └── FILE_ARCHIVER.pkb                    # Previous version
└── log/                                     # Installation logs
```

## Status
**TESTED & VERIFIED** - Installation and rollback validated in DEV environment (2026-02-27)
- ✅ Installation: SUCCESS (8 steps)
- ✅ Rollback: SUCCESS (5 steps)
- ✅ Package compilation: ALL VALID
- ✅ Version tracking: Working correctly
- ⚠️ Prerequisite: MARS-828 column rename must be applied first

## Implementation Details

### Database Changes
- Added `A_WORKFLOW_HISTORY_KEY NUMBER` column to `CT_MRDS.A_SOURCE_FILE_RECEIVED`
- No FK constraint (workflow history record created later in processing)
- Column populated during VALIDATE_SOURCE_FILE_RECEIVED procedure
- **Migration script** (01A): Updates A_WORKFLOW_HISTORY_KEY for existing records by extracting values from ODS tables

### Package Changes
- **ENV_MANAGER v3.3.0**: Added error codes CODE_WORKFLOW_KEY_NULL (-20035) and CODE_MULTIPLE_WORKFLOW_KEYS (-20036)
- **FILE_MANAGER v3.6.0**: Enhanced VALIDATE_SOURCE_FILE_RECEIVED to extract and validate A_WORKFLOW_HISTORY_KEY from external tables

### Validation Rules
- **NULL values**: Fatal error - file must contain A_WORKFLOW_HISTORY_KEY
- **Multiple values**: Fatal error - each file must have exactly one workflow execution key
- **Single value**: Value extracted and stored in A_SOURCE_FILE_RECEIVED

### Migration of Existing Data (01A Script)
The installation includes automatic migration of A_WORKFLOW_HISTORY_KEY for existing records:
- **Scope**: Updates records with status VALIDATED, READY_FOR_INGESTION, INGESTED, ARCHIVED*
- **Method**: Extracts A_WORKFLOW_HISTORY_KEY from ODS tables by matching file$name with SOURCE_FILE_NAME
- **Safety**: Uses TRY-CATCH for each configuration - continues if ODS table doesn't exist
- **Logging**: Detailed output showing success/failure for each configuration
- **Expected behavior**: Some records may remain NULL if:
  - Files not yet ingested into ODS tables
  - Files with status RECEIVED or VALIDATION_FAILED
  - ODS tables don't exist or have different structure
- These NULL records will be populated when files are reprocessed

## Next Steps
1. **Test installation** in DEV environment:
   ```sql
   @install_mars1409.sql
   ```
2. **Review migration results**: Check how many existing records were updated
3. **Validate new files**: Test with sample files containing A_WORKFLOW_HISTORY_KEY
4. **Test rollback** procedure to ensure clean restoration
5. **Deploy to higher environments** after successful DEV validation

## Installation Flow
```
install_mars1409.sql (MASTER - 8 steps)
  │
  ├─ 01: Add A_WORKFLOW_HISTORY_KEY column (DDL)
  ├─ 01A: Update existing records (Migration)
  ├─ 01B: Install ENV_MANAGER.pkg (v3.3.0)
  ├─ 01C: Install ENV_MANAGER.pkb (v3.3.0)
  ├─ 02: Install FILE_MANAGER.pkg (v3.6.0)
  ├─ 03: Install FILE_MANAGER.pkb (v3.6.0)
  ├─ 04: Verify installation
  └─ 05: Track package versions
```

## Rollback Flow
```
rollback_mars1409.sql (MASTER - 5 steps)
  │
  ├─ 01: Restore FILE_MANAGER.pkb (v3.5.1)
  ├─ 02: Restore FILE_MANAGER.pkg (v3.5.1)
  ├─ 02A: Restore ENV_MANAGER.pkb (v3.2.0)
  ├─ 02B: Restore ENV_MANAGER.pkg (v3.2.0)
  ├─ 03: Clear A_WORKFLOW_HISTORY_KEY values
  ├─ 04: Drop A_WORKFLOW_HISTORY_KEY column
  └─ 05: Verify rollback
```

## Post-Installation Verification

### Check migration results:
```sql
-- Count updated records
SELECT 
    CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END as STATUS,
    COUNT(*) as RECORD_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
GROUP BY CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END;

-- Check by configuration
SELECT 
    sfc.A_SOURCE_KEY,
    sfc.SOURCE_FILE_ID,
    sfc.TABLE_ID,
    COUNT(*) as TOTAL_FILES,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 1 ELSE 0 END) as POPULATED,
    SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NULL THEN 1 ELSE 0 END) as NULL_COUNT
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
GROUP BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID, sfc.TABLE_ID
ORDER BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID;
```

### Test new file processing:
```sql
-- Process a test file (will populate A_WORKFLOW_HISTORY_KEY automatically)
EXEC FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/TEST_FILE/TEST_TABLE/test_file.csv');

-- Verify A_WORKFLOW_HISTORY_KEY was populated
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, A_WORKFLOW_HISTORY_KEY, PROCESSING_STATUS
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE SOURCE_FILE_NAME LIKE '%test_file.csv%';
```

## Related Tickets
- Based on MARS-828 package structure
- Supports FILE_ARCHIVER workflow tracking improvements
## Test Results (2026-02-27)

### ✅ Installation Test
**Environment**: DEV (ggmichalski_high)  
**Status**: SUCCESS

**Installation Steps**:
1. Step 1: Add A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
2. Step 1A: Migrate existing records - ✅ SUCCESS (0 updated, 47 NULL expected)
3. Step 1B: Install ENV_MANAGER v3.3.0 specification - ✅ SUCCESS
4. Step 1C: Install ENV_MANAGER v3.3.0 body - ✅ SUCCESS
5. Step 2: Install FILE_MANAGER v3.6.0 specification - ✅ SUCCESS
6. Step 3: Install FILE_MANAGER v3.6.0 body - ✅ SUCCESS
7. Step 4: Verification - ✅ SUCCESS
8. Step 5: Version tracking - ✅ SUCCESS

**Post-Installation State**:
- FILE_MANAGER version: 3.6.0
- ENV_MANAGER version: 3.3.0
- A_WORKFLOW_HISTORY_KEY column: EXISTS (NUMBER, NULLABLE)
- Package compilation: ALL VALID
- Migration results: 0 records updated (expected - no data in ODS tables)

### ✅ Rollback Test
**Status**: SUCCESS

**Rollback Steps**:
1. Step 1: Restore FILE_MANAGER body v3.5.1 - ✅ SUCCESS
2. Step 2: Restore FILE_MANAGER spec v3.5.1 - ✅ SUCCESS
3. Step 2A: Restore ENV_MANAGER body v3.2.0 - ✅ SUCCESS
4. Step 2B: Restore ENV_MANAGER spec v3.2.0 - ✅ SUCCESS
5. Step 3: Clear A_WORKFLOW_HISTORY_KEY values - ✅ SUCCESS (0 cleared, 47 already NULL)
6. Step 4: Drop A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
7. Step 5: Verification - ✅ SUCCESS

**Post-Rollback State**:
- FILE_MANAGER version: 3.5.1 (restored)
- ENV_MANAGER version: 3.2.0 (restored)
- A_WORKFLOW_HISTORY_KEY column: REMOVED
- Package compilation: ALL VALID

**Critical Findings**:
- ⚠️ **Prerequisite**: MARS-828 column rename must be applied first (ARCHIVE_THRESHOLD_DAYS)
- ⚠️ **Database State**: rollback_version packages require MARS-828 naming conventions
- ✅ **Solution**: Applied MARS-828 01a script before testing - now works correctly

**Package Ready for**:
- ✅ DEV deployment (tested successfully)
- ✅ QA deployment (after DEV validation)
- ⏳ PROD deployment (pending higher environment validation)