210 lines
9.2 KiB
Markdown
210 lines
9.2 KiB
Markdown
# MARS-1409: Add A_WORKFLOW_HISTORY_KEY to A_SOURCE_FILE_RECEIVED
|
|
|
|
## Overview
|
|
Package for adding A_WORKFLOW_HISTORY_KEY column to A_SOURCE_FILE_RECEIVED table and updating FILE_MANAGER package to populate this value during file registration.
|
|
|
|
## Purpose
|
|
Direct tracking of workflow history keys in file registration for improved diagnostics and simplified archival queries.
|
|
|
|
## Structure
|
|
```
|
|
MARS-1409/
|
|
├── .gitignore
|
|
├── install_mars1409.sql # Master installation script (8 steps)
|
|
├── rollback_mars1409.sql # Master rollback script (5 steps)
|
|
├── verify_packages_version.sql # Version verification
|
|
├── track_package_versions.sql # Version tracking
|
|
├── 01_MARS_1409_add_workflow_history_key_column.sql
|
|
├── 01A_MARS_1409_update_existing_workflow_keys.sql # Update existing records
|
|
├── 01B_MARS_1409_install_CT_MRDS_ENV_MANAGER_SPEC.sql # ENV_MANAGER v3.3.0 spec
|
|
├── 01C_MARS_1409_install_CT_MRDS_ENV_MANAGER_BODY.sql # ENV_MANAGER v3.3.0 body
|
|
├── 02_MARS_1409_install_CT_MRDS_FILE_MANAGER_SPEC.sql
|
|
├── 03_MARS_1409_install_CT_MRDS_FILE_MANAGER_BODY.sql
|
|
├── 04_MARS_1409_verify_installation.sql
|
|
├── 90_MARS_1409_verify_rollback.sql
|
|
├── 91_MARS_1409_rollback_workflow_history_key_column.sql
|
|
├── 91A_MARS_1409_rollback_existing_workflow_keys.sql # Clear existing values
|
|
├── 92_MARS_1409_rollback_FILE_MANAGER_SPEC.sql
|
|
├── 92A_MARS_1409_rollback_ENV_MANAGER_SPEC.sql # ENV_MANAGER v3.2.0 spec
|
|
├── 92B_MARS_1409_rollback_ENV_MANAGER_BODY.sql # ENV_MANAGER v3.2.0 body
|
|
├── 93_MARS_1409_rollback_FILE_MANAGER_BODY.sql
|
|
├── new_version/ # Updated packages
|
|
│ ├── A_SOURCE_FILE_RECEIVED.sql # Updated table definition
|
|
│ ├── ENV_MANAGER.pkg # v3.3.0
|
|
│ ├── ENV_MANAGER.pkb # v3.3.0
|
|
│ ├── FILE_MANAGER.pkg # v3.6.0
|
|
│ ├── FILE_MANAGER.pkb # v3.6.0
|
|
│ └── FILE_ARCHIVER.pkb # Current version
|
|
├── rollback_version/ # Previous versions
|
|
│ ├── A_SOURCE_FILE_RECEIVED.sql # Original table definition
|
|
│ ├── ENV_MANAGER.pkg # v3.2.0
|
|
│ ├── ENV_MANAGER.pkb # v3.2.0
|
|
│ ├── FILE_MANAGER.pkg # v3.5.1
|
|
│ ├── FILE_MANAGER.pkb # v3.5.1
|
|
│ └── FILE_ARCHIVER.pkb # Previous version
|
|
└── log/ # Installation logs
|
|
```
|
|
|
|
## Status
|
|
**TESTED & VERIFIED** - Installation and rollback validated in DEV environment (2026-02-27)
|
|
- ✅ Installation: SUCCESS (8 steps)
|
|
- ✅ Rollback: SUCCESS (5 steps)
|
|
- ✅ Package compilation: ALL VALID
|
|
- ✅ Version tracking: Working correctly
|
|
- ⚠️ Prerequisite: MARS-828 column rename must be applied first
|
|
|
|
## Implementation Details
|
|
|
|
### Database Changes
|
|
- Added `A_WORKFLOW_HISTORY_KEY NUMBER` column to `CT_MRDS.A_SOURCE_FILE_RECEIVED`
|
|
- No FK constraint (workflow history record created later in processing)
|
|
- Column populated during VALIDATE_SOURCE_FILE_RECEIVED procedure
|
|
- **Migration script** (01A): Updates A_WORKFLOW_HISTORY_KEY for existing records by extracting values from ODS tables
|
|
|
|
### Package Changes
|
|
- **ENV_MANAGER v3.3.0**: Added error codes CODE_WORKFLOW_KEY_NULL (-20035) and CODE_MULTIPLE_WORKFLOW_KEYS (-20036)
|
|
- **FILE_MANAGER v3.6.0**: Enhanced VALIDATE_SOURCE_FILE_RECEIVED to extract and validate A_WORKFLOW_HISTORY_KEY from external tables
|
|
|
|
### Validation Rules
|
|
- **NULL values**: Fatal error - file must contain A_WORKFLOW_HISTORY_KEY
|
|
- **Multiple values**: Fatal error - each file must have exactly one workflow execution key
|
|
- **Single value**: Value extracted and stored in A_SOURCE_FILE_RECEIVED
|
|
|
|
### Migration of Existing Data (01A Script)
|
|
The installation includes automatic migration of A_WORKFLOW_HISTORY_KEY for existing records:
|
|
- **Scope**: Updates records with status VALIDATED, READY_FOR_INGESTION, INGESTED, ARCHIVED*
|
|
- **Method**: Extracts A_WORKFLOW_HISTORY_KEY from ODS tables by matching file$name with SOURCE_FILE_NAME
|
|
- **Safety**: Uses TRY-CATCH for each configuration - continues if ODS table doesn't exist
|
|
- **Logging**: Detailed output showing success/failure for each configuration
|
|
- **Expected behavior**: Some records may remain NULL if:
|
|
- Files not yet ingested into ODS tables
|
|
- Files with status RECEIVED or VALIDATION_FAILED
|
|
- ODS tables don't exist or have different structure
|
|
- These NULL records will be populated when files are reprocessed
|
|
|
|
## Next Steps
|
|
1. **Test installation** in DEV environment:
|
|
```sql
|
|
@install_mars1409.sql
|
|
```
|
|
2. **Review migration results**: Check how many existing records were updated
|
|
3. **Validate new files**: Test with sample files containing A_WORKFLOW_HISTORY_KEY
|
|
4. **Test rollback** procedure to ensure clean restoration
|
|
5. **Deploy to higher environments** after successful DEV validation
|
|
|
|
## Installation Flow
|
|
```
|
|
install_mars1409.sql (MASTER - 8 steps)
|
|
│
|
|
├─ 01: Add A_WORKFLOW_HISTORY_KEY column (DDL)
|
|
├─ 01A: Update existing records (Migration)
|
|
├─ 01B: Install ENV_MANAGER.pkg (v3.3.0)
|
|
├─ 01C: Install ENV_MANAGER.pkb (v3.3.0)
|
|
├─ 02: Install FILE_MANAGER.pkg (v3.6.0)
|
|
├─ 03: Install FILE_MANAGER.pkb (v3.6.0)
|
|
├─ 04: Verify installation
|
|
└─ 05: Track package versions
|
|
```
|
|
|
|
## Rollback Flow
|
|
```
|
|
rollback_mars1409.sql (MASTER - 5 steps)
|
|
│
|
|
├─ 01: Restore FILE_MANAGER.pkb (v3.5.1)
|
|
├─ 02: Restore FILE_MANAGER.pkg (v3.5.1)
|
|
├─ 02A: Restore ENV_MANAGER.pkb (v3.2.0)
|
|
├─ 02B: Restore ENV_MANAGER.pkg (v3.2.0)
|
|
├─ 03: Clear A_WORKFLOW_HISTORY_KEY values
|
|
├─ 04: Drop A_WORKFLOW_HISTORY_KEY column
|
|
└─ 05: Verify rollback
|
|
```
|
|
|
|
## Post-Installation Verification
|
|
|
|
### Check migration results:
|
|
```sql
|
|
-- Count updated records
|
|
SELECT
|
|
CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END as STATUS,
|
|
COUNT(*) as RECORD_COUNT
|
|
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
|
|
GROUP BY CASE WHEN A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 'POPULATED' ELSE 'NULL' END;
|
|
|
|
-- Check by configuration
|
|
SELECT
|
|
sfc.A_SOURCE_KEY,
|
|
sfc.SOURCE_FILE_ID,
|
|
sfc.TABLE_ID,
|
|
COUNT(*) as TOTAL_FILES,
|
|
SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NOT NULL THEN 1 ELSE 0 END) as POPULATED,
|
|
SUM(CASE WHEN sfr.A_WORKFLOW_HISTORY_KEY IS NULL THEN 1 ELSE 0 END) as NULL_COUNT
|
|
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
|
|
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
|
|
GROUP BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID, sfc.TABLE_ID
|
|
ORDER BY sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID;
|
|
```
|
|
|
|
### Test new file processing:
|
|
```sql
|
|
-- Process a test file (will populate A_WORKFLOW_HISTORY_KEY automatically)
|
|
EXEC FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/TEST_FILE/TEST_TABLE/test_file.csv');
|
|
|
|
-- Verify A_WORKFLOW_HISTORY_KEY was populated
|
|
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, A_WORKFLOW_HISTORY_KEY, PROCESSING_STATUS
|
|
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
|
|
WHERE SOURCE_FILE_NAME LIKE '%test_file.csv%';
|
|
```
|
|
|
|
## Related Tickets
|
|
- Based on MARS-828 package structure
|
|
- Supports FILE_ARCHIVER workflow tracking improvements
|
|
## Test Results (2026-02-27)
|
|
|
|
### ✅ Installation Test
|
|
**Environment**: DEV (ggmichalski_high)
|
|
**Status**: SUCCESS
|
|
|
|
**Installation Steps**:
|
|
1. Step 1: Add A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
|
|
2. Step 1A: Migrate existing records - ✅ SUCCESS (0 updated, 47 NULL expected)
|
|
3. Step 1B: Install ENV_MANAGER v3.3.0 specification - ✅ SUCCESS
|
|
4. Step 1C: Install ENV_MANAGER v3.3.0 body - ✅ SUCCESS
|
|
5. Step 2: Install FILE_MANAGER v3.6.0 specification - ✅ SUCCESS
|
|
6. Step 3: Install FILE_MANAGER v3.6.0 body - ✅ SUCCESS
|
|
7. Step 4: Verification - ✅ SUCCESS
|
|
8. Step 5: Version tracking - ✅ SUCCESS
|
|
|
|
**Post-Installation State**:
|
|
- FILE_MANAGER version: 3.6.0
|
|
- ENV_MANAGER version: 3.3.0
|
|
- A_WORKFLOW_HISTORY_KEY column: EXISTS (NUMBER, NULLABLE)
|
|
- Package compilation: ALL VALID
|
|
- Migration results: 0 records updated (expected - no data in ODS tables)
|
|
|
|
### ✅ Rollback Test
|
|
**Status**: SUCCESS
|
|
|
|
**Rollback Steps**:
|
|
1. Step 1: Restore FILE_MANAGER body v3.5.1 - ✅ SUCCESS
|
|
2. Step 2: Restore FILE_MANAGER spec v3.5.1 - ✅ SUCCESS
|
|
3. Step 2A: Restore ENV_MANAGER body v3.2.0 - ✅ SUCCESS
|
|
4. Step 2B: Restore ENV_MANAGER spec v3.2.0 - ✅ SUCCESS
|
|
5. Step 3: Clear A_WORKFLOW_HISTORY_KEY values - ✅ SUCCESS (0 cleared, 47 already NULL)
|
|
6. Step 4: Drop A_WORKFLOW_HISTORY_KEY column - ✅ SUCCESS
|
|
7. Step 5: Verification - ✅ SUCCESS
|
|
|
|
**Post-Rollback State**:
|
|
- FILE_MANAGER version: 3.5.1 (restored)
|
|
- ENV_MANAGER version: 3.2.0 (restored)
|
|
- A_WORKFLOW_HISTORY_KEY column: REMOVED
|
|
- Package compilation: ALL VALID
|
|
|
|
**Critical Findings**:
|
|
- ⚠️ **Prerequisite**: MARS-828 column rename must be applied first (ARCHIVE_THRESHOLD_DAYS)
|
|
- ⚠️ **Database State**: rollback_version packages require MARS-828 naming conventions
|
|
- ✅ **Solution**: Applied MARS-828 01a script before testing - now works correctly
|
|
|
|
**Package Ready for**:
|
|
- ✅ DEV deployment (tested successfully)
|
|
- ✅ QA deployment (after DEV validation)
|
|
- ⏳ PROD deployment (pending higher environment validation) |