mars/MARS_Packages/REL01_ADDITIONS/MARS-826-PREHOOK/README.md

# MARS-826-PREHOOK: DATA_EXPORTER Package Update - Column Rename Support

## Overview
**Purpose**: Update DATA_EXPORTER package to support renamed column in CT_ODS.A_LOAD_HISTORY  
**Type**: Pre-Hook Deployment (Required before MARS-826)  
**Target Schema**: CT_MRDS  
**Database Objects**: DATA_EXPORTER package body

## Background

### Issue
The CT_ODS.A_LOAD_HISTORY table has column `A_ETL_LOAD_SET_KEY` (PRIMARY KEY), but the DATA_EXPORTER package had hardcoded references to the old column name `A_WORKFLOW_HISTORY_KEY`, causing JOIN failures.

### Root Cause
Column name mismatch discovered during MARS-826 testing:
- **Database DDL**: `A_ETL_LOAD_SET_KEY` (correct, as per source definition)
- **DATA_EXPORTER package**: `L.A_WORKFLOW_HISTORY_KEY` (incorrect, outdated reference)
- **Impact**: All export procedures failed with `ORA-00904: "L"."A_WORKFLOW_HISTORY_KEY": invalid identifier`

### Solution
Updated DATA_EXPORTER.pkb to use correct column name `A_ETL_LOAD_SET_KEY` in all JOIN operations with CT_ODS.A_LOAD_HISTORY.

## Changes Made

### Package Version Update
**Previous Version**: 2.1.0  
**New Version**: 2.1.1  
**Change Type**: PATCH (Bug Fix)  
**Build Date**: 2025-12-04 13:10:00

### Modified Files
1. **DATA_EXPORTER.pkg** (Package Specification)
   - Updated PACKAGE_VERSION: '2.1.0' → '2.1.1'
   - Updated PACKAGE_BUILD_DATE: '2025-12-04 13:10:00'
   - Updated VERSION_HISTORY with v2.1.1 entry

2. **DATA_EXPORTER.pkb** (Package Body)
   - Updated 4 JOIN clauses in dynamic SQL
   - Updated example in documentation comment

### Specific Changes

**Location 1**: Line ~326 (EXPORT_TABLE_DATA_BY_DATE - partition query)
```sql
-- BEFORE:
WHERE T.' || ... || ' = L.A_WORKFLOW_HISTORY_KEY

-- AFTER:
WHERE T.' || ... || ' = L.A_ETL_LOAD_SET_KEY
```

**Location 2**: Line ~340 (EXPORT_TABLE_DATA_BY_DATE - export query)
```sql
-- BEFORE:
WHERE T.' || ... || ' = L.A_WORKFLOW_HISTORY_KEY

-- AFTER:
WHERE T.' || ... || ' = L.A_ETL_LOAD_SET_KEY
```

**Location 3**: Line ~613 (EXPORT_TABLE_DATA_TO_CSV_BY_DATE - partition query)
```sql
-- BEFORE:
WHERE T.' || ... || ' = L.A_WORKFLOW_HISTORY_KEY

-- AFTER:
WHERE T.' || ... || ' = L.A_ETL_LOAD_SET_KEY
```

**Location 4**: Line ~629 (EXPORT_TABLE_DATA_TO_CSV_BY_DATE - export query)
```sql
-- BEFORE:
WHERE T.' || ... || ' = L.A_WORKFLOW_HISTORY_KEY

-- AFTER:
WHERE T.' || ... || ' = L.A_ETL_LOAD_SET_KEY
```

**Location 5**: Line ~398 (Documentation example)
```sql
-- BEFORE:
pKeyColumnName  => 'A_WORKFLOW_HISTORY_KEY',

-- AFTER:
pKeyColumnName  => 'A_ETL_LOAD_SET_KEY_FK',
```

## Deployment

### Prerequisites
- Database connection: CT_MRDS schema
- Required privileges: ALTER PACKAGE on CT_MRDS.DATA_EXPORTER
- Estimated time: 1-2 minutes

### Installation Steps

The master installation script performs these operations in sequence:

1. **Deploy Package Specification** - Install updated DATA_EXPORTER.pkg from new_version/ (v2.1.1)
2. **Deploy Package Body** - Install updated DATA_EXPORTER.pkb from new_version/ with column reference fixes
3. **Track Version** - Record package version using universal tracking script (Standard)
4. **Verify Status** - Check all tracked packages for untracked changes using universal verification script (Standard)

**Version Folders:**
- **current_version/** - Contains DATA_EXPORTER v2.1.0 (backup from MARS-846 for rollback)
- **new_version/** - Contains DATA_EXPORTER v2.1.1 (updated with A_ETL_LOAD_SET_KEY fixes)

### Execution
1. Connect to database as CT_MRDS user
2. Execute `install_mars826_prehook.sql`
3. Verify package compilation status
4. Test export functionality

### Verification
```sql
-- Check package compilation status
SELECT object_name, object_type, status 
FROM user_objects 
WHERE object_name = 'DATA_EXPORTER' 
  AND object_type = 'PACKAGE BODY';

-- Verify package version
SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL;

-- Test export with new column reference
BEGIN
    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
        pSchemaName => 'OU_LM',
        pTableName => 'ADHOC_ADJ_HEADER',
        pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
        pBucketArea => 'ARCHIVE',
        pFolderName => 'TEST_EXPORT'
    );
END;
/
```

### Rollback
Execute `rollback_mars826_prehook.sql` to restore previous package version.

## Dependencies

### Required by MARS-826
**CRITICAL**: This package must be deployed **BEFORE** MARS-826 execution. MARS-826 export scripts depend on this updated DATA_EXPORTER package.

### Database Objects
- **Table**: CT_ODS.A_LOAD_HISTORY (must have column A_ETL_LOAD_SET_KEY)
- **Package**: CT_MRDS.ENV_MANAGER (for error handling and logging)
- **Package**: CT_MRDS.FILE_MANAGER (for bucket URI resolution)

## Configuration Changes

### Bucket Configuration
**Note**: During testing, discovered bucket configuration issue:
- Original: `ArchiveBucketName = 'archive'` (bucket did not exist)
- Fixed: `ArchiveBucketName = 'history'` (correct bucket name)

This configuration fix is **NOT** part of this MARS package but was applied directly to database:
```sql
UPDATE CT_MRDS.A_FILE_MANAGER_CONFIG 
SET CONFIG_VARIABLE_VALUE = 'history' 
WHERE CONFIG_VARIABLE = 'ArchiveBucketName';
COMMIT;
```

## Testing Results

### Test Case 1: Single Table Export
**Table**: OU_LM.ADHOC_ADJ_HEADER  
**Result**: ✅ SUCCESS  
**Output**: 2 Parquet files created with Hive-style partitioning
```
LM_ADHOC_ADJUSTMENTS_HEADER/PARTITION_YEAR=2025/PARTITION_MONTH=08/202508_1_*.parquet (1,433 bytes)
LM_ADHOC_ADJUSTMENTS_HEADER/PARTITION_YEAR=2025/PARTITION_MONTH=09/202509_1_*.parquet (1,432 bytes)
```

### Test Case 2: Package Compilation
**Status**: ✅ VALID  
**Compilation Time**: < 1 second  
**Errors**: None

## Impact Analysis

### Affected Procedures
1. `EXPORT_TABLE_DATA_BY_DATE` - Main export with Parquet partitioning
2. `EXPORT_TABLE_DATA_TO_CSV_BY_DATE` - CSV export with date filtering

### Tables Using A_ETL_LOAD_SET_KEY_FK
All tables in MARS-826 export scope (19 tables):
- OU_LM: 17 tables (ADHOC_ADJ_*, BALANCESHEET_*, CSM_ADJ_*, FORECAST_*, QR_ADJ_*, STANDING_FACILITY*, TTS_*)
- OU_MRR: 2 tables (IND_CURRENT_ACCOUNT, IND_OVERNIGHT_DEPOSITS)

### Backward Compatibility
⚠️ **BREAKING CHANGE**: This update is NOT backward compatible with databases where A_LOAD_HISTORY still uses `A_WORKFLOW_HISTORY_KEY` column name.

## Files Included

1. **README.md** - This documentation file
2. **.gitignore** - Git exclusions (confluence/, log/, test/, mock_data/)
3. **install_mars826_prehook.sql** - Master installation script with SPOOL logging (5 steps)
4. **rollback_mars826_prehook.sql** - Master rollback script (2 steps)
5. **00_MARS_826_PREHOOK_install_DATA_EXPORTER_SPEC.sql** - Deploy updated package specification
6. **01_MARS_826_PREHOOK_install_DATA_EXPORTER_BODY.sql** - Deploy updated package body
7. **02_MARS_826_PREHOOK_verify_package.sql** - Verify package compilation and test export
8. **track_package_versions.sql** - Universal version tracking script (Standard)
9. **verify_packages_version.sql** - Universal package verification script (Standard)
10. **91_MARS_826_PREHOOK_rollback_DATA_EXPORTER_BODY.sql** - Restore previous package body
11. **92_MARS_826_PREHOOK_rollback_DATA_EXPORTER_SPEC.sql** - Restore previous package specification
12. **current_version/** - Backup of DATA_EXPORTER v2.1.0 (from MARS-846)
13. **new_version/** - Updated DATA_EXPORTER v2.1.1 (with A_ETL_LOAD_SET_KEY fixes)

## Version History
- **v2.1.1** (2025-12-04): Fixed JOIN column reference A_WORKFLOW_HISTORY_KEY → A_ETL_LOAD_SET_KEY
- **v2.1.0** (2025-10-22): Added version tracking and PARTITION_YEAR/PARTITION_MONTH support
- **v2.0.0** (2025-10-01): Separated export functionality from FILE_MANAGER package

## Related JIRA Issues
- **MARS-826**: Export CSDB historical data to HIST bucket (requires this pre-hook)

## Author
Created by: Grzegorz Michalski
Date: 2025-12-04  
Schema: CT_MRDS