# MARS-828: Enhanced Archival Strategies

## Overview

Implementation of flexible archival strategies for FILE_ARCHIVER package to support business requirements for different data retention policies across source systems.

### Background

**Issue**: Current FILE_ARCHIVER v2.0.0 uses fixed threshold-based archival (DAYS_FOR_ARCHIVE_THRESHOLD) which cannot accommodate:
1. **LM/TOP sources**: Archive all data except current month
2. **CSDB sources**: Archive only data older than 6 months

**Root Cause**: Hardcoded WHERE clause in ARCHIVE_TABLE_DATA procedure:
```sql
WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD
```

**Solution**: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:
- `THRESHOLD_BASED` - Days-based threshold (backward compatible)
- `CURRENT_MONTH_ONLY` - Keep only current month data
- `MINIMUM_AGE_MONTHS` - Archive data older than X months
- `HYBRID` - Combination of current month and minimum age

## Changes Made

### Database Schema Changes
**Table**: CT_MRDS.A_SOURCE_FILE_CONFIG

**Before**:
```sql
-- Only threshold-based configuration available
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30
```

**After**:
```sql
-- Flexible strategy-based configuration
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3),
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
```

**New Trigger**: TRG_BI_ARCHIVAL_STRATEGY_VAL
- Validates MINIMUM_AGE_MONTHS required for strategies that need it
- Ensures data integrity before insert/update

### Package Changes
**Package**: CT_MRDS.FILE_ARCHIVER

**Version**: 2.0.0 → 3.0.0 (MAJOR - breaking changes to internal logic)

**Specification Changes**:
- Updated PACKAGE_VERSION: '2.0.0' → '3.0.0'
- Updated VERSION_HISTORY with MARS-828 entry
- Updated PACKAGE_BUILD_DATE to deployment date

**Body Changes**:
1. **New Function**: GET_ARCHIVAL_WHERE_CLAUSE(pSourceFileConfig) - Returns strategy-specific WHERE clause
2. **Updated**: ARCHIVE_TABLE_DATA - Uses GET_ARCHIVAL_WHERE_CLAUSE for flexible filtering
3. **Updated**: GATHER_TABLE_STAT - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation

**File Structure**:
- **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
- **new_version/** - Updated FILE_ARCHIVER v3.0.0 (with strategy support)

## Archival Strategies

| Strategy | WHERE Clause Logic | Configuration Required | Use Case |
|----------|-------------------|----------------------|----------|
| `THRESHOLD_BASED` | `extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD` | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility |
| `CURRENT_MONTH_ONLY` | `TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')` | None | General sources (LM, TOP) |
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X)` | MINIMUM_AGE_MONTHS | CSDB (6 months retention) |
| `HYBRID` | Both CURRENT_MONTH_ONLY AND MINIMUM_AGE_MONTHS | MINIMUM_AGE_MONTHS | Advanced scenarios |

## Configuration Examples

```sql
-- LM/TOP sources: Archive everything except current month
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
    MINIMUM_AGE_MONTHS = NULL
WHERE A_SOURCE_KEY IN ('LM', 'TOP');

-- CSDB: Archive only data older than 6 months
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
    MINIMUM_AGE_MONTHS = 6
WHERE A_SOURCE_KEY = 'CSDB' 
  AND TABLE_ID IN ('DEBT', 'DEBT_DAILY');

-- Legacy sources: Keep existing threshold behavior
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
    DAYS_FOR_ARCHIVE_THRESHOLD = 30
WHERE A_SOURCE_KEY = 'C2D';

-- Advanced hybrid: Current month + 3 months minimum
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'HYBRID',
    MINIMUM_AGE_MONTHS = 3
WHERE A_SOURCE_KEY = 'SPECIAL';
```

## Deployment

### Prerequisites
- **User**: ADMIN with full privileges
- **Database**: Oracle 23ai
- **ENV_MANAGER**: v3.x or higher
- **FILE_MANAGER**: v3.x or higher  
- **FILE_ARCHIVER**: v2.0.0 (will upgrade to v3.0.0)
- **Table**: A_SOURCE_FILE_CONFIG must exist

### Installation Steps

**Option 1: Master Script (Recommended)**
```powershell
# Navigate to MARS-828 directory
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828

# Execute installation (requires ADMIN user)
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"

# Log file created: log/INSTALL_MARS_828_<PDB>_<timestamp>.log
```

**Installation Workflow**:
1. **Add Columns** - ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS to A_SOURCE_FILE_CONFIG
2. **Create Trigger** - Validation trigger TRG_BI_ARCHIVAL_STRATEGY_VAL
3. **Deploy Package Spec** - FILE_ARCHIVER v3.0.0 specification
4. **Deploy Package Body** - FILE_ARCHIVER v3.0.0 body with GET_ARCHIVAL_WHERE_CLAUSE
5. **Verify Installation** - Check package compilation and structure
6. **Track Version** - Record v3.0.0 in ENV_MANAGER
7. **Verify Packages** - Check for untracked changes using ENV_MANAGER hash verification

**Option 2: Using Get-Content**
```powershell
Get-Content "MARS_Packages\REL01_ADDITIONS\MARS-828\install_mars828.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```

### Verification
```sql
-- Check package compilation status
SELECT object_name, object_type, status 
FROM all_objects 
WHERE owner = 'CT_MRDS'
  AND object_name = 'FILE_ARCHIVER' 
  AND object_type IN ('PACKAGE', 'PACKAGE BODY');

-- Verify package version
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
-- Expected: 3.0.0

-- Check new columns exist
SELECT column_name, data_type, data_default 
FROM all_tab_columns 
WHERE owner = 'CT_MRDS' 
  AND table_name = 'A_SOURCE_FILE_CONFIG'
  AND column_name IN ('ARCHIVAL_STRATEGY', 'MINIMUM_AGE_MONTHS');

-- Test strategy configuration
SELECT A_SOURCE_KEY, TABLE_ID, ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
ORDER BY A_SOURCE_KEY, TABLE_ID;
```

### Rollback
```powershell
# Execute rollback script (requires ADMIN user)
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"

# Log file created: log/ROLLBACK_MARS_828_<PDB>_<timestamp>.log
```

**Rollback Workflow**:
1. **Restore Package Body** - FILE_ARCHIVER v2.0.0 body
2. **Restore Package Spec** - FILE_ARCHIVER v2.0.0 specification
3. **Drop Trigger** - Remove TRG_BI_ARCHIVAL_STRATEGY_VAL
4. **Drop Columns** - Remove ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
5. **Track Rollback** - Record v2.0.0 restoration in ENV_MANAGER
6. **Verify Packages** - Check package hash consistency

## Testing

**Test Scripts**: Located in `test/` folder

**Main Test Script**: test/test_archival_strategies.sql

```sql
-- Test 1: CURRENT_MONTH_ONLY strategy
-- Expected: Archives data from previous months only
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');

-- Test 2: MINIMUM_AGE_MONTHS strategy (6 months)
-- Expected: Archives data older than 6 months
SELECT COUNT(*) FROM table WHERE workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -6);

-- Test 3: HYBRID strategy
-- Expected: Archives data from previous months AND older than X months
```

## Dependencies

### Required Packages
- **CT_MRDS.ENV_MANAGER** v3.x - Error handling, logging, version tracking
- **CT_MRDS.FILE_MANAGER** v3.x - Bucket URI resolution
- **MRDS_LOADER.cloud_wrapper** - DBMS_CLOUD operations

### Database Objects
- **Table**: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
- **Table**: CT_ODS.A_LOAD_HISTORY - Workflow tracking
- **Credential**: DEF_CRED_ARN - OCI bucket access

## Files Included

1. **README.md** - This documentation file
2. **.gitignore** - Git exclusions (confluence/, log/, test/, mock_data/)
3. **install_mars828.sql** - Master installation script with SPOOL logging (7 steps)
4. **rollback_mars828.sql** - Master rollback script (6 steps)
5. **01_MARS_828_install_add_archival_strategy_columns.sql** - ALTER TABLE DDL
6. **02_MARS_828_install_archival_strategy_trigger.sql** - CREATE TRIGGER DDL
7. **03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql** - Deploy package specification v3.0.0
8. **04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql** - Deploy package body v3.0.0
9. **05_MARS_828_verify_installation.sql** - Verification queries
10. **91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql** - Restore package specification v2.0.0
11. **92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql** - Restore package body v2.0.0
12. **93_MARS_828_rollback_trigger.sql** - DROP TRIGGER
13. **94_MARS_828_rollback_columns.sql** - ALTER TABLE DROP COLUMN
14. **track_package_versions.sql** - Universal version tracking script (Standard)
15. **verify_packages_version.sql** - Universal package verification script (Standard)
16. **test/** - Test scripts and validation scenarios
17. **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
18. **new_version/** - Updated FILE_ARCHIVER v3.0.0 (deployment source)

## Impact Analysis

### Backward Compatibility
✅ **FULLY BACKWARD COMPATIBLE**: Default ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' maintains existing behavior for all sources without configuration changes.

### Affected Procedures
1. **ARCHIVE_TABLE_DATA** - Uses GET_ARCHIVAL_WHERE_CLAUSE for WHERE clause generation
2. **GATHER_TABLE_STAT** - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation

### Configuration Migration
No automatic migration required. New columns have sensible defaults:
- ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' (maintains current behavior)
- MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)

## Version History
- **v3.0.0** (2026-01-27): Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration
- **v2.0.0** (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival

## Related JIRA Issues
- **MARS-828**: Enhanced Archival Strategies implementation

## Author
Created by: Grzegorz Michalski  
Date: 2026-01-27  
Schema: CT_MRDS  
Package: FILE_ARCHIVER


Grzegorz Michalski

## Date

2026-01-27