MARS-828
This commit is contained in:
Grzegorz Michalski
269
MARS_Packages/REL01_ADDITIONS/MARS-828/README.md
Normal file
269
MARS_Packages/REL01_ADDITIONS/MARS-828/README.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# MARS-828: Enhanced Archival Strategies
|
||||
|
||||
## Overview
|
||||
|
||||
Implementation of flexible archival strategies for FILE_ARCHIVER package to support business requirements for different data retention policies across source systems.
|
||||
|
||||
### Background
|
||||
|
||||
**Issue**: Current FILE_ARCHIVER v2.0.0 uses fixed threshold-based archival (DAYS_FOR_ARCHIVE_THRESHOLD) which cannot accommodate:
|
||||
1. **LM/TOP sources**: Archive all data except current month
|
||||
2. **CSDB sources**: Archive only data older than 6 months
|
||||
|
||||
**Root Cause**: Hardcoded WHERE clause in ARCHIVE_TABLE_DATA procedure:
|
||||
```sql
|
||||
WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD
|
||||
```
|
||||
|
||||
**Solution**: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:
|
||||
- `THRESHOLD_BASED` - Days-based threshold (backward compatible)
|
||||
- `CURRENT_MONTH_ONLY` - Keep only current month data
|
||||
- `MINIMUM_AGE_MONTHS` - Archive data older than X months
|
||||
- `HYBRID` - Combination of current month and minimum age
|
||||
|
||||
## Changes Made
|
||||
|
||||
### Database Schema Changes
|
||||
**Table**: CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
|
||||
**Before**:
|
||||
```sql
|
||||
-- Only threshold-based configuration available
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30
|
||||
```
|
||||
|
||||
**After**:
|
||||
```sql
|
||||
-- Flexible strategy-based configuration
|
||||
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
|
||||
MINIMUM_AGE_MONTHS NUMBER(3),
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
|
||||
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
|
||||
```
|
||||
|
||||
**New Trigger**: TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
- Validates MINIMUM_AGE_MONTHS required for strategies that need it
|
||||
- Ensures data integrity before insert/update
|
||||
|
||||
### Package Changes
|
||||
**Package**: CT_MRDS.FILE_ARCHIVER
|
||||
|
||||
**Version**: 2.0.0 → 3.0.0 (MAJOR - breaking changes to internal logic)
|
||||
|
||||
**Specification Changes**:
|
||||
- Updated PACKAGE_VERSION: '2.0.0' → '3.0.0'
|
||||
- Updated VERSION_HISTORY with MARS-828 entry
|
||||
- Updated PACKAGE_BUILD_DATE to deployment date
|
||||
|
||||
**Body Changes**:
|
||||
1. **New Function**: GET_ARCHIVAL_WHERE_CLAUSE(pSourceFileConfig) - Returns strategy-specific WHERE clause
|
||||
2. **Updated**: ARCHIVE_TABLE_DATA - Uses GET_ARCHIVAL_WHERE_CLAUSE for flexible filtering
|
||||
3. **Updated**: GATHER_TABLE_STAT - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation
|
||||
|
||||
**File Structure**:
|
||||
- **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
|
||||
- **new_version/** - Updated FILE_ARCHIVER v3.0.0 (with strategy support)
|
||||
|
||||
## Archival Strategies
|
||||
|
||||
| Strategy | WHERE Clause Logic | Configuration Required | Use Case |
|
||||
|----------|-------------------|----------------------|----------|
|
||||
| `THRESHOLD_BASED` | `extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD` | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility |
|
||||
| `CURRENT_MONTH_ONLY` | `TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')` | None | General sources (LM, TOP) |
|
||||
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X)` | MINIMUM_AGE_MONTHS | CSDB (6 months retention) |
|
||||
| `HYBRID` | Both CURRENT_MONTH_ONLY AND MINIMUM_AGE_MONTHS | MINIMUM_AGE_MONTHS | Advanced scenarios |
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
```sql
|
||||
-- LM/TOP sources: Archive everything except current month
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL
|
||||
WHERE A_SOURCE_KEY IN ('LM', 'TOP');
|
||||
|
||||
-- CSDB: Archive only data older than 6 months
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 6
|
||||
WHERE A_SOURCE_KEY = 'CSDB'
|
||||
AND TABLE_ID IN ('DEBT', 'DEBT_DAILY');
|
||||
|
||||
-- Legacy sources: Keep existing threshold behavior
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD = 30
|
||||
WHERE A_SOURCE_KEY = 'C2D';
|
||||
|
||||
-- Advanced hybrid: Current month + 3 months minimum
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'HYBRID',
|
||||
MINIMUM_AGE_MONTHS = 3
|
||||
WHERE A_SOURCE_KEY = 'SPECIAL';
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Prerequisites
|
||||
- **User**: ADMIN with full privileges
|
||||
- **Database**: Oracle 23ai
|
||||
- **ENV_MANAGER**: v3.x or higher
|
||||
- **FILE_MANAGER**: v3.x or higher
|
||||
- **FILE_ARCHIVER**: v2.0.0 (will upgrade to v3.0.0)
|
||||
- **Table**: A_SOURCE_FILE_CONFIG must exist
|
||||
|
||||
### Installation Steps
|
||||
|
||||
**Option 1: Master Script (Recommended)**
|
||||
```powershell
|
||||
# Navigate to MARS-828 directory
|
||||
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
|
||||
|
||||
# Execute installation (requires ADMIN user)
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"
|
||||
|
||||
# Log file created: log/INSTALL_MARS_828_<PDB>_<timestamp>.log
|
||||
```
|
||||
|
||||
**Installation Workflow**:
|
||||
1. **Add Columns** - ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS to A_SOURCE_FILE_CONFIG
|
||||
2. **Create Trigger** - Validation trigger TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
3. **Deploy Package Spec** - FILE_ARCHIVER v3.0.0 specification
|
||||
4. **Deploy Package Body** - FILE_ARCHIVER v3.0.0 body with GET_ARCHIVAL_WHERE_CLAUSE
|
||||
5. **Verify Installation** - Check package compilation and structure
|
||||
6. **Track Version** - Record v3.0.0 in ENV_MANAGER
|
||||
7. **Verify Packages** - Check for untracked changes using ENV_MANAGER hash verification
|
||||
|
||||
**Option 2: Using Get-Content**
|
||||
```powershell
|
||||
Get-Content "MARS_Packages\REL01_ADDITIONS\MARS-828\install_mars828.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
||||
```
|
||||
|
||||
### Verification
|
||||
```sql
|
||||
-- Check package compilation status
|
||||
SELECT object_name, object_type, status
|
||||
FROM all_objects
|
||||
WHERE owner = 'CT_MRDS'
|
||||
AND object_name = 'FILE_ARCHIVER'
|
||||
AND object_type IN ('PACKAGE', 'PACKAGE BODY');
|
||||
|
||||
-- Verify package version
|
||||
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
|
||||
-- Expected: 3.0.0
|
||||
|
||||
-- Check new columns exist
|
||||
SELECT column_name, data_type, data_default
|
||||
FROM all_tab_columns
|
||||
WHERE owner = 'CT_MRDS'
|
||||
AND table_name = 'A_SOURCE_FILE_CONFIG'
|
||||
AND column_name IN ('ARCHIVAL_STRATEGY', 'MINIMUM_AGE_MONTHS');
|
||||
|
||||
-- Test strategy configuration
|
||||
SELECT A_SOURCE_KEY, TABLE_ID, ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
|
||||
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
ORDER BY A_SOURCE_KEY, TABLE_ID;
|
||||
```
|
||||
|
||||
### Rollback
|
||||
```powershell
|
||||
# Execute rollback script (requires ADMIN user)
|
||||
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
|
||||
|
||||
# Log file created: log/ROLLBACK_MARS_828_<PDB>_<timestamp>.log
|
||||
```
|
||||
|
||||
**Rollback Workflow**:
|
||||
1. **Restore Package Body** - FILE_ARCHIVER v2.0.0 body
|
||||
2. **Restore Package Spec** - FILE_ARCHIVER v2.0.0 specification
|
||||
3. **Drop Trigger** - Remove TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
4. **Drop Columns** - Remove ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
|
||||
5. **Track Rollback** - Record v2.0.0 restoration in ENV_MANAGER
|
||||
6. **Verify Packages** - Check package hash consistency
|
||||
|
||||
## Testing
|
||||
|
||||
**Test Scripts**: Located in `test/` folder
|
||||
|
||||
**Main Test Script**: test/test_archival_strategies.sql
|
||||
|
||||
```sql
|
||||
-- Test 1: CURRENT_MONTH_ONLY strategy
|
||||
-- Expected: Archives data from previous months only
|
||||
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');
|
||||
|
||||
-- Test 2: MINIMUM_AGE_MONTHS strategy (6 months)
|
||||
-- Expected: Archives data older than 6 months
|
||||
SELECT COUNT(*) FROM table WHERE workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -6);
|
||||
|
||||
-- Test 3: HYBRID strategy
|
||||
-- Expected: Archives data from previous months AND older than X months
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Required Packages
|
||||
- **CT_MRDS.ENV_MANAGER** v3.x - Error handling, logging, version tracking
|
||||
- **CT_MRDS.FILE_MANAGER** v3.x - Bucket URI resolution
|
||||
- **MRDS_LOADER.cloud_wrapper** - DBMS_CLOUD operations
|
||||
|
||||
### Database Objects
|
||||
- **Table**: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
|
||||
- **Table**: CT_ODS.A_LOAD_HISTORY - Workflow tracking
|
||||
- **Credential**: DEF_CRED_ARN - OCI bucket access
|
||||
|
||||
## Files Included
|
||||
|
||||
1. **README.md** - This documentation file
|
||||
2. **.gitignore** - Git exclusions (confluence/, log/, test/, mock_data/)
|
||||
3. **install_mars828.sql** - Master installation script with SPOOL logging (7 steps)
|
||||
4. **rollback_mars828.sql** - Master rollback script (6 steps)
|
||||
5. **01_MARS_828_install_add_archival_strategy_columns.sql** - ALTER TABLE DDL
|
||||
6. **02_MARS_828_install_archival_strategy_trigger.sql** - CREATE TRIGGER DDL
|
||||
7. **03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql** - Deploy package specification v3.0.0
|
||||
8. **04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql** - Deploy package body v3.0.0
|
||||
9. **05_MARS_828_verify_installation.sql** - Verification queries
|
||||
10. **91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql** - Restore package specification v2.0.0
|
||||
11. **92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql** - Restore package body v2.0.0
|
||||
12. **93_MARS_828_rollback_trigger.sql** - DROP TRIGGER
|
||||
13. **94_MARS_828_rollback_columns.sql** - ALTER TABLE DROP COLUMN
|
||||
14. **track_package_versions.sql** - Universal version tracking script (Standard)
|
||||
15. **verify_packages_version.sql** - Universal package verification script (Standard)
|
||||
16. **test/** - Test scripts and validation scenarios
|
||||
17. **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
|
||||
18. **new_version/** - Updated FILE_ARCHIVER v3.0.0 (deployment source)
|
||||
|
||||
## Impact Analysis
|
||||
|
||||
### Backward Compatibility
|
||||
✅ **FULLY BACKWARD COMPATIBLE**: Default ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' maintains existing behavior for all sources without configuration changes.
|
||||
|
||||
### Affected Procedures
|
||||
1. **ARCHIVE_TABLE_DATA** - Uses GET_ARCHIVAL_WHERE_CLAUSE for WHERE clause generation
|
||||
2. **GATHER_TABLE_STAT** - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation
|
||||
|
||||
### Configuration Migration
|
||||
No automatic migration required. New columns have sensible defaults:
|
||||
- ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' (maintains current behavior)
|
||||
- MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)
|
||||
|
||||
## Version History
|
||||
- **v3.0.0** (2026-01-27): Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration
|
||||
- **v2.0.0** (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival
|
||||
|
||||
## Related JIRA Issues
|
||||
- **MARS-828**: Enhanced Archival Strategies implementation
|
||||
|
||||
## Author
|
||||
Created by: Grzegorz Michalski
|
||||
Date: 2026-01-27
|
||||
Schema: CT_MRDS
|
||||
Package: FILE_ARCHIVER
|
||||
|
||||
|
||||
Grzegorz Michalski
|
||||
|
||||
## Date
|
||||
|
||||
2026-01-27
|
||||
Reference in New Issue
Block a user