admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7864c568d72d02931b317915d96e5c9e4d6e698d
mars/MARS_Packages/REL01_ADDITIONS/MARS-828/README.md
Grzegorz Michalski 64965fb8c8 MARS-828
2026-02-03 13:51:13 +01:00

9.9 KiB
Raw Blame History

MARS-828: Enhanced Archival Strategies

Overview

Implementation of flexible archival strategies for FILE_ARCHIVER package to support business requirements for different data retention policies across source systems.

Background

Issue: Current FILE_ARCHIVER v2.0.0 uses fixed threshold-based archival (DAYS_FOR_ARCHIVE_THRESHOLD) which cannot accommodate:

  1. LM/TOP sources: Archive all data except current month
  2. CSDB sources: Archive only data older than 6 months

Root Cause: Hardcoded WHERE clause in ARCHIVE_TABLE_DATA procedure:

WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD

Solution: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:

  • THRESHOLD_BASED - Days-based threshold (backward compatible)
  • CURRENT_MONTH_ONLY - Keep only current month data
  • MINIMUM_AGE_MONTHS - Archive data older than X months
  • HYBRID - Combination of current month and minimum age

Changes Made

Database Schema Changes

Table: CT_MRDS.A_SOURCE_FILE_CONFIG

Before:

-- Only threshold-based configuration available
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30

After:

-- Flexible strategy-based configuration
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3),
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID'))

New Trigger: TRG_BI_ARCHIVAL_STRATEGY_VAL

  • Validates MINIMUM_AGE_MONTHS required for strategies that need it
  • Ensures data integrity before insert/update

Package Changes

Package: CT_MRDS.FILE_ARCHIVER

Version: 2.0.0 → 3.0.0 (MAJOR - breaking changes to internal logic)

Specification Changes:

  • Updated PACKAGE_VERSION: '2.0.0' → '3.0.0'
  • Updated VERSION_HISTORY with MARS-828 entry
  • Updated PACKAGE_BUILD_DATE to deployment date

Body Changes:

  1. New Function: GET_ARCHIVAL_WHERE_CLAUSE(pSourceFileConfig) - Returns strategy-specific WHERE clause
  2. Updated: ARCHIVE_TABLE_DATA - Uses GET_ARCHIVAL_WHERE_CLAUSE for flexible filtering
  3. Updated: GATHER_TABLE_STAT - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation

File Structure:

  • rollback/ - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
  • new_version/ - Updated FILE_ARCHIVER v3.0.0 (with strategy support)

Archival Strategies

Strategy WHERE Clause Logic Configuration Required Use Case
THRESHOLD_BASED extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD DAYS_FOR_ARCHIVE_THRESHOLD Legacy compatibility
CURRENT_MONTH_ONLY TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM') None General sources (LM, TOP)
MINIMUM_AGE_MONTHS workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X) MINIMUM_AGE_MONTHS CSDB (6 months retention)
HYBRID Both CURRENT_MONTH_ONLY AND MINIMUM_AGE_MONTHS MINIMUM_AGE_MONTHS Advanced scenarios

Configuration Examples

-- LM/TOP sources: Archive everything except current month
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
    MINIMUM_AGE_MONTHS = NULL
WHERE A_SOURCE_KEY IN ('LM', 'TOP');

-- CSDB: Archive only data older than 6 months
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
    MINIMUM_AGE_MONTHS = 6
WHERE A_SOURCE_KEY = 'CSDB' 
  AND TABLE_ID IN ('DEBT', 'DEBT_DAILY');

-- Legacy sources: Keep existing threshold behavior
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
    DAYS_FOR_ARCHIVE_THRESHOLD = 30
WHERE A_SOURCE_KEY = 'C2D';

-- Advanced hybrid: Current month + 3 months minimum
UPDATE A_SOURCE_FILE_CONFIG 
SET ARCHIVAL_STRATEGY = 'HYBRID',
    MINIMUM_AGE_MONTHS = 3
WHERE A_SOURCE_KEY = 'SPECIAL';

Deployment

Prerequisites

  • User: ADMIN with full privileges
  • Database: Oracle 23ai
  • ENV_MANAGER: v3.x or higher
  • FILE_MANAGER: v3.x or higher
  • FILE_ARCHIVER: v2.0.0 (will upgrade to v3.0.0)
  • Table: A_SOURCE_FILE_CONFIG must exist

Installation Steps

Option 1: Master Script (Recommended)

# Navigate to MARS-828 directory
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828

# Execute installation (requires ADMIN user)
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"

# Log file created: log/INSTALL_MARS_828_<PDB>_<timestamp>.log

Installation Workflow:

  1. Add Columns - ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS to A_SOURCE_FILE_CONFIG
  2. Create Trigger - Validation trigger TRG_BI_ARCHIVAL_STRATEGY_VAL
  3. Deploy Package Spec - FILE_ARCHIVER v3.0.0 specification
  4. Deploy Package Body - FILE_ARCHIVER v3.0.0 body with GET_ARCHIVAL_WHERE_CLAUSE
  5. Verify Installation - Check package compilation and structure
  6. Track Version - Record v3.0.0 in ENV_MANAGER
  7. Verify Packages - Check for untracked changes using ENV_MANAGER hash verification

Option 2: Using Get-Content

Get-Content "MARS_Packages\REL01_ADDITIONS\MARS-828\install_mars828.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"

Verification

-- Check package compilation status
SELECT object_name, object_type, status 
FROM all_objects 
WHERE owner = 'CT_MRDS'
  AND object_name = 'FILE_ARCHIVER' 
  AND object_type IN ('PACKAGE', 'PACKAGE BODY');

-- Verify package version
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
-- Expected: 3.0.0

-- Check new columns exist
SELECT column_name, data_type, data_default 
FROM all_tab_columns 
WHERE owner = 'CT_MRDS' 
  AND table_name = 'A_SOURCE_FILE_CONFIG'
  AND column_name IN ('ARCHIVAL_STRATEGY', 'MINIMUM_AGE_MONTHS');

-- Test strategy configuration
SELECT A_SOURCE_KEY, TABLE_ID, ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
ORDER BY A_SOURCE_KEY, TABLE_ID;

Rollback

# Execute rollback script (requires ADMIN user)
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"

# Log file created: log/ROLLBACK_MARS_828_<PDB>_<timestamp>.log

Rollback Workflow:

  1. Restore Package Body - FILE_ARCHIVER v2.0.0 body
  2. Restore Package Spec - FILE_ARCHIVER v2.0.0 specification
  3. Drop Trigger - Remove TRG_BI_ARCHIVAL_STRATEGY_VAL
  4. Drop Columns - Remove ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
  5. Track Rollback - Record v2.0.0 restoration in ENV_MANAGER
  6. Verify Packages - Check package hash consistency

Testing

Test Scripts: Located in test/ folder

Main Test Script: test/test_archival_strategies.sql

-- Test 1: CURRENT_MONTH_ONLY strategy
-- Expected: Archives data from previous months only
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');

-- Test 2: MINIMUM_AGE_MONTHS strategy (6 months)
-- Expected: Archives data older than 6 months
SELECT COUNT(*) FROM table WHERE workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -6);

-- Test 3: HYBRID strategy
-- Expected: Archives data from previous months AND older than X months

Dependencies

Required Packages

  • CT_MRDS.ENV_MANAGER v3.x - Error handling, logging, version tracking
  • CT_MRDS.FILE_MANAGER v3.x - Bucket URI resolution
  • MRDS_LOADER.cloud_wrapper - DBMS_CLOUD operations

Database Objects

  • Table: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
  • Table: CT_ODS.A_LOAD_HISTORY - Workflow tracking
  • Credential: DEF_CRED_ARN - OCI bucket access

Files Included

  1. README.md - This documentation file
  2. .gitignore - Git exclusions (confluence/, log/, test/, mock_data/)
  3. install_mars828.sql - Master installation script with SPOOL logging (7 steps)
  4. rollback_mars828.sql - Master rollback script (6 steps)
  5. 01_MARS_828_install_add_archival_strategy_columns.sql - ALTER TABLE DDL
  6. 02_MARS_828_install_archival_strategy_trigger.sql - CREATE TRIGGER DDL
  7. 03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql - Deploy package specification v3.0.0
  8. 04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql - Deploy package body v3.0.0
  9. 05_MARS_828_verify_installation.sql - Verification queries
  10. 91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql - Restore package specification v2.0.0
  11. 92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql - Restore package body v2.0.0
  12. 93_MARS_828_rollback_trigger.sql - DROP TRIGGER
  13. 94_MARS_828_rollback_columns.sql - ALTER TABLE DROP COLUMN
  14. track_package_versions.sql - Universal version tracking script (Standard)
  15. verify_packages_version.sql - Universal package verification script (Standard)
  16. test/ - Test scripts and validation scenarios
  17. rollback/ - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
  18. new_version/ - Updated FILE_ARCHIVER v3.0.0 (deployment source)

Impact Analysis

Backward Compatibility

✅ FULLY BACKWARD COMPATIBLE: Default ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' maintains existing behavior for all sources without configuration changes.

Affected Procedures

  1. ARCHIVE_TABLE_DATA - Uses GET_ARCHIVAL_WHERE_CLAUSE for WHERE clause generation
  2. GATHER_TABLE_STAT - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation

Configuration Migration

No automatic migration required. New columns have sensible defaults:

  • ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' (maintains current behavior)
  • MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)

Version History

  • v3.0.0 (2026-01-27): Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration
  • v2.0.0 (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival

Related JIRA Issues

  • MARS-828: Enhanced Archival Strategies implementation

Author

Created by: Grzegorz Michalski
Date: 2026-01-27
Schema: CT_MRDS
Package: FILE_ARCHIVER

Grzegorz Michalski

Date

2026-01-27

Reference in New Issue View Git Blame Copy Permalink