Refactor archival strategies in FILE_ARCHIVER package
- Removed CURRENT_MONTH_ONLY strategy, replacing it with MINIMUM_AGE_MONTHS = 0 for current month retention. - Updated validation logic to allow MINIMUM_AGE_MONTHS to be non-negative. - Consolidated documentation to reflect changes in archival strategies. - Adjusted rollback script to reset all archival parameters to NULL for 25 Release 01 tables. - Enhanced README and installation scripts to clarify new configuration requirements and usage examples. - Updated version history to indicate breaking changes and migration steps for existing configurations.
This commit is contained in:
Grzegorz Michalski
@@ -8,7 +8,7 @@ The FILE_ARCHIVER package provides flexible archival strategies that accommodate
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Four Archival Strategies**: THRESHOLD_BASED, CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID
|
||||
- **Three Archival Strategies**: THRESHOLD_BASED, MINIMUM_AGE_MONTHS (with 0=current month only), HYBRID
|
||||
- **Flexible Configuration**: Per-table archival strategy configuration via A_SOURCE_FILE_CONFIG
|
||||
- **Backward Compatible**: Default THRESHOLD_BASED strategy maintains existing behavior
|
||||
- **Validation**: Automatic validation of strategy-specific configuration requirements
|
||||
@@ -39,9 +39,8 @@ The FILE_ARCHIVER package provides flexible archival strategies that accommodate
|
||||
| Strategy | WHERE Clause Logic | Configuration Required | Primary Use Case |
|
||||
|----------|-------------------|----------------------|------------------|
|
||||
| `THRESHOLD_BASED` | Days since workflow start > threshold | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility, simple time-based archival |
|
||||
| `CURRENT_MONTH_ONLY` | Archive all data from previous months | None | LM/TOP sources - keep only current month active |
|
||||
| `MINIMUM_AGE_MONTHS` | Archive data older than X months | MINIMUM_AGE_MONTHS | CSDB sources - retain data for specific period |
|
||||
| `HYBRID` | Combines CURRENT_MONTH_ONLY + MINIMUM_AGE_MONTHS | MINIMUM_AGE_MONTHS | Advanced retention scenarios |
|
||||
| `MINIMUM_AGE_MONTHS` | Archive data older than X months (0=current month only) | MINIMUM_AGE_MONTHS (≥0) | All sources - flexible retention (0 for LM, 6 for CSDB) |
|
||||
| `HYBRID` | Combines month boundary + minimum age | MINIMUM_AGE_MONTHS | Advanced retention scenarios |
|
||||
|
||||
### 1. THRESHOLD_BASED (Default)
|
||||
|
||||
@@ -65,48 +64,27 @@ WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
|
||||
**Use Case**: Simple time-based archival, backward compatible with FILE_ARCHIVER v2.0.0 behavior.
|
||||
|
||||
### 2. CURRENT_MONTH_ONLY
|
||||
### 2. MINIMUM_AGE_MONTHS
|
||||
|
||||
Archives all data from previous months, keeping only current month data in ODS bucket.
|
||||
|
||||
**WHERE Clause**:
|
||||
```sql
|
||||
TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
```sql
|
||||
-- LM/TOP sources: Archive everything except current month
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL
|
||||
WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
|
||||
AND TABLE_ID = 'LM_STANDING_FACILITIES';
|
||||
```
|
||||
|
||||
**Use Case**:
|
||||
- LM dissemination feeds (daily/intraday updates)
|
||||
- TOP operational data requiring current month access
|
||||
- Sources where historical data is rarely accessed
|
||||
|
||||
**Behavior**:
|
||||
- January data: Archived on February 1st
|
||||
- February data: Remains in ODS bucket during February
|
||||
- March 1st: February data archived, March data active
|
||||
|
||||
### 3. MINIMUM_AGE_MONTHS
|
||||
|
||||
Archives only data older than specified number of months, allowing multi-month retention in ODS bucket.
|
||||
Archives data older than specified number of months. **Special case**: MINIMUM_AGE_MONTHS = 0 archives all data before current month (replaces deprecated CURRENT_MONTH_ONLY strategy).
|
||||
|
||||
**WHERE Clause**:
|
||||
```sql
|
||||
workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -MINIMUM_AGE_MONTHS)
|
||||
-- When MINIMUM_AGE_MONTHS = 0: workflow_start < TRUNC(SYSDATE, 'MM')
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
**Configuration Examples**:
|
||||
```sql
|
||||
-- CSDB: Archive only data older than 6 months
|
||||
-- LM: Keep only current month data (MINIMUM_AGE_MONTHS = 0)
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 0
|
||||
WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
|
||||
AND TABLE_ID = 'LM_STANDING_FACILITIES';
|
||||
|
||||
-- CSDB: Retain 6 months of data (MINIMUM_AGE_MONTHS = 6)
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 6
|
||||
@@ -115,19 +93,25 @@ WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
|
||||
```
|
||||
|
||||
**Use Case**:
|
||||
- CSDB securities/ratings data requiring 6-month retention
|
||||
- Regulatory compliance with specific retention periods
|
||||
- Analytical workloads needing multi-month historical access
|
||||
**Use Cases**:
|
||||
- **MINIMUM_AGE_MONTHS = 0**: LM dissemination feeds requiring current month only (daily/intraday updates)
|
||||
- **MINIMUM_AGE_MONTHS = 6**: CSDB securities/ratings data requiring 6-month retention
|
||||
- **MINIMUM_AGE_MONTHS = N**: Regulatory compliance with specific N-month retention periods
|
||||
|
||||
**Behavior** (with MINIMUM_AGE_MONTHS = 6):
|
||||
- February 2026: Archives data from July 2025 and earlier
|
||||
- March 2026: Archives data from August 2025 and earlier
|
||||
- Keeps 6 months of data always available in ODS bucket
|
||||
**Behavior Examples**:
|
||||
- **With MINIMUM_AGE_MONTHS = 0**:
|
||||
- January data: Archived on February 1st
|
||||
- February data: Remains in ODS bucket during February
|
||||
- March 1st: February data archived, March data active
|
||||
|
||||
- **With MINIMUM_AGE_MONTHS = 6**:
|
||||
- February 2026: Archives data from July 2025 and earlier
|
||||
- March 2026: Archives data from August 2025 and earlier
|
||||
- Keeps current month + 6 previous months (7 months total) in ODS bucket
|
||||
|
||||
### 4. HYBRID
|
||||
### 3. HYBRID
|
||||
|
||||
Combines CURRENT_MONTH_ONLY and MINIMUM_AGE_MONTHS logic - archives data from previous months AND older than minimum age.
|
||||
Combines month boundary check with minimum age threshold - archives data from previous months AND older than minimum age.
|
||||
|
||||
**WHERE Clause**:
|
||||
```sql
|
||||
@@ -157,13 +141,10 @@ WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
Automatically validates archival configuration on INSERT/UPDATE to A_SOURCE_FILE_CONFIG:
|
||||
|
||||
**Validation Rules**:
|
||||
1. **CURRENT_MONTH_ONLY**: Requires `MINIMUM_AGE_MONTHS IS NULL`
|
||||
- Error: "Strategy CURRENT_MONTH_ONLY requires MINIMUM_AGE_MONTHS to be NULL"
|
||||
1. **MINIMUM_AGE_MONTHS**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL AND MINIMUM_AGE_MONTHS >= 0`
|
||||
- Error: "Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set (≥0)"
|
||||
|
||||
2. **MINIMUM_AGE_MONTHS**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL`
|
||||
- Error: "Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set"
|
||||
|
||||
3. **HYBRID**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL`
|
||||
2. **HYBRID**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL`
|
||||
- Error: "Strategy HYBRID requires MINIMUM_AGE_MONTHS to be set"
|
||||
|
||||
**Example Validation Error**:
|
||||
@@ -233,13 +214,13 @@ END;
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
### Example 1: Configure LM Standing Facilities (CURRENT_MONTH_ONLY)
|
||||
### Example 1: Configure LM Standing Facilities (Current Month Only)
|
||||
|
||||
```sql
|
||||
-- Keep only current month data in ODS bucket
|
||||
-- Keep only current month data in ODS bucket (MINIMUM_AGE_MONTHS = 0)
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 0 -- 0 = archives all data before current month
|
||||
WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
|
||||
AND TABLE_ID = 'LM_STANDING_FACILITIES';
|
||||
@@ -280,10 +261,10 @@ WHERE TABLE_ID = 'CSDB_DEBT';
|
||||
### Example 3: Bulk Configuration for LM Source
|
||||
|
||||
```sql
|
||||
-- Configure all 19 LM tables with CURRENT_MONTH_ONLY strategy
|
||||
-- Configure all 19 LM tables with MINIMUM_AGE_MONTHS = 0 (current month only)
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 0 -- 0 = keep only current month
|
||||
WHERE SOURCE_FILE_TYPE = 'INPUT'
|
||||
AND SOURCE_FILE_ID IN (
|
||||
'DistributeStandingFacilities',
|
||||
@@ -342,7 +323,7 @@ ORDER BY ARCHIVAL_STRATEGY;
|
||||
|
||||
The following 25 Release 01 tables were configured with archival strategies:
|
||||
|
||||
**LM Tables (19 total) - CURRENT_MONTH_ONLY**:
|
||||
**LM Tables (19 total) - MINIMUM_AGE_MONTHS = 0 (current month only)**:
|
||||
- LM_STANDING_FACILITIES
|
||||
- LM_STANDING_FACILITIES_HEADER
|
||||
- LM_TTS_HEADER
|
||||
@@ -365,11 +346,11 @@ The following 25 Release 01 tables were configured with archival strategies:
|
||||
|
||||
**CSDB Tables (6 total)**:
|
||||
|
||||
*MINIMUM_AGE_MONTHS = 6*:
|
||||
*MINIMUM_AGE_MONTHS = 6 (6-month retention)*:
|
||||
- CSDB_DEBT
|
||||
- CSDB_DEBT_DAILY
|
||||
|
||||
*CURRENT_MONTH_ONLY*:
|
||||
*MINIMUM_AGE_MONTHS = 0 (current month only)*:
|
||||
- CSDB_INSTR_RAT_FULL
|
||||
- CSDB_INSTR_DESC_FULL
|
||||
- CSDB_ISSUER_RAT_FULL
|
||||
@@ -426,25 +407,7 @@ SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
WHERE ...;
|
||||
```
|
||||
|
||||
#### Issue 2: Validation Error on CURRENT_MONTH_ONLY
|
||||
|
||||
**Error**:
|
||||
```
|
||||
ORA-20001: Strategy CURRENT_MONTH_ONLY requires MINIMUM_AGE_MONTHS to be NULL
|
||||
```
|
||||
|
||||
**Cause**: MINIMUM_AGE_MONTHS has value but strategy doesn't use it
|
||||
|
||||
**Solution**:
|
||||
```sql
|
||||
-- Set MINIMUM_AGE_MONTHS to NULL for CURRENT_MONTH_ONLY
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL -- Required to be NULL
|
||||
WHERE ...;
|
||||
```
|
||||
|
||||
#### Issue 3: Archival Not Working as Expected
|
||||
#### Issue 2: Archival Not Working as Expected
|
||||
|
||||
**Symptoms**: Data not being archived according to strategy
|
||||
|
||||
@@ -492,7 +455,7 @@ END;
|
||||
/
|
||||
```
|
||||
|
||||
#### Issue 4: Package Compilation Errors After Upgrade
|
||||
#### Issue 3: Package Compilation Errors After Upgrade
|
||||
|
||||
**Symptoms**: FILE_ARCHIVER package shows INVALID status
|
||||
|
||||
@@ -516,16 +479,19 @@ WHERE object_name = 'FILE_ARCHIVER';
|
||||
|
||||
## Version History
|
||||
|
||||
### v3.1.0 (Current)
|
||||
- Enhanced archival strategies
|
||||
- Improved error handling
|
||||
- Additional validation features
|
||||
### v3.1.0 (Current - 2026-02-05)
|
||||
- **BREAKING CHANGE**: Removed CURRENT_MONTH_ONLY strategy (replaced by MINIMUM_AGE_MONTHS = 0)
|
||||
- Mathematical equivalence: CURRENT_MONTH_ONLY ≡ MINIMUM_AGE_MONTHS = 0
|
||||
- Updated trigger validation to allow MINIMUM_AGE_MONTHS >= 0 (previously >= 1)
|
||||
- Simplified architecture from 4 strategies to 3
|
||||
- Enhanced error handling
|
||||
- All 25 Release 01 tables migrated to MINIMUM_AGE_MONTHS (23 with value 0, 2 with value 6)
|
||||
|
||||
### v3.0.0 (MARS-828)
|
||||
### v3.0.0 (MARS-828 - 2026-02-04)
|
||||
- Added ARCHIVAL_STRATEGY configuration column
|
||||
- Implemented four archival strategies:
|
||||
- Implemented four archival strategies (later reduced to three in v3.1.0):
|
||||
- THRESHOLD_BASED (backward compatible)
|
||||
- CURRENT_MONTH_ONLY
|
||||
- CURRENT_MONTH_ONLY (deprecated in v3.1.0, use MINIMUM_AGE_MONTHS = 0)
|
||||
- MINIMUM_AGE_MONTHS
|
||||
- HYBRID
|
||||
- Added GET_ARCHIVAL_WHERE_CLAUSE function
|
||||
@@ -567,26 +533,26 @@ WHERE object_name = 'FILE_ARCHIVER';
|
||||
|
||||
### Strategy Selection Guidelines
|
||||
|
||||
1. **Use CURRENT_MONTH_ONLY when**:
|
||||
- Data is updated frequently (daily/intraday)
|
||||
- Historical data access is rare
|
||||
- ODS bucket space is limited
|
||||
- Example: LM dissemination feeds
|
||||
1. **Use MINIMUM_AGE_MONTHS when**:
|
||||
- **MINIMUM_AGE_MONTHS = 0**: Current month only retention
|
||||
- Data updated frequently (daily/intraday)
|
||||
- Historical data access is rare
|
||||
- ODS bucket space is limited
|
||||
- Example: LM dissemination feeds
|
||||
- **MINIMUM_AGE_MONTHS = N (N > 0)**: Multi-month retention
|
||||
- Regulatory compliance requires specific retention period
|
||||
- Analytical workloads need N-month access
|
||||
- Data updates are infrequent
|
||||
- Example: CSDB securities data (MINIMUM_AGE_MONTHS = 6)
|
||||
|
||||
2. **Use MINIMUM_AGE_MONTHS when**:
|
||||
- Regulatory compliance requires specific retention period
|
||||
- Analytical workloads need multi-month access
|
||||
- Data updates are infrequent
|
||||
- Example: CSDB securities data (6-month retention)
|
||||
|
||||
3. **Use THRESHOLD_BASED when**:
|
||||
2. **Use THRESHOLD_BASED when**:
|
||||
- Maintaining backward compatibility with legacy behavior
|
||||
- Simple time-based archival is sufficient
|
||||
- Migration from FILE_ARCHIVER v2.0.0
|
||||
|
||||
4. **Use HYBRID when**:
|
||||
3. **Use HYBRID when**:
|
||||
- Complex retention requirements
|
||||
- Combining current month access with minimum age threshold
|
||||
- Combining month boundary check with minimum age threshold
|
||||
- Advanced scenarios not covered by other strategies
|
||||
|
||||
### Configuration Best Practices
|
||||
@@ -595,8 +561,8 @@ WHERE object_name = 'FILE_ARCHIVER';
|
||||
```sql
|
||||
-- Test on single table first
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
|
||||
MINIMUM_AGE_MONTHS = NULL
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 0 -- 0 = current month only
|
||||
WHERE SOURCE_FILE_ID = 'TEST_FILE'
|
||||
AND TABLE_ID = 'TEST_TABLE';
|
||||
|
||||
@@ -610,8 +576,8 @@ WHERE object_name = 'FILE_ARCHIVER';
|
||||
SELECT
|
||||
SOURCE_FILE_ID,
|
||||
TABLE_ID,
|
||||
'CURRENT_MONTH_ONLY' AS NEW_STRATEGY,
|
||||
NULL AS NEW_MIN_AGE,
|
||||
'MINIMUM_AGE_MONTHS' AS NEW_STRATEGY,
|
||||
0 AS NEW_MIN_AGE, -- 0 = current month only
|
||||
ARCHIVAL_STRATEGY AS OLD_STRATEGY,
|
||||
MINIMUM_AGE_MONTHS AS OLD_MIN_AGE
|
||||
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
|
||||
Reference in New Issue
Block a user