Compare commits
2 Commits
e6ab189dc9
...
974f6eca25
| Author | SHA1 | Date | |
|---|---|---|---|
|
974f6eca25 | ||
|
|
c4f5ba48bc |
@@ -4,6 +4,7 @@
|
||||
# Confluence documentation (generated, not source)
|
||||
confluence/
|
||||
|
||||
patches/
|
||||
# Log files from SPOOL operations
|
||||
log/
|
||||
*.log
|
||||
|
||||
@@ -1,269 +0,0 @@
|
||||
# MARS-828: Enhanced Archival Strategies
|
||||
|
||||
## Overview
|
||||
|
||||
Implementation of flexible archival strategies for FILE_ARCHIVER package to support business requirements for different data retention policies across source systems.
|
||||
|
||||
### Background
|
||||
|
||||
**Issue**: Current FILE_ARCHIVER v2.0.0 uses fixed threshold-based archival (DAYS_FOR_ARCHIVE_THRESHOLD) which cannot accommodate:
|
||||
1. **LM/TOP sources**: Archive all data except current month
|
||||
2. **CSDB sources**: Archive only data older than 6 months
|
||||
|
||||
**Root Cause**: Hardcoded WHERE clause in ARCHIVE_TABLE_DATA procedure:
|
||||
```sql
|
||||
WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD
|
||||
```
|
||||
|
||||
**Solution**: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:
|
||||
- `THRESHOLD_BASED` - Days-based threshold (backward compatible)
|
||||
- `MINIMUM_AGE_MONTHS` - Retain data for specified months (0 = current month only)
|
||||
- `MINIMUM_AGE_MONTHS` - Archive data older than X months
|
||||
- `HYBRID` - Combination of current month and minimum age
|
||||
|
||||
## Changes Made
|
||||
|
||||
### Database Schema Changes
|
||||
**Table**: CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
|
||||
**Before**:
|
||||
```sql
|
||||
-- Only threshold-based configuration available
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30
|
||||
```
|
||||
|
||||
**After**:
|
||||
```sql
|
||||
-- Flexible strategy-based configuration
|
||||
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
|
||||
MINIMUM_AGE_MONTHS NUMBER(3),
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
|
||||
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
|
||||
```
|
||||
|
||||
**New Trigger**: TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
- Validates MINIMUM_AGE_MONTHS required for strategies that need it
|
||||
- Ensures data integrity before insert/update
|
||||
|
||||
### Package Changes
|
||||
**Package**: CT_MRDS.FILE_ARCHIVER
|
||||
|
||||
**Version**: 2.0.0 → 3.0.0 (MAJOR - breaking changes to internal logic)
|
||||
|
||||
**Specification Changes**:
|
||||
- Updated PACKAGE_VERSION: '2.0.0' → '3.0.0'
|
||||
- Updated VERSION_HISTORY with MARS-828 entry
|
||||
- Updated PACKAGE_BUILD_DATE to deployment date
|
||||
|
||||
**Body Changes**:
|
||||
1. **New Function**: GET_ARCHIVAL_WHERE_CLAUSE(pSourceFileConfig) - Returns strategy-specific WHERE clause
|
||||
2. **Updated**: ARCHIVE_TABLE_DATA - Uses GET_ARCHIVAL_WHERE_CLAUSE for flexible filtering
|
||||
3. **Updated**: GATHER_TABLE_STAT - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation
|
||||
|
||||
**File Structure**:
|
||||
- **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
|
||||
- **new_version/** - Updated FILE_ARCHIVER v3.0.0 (with strategy support)
|
||||
|
||||
## Archival Strategies
|
||||
|
||||
| Strategy | WHERE Clause Logic | Configuration Required | Use Case |
|
||||
|----------|-------------------|----------------------|----------|
|
||||
| `THRESHOLD_BASED` | `extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD` | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility |
|
||||
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -N)` | MINIMUM_AGE_MONTHS (0=current month) | All sources |
|
||||
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X)` | MINIMUM_AGE_MONTHS | CSDB (6 months retention) |
|
||||
| `HYBRID` | Combines month truncation AND minimum age | MINIMUM_AGE_MONTHS | Advanced scenarios |
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
```sql
|
||||
-- LM/TOP sources: Archive everything except current month
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
|
||||
WHERE A_SOURCE_KEY IN ('LM', 'TOP');
|
||||
|
||||
-- CSDB: Archive only data older than 6 months
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
|
||||
MINIMUM_AGE_MONTHS = 6
|
||||
WHERE A_SOURCE_KEY = 'CSDB'
|
||||
AND TABLE_ID IN ('DEBT', 'DEBT_DAILY');
|
||||
|
||||
-- Legacy sources: Keep existing threshold behavior
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
|
||||
DAYS_FOR_ARCHIVE_THRESHOLD = 30
|
||||
WHERE A_SOURCE_KEY = 'C2D';
|
||||
|
||||
-- Advanced hybrid: Current month + 3 months minimum
|
||||
UPDATE A_SOURCE_FILE_CONFIG
|
||||
SET ARCHIVAL_STRATEGY = 'HYBRID',
|
||||
MINIMUM_AGE_MONTHS = 3
|
||||
WHERE A_SOURCE_KEY = 'SPECIAL';
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Prerequisites
|
||||
- **User**: ADMIN with full privileges
|
||||
- **Database**: Oracle 23ai
|
||||
- **ENV_MANAGER**: v3.x or higher
|
||||
- **FILE_MANAGER**: v3.x or higher
|
||||
- **FILE_ARCHIVER**: v2.0.0 (will upgrade to v3.0.0)
|
||||
- **Table**: A_SOURCE_FILE_CONFIG must exist
|
||||
|
||||
### Installation Steps
|
||||
|
||||
**Option 1: Master Script (Recommended)**
|
||||
```powershell
|
||||
# Navigate to MARS-828 directory
|
||||
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
|
||||
|
||||
# Execute installation (requires ADMIN user)
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"
|
||||
|
||||
# Log file created: log/INSTALL_MARS_828_<PDB>_<timestamp>.log
|
||||
```
|
||||
|
||||
**Installation Workflow**:
|
||||
1. **Add Columns** - ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS to A_SOURCE_FILE_CONFIG
|
||||
2. **Create Trigger** - Validation trigger TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
3. **Deploy Package Spec** - FILE_ARCHIVER v3.0.0 specification
|
||||
4. **Deploy Package Body** - FILE_ARCHIVER v3.0.0 body with GET_ARCHIVAL_WHERE_CLAUSE
|
||||
5. **Verify Installation** - Check package compilation and structure
|
||||
6. **Track Version** - Record v3.0.0 in ENV_MANAGER
|
||||
7. **Verify Packages** - Check for untracked changes using ENV_MANAGER hash verification
|
||||
|
||||
**Option 2: Using Get-Content**
|
||||
```powershell
|
||||
Get-Content "MARS_Packages\REL01_ADDITIONS\MARS-828\install_mars828.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
||||
```
|
||||
|
||||
### Verification
|
||||
```sql
|
||||
-- Check package compilation status
|
||||
SELECT object_name, object_type, status
|
||||
FROM all_objects
|
||||
WHERE owner = 'CT_MRDS'
|
||||
AND object_name = 'FILE_ARCHIVER'
|
||||
AND object_type IN ('PACKAGE', 'PACKAGE BODY');
|
||||
|
||||
-- Verify package version
|
||||
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
|
||||
-- Expected: 3.0.0
|
||||
|
||||
-- Check new columns exist
|
||||
SELECT column_name, data_type, data_default
|
||||
FROM all_tab_columns
|
||||
WHERE owner = 'CT_MRDS'
|
||||
AND table_name = 'A_SOURCE_FILE_CONFIG'
|
||||
AND column_name IN ('ARCHIVAL_STRATEGY', 'MINIMUM_AGE_MONTHS');
|
||||
|
||||
-- Test strategy configuration
|
||||
SELECT A_SOURCE_KEY, TABLE_ID, ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
|
||||
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
|
||||
ORDER BY A_SOURCE_KEY, TABLE_ID;
|
||||
```
|
||||
|
||||
### Rollback
|
||||
```powershell
|
||||
# Execute rollback script (requires ADMIN user)
|
||||
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
|
||||
|
||||
# Log file created: log/ROLLBACK_MARS_828_<PDB>_<timestamp>.log
|
||||
```
|
||||
|
||||
**Rollback Workflow**:
|
||||
1. **Restore Package Body** - FILE_ARCHIVER v2.0.0 body
|
||||
2. **Restore Package Spec** - FILE_ARCHIVER v2.0.0 specification
|
||||
3. **Drop Trigger** - Remove TRG_BI_ARCHIVAL_STRATEGY_VAL
|
||||
4. **Drop Columns** - Remove ARCHIVAL_STRATEGY, MINIMUM_AGE_MONTHS
|
||||
5. **Track Rollback** - Record v2.0.0 restoration in ENV_MANAGER
|
||||
6. **Verify Packages** - Check package hash consistency
|
||||
|
||||
## Testing
|
||||
|
||||
**Test Scripts**: Located in `test/` folder
|
||||
|
||||
**Main Test Script**: test/test_archival_strategies.sql
|
||||
|
||||
```sql
|
||||
-- Test 1: MINIMUM_AGE_MONTHS=0 strategy (current month only)
|
||||
-- Expected: Archives data from previous months only
|
||||
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');
|
||||
|
||||
-- Test 2: MINIMUM_AGE_MONTHS strategy (6 months)
|
||||
-- Expected: Archives data older than 6 months
|
||||
SELECT COUNT(*) FROM table WHERE workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -6);
|
||||
|
||||
-- Test 3: HYBRID strategy
|
||||
-- Expected: Archives data from previous months AND older than X months
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Required Packages
|
||||
- **CT_MRDS.ENV_MANAGER** v3.x - Error handling, logging, version tracking
|
||||
- **CT_MRDS.FILE_MANAGER** v3.x - Bucket URI resolution
|
||||
- **MRDS_LOADER.cloud_wrapper** - DBMS_CLOUD operations
|
||||
|
||||
### Database Objects
|
||||
- **Table**: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
|
||||
- **Table**: CT_ODS.A_LOAD_HISTORY - Workflow tracking
|
||||
- **Credential**: DEF_CRED_ARN - OCI bucket access
|
||||
|
||||
## Files Included
|
||||
|
||||
1. **README.md** - This documentation file
|
||||
2. **.gitignore** - Git exclusions (confluence/, log/, test/, mock_data/)
|
||||
3. **install_mars828.sql** - Master installation script with SPOOL logging (7 steps)
|
||||
4. **rollback_mars828.sql** - Master rollback script (6 steps)
|
||||
5. **01_MARS_828_install_add_archival_strategy_columns.sql** - ALTER TABLE DDL
|
||||
6. **02_MARS_828_install_archival_strategy_trigger.sql** - CREATE TRIGGER DDL
|
||||
7. **03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql** - Deploy package specification v3.0.0
|
||||
8. **04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql** - Deploy package body v3.0.0
|
||||
9. **05_MARS_828_verify_installation.sql** - Verification queries
|
||||
10. **91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql** - Restore package specification v2.0.0
|
||||
11. **92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql** - Restore package body v2.0.0
|
||||
12. **93_MARS_828_rollback_trigger.sql** - DROP TRIGGER
|
||||
13. **94_MARS_828_rollback_columns.sql** - ALTER TABLE DROP COLUMN
|
||||
14. **track_package_versions.sql** - Universal version tracking script (Standard)
|
||||
15. **verify_packages_version.sql** - Universal package verification script (Standard)
|
||||
16. **test/** - Test scripts and validation scenarios
|
||||
17. **rollback/** - Backup of FILE_ARCHIVER v2.0.0 (for rollback)
|
||||
18. **new_version/** - Updated FILE_ARCHIVER v3.0.0 (deployment source)
|
||||
|
||||
## Impact Analysis
|
||||
|
||||
### Backward Compatibility
|
||||
✅ **FULLY BACKWARD COMPATIBLE**: Default ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' maintains existing behavior for all sources without configuration changes.
|
||||
|
||||
### Affected Procedures
|
||||
1. **ARCHIVE_TABLE_DATA** - Uses GET_ARCHIVAL_WHERE_CLAUSE for WHERE clause generation
|
||||
2. **GATHER_TABLE_STAT** - Uses GET_ARCHIVAL_WHERE_CLAUSE for statistics calculation
|
||||
|
||||
### Configuration Migration
|
||||
No automatic migration required. New columns have sensible defaults:
|
||||
- ARCHIVAL_STRATEGY = 'THRESHOLD_BASED' (maintains current behavior)
|
||||
- MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)
|
||||
|
||||
## Version History
|
||||
- **v3.0.0** (2026-01-27): Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration
|
||||
- **v2.0.0** (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival
|
||||
|
||||
## Related JIRA Issues
|
||||
- **MARS-828**: Enhanced Archival Strategies implementation
|
||||
|
||||
## Author
|
||||
Created by: Grzegorz Michalski
|
||||
Date: 2026-01-27
|
||||
Schema: CT_MRDS
|
||||
Package: FILE_ARCHIVER
|
||||
|
||||
|
||||
Grzegorz Michalski
|
||||
|
||||
## Date
|
||||
|
||||
2026-01-27
|
||||
@@ -37,7 +37,7 @@ PROMPT =========================================================================
|
||||
PROMPT Package: CT_MRDS.FILE_ARCHIVER
|
||||
PROMPT Change: Enhanced archival strategies (MINIMUM_AGE_MONTHS, HYBRID)
|
||||
PROMPT Purpose: Flexible archival policies per data source
|
||||
PROMPT Steps: 8 (DDL, Trigger, Packages, Verify, Track, Verify, Configure)
|
||||
PROMPT Steps: 8 (DDL, Trigger, Package v3.1.2, Verify, Track, Configure)
|
||||
PROMPT Timestamp:
|
||||
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_start FROM DUAL;
|
||||
PROMPT ============================================================================
|
||||
@@ -65,12 +65,12 @@ PROMPT ======================================
|
||||
@@02_MARS_828_install_archival_strategy_trigger.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 3/8: Deploying FILE_ARCHIVER Package Specification v3.1.0
|
||||
PROMPT ===============================================================
|
||||
PROMPT Step 3/8: Deploying FILE_ARCHIVER Package Specification v3.1.2
|
||||
PROMPT ================================================================
|
||||
@@03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 4/8: Deploying FILE_ARCHIVER Package Body v3.1.0
|
||||
PROMPT Step 4/8: Deploying FILE_ARCHIVER Package Body v3.1.2
|
||||
PROMPT ======================================================
|
||||
@@04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql
|
||||
|
||||
@@ -91,7 +91,7 @@ PROMPT =====================================
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 8/8: Configuring Release 01 tables archival strategies
|
||||
PROMPT ==============================================================
|
||||
PROMPT ============================================================
|
||||
@@06_MARS_828_configure_release01_tables.sql
|
||||
|
||||
PROMPT
|
||||
@@ -103,10 +103,16 @@ SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_end FROM DUAL;
|
||||
PROMPT
|
||||
PROMPT Installation Summary:
|
||||
PROMPT - Package: CT_MRDS.FILE_ARCHIVER
|
||||
PROMPT - Version: 3.1.0 (BREAKING CHANGE - CURRENT_MONTH_ONLY removed)
|
||||
PROMPT - Version: 3.1.2 (final - includes all fixes)
|
||||
PROMPT - Strategies: THRESHOLD_BASED (default), MINIMUM_AGE_MONTHS (0=current month), HYBRID
|
||||
PROMPT - Backward Compatible: Yes (default THRESHOLD_BASED preserved)
|
||||
PROMPT - Configured Tables: 25 Release 01 tables (19 LM + 6 CSDB)
|
||||
PROMPT - Includes Fixes:
|
||||
PROMPT * v3.1.1: ORA-01422 for multiple parquet files
|
||||
PROMPT * v3.1.2: PARTITION_YEAR/PARTITION_MONTH assignments
|
||||
PROMPT * v3.1.2: Export query circular dependency
|
||||
PROMPT
|
||||
PROMPT Note: Incremental patches (v3.1.0->v3.1.1->v3.1.2) available in patches/
|
||||
PROMPT
|
||||
PROMPT Log file: &_filename
|
||||
PROMPT ============================================================================
|
||||
|
||||
@@ -154,15 +154,15 @@ AS
|
||||
vQuery:=
|
||||
'select
|
||||
s.*
|
||||
-- ,r.partition_year
|
||||
-- ,r.partition_month
|
||||
from '|| vTableName ||' s
|
||||
join CT_MRDS.A_SOURCE_FILE_RECEIVED r
|
||||
on s.file$name = r.source_file_name
|
||||
and r.a_source_file_config_key = '||pSourceFileConfigKey||'
|
||||
and r.partition_year='''||ym_loop.year||'''
|
||||
and r.partition_month='''||ym_loop.month||'''
|
||||
and r.PROCESSING_STATUS = ''INGESTED''
|
||||
join CT_MRDS.a_workflow_history h
|
||||
on s.a_workflow_history_key = h.a_workflow_history_key
|
||||
and to_char(h.workflow_start,''yyyy'') = '''||ym_loop.year||'''
|
||||
and to_char(h.workflow_start,''mm'') = '''||ym_loop.month||'''
|
||||
'
|
||||
;
|
||||
vUri := FILE_MANAGER.GET_BUCKET_URI('ARCHIVE')||vSourceFileConfig.A_SOURCE_KEY||'/'||vSourceFileConfig.TABLE_ID||'/PARTITION_YEAR='||ym_loop.year||'/PARTITION_MONTH='||ym_loop.month||'/';
|
||||
@@ -204,20 +204,14 @@ AS
|
||||
RAISE_APPLICATION_ERROR(ENV_MANAGER.CODE_EXP_DATA_FOR_ARCH_FAILED,
|
||||
ENV_MANAGER.MSG_EXP_DATA_FOR_ARCH_FAILED ||cgBL|| ' Zero rows were exported.');
|
||||
ELSE
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Data exported to archival file for YEAR_MONTH: 2025_01','INFO', vParameters);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Data exported to archival file for YEAR_MONTH: '||ym_loop.year||'_'||ym_loop.month,'INFO', vParameters);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT(FILE_MANAGER.GET_DET_USER_LOAD_OPERATIONS (pOperationId => vOperationId),'DEBUG', vParameters);
|
||||
END IF;
|
||||
|
||||
SELECT
|
||||
object_name
|
||||
into vFilename
|
||||
from DBMS_CLOUD.LIST_OBJECTS(
|
||||
credential_name => 'OCI$RESOURCE_PRINCIPAL',
|
||||
location_uri => vUri)
|
||||
where TO_UTC_TIMESTAMP_TZ(REGEXP_REPLACE(object_name, '.*(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})(\d{6})Z\.parquet$', '\1-\2-\3T\4:\5:\6.\7'))
|
||||
between vUserLoadOperations.START_TIME
|
||||
and vUserLoadOperations.UPDATE_TIME
|
||||
;
|
||||
-- Note: DBMS_CLOUD.EXPORT_DATA may create multiple parquet files (parallel execution)
|
||||
-- Instead of tracking individual files, we store the archive directory prefix
|
||||
-- ARCH_FILE_NAME will contain the directory URI where all parquet files are located
|
||||
vFilename := vUri; -- Store directory prefix instead of individual filename
|
||||
|
||||
-- Try to drop EXPORTED FILES ("regular data files")
|
||||
BEGIN
|
||||
@@ -227,7 +221,9 @@ AS
|
||||
BEGIN
|
||||
UPDATE CT_MRDS.A_SOURCE_FILE_RECEIVED r
|
||||
SET PROCESSING_STATUS = 'ARCHIVED'
|
||||
,ARCH_FILE_NAME = vUri||vFilename
|
||||
,ARCH_FILE_NAME = vFilename -- Now contains directory prefix, not individual file
|
||||
,PARTITION_YEAR = ym_loop.year -- Record which partition year the data was archived to
|
||||
,PARTITION_MONTH = ym_loop.month -- Record which partition month the data was archived to
|
||||
WHERE r.a_source_file_config_key= pSourceFileConfigKey
|
||||
AND r.source_file_name = f.filename
|
||||
AND r.processing_status = 'INGESTED'
|
||||
@@ -303,16 +299,38 @@ AS
|
||||
END;
|
||||
END LOOP;
|
||||
|
||||
DBMS_CLOUD.DELETE_OBJECT(credential_name => ENV_MANAGER.gvCredentialName,
|
||||
object_uri => vUri||vFilename);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('ROLLBACK operation: Archival PARQUET file dropped.','DEBUG', vUri||vFilename);
|
||||
-- ROLLBACK: Delete all parquet files from archive directory
|
||||
FOR arch_file IN (
|
||||
SELECT object_name
|
||||
FROM DBMS_CLOUD.LIST_OBJECTS(
|
||||
credential_name => ENV_MANAGER.gvCredentialName,
|
||||
location_uri => vFilename -- vFilename now contains directory prefix
|
||||
)
|
||||
) LOOP
|
||||
DBMS_CLOUD.DELETE_OBJECT(
|
||||
credential_name => ENV_MANAGER.gvCredentialName,
|
||||
object_uri => vFilename || arch_file.object_name
|
||||
);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('ROLLBACK operation: Archival PARQUET file dropped.','DEBUG', vFilename || arch_file.object_name);
|
||||
END LOOP;
|
||||
RAISE_APPLICATION_ERROR(ENV_MANAGER.CODE_MOVE_FILE_TO_TRASH_FAILED, ENV_MANAGER.MSG_MOVE_FILE_TO_TRASH_FAILED);
|
||||
|
||||
ELSIF vProcessControlStatus = 'CHANGE_STATUS_TO_ARCHIVED_FAILURE' THEN
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT(ENV_MANAGER.MSG_CHANGE_STAT_TO_ARCHIVED_FAILED, 'ERROR', vParameters);
|
||||
DBMS_CLOUD.DELETE_OBJECT(credential_name => ENV_MANAGER.gvCredentialName,
|
||||
object_uri => vUri||vFilename);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Archival PARQUET file dropped.','DEBUG', vUri||vFilename);
|
||||
-- ROLLBACK: Delete all parquet files from archive directory
|
||||
FOR arch_file IN (
|
||||
SELECT object_name
|
||||
FROM DBMS_CLOUD.LIST_OBJECTS(
|
||||
credential_name => ENV_MANAGER.gvCredentialName,
|
||||
location_uri => vFilename -- vFilename now contains directory prefix
|
||||
)
|
||||
) LOOP
|
||||
DBMS_CLOUD.DELETE_OBJECT(
|
||||
credential_name => ENV_MANAGER.gvCredentialName,
|
||||
object_uri => vFilename || arch_file.object_name
|
||||
);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Archival PARQUET file dropped.','DEBUG', vFilename || arch_file.object_name);
|
||||
END LOOP;
|
||||
RAISE_APPLICATION_ERROR(ENV_MANAGER.CODE_CHANGE_STAT_TO_ARCHIVED_FAILED, ENV_MANAGER.MSG_CHANGE_STAT_TO_ARCHIVED_FAILED);
|
||||
|
||||
ELSIF vProcessControlStatus = 'RESTORE_FILE_FROM_TRASH_FAILURE' THEN
|
||||
|
||||
@@ -17,12 +17,14 @@ AS
|
||||
**/
|
||||
|
||||
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.0';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-01-29 21:00:00';
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.2';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-02-06 11:30:00';
|
||||
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
|
||||
|
||||
-- Version History (Latest changes first)
|
||||
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
|
||||
'3.1.2 (2026-02-06): Fixed missing PARTITION_YEAR/PARTITION_MONTH assignments in UPDATE statement and export query circular dependency (now filters by workflow_start instead of partition fields)' || CHR(13)||CHR(10) ||
|
||||
'3.1.1 (2026-02-06): Fixed ORA-01422 error when DBMS_CLOUD.EXPORT_DATA creates multiple parquet files (parallel execution). Now stores archive directory prefix instead of individual filenames' || CHR(13)||CHR(10) ||
|
||||
'3.1.0 (2026-01-29): Added function overloads for ARCHIVE_TABLE_DATA and GATHER_TABLE_STAT returning SQLCODE for Python library integration' || CHR(13)||CHR(10) ||
|
||||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
|
||||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
|
||||
|
||||
Reference in New Issue
Block a user