admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: admin:76c8ce6cc7ece702bcdd36744573484e631ed04a
Branches Tags
admin:main admin:develop
..
compare: admin:e6ab189dc94905ec0609c7e416360238cba0ff4e
Branches Tags
admin:main admin:develop

9 Commits
76c8ce6cc7 ... e6ab189dc9

Author SHA1 Message Date
Grzegorz Michalski
e6ab189dc9 Refactor archival strategies in FILE_ARCHIVER package 
- Removed CURRENT_MONTH_ONLY strategy, replacing it with MINIMUM_AGE_MONTHS = 0 for current month retention.
- Updated validation logic to allow MINIMUM_AGE_MONTHS to be non-negative.
- Consolidated documentation to reflect changes in archival strategies.
- Adjusted rollback script to reset all archival parameters to NULL for 25 Release 01 tables.
- Enhanced README and installation scripts to clarify new configuration requirements and usage examples.
- Updated version history to indicate breaking changes and migration steps for existing configurations.
 2026-02-05 20:17:51 +01:00
Grzegorz Michalski
e93140e962 zmiany w pakiecie 2026-02-05 11:03:15 +01:00
Grzegorz Michalski
f42777f480 dokumentacja 2026-02-05 10:54:48 +01:00
Grzegorz Michalski
19925603fa aktualizacja definicji 2026-02-04 14:41:30 +01:00
Grzegorz Michalski
569c647b6c dopracowanie paczki 2026-02-04 12:22:33 +01:00
Grzegorz Michalski
8a162ce9bf dok 2026-02-04 12:21:48 +01:00
Grzegorz Michalski
c508e1e4e2 Refactor archival strategy rollback scripts for Release 01 tables 2026-02-03 19:39:08 +01:00
Grzegorz Michalski
d6fd0b3dbd Add scripts for configuring and rolling back archival strategies for Release 01 tables 2026-02-03 19:02:32 +01:00
Grzegorz Michalski
32c7a5dcee wk 2026-02-03 18:38:17 +01:00
24 changed files with 3007 additions and 1111 deletions
Expand all files Collapse all files

4
MARS_Packages/REL01_ADDITIONS/MARS-828/01_MARS_828_install_add_archival_strategy_columns.sql
View File

@@ -16,12 +16,12 @@ ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
-- Add check constraint for valid strategies
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD CONSTRAINT
CHK_ARCHIVAL_STRATEGY CHECK (
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'MINIMUM_AGE_MONTHS', 'HYBRID')
);
-- Add comments
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ARCHIVAL_STRATEGY IS
'Archival strategy: THRESHOLD_BASED (days), CURRENT_MONTH_ONLY (exclude current month), MINIMUM_AGE_MONTHS (minimum age), HYBRID (combination)';
'Archival strategy: THRESHOLD_BASED (days), MINIMUM_AGE_MONTHS (0=current month, N=retain N months), HYBRID (combination)';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.MINIMUM_AGE_MONTHS IS
'Minimum age in months for archival (used with MINIMUM_AGE_MONTHS or HYBRID strategies)';

325
MARS_Packages/REL01_ADDITIONS/MARS-828/06_MARS_828_configure_release01_tables.sql Normal file
View File

@@ -0,0 +1,325 @@
-- =====================================================================
-- Script: 06_MARS_828_configure_release01_tables.sql
-- MARS Issue: MARS-828
-- Purpose: Configure COMPLETE archival parameters for Release 01 tables
-- Author: Grzegorz Michalski
-- Date: 2026-02-05
--
-- Description:
-- Configures ALL archival parameters for 25 tables:
-- - ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS
-- - Archival triggers (FILES/ROWS/BYTES thresholds)
-- - Statistics expiration (HOURS_TO_EXPIRE_STATISTICS)
--
-- Configuration by group:
-- - 19 LM tables: MINIMUM_AGE_MONTHS=0 (current month only), 10 files OR 100K rows OR 1GB, 24h stats
-- - 2 CSDB DEBT: MINIMUM_AGE_MONTHS=6, 5 files OR 50K rows OR 512MB, 48h stats
-- - 4 CSDB ratings: MINIMUM_AGE_MONTHS=0 (current month only), 10 files OR 20K rows OR 256MB, 72h stats
--
-- Dependencies:
-- - A_SOURCE_FILE_CONFIG table with all archival columns
-- - TRG_BI_A_SRC_FILE_CFG_ARCH_VAL trigger
-- =====================================================================
PROMPT =====================================================================
PROMPT MARS-828: Enhanced Configuration - Complete Archival Parameters
PROMPT =====================================================================
PROMPT
PROMPT This script configures COMPLETE archival parameters for 25 tables
PROMPT
PROMPT LM Tables (19):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 0 (current month only)
PROMPT - Triggers: 10 files OR 100,000 rows OR 1 GB
PROMPT - Stats Expiration: 24 hours
PROMPT
PROMPT CSDB DEBT Tables (2):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 6
PROMPT - Triggers: 5 files OR 50,000 rows OR 512 MB
PROMPT - Stats Expiration: 48 hours
PROMPT
PROMPT CSDB Rating/Description Tables (4):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 0 (current month only)
PROMPT - Triggers: 10 files OR 20,000 rows OR 256 MB
PROMPT - Stats Expiration: 72 hours
PROMPT
PROMPT Current timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS CURRENT_TIME FROM DUAL;
PROMPT
PROMPT =====================================================================
PROMPT SECTION 1: LM Tables Configuration (MINIMUM_AGE_MONTHS = 0)
PROMPT =====================================================================
PROMPT Thresholds: 10 files OR 100K rows OR 1GB
PROMPT Stats expire: 24 hours
PROMPT =====================================================================
-- Update all 19 LM tables in a single statement
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 100000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824, -- 1 GB
HOURS_TO_EXPIRE_STATISTICS = 24
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'LM'
AND TABLE_ID IN (
'LM_STANDING_FACILITIES',
'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER',
'LM_TTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_HEADER',
'LM_ADHOC_ADJUSTMENTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
'LM_BALANCESHEET_HEADER',
'LM_BALANCESHEET_ITEM',
'LM_CSM_ADJUSTMENTS_HEADER',
'LM_CSM_ADJUSTMENTS_ITEM',
'LM_CSM_ADJUSTMENTS_ITEM_HEADER',
'LM_CURRENT_ACCOUNTS_HEADER',
'LM_CURRENT_ACCOUNTS_ITEM',
'LM_FORECAST_HEADER',
'LM_FORECAST_ITEM',
'LM_QRE_ADJUSTMENTS_HEADER',
'LM_QRE_ADJUSTMENTS_ITEM',
'LM_QRE_ADJUSTMENTS_ITEM_HEADER'
);
PROMPT
PROMPT LM tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 2: CSDB DEBT Tables (MINIMUM_AGE_MONTHS = 6)
PROMPT =====================================================================
PROMPT Thresholds: 5 files OR 50K rows OR 512MB
PROMPT Stats expire: 48 hours
PROMPT =====================================================================
-- Update CSDB DEBT tables (6-month retention)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 5,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 50000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 536870912, -- 512 MB
HOURS_TO_EXPIRE_STATISTICS = 48
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
PROMPT
PROMPT CSDB DEBT tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 3: CSDB Rating/Description Tables (MINIMUM_AGE_MONTHS = 0)
PROMPT =====================================================================
PROMPT Thresholds: 10 files OR 20K rows OR 256MB
PROMPT Stats expire: 72 hours
PROMPT =====================================================================
-- Update CSDB rating/description tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 20000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 268435456, -- 256 MB
HOURS_TO_EXPIRE_STATISTICS = 72
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN (
'CSDB_INSTR_RAT_FULL',
'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL',
'CSDB_ISSUER_DESC_FULL'
);
PROMPT
PROMPT CSDB rating/description tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT Committing configuration changes...
PROMPT =====================================================================
COMMIT;
PROMPT
PROMPT Configuration committed successfully.
PROMPT
PROMPT =====================================================================
PROMPT VERIFICATION: Complete Archival Configuration
PROMPT =====================================================================
PROMPT
PROMPT LM Tables (MINIMUM_AGE_MONTHS = 0):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 0
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 100000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824
AND HOURS_TO_EXPIRE_STATISTICS = 24
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'LM'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID LIKE 'LM_%'
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB DEBT Tables (MINIMUM_AGE_MONTHS = 6):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 6
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 5
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 50000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 536870912
AND HOURS_TO_EXPIRE_STATISTICS = 48
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY')
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB Rating/Description Tables (MINIMUM_AGE_MONTHS = 0):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 0
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 20000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 268435456
AND HOURS_TO_EXPIRE_STATISTICS = 72
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL', 'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL')
ORDER BY TABLE_ID;
PROMPT
PROMPT =====================================================================
PROMPT Summary: Total Configured Tables with Full Parameters
PROMPT =====================================================================
SELECT
COUNT(*) AS TOTAL_CONFIGURED,
SUM(CASE WHEN MINIMUM_AGE_MONTHS = 0 THEN 1 ELSE 0 END) AS CURRENT_MONTH_ONLY,
SUM(CASE WHEN MINIMUM_AGE_MONTHS > 0 THEN 1 ELSE 0 END) AS MULTI_MONTH_RETENTION,
SUM(CASE WHEN FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_FILE_THRESHOLD,
SUM(CASE WHEN ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_ROWS_THRESHOLD,
SUM(CASE WHEN BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_BYTES_THRESHOLD,
SUM(CASE WHEN HOURS_TO_EXPIRE_STATISTICS IS NOT NULL THEN 1 ELSE 0 END) AS WITH_STATS_EXPIRY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)));
PROMPT
PROMPT Expected:
PROMPT - TOTAL_CONFIGURED = 25
PROMPT - WITH_FILE_THRESHOLD = 25
PROMPT - WITH_ROWS_THRESHOLD = 25
PROMPT - WITH_BYTES_THRESHOLD = 25
PROMPT - WITH_STATS_EXPIRY = 25
PROMPT
PROMPT =====================================================================
PROMPT Parameter Configuration Summary by Group
PROMPT =====================================================================
SELECT
CASE
WHEN A_SOURCE_KEY = 'LM' THEN 'LM Tables'
WHEN TABLE_ID LIKE 'CSDB_DEBT%' THEN 'CSDB DEBT'
ELSE 'CSDB Ratings'
END AS GROUP_NAME,
COUNT(*) AS TABLE_COUNT,
MAX(ARCHIVAL_STRATEGY) AS STRATEGY,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE,
MAX(FILES_COUNT_OVER_ARCHIVE_THRESHOLD) AS FILES_THRESHOLD,
MAX(ROWS_COUNT_OVER_ARCHIVE_THRESHOLD) AS ROWS_THRESHOLD,
ROUND(MAX(BYTES_SUM_OVER_ARCHIVE_THRESHOLD)/1048576, 0) || ' MB' AS BYTES_THRESHOLD,
MAX(HOURS_TO_EXPIRE_STATISTICS) AS STATS_HOURS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)))
GROUP BY
CASE
WHEN A_SOURCE_KEY = 'LM' THEN 'LM Tables'
WHEN TABLE_ID LIKE 'CSDB_DEBT%' THEN 'CSDB DEBT'
ELSE 'CSDB Ratings'
END
ORDER BY GROUP_NAME;
PROMPT
PROMPT =====================================================================
PROMPT Configuration Complete
PROMPT =====================================================================
PROMPT
PROMPT Next Steps:
PROMPT 1. Review verification results above
PROMPT 2. All 25 tables should show STATUS = 'OK'
PROMPT 3. Verify parameter configuration summary matches expectations
PROMPT 4. Monitor archival process with new thresholds
PROMPT
PROMPT Archival Trigger Logic (OR condition):
PROMPT - Archival starts when ANY threshold is exceeded
PROMPT - Example: 10 files OR 100K rows OR 1GB triggers archival
PROMPT
PROMPT Statistics Refresh:
PROMPT - LM: Every 24 hours (daily updates for frequent changes)
PROMPT - CSDB DEBT: Every 48 hours (bi-daily for larger datasets)
PROMPT - CSDB Ratings: Every 72 hours (every 3 days for stable data)
PROMPT
PROMPT Log file: 06_MARS_828_configure_release01_tables.log
PROMPT =====================================================================

271
MARS_Packages/REL01_ADDITIONS/MARS-828/96_MARS_828_rollback_release01_configuration.sql Normal file
View File

@@ -0,0 +1,271 @@
-- =====================================================================
-- Script: 96_MARS_828_rollback_release01_configuration.sql
-- MARS Issue: MARS-828
-- Purpose: ROLLBACK archival configuration for Release 01 tables
-- Author: Grzegorz Michalski
-- Date: 2026-02-05
--
-- Description:
-- Rolls back archival configuration for 25 tables by setting all
-- archival parameters back to NULL (unconfigured state):
-- - ARCHIVAL_STRATEGY
-- - MINIMUM_AGE_MONTHS
-- - FILES_COUNT_OVER_ARCHIVE_THRESHOLD
-- - ROWS_COUNT_OVER_ARCHIVE_THRESHOLD
-- - BYTES_SUM_OVER_ARCHIVE_THRESHOLD
-- - HOURS_TO_EXPIRE_STATISTICS
--
-- This script reverts changes made by:
-- - 06_MARS_828_configure_release01_tables.sql
--
-- Dependencies:
-- - A_SOURCE_FILE_CONFIG table with archival columns
-- =====================================================================
PROMPT =====================================================================
PROMPT MARS-828: ROLLBACK - Remove Archival Configuration
PROMPT =====================================================================
PROMPT
PROMPT This script will REMOVE archival configuration from 25 tables
PROMPT
PROMPT WARNING: This will reset ALL archival parameters to NULL
PROMPT
PROMPT Tables affected:
PROMPT - 19 LM tables
PROMPT - 2 CSDB DEBT tables
PROMPT - 4 CSDB rating/description tables
PROMPT
PROMPT Current timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS CURRENT_TIME FROM DUAL;
PROMPT
PROMPT =====================================================================
PROMPT SECTION 1: Rollback LM Tables Configuration
PROMPT =====================================================================
-- Rollback all 19 LM tables in a single statement
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'LM'
AND TABLE_ID IN (
'LM_STANDING_FACILITIES',
'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER',
'LM_TTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_HEADER',
'LM_ADHOC_ADJUSTMENTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
'LM_BALANCESHEET_HEADER',
'LM_BALANCESHEET_ITEM',
'LM_CSM_ADJUSTMENTS_HEADER',
'LM_CSM_ADJUSTMENTS_ITEM',
'LM_CSM_ADJUSTMENTS_ITEM_HEADER',
'LM_CURRENT_ACCOUNTS_HEADER',
'LM_CURRENT_ACCOUNTS_ITEM',
'LM_FORECAST_HEADER',
'LM_FORECAST_ITEM',
'LM_QRE_ADJUSTMENTS_HEADER',
'LM_QRE_ADJUSTMENTS_ITEM',
'LM_QRE_ADJUSTMENTS_ITEM_HEADER'
);
PROMPT
PROMPT LM tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 2: Rollback CSDB DEBT Tables Configuration
PROMPT =====================================================================
-- Rollback CSDB DEBT tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
PROMPT
PROMPT CSDB DEBT tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 3: Rollback CSDB Rating/Description Tables Configuration
PROMPT =====================================================================
-- Rollback CSDB rating/description tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN (
'CSDB_INSTR_RAT_FULL',
'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL',
'CSDB_ISSUER_DESC_FULL'
);
PROMPT
PROMPT CSDB rating/description tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT Committing rollback changes...
PROMPT =====================================================================
COMMIT;
PROMPT
PROMPT Rollback committed successfully.
PROMPT
PROMPT =====================================================================
PROMPT VERIFICATION: Archival Configuration Removed
PROMPT =====================================================================
PROMPT
PROMPT LM Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'LM'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID LIKE 'LM_%'
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB DEBT Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY')
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB Rating/Description Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL', 'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL')
ORDER BY TABLE_ID;
PROMPT
PROMPT =====================================================================
PROMPT Summary: Total Tables with Rollback Applied
PROMPT =====================================================================
SELECT
COUNT(*) AS TOTAL_TABLES,
SUM(CASE WHEN ARCHIVAL_STRATEGY IS NULL THEN 1 ELSE 0 END) AS STRATEGY_NULL,
SUM(CASE WHEN MINIMUM_AGE_MONTHS IS NULL THEN 1 ELSE 0 END) AS MIN_AGE_NULL,
SUM(CASE WHEN FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS FILE_THR_NULL,
SUM(CASE WHEN ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS ROWS_THR_NULL,
SUM(CASE WHEN BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS BYTES_THR_NULL,
SUM(CASE WHEN HOURS_TO_EXPIRE_STATISTICS IS NULL THEN 1 ELSE 0 END) AS STATS_NULL
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)));
PROMPT
PROMPT Expected: All counts should be 25 (all parameters NULL for all tables)
PROMPT - TOTAL_TABLES = 25
PROMPT - STRATEGY_NULL = 25
PROMPT - MIN_AGE_NULL = 25
PROMPT - FILE_THR_NULL = 25
PROMPT - ROWS_THR_NULL = 25
PROMPT - BYTES_THR_NULL = 25
PROMPT - STATS_NULL = 25
PROMPT
PROMPT =====================================================================
PROMPT Rollback Complete
PROMPT =====================================================================
PROMPT
PROMPT Next Steps:
PROMPT 1. Review verification results above
PROMPT 2. All 25 tables should show STATUS = 'OK'
PROMPT 3. All archival parameters should be NULL
PROMPT 4. Tables are now in unconfigured state (no archival strategy)
PROMPT
PROMPT To restore configuration, re-run:
PROMPT @06_MARS_828_configure_release01_tables.sql
PROMPT
PROMPT Log file: 96_MARS_828_rollback_release01_configuration.log
PROMPT =====================================================================

16
MARS_Packages/REL01_ADDITIONS/MARS-828/README.md
View File

@@ -17,7 +17,7 @@ WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRES
**Solution**: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:
- `THRESHOLD_BASED` - Days-based threshold (backward compatible)
- `CURRENT_MONTH_ONLY` - Keep only current month data
- `MINIMUM_AGE_MONTHS` - Retain data for specified months (0 = current month only)
- `MINIMUM_AGE_MONTHS` - Archive data older than X months
- `HYBRID` - Combination of current month and minimum age
@@ -38,7 +38,7 @@ DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3),
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
```
**New Trigger**: TRG_BI_ARCHIVAL_STRATEGY_VAL
@@ -69,17 +69,17 @@ CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED',
| Strategy | WHERE Clause Logic | Configuration Required | Use Case |
|----------|-------------------|----------------------|----------|
| `THRESHOLD_BASED` | `extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD` | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility |
| `CURRENT_MONTH_ONLY` | `TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')` | None | General sources (LM, TOP) |
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -N)` | MINIMUM_AGE_MONTHS (0=current month) | All sources |
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X)` | MINIMUM_AGE_MONTHS | CSDB (6 months retention) |
| `HYBRID` | Both CURRENT_MONTH_ONLY AND MINIMUM_AGE_MONTHS | MINIMUM_AGE_MONTHS | Advanced scenarios |
| `HYBRID` | Combines month truncation AND minimum age | MINIMUM_AGE_MONTHS | Advanced scenarios |
## Configuration Examples
```sql
-- LM/TOP sources: Archive everything except current month
UPDATE A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
MINIMUM_AGE_MONTHS = NULL
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
WHERE A_SOURCE_KEY IN ('LM', 'TOP');
-- CSDB: Archive only data older than 6 months
@@ -189,7 +189,7 @@ sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
**Main Test Script**: test/test_archival_strategies.sql
```sql
-- Test 1: CURRENT_MONTH_ONLY strategy
-- Test 1: MINIMUM_AGE_MONTHS=0 strategy (current month only)
-- Expected: Archives data from previous months only
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');
@@ -249,7 +249,7 @@ No automatic migration required. New columns have sensible defaults:
- MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)
## Version History
- **v3.0.0** (2026-01-27): Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration
- **v3.0.0** (2026-01-27): Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration
- **v2.0.0** (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival
## Related JIRA Issues

38
MARS_Packages/REL01_ADDITIONS/MARS-828/install_mars828.sql
View File

@@ -35,9 +35,9 @@ PROMPT =========================================================================
PROMPT MARS-828 Installation Starting
PROMPT ============================================================================
PROMPT Package: CT_MRDS.FILE_ARCHIVER
PROMPT Change: Enhanced archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID)
PROMPT Change: Enhanced archival strategies (MINIMUM_AGE_MONTHS, HYBRID)
PROMPT Purpose: Flexible archival policies per data source
PROMPT Steps: 7 (DDL, Trigger, Packages, Verify, Track, Verify)
PROMPT Steps: 8 (DDL, Trigger, Packages, Verify, Track, Verify, Configure)
PROMPT Timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_start FROM DUAL;
PROMPT ============================================================================
@@ -54,40 +54,46 @@ END;
WHENEVER SQLERROR CONTINUE
-- Installation steps
PROMPT7: Adding archival strategy columns to A_SOURCE_FILE_CONFIG
PROMPT
PROMPT Step 1/8: Adding archival strategy columns to A_SOURCE_FILE_CONFIG
PROMPT ===================================================================
@@01_MARS_828_install_add_archival_strategy_columns.sql
PROMPT
PROMPT Step 2/7: Creating validation trigger
PROMPT Step 2/8: Creating validation trigger
PROMPT ======================================
@@02_MARS_828_install_archival_strategy_trigger.sql
PROMPT
PROMPT Step 3/7: Deploying FILE_ARCHIVER Package Specification v3.0.0
PROMPT Step 3/8: Deploying FILE_ARCHIVER Package Specification v3.1.0
PROMPT ===============================================================
@@03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 4/7: Deploying FILE_ARCHIVER Package Body v3.0.0
PROMPT Step 4/8: Deploying FILE_ARCHIVER Package Body v3.1.0
PROMPT ======================================================
@@04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 5/7: Verifying installation
PROMPT Step 5/8: Verifying installation
PROMPT =================================
@@05_MARS_828_verify_installation.sql
PROMPT
PROMPT Step 6/7: Tracking package versions
PROMPT Step 6/8: Tracking package versions
PROMPT ====================================
@@track_package_versions.sql
PROMPT
PROMPT Step 7/7: Verifying tracked packages
PROMPT Step 7/8: Verifying tracked packages
PROMPT =====================================
@@verify_packages_version.sql
PROMPT
PROMPT Step 8/8: Configuring Release 01 tables archival strategies
PROMPT ==============================================================
@@06_MARS_828_configure_release01_tables.sql
PROMPT
PROMPT ============================================================================
PROMPT MARS-828 Installation Completed
@@ -97,16 +103,10 @@ SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_end FROM DUAL;
PROMPT
PROMPT Installation Summary:
PROMPT - Package: CT_MRDS.FILE_ARCHIVER
PROMPT - Version: 2.0.0 -> 3.0.0 (MAJOR)
PROMPT - New Strategies: CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID
PROMPT - Backward Compatible: THRESHOLD_BASED (default)
PROMPT
PROMPT Next Steps:
PROMPT 1. Configure archival strategies per source:
PROMPT UPDATE A_SOURCE_FILE_CONFIG SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY' WHERE A_SOURCE_KEY = 'LM';
PROMPT UPDATE A_SOURCE_FILE_CONFIG SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS', MINIMUM_AGE_MONTHS = 6 WHERE A_SOURCE_KEY = 'CSDB';
PROMPT 2. Test strategies using test_archival_strategies.sql
PROMPT 3. Monitor first archival run
PROMPT - Version: 3.1.0 (BREAKING CHANGE - CURRENT_MONTH_ONLY removed)
PROMPT - Strategies: THRESHOLD_BASED (default), MINIMUM_AGE_MONTHS (0=current month), HYBRID
PROMPT - Backward Compatible: Yes (default THRESHOLD_BASED preserved)
PROMPT - Configured Tables: 25 Release 01 tables (19 LM + 6 CSDB)
PROMPT
PROMPT Log file: &_filename
PROMPT ============================================================================

310
MARS_Packages/REL01_ADDITIONS/MARS-828/log/INSTALL_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260128_064911.log
View File

@@ -1,310 +0,0 @@
============================================================================
MARS-828 Installation Starting (AUTO MODE)
============================================================================
INSTALL_START
______________________
2026-01-28 06:49:11
1 row selected.
============================================================================
Step 1/7: Adding archival strategy columns
===================================================================
========================================
MARS-828: Adding archival strategy columns
========================================
Error starting at line : 11 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\01_MARS_828_install_add_archival_strategy_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3) DEFAULT NULL
)
Error report -
ORA-01430: column being added already exists in table
https://docs.oracle.com/error-help/db/ora-01430/
01430. 00000 - "column being added already exists in table"
*Cause: An ALTER TABLE ADD statement specified the name of a
column that was already in the table. All column names must be
unique within a table.
*Action: Specify a unique name for the new column, then
re-execute the statement.
Error starting at line : 17 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\01_MARS_828_install_add_archival_strategy_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD CONSTRAINT
CHK_ARCHIVAL_STRATEGY CHECK (
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
)
Error report -
ORA-02264: name already used by an existing constraint
https://docs.oracle.com/error-help/db/ora-02264/
02264. 00000 - "name already used by an existing constraint"
*Cause: The specified constraint name has to be unique.
*Action: Specify a unique constraint name for the constraint.
Comment created.
Comment created.
COLUMN_NAME DATA_TYPE DATA_LENGTH NULLABLE DATA_DEFAULT
_____________________ ____________ ______________ ___________ _____________________
ARCHIVAL_STRATEGY VARCHAR2 30 N 'THRESHOLD_BASED'
MINIMUM_AGE_MONTHS NUMBER 22 Y NULL
2 rows selected.
========================================
Archival strategy columns added successfully
========================================
Step 2/7: Creating validation trigger
======================================
========================================
MARS-828: Creating archival strategy validation trigger
========================================
Trigger CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL compiled
TRIGGER_NAME STATUS TRIGGER_TYPE TRIGGERING_EVENT
_________________________________ __________ __________________ ___________________
TRG_BI_A_SRC_FILE_CFG_ARCH_VAL ENABLED BEFORE EACH ROW INSERT OR UPDATE
1 row selected.
========================================
Archival strategy validation trigger created successfully
========================================
Step 3/7: Deploying FILE_ARCHIVER Spec v3.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
========================================
FILE_ARCHIVER Specification v3.0.0 ready for installation
========================================
Step 4/7: Deploying FILE_ARCHIVER Body v3.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
LINE/COL ERROR
--------- -------------------------------------------------------------
36/10 PLS-00103: Encountered the symbol "\" when expecting one of the following: ( begin case declare else end exit for goto if loop mod null pragma raise return select update when while with <an identifier> <a double-quoted delimited-identifier> <a bind variable> << continue close current delete fetch lock insert open rollback savepoint set sql execute commit forall merge pipe purge json_object The symbol "case was inserted before "\" to continue.
49/8 PLS-00103: Encountered the symbol "GET_ARCHIVAL_WHERE_CLAUSE" when expecting one of the following: case
67/9 PLS-00103: Encountered the symbol "JOIN" when expecting one of the following: , ; for group having intersect minus order start union where connect
Errors: check compiler log
Step 5/7: Verifying installation
=================================
========================================
MARS-828: Verification Script
========================================
1. Verifying A_SOURCE_FILE_CONFIG columns...
COLUMN_NAME DATA_TYPE NULLABLE DATA_DEFAULT
_____________________ ____________ ___________ _____________________
ARCHIVAL_STRATEGY VARCHAR2 N 'THRESHOLD_BASED'
MINIMUM_AGE_MONTHS NUMBER Y NULL
2 rows selected.
2. Verifying check constraint...
CONSTRAINT_NAME CONSTRAINT_TYPE SEARCH_CONDITION
________________________ __________________ _________________________________________________________________________________________________
CHK_ARCHIVAL_STRATEGY C ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
1 row selected.
3. Verifying validation trigger...
TRIGGER_NAME STATUS TRIGGER_TYPE
_________________________________ __________ __________________
TRG_BI_A_SRC_FILE_CFG_ARCH_VAL ENABLED BEFORE EACH ROW
1 row selected.
4. Checking FILE_ARCHIVER package status...
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ __________ ______________________
FILE_ARCHIVER PACKAGE VALID 2026-01-28 06:49:14
FILE_ARCHIVER PACKAGE BODY INVALID 2026-01-28 06:49:15
2 rows selected.
5. Checking for compilation errors...
NAME TYPE LINE POSITION TEXT
________________ _______________ _______ ___________ _____________________________________________________________________________
FILE_ARCHIVER PACKAGE BODY 36 10 PLS-00103: Encountered the symbol "\" when expecting one of the following:
( begin case declare else end exit for goto if loop mod null
pragma raise return select update when while with
<an identifier> <a double-quoted delimited-identifier>
<a bind variable> << continue close current delete fetch lock
insert open rollback savepoint set sql execute commit forall
merge pipe purge json_object
The symbol "case was inserted before "\" to continue.
FILE_ARCHIVER PACKAGE BODY 49 8 PLS-00103: Encountered the symbol "GET_ARCHIVAL_WHERE_CLAUSE" when expecting one of the following:
case
FILE_ARCHIVER PACKAGE BODY 67 9 PLS-00103: Encountered the symbol "JOIN" when expecting one of the following:
, ; for group having intersect minus order start union where
connect
3 rows selected.
6. Verifying FILE_ARCHIVER version...
Error starting at line : 79 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\05_MARS_828_verify_installation.sql
In command -
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() as package_version FROM DUAL
Error at Command Line : 79 Column : 68 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\05_MARS_828_verify_installation.sql
Error report -
SQL Error: ORA-04063: package body "CT_MRDS.FILE_ARCHIVER" has errors
https://docs.oracle.com/error-help/db/ora-04063/04063. 00000 - "%s has errors"
*Cause: Attempt to execute a stored procedure or use a view that has
errors. For stored procedures, the problem could be syntax errors
or references to other, non-existent procedures. For views,
the problem could be a reference in the view's defining query to
a non-existent table.
Can also be a table which has references to non-existent or
inaccessible types.
*Action: Fix the errors and/or create referenced objects as necessary.
More Details :
https://docs.oracle.com/error-help/db/ora-04063/
7. Testing trigger validation (should fail)...
SUCCESS: Trigger validation working correctly
Expected error: ORA-20999: MINIMUM_AGE_MONTHS is required for MINIMUM_AGE_MONTHS strategy
ORA-06512: at "CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL", line 26
ORA-06512: at "CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL", line 8
ORA-04088: error during execution of trigger 'CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL'
PL/SQL procedure successfully completed.
========================================
MARS-828: Verification Complete
========================================
Step 6/7: Tracking package versions
====================================
========================================
Package Version Tracking
========================================
Summary:
--------
Packages tracked: 0/1
========================================
PL/SQL procedure successfully completed.
Step 7/7: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ _______________________________________________________
CT_MRDS DATA_EXPORTER 2.5.0 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-26 17:15:41
Version: 2.5.0
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-26 17:15:41
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 WARNING: Package CT_MRDS.FILE_ARCHIVER has changed!
========================================
Last Tracked Version: 2.0.0
Last Tracked Date: 2025-11-25 16:00:36
SPECIFICATION Changed:
Current Hash: 71a835e531971ca7...
Last Hash: 836641155e237fc5...
BODY Changed:
Current Hash: 6f87dff6b0394529...
Last Hash: 9bf9b3c0e059493c...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
EnvironmentID set to: dev
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
============================================================================
MARS-828 Installation Completed
============================================================================
INSTALL_END
______________________
2026-01-28 06:49:23
1 row selected.
============================================================================

194
MARS_Packages/REL01_ADDITIONS/MARS-828/log/ROLLBACK_MARS_828_AUTO_G45C5E88148E17E_GGMICHALSKI_20260129_195230.log
View File

@@ -1,194 +0,0 @@
============================================================================
MARS-828 Rollback Starting (AUTO MODE - No Confirmation)
============================================================================
ROLLBACK_START
______________________
2026-01-29 19:52:30
Elapsed: 00:00:00.065
============================================================================
Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.110
Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.133
Step 3/6: Dropping validation trigger
======================================
========================================
MARS-828: Dropping archival strategy validation trigger
========================================
Error starting at line : 10 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\93_MARS_828_rollback_trigger.sql
In command -
DROP TRIGGER CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL
Error report -
ORA-04080: trigger 'TRG_BI_A_SRC_FILE_CFG_ARCH_VAL' does not exist
https://docs.oracle.com/error-help/db/ora-04080/
04080. 00000 - "trigger '%s' does not exist"
*Cause: The TRIGGER name is invalid.
*Action: Check the trigger name.
Elapsed: 00:00:00.135
TRIGGER_COUNT
________________
0
Elapsed: 00:00:00.070
========================================
Validation trigger dropped successfully
========================================
Step 4/6: Dropping archival strategy columns
=============================================
========================================
MARS-828: Removing archival strategy columns
========================================
Error starting at line : 11 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\94_MARS_828_rollback_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG
DROP CONSTRAINT CHK_ARCHIVAL_STRATEGY
Error report -
ORA-02443: Cannot drop constraint - nonexistent constraint
https://docs.oracle.com/error-help/db/ora-02443/
02443. 00000 - "Cannot drop constraint - nonexistent constraint"
*Cause: alter table drop constraint <constraint_name>
*Action: make sure you supply correct constraint name.
Elapsed: 00:00:00.118
Error starting at line : 15 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\94_MARS_828_rollback_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP (
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
)
Error report -
ORA-00904: "MINIMUM_AGE_MONTHS": invalid identifier
https://docs.oracle.com/error-help/db/ora-00904/
00904. 00000 - "%s: invalid identifier"
*Cause: The identifier or column name entered was invalid.
*Action: Ensure the following
Elapsed: 00:00:00.113
no rows selected
Elapsed: 00:00:00.066
========================================
Archival strategy columns removed successfully
========================================
Step 5/6: Tracking rollback version
====================================
========================================
Package Version Tracking
========================================
EnvironmentID set to: dev
[2026-01-29 19:52:34] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: Start TRACK_PACKAGE_VERSION
[2026-01-29 19:52:34] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: End TRACK_PACKAGE_VERSION - Record inserted
Summary:
--------
Packages tracked: 1/1
Tracked Packages:
CT_MRDS.FILE_ARCHIVER v2.0.0
========================================
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.232
Step 6/6: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ ______________________________________________________
CT_MRDS DATA_EXPORTER 2.6.3 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 2.6.3
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 OK: Package CT_MRDS.FILE_ARCHIVER has not changed.
Last Tracked: 2026-01-29 19:52:34
Version: 2.0.0
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
Elapsed: 00:00:00.388
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
Verification: Package Compilation Status
=========================================
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ _________ ________________
FILE_ARCHIVER PACKAGE VALID 29-JAN-26
FILE_ARCHIVER PACKAGE BODY VALID 29-JAN-26
2 rows selected.
Elapsed: 00:00:00.100
============================================================================
MARS-828 Rollback Completed
============================================================================
ROLLBACK_END
______________________
2026-01-29 19:52:36
1 row selected.
Elapsed: 00:00:00.058
Log file: ../log/ROLLBACK_MARS_828_AUTO_G45C5E88148E17E_GGMICHALSKI_20260129_195230.log
============================================================================

197
MARS_Packages/REL01_ADDITIONS/MARS-828/log/ROLLBACK_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260129_195211.log
View File

@@ -1,197 +0,0 @@
============================================================================
MARS-828 Rollback Starting
============================================================================
WARNING: This will restore FILE_ARCHIVER to v2.0.0
CRITICAL IMPACT:
1. All archival strategies revert to THRESHOLD_BASED
2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
3. Validation trigger will be removed
4. Reconfigure archival thresholds after rollback
Timestamp:
ROLLBACK_START
______________________
2026-01-29 19:52:12
Elapsed: 00:00:00.068
============================================================================
Type YES to continue with rollback, or Ctrl+C to abort: YES
old:BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
new:BEGIN
IF 'YES' IS NULL OR TRIM('YES') IS NULL OR UPPER(TRIM('YES')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.096
Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.382
Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.288
Step 3/6: Dropping validation trigger
======================================
========================================
MARS-828: Dropping archival strategy validation trigger
========================================
Trigger CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL dropped.
Elapsed: 00:00:00.193
TRIGGER_COUNT
________________
0
Elapsed: 00:00:00.335
========================================
Validation trigger dropped successfully
========================================
Step 4/6: Dropping archival strategy columns
=============================================
========================================
MARS-828: Removing archival strategy columns
========================================
Table CT_MRDS.A_SOURCE_FILE_CONFIG altered.
Elapsed: 00:00:00.131
Table CT_MRDS.A_SOURCE_FILE_CONFIG altered.
Elapsed: 00:00:00.206
no rows selected
Elapsed: 00:00:00.341
========================================
Archival strategy columns removed successfully
========================================
Step 5/6: Tracking rollback version
====================================
========================================
Package Version Tracking
========================================
EnvironmentID set to: dev
[2026-01-29 19:52:16] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: Start TRACK_PACKAGE_VERSION
[2026-01-29 19:52:17] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: End TRACK_PACKAGE_VERSION - Record inserted
Summary:
--------
Packages tracked: 1/1
Tracked Packages:
CT_MRDS.FILE_ARCHIVER v2.0.0
========================================
PL/SQL procedure successfully completed.
Elapsed: 00:00:01.230
Step 6/6: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ ______________________________________________________
CT_MRDS DATA_EXPORTER 2.6.3 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 2.6.3
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 OK: Package CT_MRDS.FILE_ARCHIVER has not changed.
Last Tracked: 2026-01-29 19:52:17
Version: 2.0.0
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
Elapsed: 00:00:00.448
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
Verification: Package Compilation Status
=========================================
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ _________ ________________
FILE_ARCHIVER PACKAGE VALID 29-JAN-26
FILE_ARCHIVER PACKAGE BODY VALID 29-JAN-26
2 rows selected.
Elapsed: 00:00:00.202
============================================================================
MARS-828 Rollback Completed
============================================================================
Completion Time:
ROLLBACK_END
______________________
2026-01-29 19:52:20
1 row selected.
Elapsed: 00:00:00.061
Rollback Summary:
- Package: CT_MRDS.FILE_ARCHIVER
- Restored Version: 2.0.0 (THRESHOLD_BASED archival only)
- Removed Features: CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID strategies
Log file: log/ROLLBACK_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260129_195211.log
============================================================================

23
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/FILE_ARCHIVER.pkb
View File

@@ -23,11 +23,7 @@ AS
WHEN 'THRESHOLD_BASED' THEN
vWhereClause := 'extract(day from (systimestamp - workflow_start)) > ' || pSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD;
-- Archive all data except current month
WHEN 'CURRENT_MONTH_ONLY' THEN
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'')';
-- Archive only data older than X months
-- Archive data older than X months (0 = current month only)
WHEN 'MINIMUM_AGE_MONTHS' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for MINIMUM_AGE_MONTHS strategy');
@@ -132,7 +128,7 @@ AS
if LENGTH(vArchivalTriggeredBy)>0 THEN
ENV_MANAGER.LOG_PROCESS_EVENT('Archival Triggered By: '||vArchivalTriggeredBy,'INFO');
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
-- Use strategy-based WHERE clause (MARS-828)
vQuery := '
@@ -387,14 +383,19 @@ AS
vSourceFileConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vTableName VARCHAR2(200);
vQuery VARCHAR2(32000);
vWhereClause VARCHAR2(4000);
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
vSourceFileConfig := FILE_MANAGER.GET_SOURCE_FILE_CONFIG(pSourceFileConfigKey => pSourceFileConfigKey);
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
ENV_MANAGER.LOG_PROCESS_EVENT('vTableName','DEBUG',vTableName);
-- Get WHERE clause based on archival strategy (MARS-828)
vWhereClause := GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig);
ENV_MANAGER.LOG_PROCESS_EVENT('vWhereClause','DEBUG',vWhereClause);
-- Use strategy-based WHERE clause for statistics (MARS-828)
vQuery :=
'with tmp as (
@@ -419,12 +420,12 @@ AS
,'||pSourceFileConfigKey||' as A_SOURCE_FILE_CONFIG_KEY
,'''||vTableName||''' as TABLE_NAME
,count(*) as FILE_COUNT
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum(case when ' || vWhereClause || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum (row_count_per_file) as ROW_COUNT
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(case when ' || vWhereClause || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(r.bytes) as BYTES
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then r.bytes else 0 end) as OLD_BYTES
,'||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' as DAYS_FOR_ARCHIVE_THRESHOLD
,sum(case when ' || vWhereClause || ' then r.bytes else 0 end) as OLD_BYTES
,'||COALESCE(TO_CHAR(vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD), 'NULL')||' as DAYS_FOR_ARCHIVE_THRESHOLD
,systimestamp as CREATED
from tmp_gr t
join (SELECT * from DBMS_CLOUD.LIST_OBJECTS(

2
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/FILE_ARCHIVER.pkg
View File

@@ -24,7 +24,7 @@ AS
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.1.0 (2026-01-29): Added function overloads for ARCHIVE_TABLE_DATA and GATHER_TABLE_STAT returning SQLCODE for Python library integration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-18): Enhanced ARCHIVE_TABLE_DATA with Hive-style partitioning support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-15): Initial release with table archival and statistics gathering';

6
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/TRG_BI_A_SRC_FILE_CFG_ARCH_VAL.sql
View File

@@ -20,10 +20,10 @@ BEGIN
RAISE_APPLICATION_ERROR(-20999, vErrorMsg);
END IF;
-- Validate MINIMUM_AGE_MONTHS is positive
-- Validate MINIMUM_AGE_MONTHS is non-negative (0 = current month only)
IF :NEW.MINIMUM_AGE_MONTHS IS NOT NULL
AND :NEW.MINIMUM_AGE_MONTHS < 1 THEN
RAISE_APPLICATION_ERROR(-20998, 'MINIMUM_AGE_MONTHS must be greater than 0');
AND :NEW.MINIMUM_AGE_MONTHS < 0 THEN
RAISE_APPLICATION_ERROR(-20998, 'MINIMUM_AGE_MONTHS must be greater than or equal to 0');
END IF;
-- Warn if MINIMUM_AGE_MONTHS set but strategy doesn't use it

39
MARS_Packages/REL01_ADDITIONS/MARS-828/rollback_mars828.sql
View File

@@ -29,13 +29,13 @@ PROMPT
PROMPT ============================================================================
PROMPT MARS-828 Rollback Starting
PROMPT ============================================================================
PROMPT WARNING: This will restore FILE_ARCHIVER to v2.0.0
PROMPT This will restore FILE_ARCHIVER to v2.0.0
PROMPT
PROMPT CRITICAL IMPACT:
PROMPT 1. All archival strategies revert to THRESHOLD_BASED
PROMPT 2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
PROMPT 3. Validation trigger will be removed
PROMPT 4. Reconfigure archival thresholds after rollback
PROMPT Rollback steps:
PROMPT 1. Remove validation trigger
PROMPT 2. Drop ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns
PROMPT 3. Restore FILE_ARCHIVER package to v2.0.0
PROMPT 4. Revert all archival strategies to THRESHOLD_BASED
PROMPT
PROMPT Timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS rollback_start FROM DUAL;
@@ -54,32 +54,27 @@ WHENEVER SQLERROR CONTINUE
-- Rollback steps (in reverse order)
PROMPT
PROMPT Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
PROMPT ===============================================================
@@91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
PROMPT ======================================================
@@92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 3/6: Dropping validation trigger
PROMPT Step 1/5: Dropping validation trigger
PROMPT ======================================
@@93_MARS_828_rollback_trigger.sql
PROMPT
PROMPT Step 4/6: Dropping archival strategy columns
PROMPT Step 2/5: Dropping archival strategy columns
PROMPT =============================================
@@94_MARS_828_rollback_columns.sql
PROMPT
PROMPT Step 5/6: Tracking rollback version
PROMPT ====================================
@@track_package_versions.sql
PROMPT Step 3/5: Restoring FILE_ARCHIVER Package Specification v2.0.0
PROMPT ===============================================================
@@91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 6/6: Verifying tracked packages
PROMPT Step 4/5: Restoring FILE_ARCHIVER Package Body v2.0.0
PROMPT ======================================================
@@92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 5/5: Verifying tracked packages
PROMPT =====================================
@@verify_packages_version.sql

1144
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb
View File

File diff suppressed because it is too large Load Diff

66
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg
View File

@@ -9,22 +9,65 @@ AS
**/
-- Package Version Information
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.1.1';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2025-12-04 13:10:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.6.3';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-01-28 19:30:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team';
-- Version History (last 3-5 changes)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) ||
'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) ||
'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) ||
'v2.6.0 (2026-01-28): CRITICAL FIX - Added STATUS tracking to A_PARALLEL_EXPORT_CHUNKS table to prevent data duplication on retry. System now restarts ONLY failed partitions instead of re-exporting all data. Added ERROR_MESSAGE and EXPORT_TIMESTAMP columns for better error handling and monitoring. Prevents duplicate file creation when parallel tasks fail (e.g., 22 partitions with 16 threads, 3 failures no longer duplicates 19 successful exports).' || CHR(10) ||
'v2.5.0 (2026-01-26): Added recorddelimiter parameter with CRLF (CHR(13)||CHR(10)) for CSV exports to ensure Windows-compatible line endings. Improves cross-platform compatibility when CSV files are opened in Windows applications (Notepad, Excel).' || CHR(10) ||
'v2.4.0 (2026-01-11): Added pTemplateTableName parameter for per-column date format configuration. Implements dynamic query building with TO_CHAR for each date/timestamp column using FILE_MANAGER.GET_DATE_FORMAT. Supports 3-tier hierarchy: column-specific, template DEFAULT, global fallback. Eliminates single dateformat limitation of DBMS_CLOUD.EXPORT_DATA.' || CHR(10) ||
'v2.3.0 (2025-12-20): Added parallel partition processing using DBMS_PARALLEL_EXECUTE. New pParallelDegree parameter (1-16, default 1) for EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE procedures. Each year/month partition processed in separate thread for improved performance.' || CHR(10) ||
'v2.2.0 (2025-12-19): DRY refactoring - extracted shared helper functions (sanitizeFilename, VALIDATE_TABLE_AND_COLUMNS, GET_PARTITIONS, EXPORT_SINGLE_PARTITION worker procedure). Reduced code duplication by ~400 lines. Prepared architecture for v2.3.0 parallel processing.' || CHR(10) ||
'v2.1.1 (2025-12-04): Fixed JOIN column reference A_WORKFLOW_HISTORY_KEY -> A_ETL_LOAD_SET_KEY, added consistent column mapping and dynamic column list to EXPORT_TABLE_DATA procedure, enhanced DEBUG logging for all export operations' || CHR(10) ||
'v2.1.1 (2025-12-04): Fixed JOIN column reference A_WORKFLOW_HISTORY_KEY -> A_ETL_LOAD_SET_KEY' || CHR(10) ||
'v2.1.0 (2025-10-22): Added version tracking and PARTITION_YEAR/PARTITION_MONTH support' || CHR(10) ||
'v2.0.0 (2025-10-01): Separated export functionality from FILE_MANAGER package' || CHR(10) ||
'v1.0.0 (2025-09-15): Initial implementation within FILE_MANAGER package' || CHR(10);
'v2.0.0 (2025-10-01): Separated export functionality from FILE_MANAGER package' || CHR(10);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
-- TYPE DEFINITIONS FOR PARTITION HANDLING
---------------------------------------------------------------------------------------------------------------------------
/**
* Record type for year/month partition information
**/
TYPE partition_rec IS RECORD (
year VARCHAR2(4),
month VARCHAR2(2)
);
/**
* Table type for collection of partition records
**/
TYPE partition_tab IS TABLE OF partition_rec;
---------------------------------------------------------------------------------------------------------------------------
-- INTERNAL PARALLEL PROCESSING CALLBACK
---------------------------------------------------------------------------------------------------------------------------
/**
* @name EXPORT_PARTITION_PARALLEL
* @desc Internal callback procedure for DBMS_PARALLEL_EXECUTE.
* Processes single partition (year/month) chunk in parallel task.
* Called by DBMS_PARALLEL_EXECUTE framework for each chunk.
* This procedure is PUBLIC because DBMS_PARALLEL_EXECUTE requires it,
* but should NOT be called directly by external code.
* @param pStartId - Chunk start ID (CHUNK_ID from A_PARALLEL_EXPORT_CHUNKS table)
* @param pEndId - Chunk end ID (same as pStartId for single-row chunks)
**/
PROCEDURE EXPORT_PARTITION_PARALLEL (
pStartId IN NUMBER,
pEndId IN NUMBER
);
---------------------------------------------------------------------------------------------------------------------------
-- MAIN EXPORT PROCEDURES
---------------------------------------------------------------------------------------------------------------------------
/**
@@ -62,6 +105,7 @@ AS
* Allows specifying custom column list or uses T.* if pColumnList is NULL.
* Validates that all columns in pColumnList exist in the target table.
* Automatically adds 'T.' prefix to column names in pColumnList.
* Supports parallel partition processing via pParallelDegree parameter (default 1, range 1-16).
* pBucketArea parameter accepts: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example
* begin
@@ -73,7 +117,8 @@ AS
* pFolderName => 'parquet_exports',
* pColumnList => 'COLUMN1, COLUMN2, COLUMN3', -- Optional
* pMinDate => DATE '2024-01-01',
* pMaxDate => SYSDATE
* pMaxDate => SYSDATE,
* pParallelDegree => 8 -- Optional, default 1, range 1-16
* );
* end;
**/
@@ -86,6 +131,8 @@ AS
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pParallelDegree IN NUMBER default 1,
pTemplateTableName IN VARCHAR2 default NULL,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
);
@@ -97,6 +144,7 @@ AS
* Creates one CSV file for each year/month combination found in the data.
* Uses the same date filtering mechanism with CT_ODS.A_LOAD_HISTORY as EXPORT_TABLE_DATA_BY_DATE,
* but exports to CSV format instead of Parquet.
* Supports parallel partition processing via pParallelDegree parameter (1-16).
* File naming pattern: {pFileName}_YYYYMM.csv or {TABLENAME}_YYYYMM.csv (if pFileName is NULL)
* @example
* begin
@@ -109,7 +157,8 @@ AS
* pFolderName => 'exports',
* pFileName => 'my_export.csv',
* pMinDate => DATE '2024-01-01',
* pMaxDate => SYSDATE
* pMaxDate => SYSDATE,
* pParallelDegree => 8 -- Optional, default 1, range 1-16
* );
*
* -- With auto-generated filename (based on table name only)
@@ -136,6 +185,9 @@ AS
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pParallelDegree IN NUMBER default 1,
pTemplateTableName IN VARCHAR2 default NULL,
pMaxFileSize IN NUMBER default 104857600,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
);

6
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/ENV_MANAGER.pkb
View File

@@ -39,6 +39,8 @@ AS
Errors(CODE_MOVE_FILE_TO_TRASH_FAILED) := Error_Record(CODE_MOVE_FILE_TO_TRASH_FAILED, MSG_MOVE_FILE_TO_TRASH_FAILED); -- -20032
Errors(CODE_DROP_EXPORTED_FILES_FAILED) := Error_Record(CODE_DROP_EXPORTED_FILES_FAILED, MSG_DROP_EXPORTED_FILES_FAILED); -- -20033
Errors(CODE_INVALID_BUCKET_AREA) := Error_Record(CODE_INVALID_BUCKET_AREA, MSG_INVALID_BUCKET_AREA); -- -20034
Errors(CODE_INVALID_PARALLEL_DEGREE) := Error_Record(CODE_INVALID_PARALLEL_DEGREE, MSG_INVALID_PARALLEL_DEGREE); -- -20110
Errors(CODE_PARALLEL_EXECUTION_FAILED) := Error_Record(CODE_PARALLEL_EXECUTION_FAILED, MSG_PARALLEL_EXECUTION_FAILED); -- -20111
Errors(CODE_UNKNOWN) := Error_Record(CODE_UNKNOWN, MSG_UNKNOWN); -- -20999
@@ -1163,3 +1165,7 @@ BEGIN
INIT_VARIABLES(pEnv => gvEnv);
END ENV_MANAGER;
/
/

18
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/ENV_MANAGER.pkg
View File

@@ -17,12 +17,13 @@ AS
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 20:57:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.2.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-12-20 10:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.2.0 (2025-12-20): Added error codes for parallel execution support (CODE_INVALID_PARALLEL_DEGREE -20110, CODE_PARALLEL_EXECUTION_FAILED -20111)' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-22): Added package hash tracking and automatic change detection system (SHA256 hashing)' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-22): Added package versioning system with centralized version management functions' || CHR(13)||CHR(10) ||
'2.1.0 (2025-10-15): Added ANALYZE_VALIDATION_ERRORS function for comprehensive CSV validation analysis' || CHR(13)||CHR(10) ||
@@ -296,6 +297,18 @@ AS
PRAGMA EXCEPTION_INIT( ERR_INVALID_BUCKET_AREA
,CODE_INVALID_BUCKET_AREA);
ERR_INVALID_PARALLEL_DEGREE EXCEPTION;
CODE_INVALID_PARALLEL_DEGREE CONSTANT PLS_INTEGER := -20110;
MSG_INVALID_PARALLEL_DEGREE VARCHAR2(4000) := 'Invalid parallel degree parameter. Must be between 1 and 16';
PRAGMA EXCEPTION_INIT( ERR_INVALID_PARALLEL_DEGREE
,CODE_INVALID_PARALLEL_DEGREE);
ERR_PARALLEL_EXECUTION_FAILED EXCEPTION;
CODE_PARALLEL_EXECUTION_FAILED CONSTANT PLS_INTEGER := -20111;
MSG_PARALLEL_EXECUTION_FAILED VARCHAR2(4000) := 'Parallel execution failed';
PRAGMA EXCEPTION_INIT( ERR_PARALLEL_EXECUTION_FAILED
,CODE_PARALLEL_EXECUTION_FAILED);
ERR_UNKNOWN EXCEPTION;
CODE_UNKNOWN CONSTANT PLS_INTEGER := -20999;
MSG_UNKNOWN VARCHAR2(4000) := 'Unknown Error Occured';
@@ -609,3 +622,4 @@ AS
) RETURN VARCHAR2;
END ENV_MANAGER;
/

107
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/FILE_ARCHIVER.pkb
View File

@@ -1,6 +1,54 @@
create or replace PACKAGE BODY CT_MRDS.FILE_ARCHIVER
AS
----------------------------------------------------------------------------------------------------
-- PRIVATE FUNCTION: GET_ARCHIVAL_WHERE_CLAUSE
----------------------------------------------------------------------------------------------------
/**
* @name GET_ARCHIVAL_WHERE_CLAUSE
* @desc Private function that generates WHERE clause based on ARCHIVAL_STRATEGY configuration.
* Supports four strategies: THRESHOLD_BASED, CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID.
* @param pSourceFileConfig - Source file configuration record with ARCHIVAL_STRATEGY
* @return VARCHAR2 - WHERE clause for filtering archival candidates
**/
FUNCTION GET_ARCHIVAL_WHERE_CLAUSE(
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
) RETURN VARCHAR2
IS
vWhereClause VARCHAR2(4000);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
BEGIN
CASE pSourceFileConfig.ARCHIVAL_STRATEGY
-- Legacy threshold-based strategy (backward compatible)
WHEN 'THRESHOLD_BASED' THEN
vWhereClause := 'extract(day from (systimestamp - workflow_start)) > ' || pSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD;
-- Archive all data except current month
WHEN 'CURRENT_MONTH_ONLY' THEN
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'')';
-- Archive only data older than X months
WHEN 'MINIMUM_AGE_MONTHS' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for MINIMUM_AGE_MONTHS strategy');
END IF;
vWhereClause := 'workflow_start < ADD_MONTHS(TRUNC(SYSDATE, ''MM''), -' || pSourceFileConfig.MINIMUM_AGE_MONTHS || ')';
-- Hybrid: Current month exclusion AND minimum age requirement
WHEN 'HYBRID' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for HYBRID strategy');
END IF;
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'') ' ||
'AND workflow_start < ADD_MONTHS(TRUNC(SYSDATE, ''MM''), -' || pSourceFileConfig.MINIMUM_AGE_MONTHS || ')';
ELSE
RAISE_APPLICATION_ERROR(-20002, 'Invalid ARCHIVAL_STRATEGY: ' || pSourceFileConfig.ARCHIVAL_STRATEGY);
END CASE;
RETURN vWhereClause;
END GET_ARCHIVAL_WHERE_CLAUSE;
----------------------------------------------------------------------------------------------------
FUNCTION GET_TABLE_STAT(pSourceFileConfigKey IN NUMBER)
@@ -85,6 +133,8 @@ AS
if LENGTH(vArchivalTriggeredBy)>0 THEN
ENV_MANAGER.LOG_PROCESS_EVENT('Archival Triggered By: '||vArchivalTriggeredBy,'INFO');
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
-- Use strategy-based WHERE clause (MARS-828)
vQuery := '
select t_filename(
file$name
@@ -96,7 +146,7 @@ AS
from '||vTableName||' s
join CT_MRDS.a_workflow_history h
on s.a_workflow_history_key = h.a_workflow_history_key
where extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD
where ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig)
;
-- Get all files that will be archived into "vfiles" collection ("regular data files")
@@ -337,6 +387,7 @@ AS
vSourceFileConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vTableName VARCHAR2(200);
vQuery VARCHAR2(32000);
vWhereClause VARCHAR2(4000);
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
@@ -344,6 +395,12 @@ AS
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
ENV_MANAGER.LOG_PROCESS_EVENT('vTableName','DEBUG',vTableName);
-- Get WHERE clause based on archival strategy (MARS-828)
vWhereClause := GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig);
ENV_MANAGER.LOG_PROCESS_EVENT('vWhereClause','DEBUG',vWhereClause);
-- Use strategy-based WHERE clause for statistics (MARS-828)
vQuery :=
'with tmp as (
select
@@ -367,11 +424,11 @@ AS
,'||pSourceFileConfigKey||' as A_SOURCE_FILE_CONFIG_KEY
,'''||vTableName||''' as TABLE_NAME
,count(*) as FILE_COUNT
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then 1 else 0 end) as OLD_FILE_COUNT
,sum(case when ' || vWhereClause || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum (row_count_per_file) as ROW_COUNT
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(case when ' || vWhereClause || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(r.bytes) as BYTES
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then r.bytes else 0 end) as OLD_BYTES
,sum(case when ' || vWhereClause || ' then r.bytes else 0 end) as OLD_BYTES
,'||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' as DAYS_FOR_ARCHIVE_THRESHOLD
,systimestamp as CREATED
from tmp_gr t
@@ -438,6 +495,48 @@ AS
----------------------------------------------------------------------------------------------------
FUNCTION ARCHIVE_TABLE_DATA (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
----
ARCHIVE_TABLE_DATA(pSourceFileConfigKey => pSourceFileConfigKey);
----
ENV_MANAGER.LOG_PROCESS_EVENT('End','INFO',vParameters);
RETURN SQLCODE;
EXCEPTION
WHEN OTHERS THEN
ENV_MANAGER.LOG_PROCESS_EVENT(ENV_MANAGER.GET_ERROR_STACK(pFormat => 'TABLE', pCode=> SQLCODE), 'ERROR', vParameters);
RETURN SQLCODE;
END ARCHIVE_TABLE_DATA;
----------------------------------------------------------------------------------------------------
FUNCTION GATHER_TABLE_STAT (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
----
GATHER_TABLE_STAT(pSourceFileConfigKey => pSourceFileConfigKey);
----
ENV_MANAGER.LOG_PROCESS_EVENT('End','INFO',vParameters);
RETURN SQLCODE;
EXCEPTION
WHEN OTHERS THEN
ENV_MANAGER.LOG_PROCESS_EVENT(ENV_MANAGER.GET_ERROR_STACK(pFormat => 'TABLE', pCode=> SQLCODE), 'ERROR', vParameters);
RETURN SQLCODE;
END GATHER_TABLE_STAT;
----------------------------------------------------------------------------------------------------
END;
/

30
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/FILE_ARCHIVER.pkg
View File

@@ -17,12 +17,14 @@ AS
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.0.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 16:45:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-01-29 21:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.1.0 (2026-01-29): Added function overloads for ARCHIVE_TABLE_DATA and GATHER_TABLE_STAT returning SQLCODE for Python library integration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-18): Enhanced ARCHIVE_TABLE_DATA with Hive-style partitioning support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-15): Initial release with table archival and statistics gathering';
@@ -39,6 +41,18 @@ AS
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
);
/**
* @name ARCHIVE_TABLE_DATA
* @desc Function overload for ARCHIVE_TABLE_DATA procedure.
* Returns SQLCODE for Python library integration.
* Calls the main ARCHIVE_TABLE_DATA procedure and captures execution result.
* @example SELECT FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfigKey => 123) FROM DUAL;
* @ex_rslt 0 (success) or error code
**/
FUNCTION ARCHIVE_TABLE_DATA (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER;
/**
@@ -50,6 +64,18 @@ AS
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
);
/**
* @name GATHER_TABLE_STAT
* @desc Function overload for GATHER_TABLE_STAT procedure.
* Returns SQLCODE for Python library integration.
* Calls the main GATHER_TABLE_STAT procedure and captures execution result.
* @example SELECT FILE_ARCHIVER.GATHER_TABLE_STAT(pSourceFileConfigKey => 123) FROM DUAL;
* @ex_rslt 0 (success) or error code
**/
FUNCTION GATHER_TABLE_STAT (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------

5
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/tables/A_SOURCE_FILE_CONFIG.sql
View File

@@ -3,6 +3,7 @@
-- ====================================================================
-- Purpose: Store source file configuration and processing rules
-- MARS-1049: Added ENCODING column for CSV character set support
-- MARS-828: Added ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS for archival automation
-- ====================================================================
CREATE TABLE CT_MRDS.A_SOURCE_FILE_CONFIG (
@@ -21,6 +22,8 @@ CREATE TABLE CT_MRDS.A_SOURCE_FILE_CONFIG (
ODS_SCHEMA_NAME VARCHAR2(100),
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD NUMBER(38,0),
HOURS_TO_EXPIRE_STATISTICS NUMBER(38,3),
ARCHIVAL_STRATEGY VARCHAR2(50),
MINIMUM_AGE_MONTHS NUMBER(3,0),
ENCODING VARCHAR2(50) DEFAULT 'UTF8',
CONSTRAINT A_SOURCE_FILE_CONFIG_PK PRIMARY KEY (A_SOURCE_FILE_CONFIG_KEY),
CONSTRAINT SOURCE_FILE_TYPE_CHK CHECK (SOURCE_FILE_TYPE IN ('INPUT', 'CONTAINER', 'LOAD_CONFIG')),
@@ -40,6 +43,8 @@ ON "CT_MRDS"."A_SOURCE_FILE_CONFIG" ("SOURCE_FILE_TYPE", "SOURCE_FILE_ID", "TABL
TABLESPACE "DATA";
-- Column comments
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ARCHIVAL_STRATEGY IS 'Archival strategy: THRESHOLD_BASED, CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID. Added in MARS-828';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.MINIMUM_AGE_MONTHS IS 'Minimum age in months before archival (required for MINIMUM_AGE_MONTHS strategy). Added in MARS-828';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING IS 'Oracle character set name for CSV files (e.g., UTF8, WE8MSWIN1252, EE8ISO8859P2). Added in MARS-1049';
GRANT SELECT, INSERT, UPDATE, DELETE ON CT_MRDS.A_SOURCE_FILE_CONFIG TO MRDS_LOADER_ROLE;

7
README.md
View File

@@ -67,7 +67,14 @@ sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars835_prehook.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M835PH_arch.7z MARS-835-PREHOOK
cd .\MARS_Packages\REL03\MARS-1057
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars1057.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars1057.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M1057_arch.7z MARS-1057
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M826_arch.7z MARS-828\

618
confluence/FILE_ARCHIVER_Guide.md Normal file
View File

@@ -0,0 +1,618 @@
# FILE_ARCHIVER Configuration Guide
This document describes the archival strategies available in the FILE_ARCHIVER package for managing data lifecycle across OCI buckets (INBOX → ODS → ARCHIVE).
## Overview
The FILE_ARCHIVER package provides flexible archival strategies that accommodate different data retention policies across source systems. It manages the movement of processed data from operational storage (ODS bucket) to long-term archival storage (ARCHIVE bucket) based on configurable strategies.
### Key Features
- **Three Archival Strategies**: THRESHOLD_BASED, MINIMUM_AGE_MONTHS (with 0=current month only), HYBRID
- **Flexible Configuration**: Per-table archival strategy configuration via A_SOURCE_FILE_CONFIG
- **Backward Compatible**: Default THRESHOLD_BASED strategy maintains existing behavior
- **Validation**: Automatic validation of strategy-specific configuration requirements
- **OCI Integration**: Works seamlessly with DBMS_CLOUD operations via cloud_wrapper
### Package Information
- **Schema**: CT_MRDS
- **Package**: FILE_ARCHIVER
- **Current Version**: 3.1.0
- **Dependencies**: ENV_MANAGER, FILE_MANAGER, cloud_wrapper, A_SOURCE_FILE_CONFIG, A_SOURCE_FILE_RECEIVED, A_WORKFLOW_HISTORY
### Critical Prerequisites
⚠️ **IMPORTANT**: FILE_ARCHIVER requires data to be registered in `CT_MRDS.A_SOURCE_FILE_RECEIVED` table. This table is automatically populated when files are processed through the modern Airflow + DBT workflow via `FILE_MANAGER.PROCESS_SOURCE_FILE`.
**For legacy data migrated from Informatica + WLA system:**
- Legacy data exported using `DATA_EXPORTER` does NOT automatically create `A_SOURCE_FILE_RECEIVED` records
- Without these records, FILE_ARCHIVER **CANNOT** archive the data
- See [System Migration Guide](System_Migration_Informatica_to_Airflow_DBT.md) for workaround strategies
**Recommendation for legacy data**: Export directly to ARCHIVE bucket using `DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE` with `pBucketArea => 'ARCHIVE'` to bypass this requirement
## Archival Strategies
### Strategy Overview
| Strategy | WHERE Clause Logic | Configuration Required | Primary Use Case |
|----------|-------------------|----------------------|------------------|
| `THRESHOLD_BASED` | Days since workflow start > threshold | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility, simple time-based archival |
| `MINIMUM_AGE_MONTHS` | Archive data older than X months (0=current month only) | MINIMUM_AGE_MONTHS (≥0) | All sources - flexible retention (0 for LM, 6 for CSDB) |
| `HYBRID` | Combines month boundary + minimum age | MINIMUM_AGE_MONTHS | Advanced retention scenarios |
### 1. THRESHOLD_BASED (Default)
Archives data based on number of days since workflow start.
**WHERE Clause**:
```sql
extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD
```
**Configuration**:
```sql
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
DAYS_FOR_ARCHIVE_THRESHOLD = 30,
MINIMUM_AGE_MONTHS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'C2D_DATA'
AND TABLE_ID = 'C2D_TABLE';
```
**Use Case**: Simple time-based archival, backward compatible with FILE_ARCHIVER v2.0.0 behavior.
### 2. MINIMUM_AGE_MONTHS
Archives data older than specified number of months. **Special case**: MINIMUM_AGE_MONTHS = 0 archives all data before current month (replaces deprecated CURRENT_MONTH_ONLY strategy).
**WHERE Clause**:
```sql
workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -MINIMUM_AGE_MONTHS)
-- When MINIMUM_AGE_MONTHS = 0: workflow_start < TRUNC(SYSDATE, 'MM')
```
**Configuration Examples**:
```sql
-- LM: Keep only current month data (MINIMUM_AGE_MONTHS = 0)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
AND TABLE_ID = 'LM_STANDING_FACILITIES';
-- CSDB: Retain 6 months of data (MINIMUM_AGE_MONTHS = 6)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
```
**Use Cases**:
- **MINIMUM_AGE_MONTHS = 0**: LM dissemination feeds requiring current month only (daily/intraday updates)
- **MINIMUM_AGE_MONTHS = 6**: CSDB securities/ratings data requiring 6-month retention
- **MINIMUM_AGE_MONTHS = N**: Regulatory compliance with specific N-month retention periods
**Behavior Examples**:
- **With MINIMUM_AGE_MONTHS = 0**:
- January data: Archived on February 1st
- February data: Remains in ODS bucket during February
- March 1st: February data archived, March data active
- **With MINIMUM_AGE_MONTHS = 6**:
- February 2026: Archives data from July 2025 and earlier
- March 2026: Archives data from August 2025 and earlier
- Keeps current month + 6 previous months (7 months total) in ODS bucket
### 3. HYBRID
Combines month boundary check with minimum age threshold - archives data from previous months AND older than minimum age.
**WHERE Clause**:
```sql
TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')
AND workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -MINIMUM_AGE_MONTHS)
```
**Configuration**:
```sql
-- Advanced: Current month + 3 months minimum
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'HYBRID',
MINIMUM_AGE_MONTHS = 3
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'SPECIAL_SOURCE'
AND TABLE_ID = 'SPECIAL_TABLE';
```
**Use Case**: Advanced scenarios requiring both current month retention AND minimum age threshold.
## Configuration Validation
### Validation Trigger
**Trigger**: `TRG_BI_A_SRC_FILE_CFG_ARCH_VAL`
Automatically validates archival configuration on INSERT/UPDATE to A_SOURCE_FILE_CONFIG:
**Validation Rules**:
1. **MINIMUM_AGE_MONTHS**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL AND MINIMUM_AGE_MONTHS >= 0`
- Error: "Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set (≥0)"
2. **HYBRID**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL`
- Error: "Strategy HYBRID requires MINIMUM_AGE_MONTHS to be set"
**Example Validation Error**:
```sql
-- This will fail validation
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = NULL -- ERROR: Required for this strategy
WHERE ...;
-- Error: ORA-20001: Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set
```
## Data Lifecycle Workflow
### Standard File Processing Flow
```
┌─────────────────────────────────────────────────────────────┐
│ FILE PROCESSING LIFECYCLE │
└─────────────────────────────────────────────────────────────┘
1. INBOX Bucket (Validation)
├─ File arrives from source system
├─ FILE_MANAGER.PROCESS_SOURCE_FILE validates structure
├─ Status: RECEIVED → VALIDATED → READY_FOR_INGESTION
└─ FILE_MANAGER.MOVE_FILE relocates to ODS bucket
2. ODS Bucket (Operational Data)
├─ Active data processing (Airflow + DBT)
├─ External tables read data from bucket
├─ Status: INGESTED
└─ FILE_ARCHIVER.ARCHIVE_TABLE_DATA archives based on strategy
3. ARCHIVE Bucket (Long-term Storage)
├─ Historical data in Parquet format
├─ Hive-style partitioning: PARTITION_YEAR=/PARTITION_MONTH=
├─ Status: ARCHIVED
└─ Optimized for big data analytics (Spark, Hive)
```
### Archival Process
The FILE_ARCHIVER package automatically manages data movement from ODS to ARCHIVE:
**Key Procedures**:
- `ARCHIVE_TABLE_DATA` - Main archival procedure using strategy-specific WHERE clause
- `GET_ARCHIVAL_WHERE_CLAUSE` - Returns WHERE clause based on configured strategy
- `GATHER_TABLE_STAT` - Calculates archival statistics using strategy logic
**Archival Execution**:
```sql
-- Triggered by FILE_MANAGER or scheduled job
BEGIN
CT_MRDS.FILE_ARCHIVER.ARCHIVE_TABLE_DATA(
pSourceFileConfig => vSourceFileConfigRecord
);
END;
/
```
**Strategy-Based Filtering**:
- Package retrieves ARCHIVAL_STRATEGY from A_SOURCE_FILE_CONFIG
- GET_ARCHIVAL_WHERE_CLAUSE generates appropriate WHERE clause
- Data matching criteria moved from ODS to ARCHIVE bucket
- Parquet format with Hive-style partitioning applied
## Configuration Examples
### Example 1: Configure LM Standing Facilities (Current Month Only)
```sql
-- Keep only current month data in ODS bucket (MINIMUM_AGE_MONTHS = 0)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = archives all data before current month
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
AND TABLE_ID = 'LM_STANDING_FACILITIES';
COMMIT;
-- Verify configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'DistributeStandingFacilities';
```
### Example 2: Configure CSDB Debt (MINIMUM_AGE_MONTHS)
```sql
-- Retain 6 months of data in ODS bucket
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'CSDB'
AND TABLE_ID = 'CSDB_DEBT';
COMMIT;
-- Verify configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'CSDB_DEBT';
```
### Example 3: Bulk Configuration for LM Source
```sql
-- Configure all 19 LM tables with MINIMUM_AGE_MONTHS = 0 (current month only)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = keep only current month
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID IN (
'DistributeStandingFacilities',
'DistributeTTS',
'DistributeAdHocAdjustments',
'DistributeBalanceSheet',
'DistributeCSMAdjustments',
'DistributeCurrentAccounts',
'DistributeForecast',
'DistributeQREAdjustments'
);
COMMIT;
-- Verify bulk configuration
SELECT
SOURCE_FILE_ID,
COUNT(*) AS TABLE_COUNT,
MAX(ARCHIVAL_STRATEGY) AS STRATEGY,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID LIKE 'Distribute%'
GROUP BY SOURCE_FILE_ID
ORDER BY SOURCE_FILE_ID;
```
### Example 4: View Current Archival Configuration
```sql
-- All configured tables with their archival strategies
SELECT
A_SOURCE_KEY,
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
DAYS_FOR_ARCHIVE_THRESHOLD
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
ORDER BY A_SOURCE_KEY, SOURCE_FILE_ID, TABLE_ID;
-- Summary by strategy
SELECT
ARCHIVAL_STRATEGY,
COUNT(*) AS TABLE_COUNT,
MIN(MINIMUM_AGE_MONTHS) AS MIN_AGE_MIN,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE_MAX
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
GROUP BY ARCHIVAL_STRATEGY
ORDER BY ARCHIVAL_STRATEGY;
```
## Release 01 Configuration
### Configured Tables (MARS-828)
The following 25 Release 01 tables were configured with archival strategies:
**LM Tables (19 total) - MINIMUM_AGE_MONTHS = 0 (current month only)**:
- LM_STANDING_FACILITIES
- LM_STANDING_FACILITIES_HEADER
- LM_TTS_HEADER
- LM_TTS_ITEM
- LM_ADHOC_ADJUSTMENTS_HEADER
- LM_ADHOC_ADJUSTMENTS_ITEM
- LM_ADHOC_ADJUSTMENTS_ITEM_HEADER
- LM_BALANCESHEET_HEADER
- LM_BALANCESHEET_ITEM
- LM_CSM_ADJUSTMENTS_HEADER
- LM_CSM_ADJUSTMENTS_ITEM
- LM_CSM_ADJUSTMENTS_ITEM_HEADER
- LM_CURRENT_ACCOUNTS_HEADER
- LM_CURRENT_ACCOUNTS_ITEM
- LM_FORECAST_HEADER
- LM_FORECAST_ITEM
- LM_QRE_ADJUSTMENTS_HEADER
- LM_QRE_ADJUSTMENTS_ITEM
- LM_QRE_ADJUSTMENTS_ITEM_HEADER
**CSDB Tables (6 total)**:
*MINIMUM_AGE_MONTHS = 6 (6-month retention)*:
- CSDB_DEBT
- CSDB_DEBT_DAILY
*MINIMUM_AGE_MONTHS = 0 (current month only)*:
- CSDB_INSTR_RAT_FULL
- CSDB_INSTR_DESC_FULL
- CSDB_ISSUER_RAT_FULL
- CSDB_ISSUER_DESC_FULL
**Verification Query**:
```sql
-- Check Release 01 configuration
SELECT
CASE
WHEN TABLE_ID LIKE 'LM_%' THEN 'LM'
WHEN TABLE_ID LIKE 'CSDB_%' THEN 'CSDB'
END AS SOURCE_GROUP,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
COUNT(*) AS TABLE_COUNT
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN (
-- 25 Release 01 tables
'LM_STANDING_FACILITIES', 'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER', 'LM_TTS_ITEM',
-- ... other tables
)
GROUP BY
CASE
WHEN TABLE_ID LIKE 'LM_%' THEN 'LM'
WHEN TABLE_ID LIKE 'CSDB_%' THEN 'CSDB'
END,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
ORDER BY SOURCE_GROUP, ARCHIVAL_STRATEGY;
```
## Troubleshooting
### Common Issues
#### Issue 1: Validation Error on Configuration Update
**Error**:
```
ORA-20001: Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set
```
**Cause**: Trigger validation failed - strategy requires MINIMUM_AGE_MONTHS but value is NULL
**Solution**:
```sql
-- Provide required MINIMUM_AGE_MONTHS value
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6 -- Required for this strategy
WHERE ...;
```
#### Issue 2: Archival Not Working as Expected
**Symptoms**: Data not being archived according to strategy
**Diagnostic Steps**:
```sql
-- 1. Check configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
DAYS_FOR_ARCHIVE_THRESHOLD
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'YOUR_TABLE';
-- 2. Check package version
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
-- Expected: 3.0.0 or higher
-- 3. Check process logs
SELECT
PROCESS_LOG_KEY,
PROCESS_NAME,
LOG_MESSAGE,
LOG_LEVEL,
LOG_TIMESTAMP
FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCESS_NAME LIKE '%ARCHIVE%'
ORDER BY LOG_TIMESTAMP DESC
FETCH FIRST 20 ROWS ONLY;
-- 4. Test WHERE clause generation
DECLARE
vConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vWhereClause VARCHAR2(4000);
BEGIN
SELECT * INTO vConfig
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'YOUR_TABLE'
AND ROWNUM = 1;
vWhereClause := CT_MRDS.FILE_ARCHIVER.GET_ARCHIVAL_WHERE_CLAUSE(vConfig);
DBMS_OUTPUT.PUT_LINE('WHERE Clause: ' || vWhereClause);
END;
/
```
#### Issue 3: Package Compilation Errors After Upgrade
**Symptoms**: FILE_ARCHIVER package shows INVALID status
**Solution**:
```sql
-- Check compilation errors
SELECT * FROM USER_ERRORS
WHERE NAME = 'FILE_ARCHIVER'
AND TYPE IN ('PACKAGE', 'PACKAGE BODY')
ORDER BY SEQUENCE;
-- Recompile package
ALTER PACKAGE CT_MRDS.FILE_ARCHIVER COMPILE SPECIFICATION;
ALTER PACKAGE CT_MRDS.FILE_ARCHIVER COMPILE BODY;
-- Verify status
SELECT object_name, object_type, status
FROM user_objects
WHERE object_name = 'FILE_ARCHIVER';
```
## Version History
### v3.1.0 (Current - 2026-02-05)
- **BREAKING CHANGE**: Removed CURRENT_MONTH_ONLY strategy (replaced by MINIMUM_AGE_MONTHS = 0)
- Mathematical equivalence: CURRENT_MONTH_ONLY ≡ MINIMUM_AGE_MONTHS = 0
- Updated trigger validation to allow MINIMUM_AGE_MONTHS >= 0 (previously >= 1)
- Simplified architecture from 4 strategies to 3
- Enhanced error handling
- All 25 Release 01 tables migrated to MINIMUM_AGE_MONTHS (23 with value 0, 2 with value 6)
### v3.0.0 (MARS-828 - 2026-02-04)
- Added ARCHIVAL_STRATEGY configuration column
- Implemented four archival strategies (later reduced to three in v3.1.0):
- THRESHOLD_BASED (backward compatible)
- CURRENT_MONTH_ONLY (deprecated in v3.1.0, use MINIMUM_AGE_MONTHS = 0)
- MINIMUM_AGE_MONTHS
- HYBRID
- Added GET_ARCHIVAL_WHERE_CLAUSE function
- Created validation trigger TRG_BI_A_SRC_FILE_CFG_ARCH_VAL
- Configured 25 Release 01 tables with appropriate strategies
### v2.0.0 (Legacy)
- Initial FILE_ARCHIVER package
- THRESHOLD_BASED archival only
- Fixed DAYS_FOR_ARCHIVE_THRESHOLD configuration
## Related Documentation
- [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md) - File processing and validation
- [Package Deployment Guide](Package_Deployment_Guide.md) - Package deployment standards
- [Universal Package Tracking System](Universal_Package_Tracking_System.md) - Version tracking
- [MARS-828 README](../MARS_Packages/REL01_ADDITIONS/MARS-828/README.md) - Detailed implementation notes
## Dependencies
### Required Packages
- **CT_MRDS.ENV_MANAGER** v3.x - Error handling, logging, version tracking
- **CT_MRDS.FILE_MANAGER** v3.x - Bucket URI resolution, file processing
- **MRDS_LOADER.cloud_wrapper** - DBMS_CLOUD operations wrapper
### Database Objects
- **Table**: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
- **Table**: CT_MRDS.A_SOURCE_FILE_RECEIVED - File processing tracking
- **Table**: CT_MRDS.A_WORKFLOW_HISTORY - Workflow execution tracking (Airflow + DBT)
- **Trigger**: TRG_BI_A_SRC_FILE_CFG_ARCH_VAL - Configuration validation
- **Credential**: DEF_CRED_ARN - OCI bucket access
### OCI Buckets
- **INBOX**: Incoming file validation (`'INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/'`)
- **ODS/DATA**: Operational data processing (`'ODS/{SOURCE}/{TABLE_NAME}/'`)
- **ARCHIVE**: Historical data storage (`'ARCHIVE/{SOURCE}/{TABLE_NAME}/PARTITION_YEAR=/PARTITION_MONTH=/'`)
## Best Practices
### Strategy Selection Guidelines
1. **Use MINIMUM_AGE_MONTHS when**:
- **MINIMUM_AGE_MONTHS = 0**: Current month only retention
- Data updated frequently (daily/intraday)
- Historical data access is rare
- ODS bucket space is limited
- Example: LM dissemination feeds
- **MINIMUM_AGE_MONTHS = N (N > 0)**: Multi-month retention
- Regulatory compliance requires specific retention period
- Analytical workloads need N-month access
- Data updates are infrequent
- Example: CSDB securities data (MINIMUM_AGE_MONTHS = 6)
2. **Use THRESHOLD_BASED when**:
- Maintaining backward compatibility with legacy behavior
- Simple time-based archival is sufficient
- Migration from FILE_ARCHIVER v2.0.0
3. **Use HYBRID when**:
- Complex retention requirements
- Combining month boundary check with minimum age threshold
- Advanced scenarios not covered by other strategies
### Configuration Best Practices
1. **Test Configuration Changes**:
```sql
-- Test on single table first
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = current month only
WHERE SOURCE_FILE_ID = 'TEST_FILE'
AND TABLE_ID = 'TEST_TABLE';
-- Monitor archival behavior
-- Expand to other tables after validation
```
2. **Verify Before Bulk Updates**:
```sql
-- Preview changes with SELECT
SELECT
SOURCE_FILE_ID,
TABLE_ID,
'MINIMUM_AGE_MONTHS' AS NEW_STRATEGY,
0 AS NEW_MIN_AGE, -- 0 = current month only
ARCHIVAL_STRATEGY AS OLD_STRATEGY,
MINIMUM_AGE_MONTHS AS OLD_MIN_AGE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID LIKE 'Distribute%';
-- Then execute UPDATE
```
3. **Document Configuration Decisions**:
- Record why specific strategy was chosen
- Note business requirements driving retention policy
- Track configuration changes in version control
4. **Monitor Archival Performance**:
```sql
-- Check archival execution logs
SELECT
PROCESS_NAME,
LOG_MESSAGE,
LOG_TIMESTAMP
FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCESS_NAME LIKE '%ARCHIVE%'
AND LOG_TIMESTAMP > SYSDATE - 7
ORDER BY LOG_TIMESTAMP DESC;
```
5. **Regular Configuration Reviews**:
- Verify strategies still match business requirements
- Check for tables without archival configuration
- Optimize MINIMUM_AGE_MONTHS based on actual usage patterns
## Author
Created by: Grzegorz Michalski
Date: 2026-02-04
Schema: CT_MRDS
Package: FILE_ARCHIVER
Version: 3.1.0

5
confluence/PROCESS_SOURCE_FILE_Guide.md
View File

@@ -15,6 +15,11 @@ This document provides comprehensive documentation for the `FILE_MANAGER.PROCESS
- **Error Resilient**: Comprehensive error handling and logging for validation and file operations
- **Status Tracking**: Updates file processing status throughout validation and preparation workflow
**Migration Context:**
- This procedure is part of the **modern Airflow + DBT system** architecture
- Creates records in `CT_MRDS.A_SOURCE_FILE_RECEIVED` (modern control table)
- For legacy Informatica + WLA data migration, see [System Migration Guide](System_Migration_Informatica_to_Airflow_DBT.md)
## Procedure Signatures
The procedure is available in two variants:

149
confluence/Package_Deployment_Guide.md
View File

@@ -10,6 +10,122 @@ This guide provides step-by-step instructions for developers on how to properly
---
## MANDATORY STANDARDS (NEVER SKIP!)
### 🔴 Critical: Author Field
**ALL scripts MUST have:**
```sql
-- Author: Grzegorz Michalski
```
**❌ WRONG:**
```sql
-- Author: System
-- Author: Developer
-- Author: [your name here]
```
**✅ CORRECT:**
```sql
-- Author: Grzegorz Michalski
```
### 📅 Date Format Standard
**ISO 8601 format only:**
```sql
-- Date: 2026-02-03 (YYYY-MM-DD)
```
### 📝 Script Header Template
**Copy-paste this template for ALL scripts:**
```sql
-- =====================================================================
-- Script: XX_MARS_XXX_description.sql
-- MARS Issue: MARS-XXX
-- Purpose: Brief description of what this script does
-- Author: Grzegorz Michalski
-- Date: YYYY-MM-DD
-- =====================================================================
```
### 🔐 Database User Requirements
**Execute installations as ADMIN:**
```powershell
# ✅ CORRECT
Get-Content "install_marsXXX.sql" | sql "ADMIN/password@service"
# ❌ WRONG
Get-Content "install_marsXXX.sql" | sql "CT_MRDS/password@service"
```
### 🔍 Data Dictionary Views
**Use ALL_* views when installing as ADMIN:**
```sql
-- ✅ CORRECT (ADMIN user)
SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PACKAGE_NAME';
-- ❌ WRONG (shows ADMIN schema, not CT_MRDS)
SELECT * FROM USER_ERRORS WHERE NAME = 'PACKAGE_NAME';
```
### 📂 Log Directory Management
**MANDATORY in all master scripts:**
```sql
-- Create log/ directory before SPOOL (prevents failures)
host mkdir log 2>nul
-- Then configure dynamic SPOOL filename
var filename VARCHAR2(100)
BEGIN
:filename := 'log/INSTALL_MARS_XXX_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
```
### ✋ User Confirmation
**MANDATORY ACCEPT validation in master scripts:**
```sql
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
```
### 🚪 Clean Exit
**End all master scripts with:**
```sql
spool off
quit; -- ← MANDATORY for clean SQLcl/SQL*Plus exit
```
### 📋 Quick Compliance Checklist
Before committing any script, verify:
- [ ] `Author: Grzegorz Michalski` in header
- [ ] Date in ISO 8601 format (YYYY-MM-DD)
- [ ] Master scripts use `host mkdir log 2>nul`
- [ ] Master scripts require ACCEPT validation
- [ ] Master scripts end with `quit;`
- [ ] Verification queries use ALL_* views with OWNER filter
- [ ] Installation instructions specify ADMIN user
**💡 Remember:** These standards prevent common mistakes and ensure consistency across all MARS packages!
---
## Table of Contents
1. [Before You Start](#before-you-start)
@@ -1225,7 +1341,12 @@ SET PAUSE OFF
PROMPT =========================================================================
PROMPT MARS-XXXX: Rollback Package
PROMPT =========================================================================
PROMPT WARNING: This will reverse all changes from MARS-XXXX installation!
PROMPT This will reverse all changes from MARS-XXXX installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Restore previous package version
PROMPT 2. Remove added columns/tables
PROMPT 3. Revert configuration changes
PROMPT =========================================================================
-- Confirm rollback with user
@@ -1259,6 +1380,32 @@ spool off
- Revert configuration changes
- Include verification step after rollback (@@verify_packages_version.sql)
**Rollback Communication Style:**
- ✅ **DO**: Use calm, professional tone describing what will be done
- ✅ **DO**: List specific rollback steps clearly
- ❌ **DON'T**: Use dramatic words like "WARNING", "CRITICAL IMPACT", "DANGER"
- ❌ **DON'T**: Create panic - rollback is a normal part of deployment process
- **Rationale**: Rollbacks happen when installations fail or need to be reversed. This is a standard procedure, not an emergency.
**Example - Professional vs. Dramatic:**
```sql
-- ✅ CORRECT - Professional and Clear
PROMPT This will reverse all changes from MARS-828 installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Remove validation trigger
PROMPT 2. Drop ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns
PROMPT 3. Restore FILE_ARCHIVER package to v2.0.0
-- ❌ WRONG - Unnecessarily Dramatic
PROMPT WARNING: This will restore FILE_ARCHIVER to v2.0.0
PROMPT
PROMPT CRITICAL IMPACT:
PROMPT 1. All archival strategies revert to THRESHOLD_BASED
PROMPT 2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
```
---
### Version Tracking Integration

446
confluence/System_Migration_Informatica_to_Airflow_DBT.md Normal file
View File

@@ -0,0 +1,446 @@
# System Migration: Informatica + WLA → Airflow + DBT
This document describes the migration from the legacy Informatica + WLA data processing system to the modern Airflow + DBT architecture, including control table differences, data export strategies, and known limitations.
## Migration Overview
The MRDS (Market Reference Data System) is undergoing a fundamental technology migration:
**Legacy System (Informatica + WLA):**
- ETL Tool: Informatica PowerCenter
- Workflow Orchestration: WLA (Workflow Automation)
- Control Schema: `CT_ODS` (Operational Data Store Control)
- Primary Control Table: `CT_ODS.A_LOAD_HISTORY`
- Key Column: `A_ETL_LOAD_SET_KEY`
**Modern System (Airflow + DBT):**
- Orchestration: Apache Airflow
- Transformation: DBT (Data Build Tool)
- Control Schema: `CT_MRDS` (MRDS Control)
- Primary Control Tables: `CT_MRDS.A_SOURCE_FILE_RECEIVED`, `CT_MRDS.A_WORKFLOW_HISTORY`
- Key Column: `A_WORKFLOW_HISTORY_KEY`
## Control Table Architecture
### Legacy System: CT_ODS.A_LOAD_HISTORY
**Purpose**: Tracks Informatica PowerCenter workflow executions
**Structure**:
```sql
DESC CT_ODS.A_LOAD_HISTORY;
Name Type
______________________ _____________________
A_ETL_LOAD_SET_KEY NUMBER(38) -- Primary key
WORKFLOW_NAME VARCHAR2(255) -- Informatica workflow name
INFA_RUN_ID NUMBER(38) -- Informatica run ID
LOAD_START TIMESTAMP(6) -- Workflow start time
LOAD_END TIMESTAMP(6) -- Workflow end time
EXDI_APPL_REQ_ID VARCHAR2(255)
EXDI_CORRELATION_ID VARCHAR2(255)
LOAD_SUCCESSFUL CHAR(1) -- Y/N success flag
WLA_RUN_ID NUMBER(28) -- WLA run ID
DQ_FLAG VARCHAR2(5) -- Data quality flag
```
**Usage Pattern**:
- Created by Informatica workflows during ETL execution
- Used for temporal partitioning in DATA_EXPORTER
- Referenced via `A_ETL_LOAD_SET_KEY_FK` foreign key in data tables
### Modern System: CT_MRDS Control Tables
#### 1. A_SOURCE_FILE_RECEIVED
**Purpose**: Tracks individual file processing through the complete lifecycle
**Key Columns**:
```sql
A_SOURCE_FILE_RECEIVED_KEY NUMBER -- Primary key
SOURCE_FILE_NAME VARCHAR2 -- Full OCI path
PROCESSING_STATUS VARCHAR2 -- Status tracking
RECEPTION_DATE DATE -- File arrival timestamp
PARTITION_YEAR VARCHAR2(4) -- Archive partition (year)
PARTITION_MONTH VARCHAR2(2) -- Archive partition (month)
ARCH_FILE_NAME VARCHAR2 -- Parquet archive file path
```
**Status Workflow**:
```
RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED
```
**Usage Pattern**:
- Created by FILE_MANAGER.PROCESS_SOURCE_FILE during file validation
- Updated throughout file lifecycle (validation, ingestion, archival)
- Required by FILE_ARCHIVER for archival operations
- Links to A_WORKFLOW_HISTORY via workflow execution
#### 2. A_WORKFLOW_HISTORY
**Purpose**: Tracks Airflow + DBT workflow executions (similar role to A_LOAD_HISTORY)
**Key Columns**:
```sql
A_WORKFLOW_HISTORY_KEY NUMBER -- Primary key
WORKFLOW_NAME VARCHAR2 -- Airflow DAG name
WORKFLOW_START TIMESTAMP -- Workflow start time
WORKFLOW_END TIMESTAMP -- Workflow end time
WORKFLOW_SUCCESSFUL CHAR(1) -- Y/N success flag
```
**Usage Pattern**:
- Created by Airflow + DBT workflows during data processing
- Used for temporal partitioning decisions in FILE_ARCHIVER
- Referenced via `A_WORKFLOW_HISTORY_KEY_FK` foreign key in data tables
## Data Export Strategies
### Scenario 1: Legacy Data Export (Informatica → ODS)
**Use Case**: One-time migration of historical data loaded by Informatica
**Package**: `DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE`
**Control Table**: Uses `CT_ODS.A_LOAD_HISTORY` for temporal partitioning
**Example**:
```sql
-- Export historical AGGREGATED_ALLOTMENT data (loaded by Informatica)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK', -- Links to A_LOAD_HISTORY
pBucketArea => 'DATA',
pFolderName => 'legacy_migration',
pMinDate => DATE '2020-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
```
**Result**: CSV files in ODS bucket (DATA area), partitioned by LOAD_START from A_LOAD_HISTORY
### Scenario 2: Modern System Data (Airflow + DBT → ODS → ARCHIVE)
**Use Case**: Ongoing processing with new Airflow + DBT system
**Workflow**:
1. **File Arrival**: FILES → INBOX bucket
2. **Validation**: FILE_MANAGER.PROCESS_SOURCE_FILE → creates A_SOURCE_FILE_RECEIVED
3. **Processing**: Airflow + DBT → creates A_WORKFLOW_HISTORY
4. **Export**: DATA_EXPORTER → moves to ODS bucket (DATA area)
5. **Archival**: FILE_ARCHIVER → moves to ARCHIVE bucket (Parquet with Hive partitioning)
**Control Tables**: Uses both A_SOURCE_FILE_RECEIVED and A_WORKFLOW_HISTORY
**Example**:
```sql
-- Archival of data processed by Airflow + DBT
BEGIN
CT_MRDS.FILE_ARCHIVER.ARCHIVE_TABLE_DATA(
pSourceFileConfig => vConfig -- Requires A_SOURCE_FILE_RECEIVED records
);
END;
/
```
## Critical Gap: Legacy Data Archival
### Problem Statement
**Scenario**: Historical data exported using DATA_EXPORTER from Informatica-loaded tables
**Issue**: FILE_ARCHIVER requires records in `A_SOURCE_FILE_RECEIVED`, but legacy exports don't create them
**Impact**: Legacy data exported to ODS/DATA bucket **CANNOT** be archived to ARCHIVE bucket using FILE_ARCHIVER
### Technical Analysis
**DATA_EXPORTER Behavior**:
```sql
-- Uses A_LOAD_HISTORY for partitioning (Informatica workflows)
SELECT DISTINCT TO_CHAR(L.LOAD_START,'YYYY') AS YR,
TO_CHAR(L.LOAD_START,'MM') AS MN
FROM OU_TOP.AGGREGATED_ALLOTMENT T, CT_ODS.A_LOAD_HISTORY L
WHERE T.A_ETL_LOAD_SET_KEY_FK = L.A_ETL_LOAD_SET_KEY
AND L.LOAD_START >= :pMinDate
AND L.LOAD_START < :pMaxDate;
-- Creates CSV files: ODS/legacy_migration/AGGREGATED_ALLOTMENT_YYYYMM.csv
-- Does NOT create A_SOURCE_FILE_RECEIVED records
```
**FILE_ARCHIVER Requirement**:
```sql
-- Joins A_SOURCE_FILE_RECEIVED with A_WORKFLOW_HISTORY
JOIN CT_MRDS.A_SOURCE_FILE_RECEIVED r
ON r.A_SOURCE_FILE_CONFIG_KEY = pSourceFileConfig.A_SOURCE_FILE_CONFIG_KEY
AND r.PROCESSING_STATUS = 'INGESTED';
-- Without A_SOURCE_FILE_RECEIVED records, archival CANNOT proceed
```
### Workaround Strategies
#### Strategy 1: Manual Registration (Recommended for Small Datasets)
Manually create `A_SOURCE_FILE_RECEIVED` records for legacy exported files:
```sql
-- Step 1: Export legacy data to ODS/DATA
BEGIN
DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
pBucketArea => 'DATA',
pFolderName => 'legacy_export',
pMinDate => DATE '2024-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
-- Step 2: List exported CSV files
SELECT object_name, time_created, bytes
FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
credential_name => 'DEF_CRED_ARN',
location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/o/'
)) WHERE object_name LIKE 'ODS/legacy_export/AGGREGATED_ALLOTMENT_%';
-- Step 3: Manually register each file in A_SOURCE_FILE_RECEIVED
-- (Requires source configuration for AGGREGATED_ALLOTMENT to exist)
INSERT INTO CT_MRDS.A_SOURCE_FILE_RECEIVED (
A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
PROCESSING_STATUS,
RECEPTION_DATE,
BYTES,
CHECKSUM,
EXTERNAL_TABLE_NAME
) VALUES (
A_SOURCE_FILE_RECEIVED_KEY_SEQ.NEXTVAL,
(SELECT A_SOURCE_FILE_CONFIG_KEY FROM A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'AGGREGATED_ALLOTMENT' AND SOURCE_FILE_TYPE = 'INPUT'),
'ODS/legacy_export/AGGREGATED_ALLOTMENT_202401.csv',
'INGESTED', -- Skip validation, mark as already ingested
DATE '2024-01-15',
1048576, -- File size in bytes
'manual_registration',
NULL -- No external table needed
);
-- Repeat for all exported CSV files
COMMIT;
-- Step 4: Now FILE_ARCHIVER can process these files
BEGIN
FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfig => vConfig);
END;
/
```
#### Strategy 2: Direct Archive Export (Bypass ODS)
Skip ODS/DATA bucket entirely - export directly to ARCHIVE bucket in Parquet format:
```sql
-- Export legacy data directly to ARCHIVE bucket
BEGIN
DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
pBucketArea => 'ARCHIVE', -- Direct to archive
pFolderName => 'legacy_direct',
pMinDate => DATE '2020-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
-- Result: Parquet files with Hive partitioning in ARCHIVE bucket
-- Files: ARCHIVE/legacy_direct/PARTITION_YEAR=2024/PARTITION_MONTH=01/*.parquet
-- No A_SOURCE_FILE_RECEIVED records needed (archival already complete)
```
**Pros**:
- Simple, no manual registration
- Data already in final archive format (Parquet)
- Hive-style partitioning automatically applied
**Cons**:
- No record in A_SOURCE_FILE_RECEIVED (tracking gap)
- Cannot use FILE_ARCHIVER features (archival strategies, status tracking)
- Mixed folder structure (legacy_direct vs. standard source/table paths)
#### Strategy 3: Hybrid Approach (Recommended for Large Datasets)
Use DATA_EXPORTER for initial export, create minimal A_SOURCE_FILE_RECEIVED records programmatically:
```sql
-- Create helper procedure to register legacy exports
CREATE OR REPLACE PROCEDURE REGISTER_LEGACY_EXPORT (
pSourceFileId VARCHAR2,
pBucketArea VARCHAR2,
pFolderName VARCHAR2,
pFilePattern VARCHAR2,
pReceptionDate DATE DEFAULT SYSDATE
) AS
vConfigKey NUMBER;
vBucketUri VARCHAR2(500);
vPrefix VARCHAR2(500);
BEGIN
-- Get source configuration
SELECT A_SOURCE_FILE_CONFIG_KEY
INTO vConfigKey
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = pSourceFileId
AND SOURCE_FILE_TYPE = 'INPUT';
-- Get bucket URI
vBucketUri := CT_MRDS.ENV_MANAGER.GET_BUCKET_URI(pBucketArea);
vPrefix := 'ODS/' || pFolderName || '/';
-- Register all matching files
FOR rec IN (
SELECT object_name, bytes, etag
FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
credential_name => 'DEF_CRED_ARN',
location_uri => vBucketUri
))
WHERE object_name LIKE vPrefix || pFilePattern
) LOOP
INSERT INTO CT_MRDS.A_SOURCE_FILE_RECEIVED (
A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
PROCESSING_STATUS,
RECEPTION_DATE,
BYTES,
CHECKSUM
) VALUES (
A_SOURCE_FILE_RECEIVED_KEY_SEQ.NEXTVAL,
vConfigKey,
rec.object_name,
'INGESTED',
pReceptionDate,
rec.bytes,
rec.etag
);
END LOOP;
COMMIT;
DBMS_OUTPUT.PUT_LINE('Registered ' || SQL%ROWCOUNT || ' legacy files');
END;
/
-- Usage:
BEGIN
-- After DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE completes
REGISTER_LEGACY_EXPORT(
pSourceFileId => 'AGGREGATED_ALLOTMENT',
pBucketArea => 'DATA',
pFolderName => 'legacy_export',
pFilePattern => 'AGGREGATED_ALLOTMENT_2024%.csv',
pReceptionDate => DATE '2024-12-31'
);
-- Now FILE_ARCHIVER can process
FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfig => vConfig);
END;
/
```
## Migration Timeline and Coexistence
### Phase 1: Legacy System Only (Before Migration)
- All data loaded via Informatica + WLA
- Control table: `CT_ODS.A_LOAD_HISTORY`
- No `A_SOURCE_FILE_RECEIVED` records
### Phase 2: Parallel Operation (During Migration)
- **Old data**: Continue using Informatica + WLA → A_LOAD_HISTORY
- **New data**: Start using Airflow + DBT → A_SOURCE_FILE_RECEIVED + A_WORKFLOW_HISTORY
- **Challenge**: Different control tables for different data vintages
### Phase 3: New System Only (After Migration)
- All new data via Airflow + DBT
- Legacy data archived (one-time export using DATA_EXPORTER)
- Control tables: `CT_MRDS.A_SOURCE_FILE_RECEIVED`, `CT_MRDS.A_WORKFLOW_HISTORY`
## Recommendations
### For New Data (Airflow + DBT)
✅ Use standard workflow:
1. FILE_MANAGER.PROCESS_SOURCE_FILE (creates A_SOURCE_FILE_RECEIVED)
2. Airflow + DBT processing (creates A_WORKFLOW_HISTORY)
3. FILE_ARCHIVER.ARCHIVE_TABLE_DATA (uses both control tables)
### For Legacy Data Migration
✅ **Small Datasets (<1000 files)**: Strategy 1 (Manual Registration)
✅ **Large Datasets (>1000 files)**: Strategy 2 (Direct to ARCHIVE) or Strategy 3 (Hybrid)
❌ **Avoid**: Exporting to ODS/DATA without registration (orphaned files, cannot archive)
### Configuration Requirements
Before archiving legacy data, ensure source configuration exists:
```sql
-- Check if configuration exists
SELECT A_SOURCE_FILE_CONFIG_KEY, SOURCE_FILE_ID, TABLE_ID, ARCHIVAL_STRATEGY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'YOUR_SOURCE_FILE_ID';
-- If missing, create configuration
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'YOUR_SOURCE',
pSourceFileType => 'INPUT',
pSourceFileId => 'YOUR_SOURCE_FILE_ID',
pSourceFileDesc => 'Legacy migrated data',
pSourceFileNamePattern => 'pattern_*.csv',
pTableId => 'YOUR_TABLE_ID',
pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE'
);
```
## Known Limitations
### 1. No Retroactive A_SOURCE_FILE_RECEIVED Creation
DATA_EXPORTER does not automatically create A_SOURCE_FILE_RECEIVED records when exporting legacy data. This is by design - it's a one-time export tool, not a file tracking system.
### 2. FILE_ARCHIVER Requires A_SOURCE_FILE_RECEIVED
FILE_ARCHIVER cannot archive data without corresponding A_SOURCE_FILE_RECEIVED records. This prevents archiving of:
- Legacy Informatica-loaded data exported via DATA_EXPORTER
- Manually uploaded files not processed through FILE_MANAGER.PROCESS_SOURCE_FILE
### 3. Mixed Control Table References
During migration period, some procedures reference A_LOAD_HISTORY (DATA_EXPORTER) while others reference A_WORKFLOW_HISTORY (FILE_ARCHIVER). This is intentional but requires careful understanding of data lineage.
### 4. A_WORKFLOW_HISTORY vs A_LOAD_HISTORY Column Mismatch
The control tables have different schemas:
- **A_LOAD_HISTORY**: `LOAD_START`, `A_ETL_LOAD_SET_KEY`
- **A_WORKFLOW_HISTORY**: `WORKFLOW_START`, `A_WORKFLOW_HISTORY_KEY`
Test scripts must be aware of which table is being used.
## Related Documentation
- [PROCESS_SOURCE_FILE Guide](PROCESS_SOURCE_FILE_Guide.md) - File validation and ingestion workflow
- [FILE_ARCHIVER Guide](FILE_ARCHIVER_Guide.md) - Archival strategies and configuration
- [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md) - System configuration
- [Package Deployment Guide](Package_Deployment_Guide.md) - Deployment procedures
## Summary
The migration from Informatica + WLA to Airflow + DBT introduces new control tables (`A_SOURCE_FILE_RECEIVED`, `A_WORKFLOW_HISTORY`) while maintaining compatibility with legacy control tables (`A_LOAD_HISTORY`). Understanding the relationship between these tables is critical for:
- **Data Lineage**: Tracking which system processed which data
- **Export Operations**: Choosing appropriate DATA_EXPORTER procedures
- **Archival Operations**: Ensuring FILE_ARCHIVER has required metadata
- **Testing**: Using correct control tables in test scenarios
The recommended approach for legacy data migration is **Strategy 2 (Direct to ARCHIVE)** for large datasets, as it avoids the complexity of manual A_SOURCE_FILE_RECEIVED registration while achieving the goal of moving historical data to long-term archival storage.