admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: admin:76c8ce6cc7ece702bcdd36744573484e631ed04a
Branches Tags
admin:main admin:develop
...
compare: admin:e6ab189dc94905ec0609c7e416360238cba0ff4e
Branches Tags
admin:main admin:develop

9 Commits
76c8ce6cc7 ... e6ab189dc9

Author SHA1 Message Date
Grzegorz Michalski
e6ab189dc9 Refactor archival strategies in FILE_ARCHIVER package 
- Removed CURRENT_MONTH_ONLY strategy, replacing it with MINIMUM_AGE_MONTHS = 0 for current month retention.
- Updated validation logic to allow MINIMUM_AGE_MONTHS to be non-negative.
- Consolidated documentation to reflect changes in archival strategies.
- Adjusted rollback script to reset all archival parameters to NULL for 25 Release 01 tables.
- Enhanced README and installation scripts to clarify new configuration requirements and usage examples.
- Updated version history to indicate breaking changes and migration steps for existing configurations.
 2026-02-05 20:17:51 +01:00
Grzegorz Michalski
e93140e962 zmiany w pakiecie 2026-02-05 11:03:15 +01:00
Grzegorz Michalski
f42777f480 dokumentacja 2026-02-05 10:54:48 +01:00
Grzegorz Michalski
19925603fa aktualizacja definicji 2026-02-04 14:41:30 +01:00
Grzegorz Michalski
569c647b6c dopracowanie paczki 2026-02-04 12:22:33 +01:00
Grzegorz Michalski
8a162ce9bf dok 2026-02-04 12:21:48 +01:00
Grzegorz Michalski
c508e1e4e2 Refactor archival strategy rollback scripts for Release 01 tables 2026-02-03 19:39:08 +01:00
Grzegorz Michalski
d6fd0b3dbd Add scripts for configuring and rolling back archival strategies for Release 01 tables 2026-02-03 19:02:32 +01:00
Grzegorz Michalski
32c7a5dcee wk 2026-02-03 18:38:17 +01:00
24 changed files with 3007 additions and 1111 deletions
Expand all files Collapse all files

4
MARS_Packages/REL01_ADDITIONS/MARS-828/01_MARS_828_install_add_archival_strategy_columns.sql
View File

@@ -16,12 +16,12 @@ ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
-- Add check constraint for valid strategies
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD CONSTRAINT
CHK_ARCHIVAL_STRATEGY CHECK (
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'MINIMUM_AGE_MONTHS', 'HYBRID')
);
-- Add comments
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ARCHIVAL_STRATEGY IS
'Archival strategy: THRESHOLD_BASED (days), CURRENT_MONTH_ONLY (exclude current month), MINIMUM_AGE_MONTHS (minimum age), HYBRID (combination)';
'Archival strategy: THRESHOLD_BASED (days), MINIMUM_AGE_MONTHS (0=current month, N=retain N months), HYBRID (combination)';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.MINIMUM_AGE_MONTHS IS
'Minimum age in months for archival (used with MINIMUM_AGE_MONTHS or HYBRID strategies)';

325
MARS_Packages/REL01_ADDITIONS/MARS-828/06_MARS_828_configure_release01_tables.sql Normal file
View File

@@ -0,0 +1,325 @@
-- =====================================================================
-- Script: 06_MARS_828_configure_release01_tables.sql
-- MARS Issue: MARS-828
-- Purpose: Configure COMPLETE archival parameters for Release 01 tables
-- Author: Grzegorz Michalski
-- Date: 2026-02-05
--
-- Description:
-- Configures ALL archival parameters for 25 tables:
-- - ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS
-- - Archival triggers (FILES/ROWS/BYTES thresholds)
-- - Statistics expiration (HOURS_TO_EXPIRE_STATISTICS)
--
-- Configuration by group:
-- - 19 LM tables: MINIMUM_AGE_MONTHS=0 (current month only), 10 files OR 100K rows OR 1GB, 24h stats
-- - 2 CSDB DEBT: MINIMUM_AGE_MONTHS=6, 5 files OR 50K rows OR 512MB, 48h stats
-- - 4 CSDB ratings: MINIMUM_AGE_MONTHS=0 (current month only), 10 files OR 20K rows OR 256MB, 72h stats
--
-- Dependencies:
-- - A_SOURCE_FILE_CONFIG table with all archival columns
-- - TRG_BI_A_SRC_FILE_CFG_ARCH_VAL trigger
-- =====================================================================
PROMPT =====================================================================
PROMPT MARS-828: Enhanced Configuration - Complete Archival Parameters
PROMPT =====================================================================
PROMPT
PROMPT This script configures COMPLETE archival parameters for 25 tables
PROMPT
PROMPT LM Tables (19):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 0 (current month only)
PROMPT - Triggers: 10 files OR 100,000 rows OR 1 GB
PROMPT - Stats Expiration: 24 hours
PROMPT
PROMPT CSDB DEBT Tables (2):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 6
PROMPT - Triggers: 5 files OR 50,000 rows OR 512 MB
PROMPT - Stats Expiration: 48 hours
PROMPT
PROMPT CSDB Rating/Description Tables (4):
PROMPT - Strategy: MINIMUM_AGE_MONTHS = 0 (current month only)
PROMPT - Triggers: 10 files OR 20,000 rows OR 256 MB
PROMPT - Stats Expiration: 72 hours
PROMPT
PROMPT Current timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS CURRENT_TIME FROM DUAL;
PROMPT
PROMPT =====================================================================
PROMPT SECTION 1: LM Tables Configuration (MINIMUM_AGE_MONTHS = 0)
PROMPT =====================================================================
PROMPT Thresholds: 10 files OR 100K rows OR 1GB
PROMPT Stats expire: 24 hours
PROMPT =====================================================================
-- Update all 19 LM tables in a single statement
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 100000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824, -- 1 GB
HOURS_TO_EXPIRE_STATISTICS = 24
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'LM'
AND TABLE_ID IN (
'LM_STANDING_FACILITIES',
'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER',
'LM_TTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_HEADER',
'LM_ADHOC_ADJUSTMENTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
'LM_BALANCESHEET_HEADER',
'LM_BALANCESHEET_ITEM',
'LM_CSM_ADJUSTMENTS_HEADER',
'LM_CSM_ADJUSTMENTS_ITEM',
'LM_CSM_ADJUSTMENTS_ITEM_HEADER',
'LM_CURRENT_ACCOUNTS_HEADER',
'LM_CURRENT_ACCOUNTS_ITEM',
'LM_FORECAST_HEADER',
'LM_FORECAST_ITEM',
'LM_QRE_ADJUSTMENTS_HEADER',
'LM_QRE_ADJUSTMENTS_ITEM',
'LM_QRE_ADJUSTMENTS_ITEM_HEADER'
);
PROMPT
PROMPT LM tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 2: CSDB DEBT Tables (MINIMUM_AGE_MONTHS = 6)
PROMPT =====================================================================
PROMPT Thresholds: 5 files OR 50K rows OR 512MB
PROMPT Stats expire: 48 hours
PROMPT =====================================================================
-- Update CSDB DEBT tables (6-month retention)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 5,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 50000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 536870912, -- 512 MB
HOURS_TO_EXPIRE_STATISTICS = 48
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
PROMPT
PROMPT CSDB DEBT tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 3: CSDB Rating/Description Tables (MINIMUM_AGE_MONTHS = 0)
PROMPT =====================================================================
PROMPT Thresholds: 10 files OR 20K rows OR 256MB
PROMPT Stats expire: 72 hours
PROMPT =====================================================================
-- Update CSDB rating/description tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 20000,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 268435456, -- 256 MB
HOURS_TO_EXPIRE_STATISTICS = 72
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN (
'CSDB_INSTR_RAT_FULL',
'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL',
'CSDB_ISSUER_DESC_FULL'
);
PROMPT
PROMPT CSDB rating/description tables configuration completed
PROMPT
PROMPT =====================================================================
PROMPT Committing configuration changes...
PROMPT =====================================================================
COMMIT;
PROMPT
PROMPT Configuration committed successfully.
PROMPT
PROMPT =====================================================================
PROMPT VERIFICATION: Complete Archival Configuration
PROMPT =====================================================================
PROMPT
PROMPT LM Tables (MINIMUM_AGE_MONTHS = 0):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 0
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 100000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824
AND HOURS_TO_EXPIRE_STATISTICS = 24
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'LM'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID LIKE 'LM_%'
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB DEBT Tables (MINIMUM_AGE_MONTHS = 6):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 6
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 5
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 50000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 536870912
AND HOURS_TO_EXPIRE_STATISTICS = 48
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY')
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB Rating/Description Tables (MINIMUM_AGE_MONTHS = 0):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS'
AND MINIMUM_AGE_MONTHS = 0
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 10
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 20000
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 268435456
AND HOURS_TO_EXPIRE_STATISTICS = 72
THEN 'OK'
ELSE 'ERROR'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL', 'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL')
ORDER BY TABLE_ID;
PROMPT
PROMPT =====================================================================
PROMPT Summary: Total Configured Tables with Full Parameters
PROMPT =====================================================================
SELECT
COUNT(*) AS TOTAL_CONFIGURED,
SUM(CASE WHEN MINIMUM_AGE_MONTHS = 0 THEN 1 ELSE 0 END) AS CURRENT_MONTH_ONLY,
SUM(CASE WHEN MINIMUM_AGE_MONTHS > 0 THEN 1 ELSE 0 END) AS MULTI_MONTH_RETENTION,
SUM(CASE WHEN FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_FILE_THRESHOLD,
SUM(CASE WHEN ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_ROWS_THRESHOLD,
SUM(CASE WHEN BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NOT NULL THEN 1 ELSE 0 END) AS WITH_BYTES_THRESHOLD,
SUM(CASE WHEN HOURS_TO_EXPIRE_STATISTICS IS NOT NULL THEN 1 ELSE 0 END) AS WITH_STATS_EXPIRY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)));
PROMPT
PROMPT Expected:
PROMPT - TOTAL_CONFIGURED = 25
PROMPT - WITH_FILE_THRESHOLD = 25
PROMPT - WITH_ROWS_THRESHOLD = 25
PROMPT - WITH_BYTES_THRESHOLD = 25
PROMPT - WITH_STATS_EXPIRY = 25
PROMPT
PROMPT =====================================================================
PROMPT Parameter Configuration Summary by Group
PROMPT =====================================================================
SELECT
CASE
WHEN A_SOURCE_KEY = 'LM' THEN 'LM Tables'
WHEN TABLE_ID LIKE 'CSDB_DEBT%' THEN 'CSDB DEBT'
ELSE 'CSDB Ratings'
END AS GROUP_NAME,
COUNT(*) AS TABLE_COUNT,
MAX(ARCHIVAL_STRATEGY) AS STRATEGY,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE,
MAX(FILES_COUNT_OVER_ARCHIVE_THRESHOLD) AS FILES_THRESHOLD,
MAX(ROWS_COUNT_OVER_ARCHIVE_THRESHOLD) AS ROWS_THRESHOLD,
ROUND(MAX(BYTES_SUM_OVER_ARCHIVE_THRESHOLD)/1048576, 0) || ' MB' AS BYTES_THRESHOLD,
MAX(HOURS_TO_EXPIRE_STATISTICS) AS STATS_HOURS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)))
GROUP BY
CASE
WHEN A_SOURCE_KEY = 'LM' THEN 'LM Tables'
WHEN TABLE_ID LIKE 'CSDB_DEBT%' THEN 'CSDB DEBT'
ELSE 'CSDB Ratings'
END
ORDER BY GROUP_NAME;
PROMPT
PROMPT =====================================================================
PROMPT Configuration Complete
PROMPT =====================================================================
PROMPT
PROMPT Next Steps:
PROMPT 1. Review verification results above
PROMPT 2. All 25 tables should show STATUS = 'OK'
PROMPT 3. Verify parameter configuration summary matches expectations
PROMPT 4. Monitor archival process with new thresholds
PROMPT
PROMPT Archival Trigger Logic (OR condition):
PROMPT - Archival starts when ANY threshold is exceeded
PROMPT - Example: 10 files OR 100K rows OR 1GB triggers archival
PROMPT
PROMPT Statistics Refresh:
PROMPT - LM: Every 24 hours (daily updates for frequent changes)
PROMPT - CSDB DEBT: Every 48 hours (bi-daily for larger datasets)
PROMPT - CSDB Ratings: Every 72 hours (every 3 days for stable data)
PROMPT
PROMPT Log file: 06_MARS_828_configure_release01_tables.log
PROMPT =====================================================================

271
MARS_Packages/REL01_ADDITIONS/MARS-828/96_MARS_828_rollback_release01_configuration.sql Normal file
View File

@@ -0,0 +1,271 @@
-- =====================================================================
-- Script: 96_MARS_828_rollback_release01_configuration.sql
-- MARS Issue: MARS-828
-- Purpose: ROLLBACK archival configuration for Release 01 tables
-- Author: Grzegorz Michalski
-- Date: 2026-02-05
--
-- Description:
-- Rolls back archival configuration for 25 tables by setting all
-- archival parameters back to NULL (unconfigured state):
-- - ARCHIVAL_STRATEGY
-- - MINIMUM_AGE_MONTHS
-- - FILES_COUNT_OVER_ARCHIVE_THRESHOLD
-- - ROWS_COUNT_OVER_ARCHIVE_THRESHOLD
-- - BYTES_SUM_OVER_ARCHIVE_THRESHOLD
-- - HOURS_TO_EXPIRE_STATISTICS
--
-- This script reverts changes made by:
-- - 06_MARS_828_configure_release01_tables.sql
--
-- Dependencies:
-- - A_SOURCE_FILE_CONFIG table with archival columns
-- =====================================================================
PROMPT =====================================================================
PROMPT MARS-828: ROLLBACK - Remove Archival Configuration
PROMPT =====================================================================
PROMPT
PROMPT This script will REMOVE archival configuration from 25 tables
PROMPT
PROMPT WARNING: This will reset ALL archival parameters to NULL
PROMPT
PROMPT Tables affected:
PROMPT - 19 LM tables
PROMPT - 2 CSDB DEBT tables
PROMPT - 4 CSDB rating/description tables
PROMPT
PROMPT Current timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS CURRENT_TIME FROM DUAL;
PROMPT
PROMPT =====================================================================
PROMPT SECTION 1: Rollback LM Tables Configuration
PROMPT =====================================================================
-- Rollback all 19 LM tables in a single statement
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'LM'
AND TABLE_ID IN (
'LM_STANDING_FACILITIES',
'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER',
'LM_TTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_HEADER',
'LM_ADHOC_ADJUSTMENTS_ITEM',
'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
'LM_BALANCESHEET_HEADER',
'LM_BALANCESHEET_ITEM',
'LM_CSM_ADJUSTMENTS_HEADER',
'LM_CSM_ADJUSTMENTS_ITEM',
'LM_CSM_ADJUSTMENTS_ITEM_HEADER',
'LM_CURRENT_ACCOUNTS_HEADER',
'LM_CURRENT_ACCOUNTS_ITEM',
'LM_FORECAST_HEADER',
'LM_FORECAST_ITEM',
'LM_QRE_ADJUSTMENTS_HEADER',
'LM_QRE_ADJUSTMENTS_ITEM',
'LM_QRE_ADJUSTMENTS_ITEM_HEADER'
);
PROMPT
PROMPT LM tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 2: Rollback CSDB DEBT Tables Configuration
PROMPT =====================================================================
-- Rollback CSDB DEBT tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
PROMPT
PROMPT CSDB DEBT tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT SECTION 3: Rollback CSDB Rating/Description Tables Configuration
PROMPT =====================================================================
-- Rollback CSDB rating/description tables
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = NULL,
MINIMUM_AGE_MONTHS = NULL,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = NULL,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD = NULL,
HOURS_TO_EXPIRE_STATISTICS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND A_SOURCE_KEY = 'CSDB'
AND TABLE_ID IN (
'CSDB_INSTR_RAT_FULL',
'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL',
'CSDB_ISSUER_DESC_FULL'
);
PROMPT
PROMPT CSDB rating/description tables rollback completed
PROMPT
PROMPT =====================================================================
PROMPT Committing rollback changes...
PROMPT =====================================================================
COMMIT;
PROMPT
PROMPT Rollback committed successfully.
PROMPT
PROMPT =====================================================================
PROMPT VERIFICATION: Archival Configuration Removed
PROMPT =====================================================================
PROMPT
PROMPT LM Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'LM'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID LIKE 'LM_%'
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB DEBT Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY')
ORDER BY TABLE_ID;
PROMPT
PROMPT CSDB Rating/Description Tables (should all be NULL):
PROMPT
SELECT
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
FILES_COUNT_OVER_ARCHIVE_THRESHOLD AS FILE_THR,
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD AS ROW_THR,
BYTES_SUM_OVER_ARCHIVE_THRESHOLD AS BYTE_THR,
HOURS_TO_EXPIRE_STATISTICS AS STATS_HRS,
CASE
WHEN ARCHIVAL_STRATEGY IS NULL
AND MINIMUM_AGE_MONTHS IS NULL
AND FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL
AND BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL
AND HOURS_TO_EXPIRE_STATISTICS IS NULL
THEN 'OK'
ELSE 'ERROR - Still configured'
END AS STATUS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'CSDB'
AND SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN ('CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL', 'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL')
ORDER BY TABLE_ID;
PROMPT
PROMPT =====================================================================
PROMPT Summary: Total Tables with Rollback Applied
PROMPT =====================================================================
SELECT
COUNT(*) AS TOTAL_TABLES,
SUM(CASE WHEN ARCHIVAL_STRATEGY IS NULL THEN 1 ELSE 0 END) AS STRATEGY_NULL,
SUM(CASE WHEN MINIMUM_AGE_MONTHS IS NULL THEN 1 ELSE 0 END) AS MIN_AGE_NULL,
SUM(CASE WHEN FILES_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS FILE_THR_NULL,
SUM(CASE WHEN ROWS_COUNT_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS ROWS_THR_NULL,
SUM(CASE WHEN BYTES_SUM_OVER_ARCHIVE_THRESHOLD IS NULL THEN 1 ELSE 0 END) AS BYTES_THR_NULL,
SUM(CASE WHEN HOURS_TO_EXPIRE_STATISTICS IS NULL THEN 1 ELSE 0 END) AS STATS_NULL
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND ((A_SOURCE_KEY = 'LM' AND TABLE_ID LIKE 'LM_%')
OR (A_SOURCE_KEY = 'CSDB' AND TABLE_ID IN (
'CSDB_DEBT', 'CSDB_DEBT_DAILY',
'CSDB_INSTR_RAT_FULL', 'CSDB_INSTR_DESC_FULL',
'CSDB_ISSUER_RAT_FULL', 'CSDB_ISSUER_DESC_FULL'
)));
PROMPT
PROMPT Expected: All counts should be 25 (all parameters NULL for all tables)
PROMPT - TOTAL_TABLES = 25
PROMPT - STRATEGY_NULL = 25
PROMPT - MIN_AGE_NULL = 25
PROMPT - FILE_THR_NULL = 25
PROMPT - ROWS_THR_NULL = 25
PROMPT - BYTES_THR_NULL = 25
PROMPT - STATS_NULL = 25
PROMPT
PROMPT =====================================================================
PROMPT Rollback Complete
PROMPT =====================================================================
PROMPT
PROMPT Next Steps:
PROMPT 1. Review verification results above
PROMPT 2. All 25 tables should show STATUS = 'OK'
PROMPT 3. All archival parameters should be NULL
PROMPT 4. Tables are now in unconfigured state (no archival strategy)
PROMPT
PROMPT To restore configuration, re-run:
PROMPT @06_MARS_828_configure_release01_tables.sql
PROMPT
PROMPT Log file: 96_MARS_828_rollback_release01_configuration.log
PROMPT =====================================================================

16
MARS_Packages/REL01_ADDITIONS/MARS-828/README.md
View File

@@ -17,7 +17,7 @@ WHERE extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRES
**Solution**: Introduce ARCHIVAL_STRATEGY configuration column with four strategies:
- `THRESHOLD_BASED` - Days-based threshold (backward compatible)
- `CURRENT_MONTH_ONLY` - Keep only current month data
- `MINIMUM_AGE_MONTHS` - Retain data for specified months (0 = current month only)
- `MINIMUM_AGE_MONTHS` - Archive data older than X months
- `HYBRID` - Combination of current month and minimum age
@@ -38,7 +38,7 @@ DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3),
DAYS_FOR_ARCHIVE_THRESHOLD NUMBER DEFAULT 30,
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'MINIMUM_AGE_MONTHS', 'HYBRID'))
```
**New Trigger**: TRG_BI_ARCHIVAL_STRATEGY_VAL
@@ -69,17 +69,17 @@ CONSTRAINT CHK_ARCHIVAL_STRATEGY CHECK (ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED',
| Strategy | WHERE Clause Logic | Configuration Required | Use Case |
|----------|-------------------|----------------------|----------|
| `THRESHOLD_BASED` | `extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD` | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility |
| `CURRENT_MONTH_ONLY` | `TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')` | None | General sources (LM, TOP) |
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -N)` | MINIMUM_AGE_MONTHS (0=current month) | All sources |
| `MINIMUM_AGE_MONTHS` | `workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -X)` | MINIMUM_AGE_MONTHS | CSDB (6 months retention) |
| `HYBRID` | Both CURRENT_MONTH_ONLY AND MINIMUM_AGE_MONTHS | MINIMUM_AGE_MONTHS | Advanced scenarios |
| `HYBRID` | Combines month truncation AND minimum age | MINIMUM_AGE_MONTHS | Advanced scenarios |
## Configuration Examples
```sql
-- LM/TOP sources: Archive everything except current month
UPDATE A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY',
MINIMUM_AGE_MONTHS = NULL
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0, -- 0 = current month only
WHERE A_SOURCE_KEY IN ('LM', 'TOP');
-- CSDB: Archive only data older than 6 months
@@ -189,7 +189,7 @@ sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
**Main Test Script**: test/test_archival_strategies.sql
```sql
-- Test 1: CURRENT_MONTH_ONLY strategy
-- Test 1: MINIMUM_AGE_MONTHS=0 strategy (current month only)
-- Expected: Archives data from previous months only
SELECT COUNT(*) FROM table WHERE TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM');
@@ -249,7 +249,7 @@ No automatic migration required. New columns have sensible defaults:
- MINIMUM_AGE_MONTHS = NULL (not required for THRESHOLD_BASED)
## Version History
- **v3.0.0** (2026-01-27): Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration
- **v3.0.0** (2026-01-27): Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration
- **v2.0.0** (2025-10-01): Initial FILE_ARCHIVER package with threshold-based archival
## Related JIRA Issues

38
MARS_Packages/REL01_ADDITIONS/MARS-828/install_mars828.sql
View File

@@ -35,9 +35,9 @@ PROMPT =========================================================================
PROMPT MARS-828 Installation Starting
PROMPT ============================================================================
PROMPT Package: CT_MRDS.FILE_ARCHIVER
PROMPT Change: Enhanced archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID)
PROMPT Change: Enhanced archival strategies (MINIMUM_AGE_MONTHS, HYBRID)
PROMPT Purpose: Flexible archival policies per data source
PROMPT Steps: 7 (DDL, Trigger, Packages, Verify, Track, Verify)
PROMPT Steps: 8 (DDL, Trigger, Packages, Verify, Track, Verify, Configure)
PROMPT Timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_start FROM DUAL;
PROMPT ============================================================================
@@ -54,40 +54,46 @@ END;
WHENEVER SQLERROR CONTINUE
-- Installation steps
PROMPT7: Adding archival strategy columns to A_SOURCE_FILE_CONFIG
PROMPT
PROMPT Step 1/8: Adding archival strategy columns to A_SOURCE_FILE_CONFIG
PROMPT ===================================================================
@@01_MARS_828_install_add_archival_strategy_columns.sql
PROMPT
PROMPT Step 2/7: Creating validation trigger
PROMPT Step 2/8: Creating validation trigger
PROMPT ======================================
@@02_MARS_828_install_archival_strategy_trigger.sql
PROMPT
PROMPT Step 3/7: Deploying FILE_ARCHIVER Package Specification v3.0.0
PROMPT Step 3/8: Deploying FILE_ARCHIVER Package Specification v3.1.0
PROMPT ===============================================================
@@03_MARS_828_install_CT_MRDS_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 4/7: Deploying FILE_ARCHIVER Package Body v3.0.0
PROMPT Step 4/8: Deploying FILE_ARCHIVER Package Body v3.1.0
PROMPT ======================================================
@@04_MARS_828_install_CT_MRDS_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 5/7: Verifying installation
PROMPT Step 5/8: Verifying installation
PROMPT =================================
@@05_MARS_828_verify_installation.sql
PROMPT
PROMPT Step 6/7: Tracking package versions
PROMPT Step 6/8: Tracking package versions
PROMPT ====================================
@@track_package_versions.sql
PROMPT
PROMPT Step 7/7: Verifying tracked packages
PROMPT Step 7/8: Verifying tracked packages
PROMPT =====================================
@@verify_packages_version.sql
PROMPT
PROMPT Step 8/8: Configuring Release 01 tables archival strategies
PROMPT ==============================================================
@@06_MARS_828_configure_release01_tables.sql
PROMPT
PROMPT ============================================================================
PROMPT MARS-828 Installation Completed
@@ -97,16 +103,10 @@ SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS install_end FROM DUAL;
PROMPT
PROMPT Installation Summary:
PROMPT - Package: CT_MRDS.FILE_ARCHIVER
PROMPT - Version: 2.0.0 -> 3.0.0 (MAJOR)
PROMPT - New Strategies: CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID
PROMPT - Backward Compatible: THRESHOLD_BASED (default)
PROMPT
PROMPT Next Steps:
PROMPT 1. Configure archival strategies per source:
PROMPT UPDATE A_SOURCE_FILE_CONFIG SET ARCHIVAL_STRATEGY = 'CURRENT_MONTH_ONLY' WHERE A_SOURCE_KEY = 'LM';
PROMPT UPDATE A_SOURCE_FILE_CONFIG SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS', MINIMUM_AGE_MONTHS = 6 WHERE A_SOURCE_KEY = 'CSDB';
PROMPT 2. Test strategies using test_archival_strategies.sql
PROMPT 3. Monitor first archival run
PROMPT - Version: 3.1.0 (BREAKING CHANGE - CURRENT_MONTH_ONLY removed)
PROMPT - Strategies: THRESHOLD_BASED (default), MINIMUM_AGE_MONTHS (0=current month), HYBRID
PROMPT - Backward Compatible: Yes (default THRESHOLD_BASED preserved)
PROMPT - Configured Tables: 25 Release 01 tables (19 LM + 6 CSDB)
PROMPT
PROMPT Log file: &_filename
PROMPT ============================================================================

310
MARS_Packages/REL01_ADDITIONS/MARS-828/log/INSTALL_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260128_064911.log
View File

@@ -1,310 +0,0 @@
============================================================================
MARS-828 Installation Starting (AUTO MODE)
============================================================================
INSTALL_START
______________________
2026-01-28 06:49:11
1 row selected.
============================================================================
Step 1/7: Adding archival strategy columns
===================================================================
========================================
MARS-828: Adding archival strategy columns
========================================
Error starting at line : 11 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\01_MARS_828_install_add_archival_strategy_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
ARCHIVAL_STRATEGY VARCHAR2(30) DEFAULT 'THRESHOLD_BASED' NOT NULL,
MINIMUM_AGE_MONTHS NUMBER(3) DEFAULT NULL
)
Error report -
ORA-01430: column being added already exists in table
https://docs.oracle.com/error-help/db/ora-01430/
01430. 00000 - "column being added already exists in table"
*Cause: An ALTER TABLE ADD statement specified the name of a
column that was already in the table. All column names must be
unique within a table.
*Action: Specify a unique name for the new column, then
re-execute the statement.
Error starting at line : 17 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\01_MARS_828_install_add_archival_strategy_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD CONSTRAINT
CHK_ARCHIVAL_STRATEGY CHECK (
ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
)
Error report -
ORA-02264: name already used by an existing constraint
https://docs.oracle.com/error-help/db/ora-02264/
02264. 00000 - "name already used by an existing constraint"
*Cause: The specified constraint name has to be unique.
*Action: Specify a unique constraint name for the constraint.
Comment created.
Comment created.
COLUMN_NAME DATA_TYPE DATA_LENGTH NULLABLE DATA_DEFAULT
_____________________ ____________ ______________ ___________ _____________________
ARCHIVAL_STRATEGY VARCHAR2 30 N 'THRESHOLD_BASED'
MINIMUM_AGE_MONTHS NUMBER 22 Y NULL
2 rows selected.
========================================
Archival strategy columns added successfully
========================================
Step 2/7: Creating validation trigger
======================================
========================================
MARS-828: Creating archival strategy validation trigger
========================================
Trigger CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL compiled
TRIGGER_NAME STATUS TRIGGER_TYPE TRIGGERING_EVENT
_________________________________ __________ __________________ ___________________
TRG_BI_A_SRC_FILE_CFG_ARCH_VAL ENABLED BEFORE EACH ROW INSERT OR UPDATE
1 row selected.
========================================
Archival strategy validation trigger created successfully
========================================
Step 3/7: Deploying FILE_ARCHIVER Spec v3.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
========================================
FILE_ARCHIVER Specification v3.0.0 ready for installation
========================================
Step 4/7: Deploying FILE_ARCHIVER Body v3.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
LINE/COL ERROR
--------- -------------------------------------------------------------
36/10 PLS-00103: Encountered the symbol "\" when expecting one of the following: ( begin case declare else end exit for goto if loop mod null pragma raise return select update when while with <an identifier> <a double-quoted delimited-identifier> <a bind variable> << continue close current delete fetch lock insert open rollback savepoint set sql execute commit forall merge pipe purge json_object The symbol "case was inserted before "\" to continue.
49/8 PLS-00103: Encountered the symbol "GET_ARCHIVAL_WHERE_CLAUSE" when expecting one of the following: case
67/9 PLS-00103: Encountered the symbol "JOIN" when expecting one of the following: , ; for group having intersect minus order start union where connect
Errors: check compiler log
Step 5/7: Verifying installation
=================================
========================================
MARS-828: Verification Script
========================================
1. Verifying A_SOURCE_FILE_CONFIG columns...
COLUMN_NAME DATA_TYPE NULLABLE DATA_DEFAULT
_____________________ ____________ ___________ _____________________
ARCHIVAL_STRATEGY VARCHAR2 N 'THRESHOLD_BASED'
MINIMUM_AGE_MONTHS NUMBER Y NULL
2 rows selected.
2. Verifying check constraint...
CONSTRAINT_NAME CONSTRAINT_TYPE SEARCH_CONDITION
________________________ __________________ _________________________________________________________________________________________________
CHK_ARCHIVAL_STRATEGY C ARCHIVAL_STRATEGY IN ('THRESHOLD_BASED', 'CURRENT_MONTH_ONLY', 'MINIMUM_AGE_MONTHS', 'HYBRID')
1 row selected.
3. Verifying validation trigger...
TRIGGER_NAME STATUS TRIGGER_TYPE
_________________________________ __________ __________________
TRG_BI_A_SRC_FILE_CFG_ARCH_VAL ENABLED BEFORE EACH ROW
1 row selected.
4. Checking FILE_ARCHIVER package status...
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ __________ ______________________
FILE_ARCHIVER PACKAGE VALID 2026-01-28 06:49:14
FILE_ARCHIVER PACKAGE BODY INVALID 2026-01-28 06:49:15
2 rows selected.
5. Checking for compilation errors...
NAME TYPE LINE POSITION TEXT
________________ _______________ _______ ___________ _____________________________________________________________________________
FILE_ARCHIVER PACKAGE BODY 36 10 PLS-00103: Encountered the symbol "\" when expecting one of the following:
( begin case declare else end exit for goto if loop mod null
pragma raise return select update when while with
<an identifier> <a double-quoted delimited-identifier>
<a bind variable> << continue close current delete fetch lock
insert open rollback savepoint set sql execute commit forall
merge pipe purge json_object
The symbol "case was inserted before "\" to continue.
FILE_ARCHIVER PACKAGE BODY 49 8 PLS-00103: Encountered the symbol "GET_ARCHIVAL_WHERE_CLAUSE" when expecting one of the following:
case
FILE_ARCHIVER PACKAGE BODY 67 9 PLS-00103: Encountered the symbol "JOIN" when expecting one of the following:
, ; for group having intersect minus order start union where
connect
3 rows selected.
6. Verifying FILE_ARCHIVER version...
Error starting at line : 79 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\05_MARS_828_verify_installation.sql
In command -
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() as package_version FROM DUAL
Error at Command Line : 79 Column : 68 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\05_MARS_828_verify_installation.sql
Error report -
SQL Error: ORA-04063: package body "CT_MRDS.FILE_ARCHIVER" has errors
https://docs.oracle.com/error-help/db/ora-04063/04063. 00000 - "%s has errors"
*Cause: Attempt to execute a stored procedure or use a view that has
errors. For stored procedures, the problem could be syntax errors
or references to other, non-existent procedures. For views,
the problem could be a reference in the view's defining query to
a non-existent table.
Can also be a table which has references to non-existent or
inaccessible types.
*Action: Fix the errors and/or create referenced objects as necessary.
More Details :
https://docs.oracle.com/error-help/db/ora-04063/
7. Testing trigger validation (should fail)...
SUCCESS: Trigger validation working correctly
Expected error: ORA-20999: MINIMUM_AGE_MONTHS is required for MINIMUM_AGE_MONTHS strategy
ORA-06512: at "CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL", line 26
ORA-06512: at "CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL", line 8
ORA-04088: error during execution of trigger 'CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL'
PL/SQL procedure successfully completed.
========================================
MARS-828: Verification Complete
========================================
Step 6/7: Tracking package versions
====================================
========================================
Package Version Tracking
========================================
Summary:
--------
Packages tracked: 0/1
========================================
PL/SQL procedure successfully completed.
Step 7/7: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ _______________________________________________________
CT_MRDS DATA_EXPORTER 2.5.0 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-26 17:15:41
Version: 2.5.0
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-26 17:15:41
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 WARNING: Package CT_MRDS.FILE_ARCHIVER has changed!
========================================
Last Tracked Version: 2.0.0
Last Tracked Date: 2025-11-25 16:00:36
SPECIFICATION Changed:
Current Hash: 71a835e531971ca7...
Last Hash: 836641155e237fc5...
BODY Changed:
Current Hash: 6f87dff6b0394529...
Last Hash: 9bf9b3c0e059493c...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
EnvironmentID set to: dev
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
============================================================================
MARS-828 Installation Completed
============================================================================
INSTALL_END
______________________
2026-01-28 06:49:23
1 row selected.
============================================================================

194
MARS_Packages/REL01_ADDITIONS/MARS-828/log/ROLLBACK_MARS_828_AUTO_G45C5E88148E17E_GGMICHALSKI_20260129_195230.log
View File

@@ -1,194 +0,0 @@
============================================================================
MARS-828 Rollback Starting (AUTO MODE - No Confirmation)
============================================================================
ROLLBACK_START
______________________
2026-01-29 19:52:30
Elapsed: 00:00:00.065
============================================================================
Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.110
Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.133
Step 3/6: Dropping validation trigger
======================================
========================================
MARS-828: Dropping archival strategy validation trigger
========================================
Error starting at line : 10 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\93_MARS_828_rollback_trigger.sql
In command -
DROP TRIGGER CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL
Error report -
ORA-04080: trigger 'TRG_BI_A_SRC_FILE_CFG_ARCH_VAL' does not exist
https://docs.oracle.com/error-help/db/ora-04080/
04080. 00000 - "trigger '%s' does not exist"
*Cause: The TRIGGER name is invalid.
*Action: Check the trigger name.
Elapsed: 00:00:00.135
TRIGGER_COUNT
________________
0
Elapsed: 00:00:00.070
========================================
Validation trigger dropped successfully
========================================
Step 4/6: Dropping archival strategy columns
=============================================
========================================
MARS-828: Removing archival strategy columns
========================================
Error starting at line : 11 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\94_MARS_828_rollback_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG
DROP CONSTRAINT CHK_ARCHIVAL_STRATEGY
Error report -
ORA-02443: Cannot drop constraint - nonexistent constraint
https://docs.oracle.com/error-help/db/ora-02443/
02443. 00000 - "Cannot drop constraint - nonexistent constraint"
*Cause: alter table drop constraint <constraint_name>
*Action: make sure you supply correct constraint name.
Elapsed: 00:00:00.118
Error starting at line : 15 File @ C:\_git\_local_rep\working_dir_02\MARS_Packages\REL01_ADDITIONS\MARS-828\94_MARS_828_rollback_columns.sql
In command -
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP (
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
)
Error report -
ORA-00904: "MINIMUM_AGE_MONTHS": invalid identifier
https://docs.oracle.com/error-help/db/ora-00904/
00904. 00000 - "%s: invalid identifier"
*Cause: The identifier or column name entered was invalid.
*Action: Ensure the following
Elapsed: 00:00:00.113
no rows selected
Elapsed: 00:00:00.066
========================================
Archival strategy columns removed successfully
========================================
Step 5/6: Tracking rollback version
====================================
========================================
Package Version Tracking
========================================
EnvironmentID set to: dev
[2026-01-29 19:52:34] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: Start TRACK_PACKAGE_VERSION
[2026-01-29 19:52:34] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: End TRACK_PACKAGE_VERSION - Record inserted
Summary:
--------
Packages tracked: 1/1
Tracked Packages:
CT_MRDS.FILE_ARCHIVER v2.0.0
========================================
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.232
Step 6/6: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ ______________________________________________________
CT_MRDS DATA_EXPORTER 2.6.3 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 2.6.3
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 OK: Package CT_MRDS.FILE_ARCHIVER has not changed.
Last Tracked: 2026-01-29 19:52:34
Version: 2.0.0
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
Elapsed: 00:00:00.388
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
Verification: Package Compilation Status
=========================================
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ _________ ________________
FILE_ARCHIVER PACKAGE VALID 29-JAN-26
FILE_ARCHIVER PACKAGE BODY VALID 29-JAN-26
2 rows selected.
Elapsed: 00:00:00.100
============================================================================
MARS-828 Rollback Completed
============================================================================
ROLLBACK_END
______________________
2026-01-29 19:52:36
1 row selected.
Elapsed: 00:00:00.058
Log file: ../log/ROLLBACK_MARS_828_AUTO_G45C5E88148E17E_GGMICHALSKI_20260129_195230.log
============================================================================

197
MARS_Packages/REL01_ADDITIONS/MARS-828/log/ROLLBACK_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260129_195211.log
View File

@@ -1,197 +0,0 @@
============================================================================
MARS-828 Rollback Starting
============================================================================
WARNING: This will restore FILE_ARCHIVER to v2.0.0
CRITICAL IMPACT:
1. All archival strategies revert to THRESHOLD_BASED
2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
3. Validation trigger will be removed
4. Reconfigure archival thresholds after rollback
Timestamp:
ROLLBACK_START
______________________
2026-01-29 19:52:12
Elapsed: 00:00:00.068
============================================================================
Type YES to continue with rollback, or Ctrl+C to abort: YES
old:BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
new:BEGIN
IF 'YES' IS NULL OR TRIM('YES') IS NULL OR UPPER(TRIM('YES')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.096
Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
===============================================================
Package CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.382
Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
======================================================
Package Body CT_MRDS.FILE_ARCHIVER compiled
Elapsed: 00:00:00.288
Step 3/6: Dropping validation trigger
======================================
========================================
MARS-828: Dropping archival strategy validation trigger
========================================
Trigger CT_MRDS.TRG_BI_A_SRC_FILE_CFG_ARCH_VAL dropped.
Elapsed: 00:00:00.193
TRIGGER_COUNT
________________
0
Elapsed: 00:00:00.335
========================================
Validation trigger dropped successfully
========================================
Step 4/6: Dropping archival strategy columns
=============================================
========================================
MARS-828: Removing archival strategy columns
========================================
Table CT_MRDS.A_SOURCE_FILE_CONFIG altered.
Elapsed: 00:00:00.131
Table CT_MRDS.A_SOURCE_FILE_CONFIG altered.
Elapsed: 00:00:00.206
no rows selected
Elapsed: 00:00:00.341
========================================
Archival strategy columns removed successfully
========================================
Step 5/6: Tracking rollback version
====================================
========================================
Package Version Tracking
========================================
EnvironmentID set to: dev
[2026-01-29 19:52:16] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: Start TRACK_PACKAGE_VERSION
[2026-01-29 19:52:17] [INFO] ENV_MANAGER.TRACK_PACKAGE_VERSION: End TRACK_PACKAGE_VERSION - Record inserted
Summary:
--------
Packages tracked: 1/1
Tracked Packages:
CT_MRDS.FILE_ARCHIVER v2.0.0
========================================
PL/SQL procedure successfully completed.
Elapsed: 00:00:01.230
Step 6/6: Verifying tracked packages
=====================================
========================================
Package Version Verification
========================================
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
________________ ___________________ __________ ______________________________________________________
CT_MRDS DATA_EXPORTER 2.6.3 OK: Package CT_MRDS.DATA_EXPORTER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 2.6.3
CT_MRDS ENV_MANAGER 3.2.0 OK: Package CT_MRDS.ENV_MANAGER has not changed.
Last Tracked: 2026-01-28 19:18:36
Version: 3.2.0
CT_MRDS FILE_ARCHIVER 2.0.0 OK: Package CT_MRDS.FILE_ARCHIVER has not changed.
Last Tracked: 2026-01-29 19:52:17
Version: 2.0.0
CT_MRDS FILE_MANAGER 3.4.0 OK: Package CT_MRDS.FILE_MANAGER has not changed.
Last Tracked: 2026-01-26 11:42:32
Version: 3.4.0
CT_MRDS WORKFLOW_MANAGER 1.7.1 OK: Package CT_MRDS.WORKFLOW_MANAGER has not changed.
Last Tracked: 2025-11-25 16:00:36
Version: 1.7.1
ODS FILE_MANAGER_ODS 2.1.0 WARNING: Package ODS.FILE_MANAGER_ODS has changed!
========================================
Last Tracked Version: 2.1.0
Last Tracked Date: 2025-11-26 08:58:57
BODY Changed:
Current Hash: 1d167a53256c10dd...
Last Hash: NULL...
RECOMMENDATION:
1. Update PACKAGE_VERSION constant
2. Update PACKAGE_BUILD_DATE constant
3. Add entry to VERSION_HISTORY
4. Call TRACK_PACKAGE_VERSION to update tracking
Elapsed: 00:00:00.448
========================================
Verification Complete
========================================
Legend:
OK - Package has not changed since last tracking
WARNING - Package code changed without version update
For detailed hash information, use:
SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL
========================================
Verification: Package Compilation Status
=========================================
OBJECT_NAME OBJECT_TYPE STATUS LAST_DDL_TIME
________________ _______________ _________ ________________
FILE_ARCHIVER PACKAGE VALID 29-JAN-26
FILE_ARCHIVER PACKAGE BODY VALID 29-JAN-26
2 rows selected.
Elapsed: 00:00:00.202
============================================================================
MARS-828 Rollback Completed
============================================================================
Completion Time:
ROLLBACK_END
______________________
2026-01-29 19:52:20
1 row selected.
Elapsed: 00:00:00.061
Rollback Summary:
- Package: CT_MRDS.FILE_ARCHIVER
- Restored Version: 2.0.0 (THRESHOLD_BASED archival only)
- Removed Features: CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID strategies
Log file: log/ROLLBACK_MARS_828_G45C5E88148E17E_GGMICHALSKI_20260129_195211.log
============================================================================

23
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/FILE_ARCHIVER.pkb
View File

@@ -23,11 +23,7 @@ AS
WHEN 'THRESHOLD_BASED' THEN
vWhereClause := 'extract(day from (systimestamp - workflow_start)) > ' || pSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD;
-- Archive all data except current month
WHEN 'CURRENT_MONTH_ONLY' THEN
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'')';
-- Archive only data older than X months
-- Archive data older than X months (0 = current month only)
WHEN 'MINIMUM_AGE_MONTHS' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for MINIMUM_AGE_MONTHS strategy');
@@ -132,7 +128,7 @@ AS
if LENGTH(vArchivalTriggeredBy)>0 THEN
ENV_MANAGER.LOG_PROCESS_EVENT('Archival Triggered By: '||vArchivalTriggeredBy,'INFO');
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
-- Use strategy-based WHERE clause (MARS-828)
vQuery := '
@@ -387,14 +383,19 @@ AS
vSourceFileConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vTableName VARCHAR2(200);
vQuery VARCHAR2(32000);
vWhereClause VARCHAR2(4000);
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
vSourceFileConfig := FILE_MANAGER.GET_SOURCE_FILE_CONFIG(pSourceFileConfigKey => pSourceFileConfigKey);
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
ENV_MANAGER.LOG_PROCESS_EVENT('vTableName','DEBUG',vTableName);
-- Get WHERE clause based on archival strategy (MARS-828)
vWhereClause := GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig);
ENV_MANAGER.LOG_PROCESS_EVENT('vWhereClause','DEBUG',vWhereClause);
-- Use strategy-based WHERE clause for statistics (MARS-828)
vQuery :=
'with tmp as (
@@ -419,12 +420,12 @@ AS
,'||pSourceFileConfigKey||' as A_SOURCE_FILE_CONFIG_KEY
,'''||vTableName||''' as TABLE_NAME
,count(*) as FILE_COUNT
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum(case when ' || vWhereClause || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum (row_count_per_file) as ROW_COUNT
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(case when ' || vWhereClause || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(r.bytes) as BYTES
,sum(case when ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig) || ' then r.bytes else 0 end) as OLD_BYTES
,'||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' as DAYS_FOR_ARCHIVE_THRESHOLD
,sum(case when ' || vWhereClause || ' then r.bytes else 0 end) as OLD_BYTES
,'||COALESCE(TO_CHAR(vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD), 'NULL')||' as DAYS_FOR_ARCHIVE_THRESHOLD
,systimestamp as CREATED
from tmp_gr t
join (SELECT * from DBMS_CLOUD.LIST_OBJECTS(

2
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/FILE_ARCHIVER.pkg
View File

@@ -24,7 +24,7 @@ AS
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.1.0 (2026-01-29): Added function overloads for ARCHIVE_TABLE_DATA and GATHER_TABLE_STAT returning SQLCODE for Python library integration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (MINIMUM_AGE_MONTHS with 0=current month, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-18): Enhanced ARCHIVE_TABLE_DATA with Hive-style partitioning support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-15): Initial release with table archival and statistics gathering';

6
MARS_Packages/REL01_ADDITIONS/MARS-828/new_version/TRG_BI_A_SRC_FILE_CFG_ARCH_VAL.sql
View File

@@ -20,10 +20,10 @@ BEGIN
RAISE_APPLICATION_ERROR(-20999, vErrorMsg);
END IF;
-- Validate MINIMUM_AGE_MONTHS is positive
-- Validate MINIMUM_AGE_MONTHS is non-negative (0 = current month only)
IF :NEW.MINIMUM_AGE_MONTHS IS NOT NULL
AND :NEW.MINIMUM_AGE_MONTHS < 1 THEN
RAISE_APPLICATION_ERROR(-20998, 'MINIMUM_AGE_MONTHS must be greater than 0');
AND :NEW.MINIMUM_AGE_MONTHS < 0 THEN
RAISE_APPLICATION_ERROR(-20998, 'MINIMUM_AGE_MONTHS must be greater than or equal to 0');
END IF;
-- Warn if MINIMUM_AGE_MONTHS set but strategy doesn't use it

39
MARS_Packages/REL01_ADDITIONS/MARS-828/rollback_mars828.sql
View File

@@ -29,13 +29,13 @@ PROMPT
PROMPT ============================================================================
PROMPT MARS-828 Rollback Starting
PROMPT ============================================================================
PROMPT WARNING: This will restore FILE_ARCHIVER to v2.0.0
PROMPT This will restore FILE_ARCHIVER to v2.0.0
PROMPT
PROMPT CRITICAL IMPACT:
PROMPT 1. All archival strategies revert to THRESHOLD_BASED
PROMPT 2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
PROMPT 3. Validation trigger will be removed
PROMPT 4. Reconfigure archival thresholds after rollback
PROMPT Rollback steps:
PROMPT 1. Remove validation trigger
PROMPT 2. Drop ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns
PROMPT 3. Restore FILE_ARCHIVER package to v2.0.0
PROMPT 4. Revert all archival strategies to THRESHOLD_BASED
PROMPT
PROMPT Timestamp:
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS rollback_start FROM DUAL;
@@ -54,32 +54,27 @@ WHENEVER SQLERROR CONTINUE
-- Rollback steps (in reverse order)
PROMPT
PROMPT Step 1/6: Restoring FILE_ARCHIVER Package Specification v2.0.0
PROMPT ===============================================================
@@91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 2/6: Restoring FILE_ARCHIVER Package Body v2.0.0
PROMPT ======================================================
@@92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 3/6: Dropping validation trigger
PROMPT Step 1/5: Dropping validation trigger
PROMPT ======================================
@@93_MARS_828_rollback_trigger.sql
PROMPT
PROMPT Step 4/6: Dropping archival strategy columns
PROMPT Step 2/5: Dropping archival strategy columns
PROMPT =============================================
@@94_MARS_828_rollback_columns.sql
PROMPT
PROMPT Step 5/6: Tracking rollback version
PROMPT ====================================
@@track_package_versions.sql
PROMPT Step 3/5: Restoring FILE_ARCHIVER Package Specification v2.0.0
PROMPT ===============================================================
@@91_MARS_828_rollback_FILE_ARCHIVER_SPEC.sql
PROMPT
PROMPT Step 6/6: Verifying tracked packages
PROMPT Step 4/5: Restoring FILE_ARCHIVER Package Body v2.0.0
PROMPT ======================================================
@@92_MARS_828_rollback_FILE_ARCHIVER_BODY.sql
PROMPT
PROMPT Step 5/5: Verifying tracked packages
PROMPT =====================================
@@verify_packages_version.sql

1196
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb
View File

File diff suppressed because it is too large Load Diff

104
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg
View File

@@ -9,22 +9,65 @@ AS
**/
-- Package Version Information
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.1.1';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2025-12-04 13:10:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.6.3';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-01-28 19:30:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team';
-- Version History (last 3-5 changes)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) ||
'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) ||
'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) ||
'v2.6.0 (2026-01-28): CRITICAL FIX - Added STATUS tracking to A_PARALLEL_EXPORT_CHUNKS table to prevent data duplication on retry. System now restarts ONLY failed partitions instead of re-exporting all data. Added ERROR_MESSAGE and EXPORT_TIMESTAMP columns for better error handling and monitoring. Prevents duplicate file creation when parallel tasks fail (e.g., 22 partitions with 16 threads, 3 failures no longer duplicates 19 successful exports).' || CHR(10) ||
'v2.5.0 (2026-01-26): Added recorddelimiter parameter with CRLF (CHR(13)||CHR(10)) for CSV exports to ensure Windows-compatible line endings. Improves cross-platform compatibility when CSV files are opened in Windows applications (Notepad, Excel).' || CHR(10) ||
'v2.4.0 (2026-01-11): Added pTemplateTableName parameter for per-column date format configuration. Implements dynamic query building with TO_CHAR for each date/timestamp column using FILE_MANAGER.GET_DATE_FORMAT. Supports 3-tier hierarchy: column-specific, template DEFAULT, global fallback. Eliminates single dateformat limitation of DBMS_CLOUD.EXPORT_DATA.' || CHR(10) ||
'v2.3.0 (2025-12-20): Added parallel partition processing using DBMS_PARALLEL_EXECUTE. New pParallelDegree parameter (1-16, default 1) for EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE procedures. Each year/month partition processed in separate thread for improved performance.' || CHR(10) ||
'v2.2.0 (2025-12-19): DRY refactoring - extracted shared helper functions (sanitizeFilename, VALIDATE_TABLE_AND_COLUMNS, GET_PARTITIONS, EXPORT_SINGLE_PARTITION worker procedure). Reduced code duplication by ~400 lines. Prepared architecture for v2.3.0 parallel processing.' || CHR(10) ||
'v2.1.1 (2025-12-04): Fixed JOIN column reference A_WORKFLOW_HISTORY_KEY -> A_ETL_LOAD_SET_KEY, added consistent column mapping and dynamic column list to EXPORT_TABLE_DATA procedure, enhanced DEBUG logging for all export operations' || CHR(10) ||
'v2.1.1 (2025-12-04): Fixed JOIN column reference A_WORKFLOW_HISTORY_KEY -> A_ETL_LOAD_SET_KEY' || CHR(10) ||
'v2.1.0 (2025-10-22): Added version tracking and PARTITION_YEAR/PARTITION_MONTH support' || CHR(10) ||
'v2.0.0 (2025-10-01): Separated export functionality from FILE_MANAGER package' || CHR(10) ||
'v1.0.0 (2025-09-15): Initial implementation within FILE_MANAGER package' || CHR(10);
'v2.0.0 (2025-10-01): Separated export functionality from FILE_MANAGER package' || CHR(10);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
-- TYPE DEFINITIONS FOR PARTITION HANDLING
---------------------------------------------------------------------------------------------------------------------------
/**
* Record type for year/month partition information
**/
TYPE partition_rec IS RECORD (
year VARCHAR2(4),
month VARCHAR2(2)
);
/**
* Table type for collection of partition records
**/
TYPE partition_tab IS TABLE OF partition_rec;
---------------------------------------------------------------------------------------------------------------------------
-- INTERNAL PARALLEL PROCESSING CALLBACK
---------------------------------------------------------------------------------------------------------------------------
/**
* @name EXPORT_PARTITION_PARALLEL
* @desc Internal callback procedure for DBMS_PARALLEL_EXECUTE.
* Processes single partition (year/month) chunk in parallel task.
* Called by DBMS_PARALLEL_EXECUTE framework for each chunk.
* This procedure is PUBLIC because DBMS_PARALLEL_EXECUTE requires it,
* but should NOT be called directly by external code.
* @param pStartId - Chunk start ID (CHUNK_ID from A_PARALLEL_EXPORT_CHUNKS table)
* @param pEndId - Chunk end ID (same as pStartId for single-row chunks)
**/
PROCEDURE EXPORT_PARTITION_PARALLEL (
pStartId IN NUMBER,
pEndId IN NUMBER
);
---------------------------------------------------------------------------------------------------------------------------
-- MAIN EXPORT PROCEDURES
---------------------------------------------------------------------------------------------------------------------------
/**
@@ -62,6 +105,7 @@ AS
* Allows specifying custom column list or uses T.* if pColumnList is NULL.
* Validates that all columns in pColumnList exist in the target table.
* Automatically adds 'T.' prefix to column names in pColumnList.
* Supports parallel partition processing via pParallelDegree parameter (default 1, range 1-16).
* pBucketArea parameter accepts: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example
* begin
@@ -73,20 +117,23 @@ AS
* pFolderName => 'parquet_exports',
* pColumnList => 'COLUMN1, COLUMN2, COLUMN3', -- Optional
* pMinDate => DATE '2024-01-01',
* pMaxDate => SYSDATE
* pMaxDate => SYSDATE,
* pParallelDegree => 8 -- Optional, default 1, range 1-16
* );
* end;
**/
PROCEDURE EXPORT_TABLE_DATA_BY_DATE (
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2,
pFolderName IN VARCHAR2,
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2,
pFolderName IN VARCHAR2,
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pParallelDegree IN NUMBER default 1,
pTemplateTableName IN VARCHAR2 default NULL,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
);
@@ -97,6 +144,7 @@ AS
* Creates one CSV file for each year/month combination found in the data.
* Uses the same date filtering mechanism with CT_ODS.A_LOAD_HISTORY as EXPORT_TABLE_DATA_BY_DATE,
* but exports to CSV format instead of Parquet.
* Supports parallel partition processing via pParallelDegree parameter (1-16).
* File naming pattern: {pFileName}_YYYYMM.csv or {TABLENAME}_YYYYMM.csv (if pFileName is NULL)
* @example
* begin
@@ -109,7 +157,8 @@ AS
* pFolderName => 'exports',
* pFileName => 'my_export.csv',
* pMinDate => DATE '2024-01-01',
* pMaxDate => SYSDATE
* pMaxDate => SYSDATE,
* pParallelDegree => 8 -- Optional, default 1, range 1-16
* );
*
* -- With auto-generated filename (based on table name only)
@@ -127,16 +176,19 @@ AS
* end;
**/
PROCEDURE EXPORT_TABLE_DATA_TO_CSV_BY_DATE (
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2,
pFolderName IN VARCHAR2,
pFileName IN VARCHAR2 DEFAULT NULL,
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2,
pFolderName IN VARCHAR2,
pFileName IN VARCHAR2 DEFAULT NULL,
pColumnList IN VARCHAR2 default NULL,
pMinDate IN DATE default DATE '1900-01-01',
pMaxDate IN DATE default SYSDATE,
pParallelDegree IN NUMBER default 1,
pTemplateTableName IN VARCHAR2 default NULL,
pMaxFileSize IN NUMBER default 104857600,
pCredentialName IN VARCHAR2 default ENV_MANAGER.gvCredentialName
);
---------------------------------------------------------------------------------------------------------------------------

6
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/ENV_MANAGER.pkb
View File

@@ -39,6 +39,8 @@ AS
Errors(CODE_MOVE_FILE_TO_TRASH_FAILED) := Error_Record(CODE_MOVE_FILE_TO_TRASH_FAILED, MSG_MOVE_FILE_TO_TRASH_FAILED); -- -20032
Errors(CODE_DROP_EXPORTED_FILES_FAILED) := Error_Record(CODE_DROP_EXPORTED_FILES_FAILED, MSG_DROP_EXPORTED_FILES_FAILED); -- -20033
Errors(CODE_INVALID_BUCKET_AREA) := Error_Record(CODE_INVALID_BUCKET_AREA, MSG_INVALID_BUCKET_AREA); -- -20034
Errors(CODE_INVALID_PARALLEL_DEGREE) := Error_Record(CODE_INVALID_PARALLEL_DEGREE, MSG_INVALID_PARALLEL_DEGREE); -- -20110
Errors(CODE_PARALLEL_EXECUTION_FAILED) := Error_Record(CODE_PARALLEL_EXECUTION_FAILED, MSG_PARALLEL_EXECUTION_FAILED); -- -20111
Errors(CODE_UNKNOWN) := Error_Record(CODE_UNKNOWN, MSG_UNKNOWN); -- -20999
@@ -1163,3 +1165,7 @@ BEGIN
INIT_VARIABLES(pEnv => gvEnv);
END ENV_MANAGER;
/
/

18
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/ENV_MANAGER.pkg
View File

@@ -17,12 +17,13 @@ AS
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 20:57:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.2.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-12-20 10:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.2.0 (2025-12-20): Added error codes for parallel execution support (CODE_INVALID_PARALLEL_DEGREE -20110, CODE_PARALLEL_EXECUTION_FAILED -20111)' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-22): Added package hash tracking and automatic change detection system (SHA256 hashing)' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-22): Added package versioning system with centralized version management functions' || CHR(13)||CHR(10) ||
'2.1.0 (2025-10-15): Added ANALYZE_VALIDATION_ERRORS function for comprehensive CSV validation analysis' || CHR(13)||CHR(10) ||
@@ -296,6 +297,18 @@ AS
PRAGMA EXCEPTION_INIT( ERR_INVALID_BUCKET_AREA
,CODE_INVALID_BUCKET_AREA);
ERR_INVALID_PARALLEL_DEGREE EXCEPTION;
CODE_INVALID_PARALLEL_DEGREE CONSTANT PLS_INTEGER := -20110;
MSG_INVALID_PARALLEL_DEGREE VARCHAR2(4000) := 'Invalid parallel degree parameter. Must be between 1 and 16';
PRAGMA EXCEPTION_INIT( ERR_INVALID_PARALLEL_DEGREE
,CODE_INVALID_PARALLEL_DEGREE);
ERR_PARALLEL_EXECUTION_FAILED EXCEPTION;
CODE_PARALLEL_EXECUTION_FAILED CONSTANT PLS_INTEGER := -20111;
MSG_PARALLEL_EXECUTION_FAILED VARCHAR2(4000) := 'Parallel execution failed';
PRAGMA EXCEPTION_INIT( ERR_PARALLEL_EXECUTION_FAILED
,CODE_PARALLEL_EXECUTION_FAILED);
ERR_UNKNOWN EXCEPTION;
CODE_UNKNOWN CONSTANT PLS_INTEGER := -20999;
MSG_UNKNOWN VARCHAR2(4000) := 'Unknown Error Occured';
@@ -609,3 +622,4 @@ AS
) RETURN VARCHAR2;
END ENV_MANAGER;
/

107
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/FILE_ARCHIVER.pkb
View File

@@ -1,6 +1,54 @@
create or replace PACKAGE BODY CT_MRDS.FILE_ARCHIVER
AS
----------------------------------------------------------------------------------------------------
-- PRIVATE FUNCTION: GET_ARCHIVAL_WHERE_CLAUSE
----------------------------------------------------------------------------------------------------
/**
* @name GET_ARCHIVAL_WHERE_CLAUSE
* @desc Private function that generates WHERE clause based on ARCHIVAL_STRATEGY configuration.
* Supports four strategies: THRESHOLD_BASED, CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID.
* @param pSourceFileConfig - Source file configuration record with ARCHIVAL_STRATEGY
* @return VARCHAR2 - WHERE clause for filtering archival candidates
**/
FUNCTION GET_ARCHIVAL_WHERE_CLAUSE(
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
) RETURN VARCHAR2
IS
vWhereClause VARCHAR2(4000);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
BEGIN
CASE pSourceFileConfig.ARCHIVAL_STRATEGY
-- Legacy threshold-based strategy (backward compatible)
WHEN 'THRESHOLD_BASED' THEN
vWhereClause := 'extract(day from (systimestamp - workflow_start)) > ' || pSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD;
-- Archive all data except current month
WHEN 'CURRENT_MONTH_ONLY' THEN
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'')';
-- Archive only data older than X months
WHEN 'MINIMUM_AGE_MONTHS' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for MINIMUM_AGE_MONTHS strategy');
END IF;
vWhereClause := 'workflow_start < ADD_MONTHS(TRUNC(SYSDATE, ''MM''), -' || pSourceFileConfig.MINIMUM_AGE_MONTHS || ')';
-- Hybrid: Current month exclusion AND minimum age requirement
WHEN 'HYBRID' THEN
IF pSourceFileConfig.MINIMUM_AGE_MONTHS IS NULL THEN
RAISE_APPLICATION_ERROR(-20001, 'MINIMUM_AGE_MONTHS must be configured for HYBRID strategy');
END IF;
vWhereClause := 'TRUNC(workflow_start, ''MM'') < TRUNC(SYSDATE, ''MM'') ' ||
'AND workflow_start < ADD_MONTHS(TRUNC(SYSDATE, ''MM''), -' || pSourceFileConfig.MINIMUM_AGE_MONTHS || ')';
ELSE
RAISE_APPLICATION_ERROR(-20002, 'Invalid ARCHIVAL_STRATEGY: ' || pSourceFileConfig.ARCHIVAL_STRATEGY);
END CASE;
RETURN vWhereClause;
END GET_ARCHIVAL_WHERE_CLAUSE;
----------------------------------------------------------------------------------------------------
FUNCTION GET_TABLE_STAT(pSourceFileConfigKey IN NUMBER)
@@ -85,6 +133,8 @@ AS
if LENGTH(vArchivalTriggeredBy)>0 THEN
ENV_MANAGER.LOG_PROCESS_EVENT('Archival Triggered By: '||vArchivalTriggeredBy,'INFO');
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
-- Use strategy-based WHERE clause (MARS-828)
vQuery := '
select t_filename(
file$name
@@ -96,7 +146,7 @@ AS
from '||vTableName||' s
join CT_MRDS.a_workflow_history h
on s.a_workflow_history_key = h.a_workflow_history_key
where extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD
where ' || GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig)
;
-- Get all files that will be archived into "vfiles" collection ("regular data files")
@@ -337,6 +387,7 @@ AS
vSourceFileConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vTableName VARCHAR2(200);
vQuery VARCHAR2(32000);
vWhereClause VARCHAR2(4000);
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
@@ -344,6 +395,12 @@ AS
vTableName := DBMS_ASSERT.SCHEMA_NAME(vSourceFileConfig.ODS_SCHEMA_NAME) || '.'||vSourceFileConfig.A_SOURCE_KEY||'_'||DBMS_ASSERT.simple_sql_name(vSourceFileConfig.TABLE_ID)||'_ODS';
ENV_MANAGER.LOG_PROCESS_EVENT('vTableName','DEBUG',vTableName);
-- Get WHERE clause based on archival strategy (MARS-828)
vWhereClause := GET_ARCHIVAL_WHERE_CLAUSE(vSourceFileConfig);
ENV_MANAGER.LOG_PROCESS_EVENT('vWhereClause','DEBUG',vWhereClause);
-- Use strategy-based WHERE clause for statistics (MARS-828)
vQuery :=
'with tmp as (
select
@@ -367,11 +424,11 @@ AS
,'||pSourceFileConfigKey||' as A_SOURCE_FILE_CONFIG_KEY
,'''||vTableName||''' as TABLE_NAME
,count(*) as FILE_COUNT
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then 1 else 0 end) as OLD_FILE_COUNT
,sum(case when ' || vWhereClause || ' then 1 else 0 end) as OLD_FILE_COUNT
,sum (row_count_per_file) as ROW_COUNT
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(case when ' || vWhereClause || ' then row_count_per_file else 0 end) as OLD_ROW_COUNT
,sum(r.bytes) as BYTES
,sum(case when extract(day from (systimestamp - workflow_start)) > '||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' then r.bytes else 0 end) as OLD_BYTES
,sum(case when ' || vWhereClause || ' then r.bytes else 0 end) as OLD_BYTES
,'||vSourceFileConfig.DAYS_FOR_ARCHIVE_THRESHOLD||' as DAYS_FOR_ARCHIVE_THRESHOLD
,systimestamp as CREATED
from tmp_gr t
@@ -438,6 +495,48 @@ AS
----------------------------------------------------------------------------------------------------
FUNCTION ARCHIVE_TABLE_DATA (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
----
ARCHIVE_TABLE_DATA(pSourceFileConfigKey => pSourceFileConfigKey);
----
ENV_MANAGER.LOG_PROCESS_EVENT('End','INFO',vParameters);
RETURN SQLCODE;
EXCEPTION
WHEN OTHERS THEN
ENV_MANAGER.LOG_PROCESS_EVENT(ENV_MANAGER.GET_ERROR_STACK(pFormat => 'TABLE', pCode=> SQLCODE), 'ERROR', vParameters);
RETURN SQLCODE;
END ARCHIVE_TABLE_DATA;
----------------------------------------------------------------------------------------------------
FUNCTION GATHER_TABLE_STAT (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST('pSourceFileConfigKey => '||nvl(to_char(pSourceFileConfigKey), 'NULL')));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
----
GATHER_TABLE_STAT(pSourceFileConfigKey => pSourceFileConfigKey);
----
ENV_MANAGER.LOG_PROCESS_EVENT('End','INFO',vParameters);
RETURN SQLCODE;
EXCEPTION
WHEN OTHERS THEN
ENV_MANAGER.LOG_PROCESS_EVENT(ENV_MANAGER.GET_ERROR_STACK(pFormat => 'TABLE', pCode=> SQLCODE), 'ERROR', vParameters);
RETURN SQLCODE;
END GATHER_TABLE_STAT;
----------------------------------------------------------------------------------------------------
END;
/

30
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/FILE_ARCHIVER.pkg
View File

@@ -17,12 +17,14 @@ AS
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.0.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 16:45:00';
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.1.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-01-29 21:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.1.0 (2026-01-29): Added function overloads for ARCHIVE_TABLE_DATA and GATHER_TABLE_STAT returning SQLCODE for Python library integration' || CHR(13)||CHR(10) ||
'3.0.0 (2026-01-27): MARS-828 - Added flexible archival strategies (CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID) via ARCHIVAL_STRATEGY configuration' || CHR(13)||CHR(10) ||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-18): Enhanced ARCHIVE_TABLE_DATA with Hive-style partitioning support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-15): Initial release with table archival and statistics gathering';
@@ -39,6 +41,18 @@ AS
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
);
/**
* @name ARCHIVE_TABLE_DATA
* @desc Function overload for ARCHIVE_TABLE_DATA procedure.
* Returns SQLCODE for Python library integration.
* Calls the main ARCHIVE_TABLE_DATA procedure and captures execution result.
* @example SELECT FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfigKey => 123) FROM DUAL;
* @ex_rslt 0 (success) or error code
**/
FUNCTION ARCHIVE_TABLE_DATA (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER;
/**
@@ -50,6 +64,18 @@ AS
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
);
/**
* @name GATHER_TABLE_STAT
* @desc Function overload for GATHER_TABLE_STAT procedure.
* Returns SQLCODE for Python library integration.
* Calls the main GATHER_TABLE_STAT procedure and captures execution result.
* @example SELECT FILE_ARCHIVER.GATHER_TABLE_STAT(pSourceFileConfigKey => 123) FROM DUAL;
* @ex_rslt 0 (success) or error code
**/
FUNCTION GATHER_TABLE_STAT (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
) RETURN PLS_INTEGER;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------

5
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/tables/A_SOURCE_FILE_CONFIG.sql
View File

@@ -3,6 +3,7 @@
-- ====================================================================
-- Purpose: Store source file configuration and processing rules
-- MARS-1049: Added ENCODING column for CSV character set support
-- MARS-828: Added ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS for archival automation
-- ====================================================================
CREATE TABLE CT_MRDS.A_SOURCE_FILE_CONFIG (
@@ -21,6 +22,8 @@ CREATE TABLE CT_MRDS.A_SOURCE_FILE_CONFIG (
ODS_SCHEMA_NAME VARCHAR2(100),
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD NUMBER(38,0),
HOURS_TO_EXPIRE_STATISTICS NUMBER(38,3),
ARCHIVAL_STRATEGY VARCHAR2(50),
MINIMUM_AGE_MONTHS NUMBER(3,0),
ENCODING VARCHAR2(50) DEFAULT 'UTF8',
CONSTRAINT A_SOURCE_FILE_CONFIG_PK PRIMARY KEY (A_SOURCE_FILE_CONFIG_KEY),
CONSTRAINT SOURCE_FILE_TYPE_CHK CHECK (SOURCE_FILE_TYPE IN ('INPUT', 'CONTAINER', 'LOAD_CONFIG')),
@@ -40,6 +43,8 @@ ON "CT_MRDS"."A_SOURCE_FILE_CONFIG" ("SOURCE_FILE_TYPE", "SOURCE_FILE_ID", "TABL
TABLESPACE "DATA";
-- Column comments
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ARCHIVAL_STRATEGY IS 'Archival strategy: THRESHOLD_BASED, CURRENT_MONTH_ONLY, MINIMUM_AGE_MONTHS, HYBRID. Added in MARS-828';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.MINIMUM_AGE_MONTHS IS 'Minimum age in months before archival (required for MINIMUM_AGE_MONTHS strategy). Added in MARS-828';
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING IS 'Oracle character set name for CSV files (e.g., UTF8, WE8MSWIN1252, EE8ISO8859P2). Added in MARS-1049';
GRANT SELECT, INSERT, UPDATE, DELETE ON CT_MRDS.A_SOURCE_FILE_CONFIG TO MRDS_LOADER_ROLE;

9
README.md
View File

@@ -67,7 +67,14 @@ sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars835_prehook.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M835PH_arch.7z MARS-835-PREHOOK
cd .\MARS_Packages\REL03\MARS-1057
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars1057.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars1057.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M1057_arch.7z MARS-1057
7z a -pMojeSuperHaslo#123 -mhe=on M1057_arch.7z MARS-1057
cd .\MARS_Packages\REL01_ADDITIONS\MARS-828
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars828.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars828.sql"
7z a -pMojeSuperHaslo#123 -mhe=on M826_arch.7z MARS-828\

618
confluence/FILE_ARCHIVER_Guide.md Normal file
View File

@@ -0,0 +1,618 @@
# FILE_ARCHIVER Configuration Guide
This document describes the archival strategies available in the FILE_ARCHIVER package for managing data lifecycle across OCI buckets (INBOX → ODS → ARCHIVE).
## Overview
The FILE_ARCHIVER package provides flexible archival strategies that accommodate different data retention policies across source systems. It manages the movement of processed data from operational storage (ODS bucket) to long-term archival storage (ARCHIVE bucket) based on configurable strategies.
### Key Features
- **Three Archival Strategies**: THRESHOLD_BASED, MINIMUM_AGE_MONTHS (with 0=current month only), HYBRID
- **Flexible Configuration**: Per-table archival strategy configuration via A_SOURCE_FILE_CONFIG
- **Backward Compatible**: Default THRESHOLD_BASED strategy maintains existing behavior
- **Validation**: Automatic validation of strategy-specific configuration requirements
- **OCI Integration**: Works seamlessly with DBMS_CLOUD operations via cloud_wrapper
### Package Information
- **Schema**: CT_MRDS
- **Package**: FILE_ARCHIVER
- **Current Version**: 3.1.0
- **Dependencies**: ENV_MANAGER, FILE_MANAGER, cloud_wrapper, A_SOURCE_FILE_CONFIG, A_SOURCE_FILE_RECEIVED, A_WORKFLOW_HISTORY
### Critical Prerequisites
⚠️ **IMPORTANT**: FILE_ARCHIVER requires data to be registered in `CT_MRDS.A_SOURCE_FILE_RECEIVED` table. This table is automatically populated when files are processed through the modern Airflow + DBT workflow via `FILE_MANAGER.PROCESS_SOURCE_FILE`.
**For legacy data migrated from Informatica + WLA system:**
- Legacy data exported using `DATA_EXPORTER` does NOT automatically create `A_SOURCE_FILE_RECEIVED` records
- Without these records, FILE_ARCHIVER **CANNOT** archive the data
- See [System Migration Guide](System_Migration_Informatica_to_Airflow_DBT.md) for workaround strategies
**Recommendation for legacy data**: Export directly to ARCHIVE bucket using `DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE` with `pBucketArea => 'ARCHIVE'` to bypass this requirement
## Archival Strategies
### Strategy Overview
| Strategy | WHERE Clause Logic | Configuration Required | Primary Use Case |
|----------|-------------------|----------------------|------------------|
| `THRESHOLD_BASED` | Days since workflow start > threshold | DAYS_FOR_ARCHIVE_THRESHOLD | Legacy compatibility, simple time-based archival |
| `MINIMUM_AGE_MONTHS` | Archive data older than X months (0=current month only) | MINIMUM_AGE_MONTHS (≥0) | All sources - flexible retention (0 for LM, 6 for CSDB) |
| `HYBRID` | Combines month boundary + minimum age | MINIMUM_AGE_MONTHS | Advanced retention scenarios |
### 1. THRESHOLD_BASED (Default)
Archives data based on number of days since workflow start.
**WHERE Clause**:
```sql
extract(day from (systimestamp - workflow_start)) > DAYS_FOR_ARCHIVE_THRESHOLD
```
**Configuration**:
```sql
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'THRESHOLD_BASED',
DAYS_FOR_ARCHIVE_THRESHOLD = 30,
MINIMUM_AGE_MONTHS = NULL
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'C2D_DATA'
AND TABLE_ID = 'C2D_TABLE';
```
**Use Case**: Simple time-based archival, backward compatible with FILE_ARCHIVER v2.0.0 behavior.
### 2. MINIMUM_AGE_MONTHS
Archives data older than specified number of months. **Special case**: MINIMUM_AGE_MONTHS = 0 archives all data before current month (replaces deprecated CURRENT_MONTH_ONLY strategy).
**WHERE Clause**:
```sql
workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -MINIMUM_AGE_MONTHS)
-- When MINIMUM_AGE_MONTHS = 0: workflow_start < TRUNC(SYSDATE, 'MM')
```
**Configuration Examples**:
```sql
-- LM: Keep only current month data (MINIMUM_AGE_MONTHS = 0)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
AND TABLE_ID = 'LM_STANDING_FACILITIES';
-- CSDB: Retain 6 months of data (MINIMUM_AGE_MONTHS = 6)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'CSDB'
AND TABLE_ID IN ('CSDB_DEBT', 'CSDB_DEBT_DAILY');
```
**Use Cases**:
- **MINIMUM_AGE_MONTHS = 0**: LM dissemination feeds requiring current month only (daily/intraday updates)
- **MINIMUM_AGE_MONTHS = 6**: CSDB securities/ratings data requiring 6-month retention
- **MINIMUM_AGE_MONTHS = N**: Regulatory compliance with specific N-month retention periods
**Behavior Examples**:
- **With MINIMUM_AGE_MONTHS = 0**:
- January data: Archived on February 1st
- February data: Remains in ODS bucket during February
- March 1st: February data archived, March data active
- **With MINIMUM_AGE_MONTHS = 6**:
- February 2026: Archives data from July 2025 and earlier
- March 2026: Archives data from August 2025 and earlier
- Keeps current month + 6 previous months (7 months total) in ODS bucket
### 3. HYBRID
Combines month boundary check with minimum age threshold - archives data from previous months AND older than minimum age.
**WHERE Clause**:
```sql
TRUNC(workflow_start, 'MM') < TRUNC(SYSDATE, 'MM')
AND workflow_start < ADD_MONTHS(TRUNC(SYSDATE, 'MM'), -MINIMUM_AGE_MONTHS)
```
**Configuration**:
```sql
-- Advanced: Current month + 3 months minimum
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'HYBRID',
MINIMUM_AGE_MONTHS = 3
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'SPECIAL_SOURCE'
AND TABLE_ID = 'SPECIAL_TABLE';
```
**Use Case**: Advanced scenarios requiring both current month retention AND minimum age threshold.
## Configuration Validation
### Validation Trigger
**Trigger**: `TRG_BI_A_SRC_FILE_CFG_ARCH_VAL`
Automatically validates archival configuration on INSERT/UPDATE to A_SOURCE_FILE_CONFIG:
**Validation Rules**:
1. **MINIMUM_AGE_MONTHS**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL AND MINIMUM_AGE_MONTHS >= 0`
- Error: "Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set (≥0)"
2. **HYBRID**: Requires `MINIMUM_AGE_MONTHS IS NOT NULL`
- Error: "Strategy HYBRID requires MINIMUM_AGE_MONTHS to be set"
**Example Validation Error**:
```sql
-- This will fail validation
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = NULL -- ERROR: Required for this strategy
WHERE ...;
-- Error: ORA-20001: Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set
```
## Data Lifecycle Workflow
### Standard File Processing Flow
```
┌─────────────────────────────────────────────────────────────┐
│ FILE PROCESSING LIFECYCLE │
└─────────────────────────────────────────────────────────────┘
1. INBOX Bucket (Validation)
├─ File arrives from source system
├─ FILE_MANAGER.PROCESS_SOURCE_FILE validates structure
├─ Status: RECEIVED → VALIDATED → READY_FOR_INGESTION
└─ FILE_MANAGER.MOVE_FILE relocates to ODS bucket
2. ODS Bucket (Operational Data)
├─ Active data processing (Airflow + DBT)
├─ External tables read data from bucket
├─ Status: INGESTED
└─ FILE_ARCHIVER.ARCHIVE_TABLE_DATA archives based on strategy
3. ARCHIVE Bucket (Long-term Storage)
├─ Historical data in Parquet format
├─ Hive-style partitioning: PARTITION_YEAR=/PARTITION_MONTH=
├─ Status: ARCHIVED
└─ Optimized for big data analytics (Spark, Hive)
```
### Archival Process
The FILE_ARCHIVER package automatically manages data movement from ODS to ARCHIVE:
**Key Procedures**:
- `ARCHIVE_TABLE_DATA` - Main archival procedure using strategy-specific WHERE clause
- `GET_ARCHIVAL_WHERE_CLAUSE` - Returns WHERE clause based on configured strategy
- `GATHER_TABLE_STAT` - Calculates archival statistics using strategy logic
**Archival Execution**:
```sql
-- Triggered by FILE_MANAGER or scheduled job
BEGIN
CT_MRDS.FILE_ARCHIVER.ARCHIVE_TABLE_DATA(
pSourceFileConfig => vSourceFileConfigRecord
);
END;
/
```
**Strategy-Based Filtering**:
- Package retrieves ARCHIVAL_STRATEGY from A_SOURCE_FILE_CONFIG
- GET_ARCHIVAL_WHERE_CLAUSE generates appropriate WHERE clause
- Data matching criteria moved from ODS to ARCHIVE bucket
- Parquet format with Hive-style partitioning applied
## Configuration Examples
### Example 1: Configure LM Standing Facilities (Current Month Only)
```sql
-- Keep only current month data in ODS bucket (MINIMUM_AGE_MONTHS = 0)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = archives all data before current month
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'DistributeStandingFacilities'
AND TABLE_ID = 'LM_STANDING_FACILITIES';
COMMIT;
-- Verify configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'DistributeStandingFacilities';
```
### Example 2: Configure CSDB Debt (MINIMUM_AGE_MONTHS)
```sql
-- Retain 6 months of data in ODS bucket
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID = 'CSDB'
AND TABLE_ID = 'CSDB_DEBT';
COMMIT;
-- Verify configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'CSDB_DEBT';
```
### Example 3: Bulk Configuration for LM Source
```sql
-- Configure all 19 LM tables with MINIMUM_AGE_MONTHS = 0 (current month only)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = keep only current month
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND SOURCE_FILE_ID IN (
'DistributeStandingFacilities',
'DistributeTTS',
'DistributeAdHocAdjustments',
'DistributeBalanceSheet',
'DistributeCSMAdjustments',
'DistributeCurrentAccounts',
'DistributeForecast',
'DistributeQREAdjustments'
);
COMMIT;
-- Verify bulk configuration
SELECT
SOURCE_FILE_ID,
COUNT(*) AS TABLE_COUNT,
MAX(ARCHIVAL_STRATEGY) AS STRATEGY,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID LIKE 'Distribute%'
GROUP BY SOURCE_FILE_ID
ORDER BY SOURCE_FILE_ID;
```
### Example 4: View Current Archival Configuration
```sql
-- All configured tables with their archival strategies
SELECT
A_SOURCE_KEY,
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
DAYS_FOR_ARCHIVE_THRESHOLD
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
ORDER BY A_SOURCE_KEY, SOURCE_FILE_ID, TABLE_ID;
-- Summary by strategy
SELECT
ARCHIVAL_STRATEGY,
COUNT(*) AS TABLE_COUNT,
MIN(MINIMUM_AGE_MONTHS) AS MIN_AGE_MIN,
MAX(MINIMUM_AGE_MONTHS) AS MIN_AGE_MAX
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
GROUP BY ARCHIVAL_STRATEGY
ORDER BY ARCHIVAL_STRATEGY;
```
## Release 01 Configuration
### Configured Tables (MARS-828)
The following 25 Release 01 tables were configured with archival strategies:
**LM Tables (19 total) - MINIMUM_AGE_MONTHS = 0 (current month only)**:
- LM_STANDING_FACILITIES
- LM_STANDING_FACILITIES_HEADER
- LM_TTS_HEADER
- LM_TTS_ITEM
- LM_ADHOC_ADJUSTMENTS_HEADER
- LM_ADHOC_ADJUSTMENTS_ITEM
- LM_ADHOC_ADJUSTMENTS_ITEM_HEADER
- LM_BALANCESHEET_HEADER
- LM_BALANCESHEET_ITEM
- LM_CSM_ADJUSTMENTS_HEADER
- LM_CSM_ADJUSTMENTS_ITEM
- LM_CSM_ADJUSTMENTS_ITEM_HEADER
- LM_CURRENT_ACCOUNTS_HEADER
- LM_CURRENT_ACCOUNTS_ITEM
- LM_FORECAST_HEADER
- LM_FORECAST_ITEM
- LM_QRE_ADJUSTMENTS_HEADER
- LM_QRE_ADJUSTMENTS_ITEM
- LM_QRE_ADJUSTMENTS_ITEM_HEADER
**CSDB Tables (6 total)**:
*MINIMUM_AGE_MONTHS = 6 (6-month retention)*:
- CSDB_DEBT
- CSDB_DEBT_DAILY
*MINIMUM_AGE_MONTHS = 0 (current month only)*:
- CSDB_INSTR_RAT_FULL
- CSDB_INSTR_DESC_FULL
- CSDB_ISSUER_RAT_FULL
- CSDB_ISSUER_DESC_FULL
**Verification Query**:
```sql
-- Check Release 01 configuration
SELECT
CASE
WHEN TABLE_ID LIKE 'LM_%' THEN 'LM'
WHEN TABLE_ID LIKE 'CSDB_%' THEN 'CSDB'
END AS SOURCE_GROUP,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
COUNT(*) AS TABLE_COUNT
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_TYPE = 'INPUT'
AND TABLE_ID IN (
-- 25 Release 01 tables
'LM_STANDING_FACILITIES', 'LM_STANDING_FACILITIES_HEADER',
'LM_TTS_HEADER', 'LM_TTS_ITEM',
-- ... other tables
)
GROUP BY
CASE
WHEN TABLE_ID LIKE 'LM_%' THEN 'LM'
WHEN TABLE_ID LIKE 'CSDB_%' THEN 'CSDB'
END,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS
ORDER BY SOURCE_GROUP, ARCHIVAL_STRATEGY;
```
## Troubleshooting
### Common Issues
#### Issue 1: Validation Error on Configuration Update
**Error**:
```
ORA-20001: Strategy MINIMUM_AGE_MONTHS requires MINIMUM_AGE_MONTHS to be set
```
**Cause**: Trigger validation failed - strategy requires MINIMUM_AGE_MONTHS but value is NULL
**Solution**:
```sql
-- Provide required MINIMUM_AGE_MONTHS value
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 6 -- Required for this strategy
WHERE ...;
```
#### Issue 2: Archival Not Working as Expected
**Symptoms**: Data not being archived according to strategy
**Diagnostic Steps**:
```sql
-- 1. Check configuration
SELECT
SOURCE_FILE_ID,
TABLE_ID,
ARCHIVAL_STRATEGY,
MINIMUM_AGE_MONTHS,
DAYS_FOR_ARCHIVE_THRESHOLD
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'YOUR_TABLE';
-- 2. Check package version
SELECT CT_MRDS.FILE_ARCHIVER.GET_VERSION() FROM DUAL;
-- Expected: 3.0.0 or higher
-- 3. Check process logs
SELECT
PROCESS_LOG_KEY,
PROCESS_NAME,
LOG_MESSAGE,
LOG_LEVEL,
LOG_TIMESTAMP
FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCESS_NAME LIKE '%ARCHIVE%'
ORDER BY LOG_TIMESTAMP DESC
FETCH FIRST 20 ROWS ONLY;
-- 4. Test WHERE clause generation
DECLARE
vConfig CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
vWhereClause VARCHAR2(4000);
BEGIN
SELECT * INTO vConfig
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TABLE_ID = 'YOUR_TABLE'
AND ROWNUM = 1;
vWhereClause := CT_MRDS.FILE_ARCHIVER.GET_ARCHIVAL_WHERE_CLAUSE(vConfig);
DBMS_OUTPUT.PUT_LINE('WHERE Clause: ' || vWhereClause);
END;
/
```
#### Issue 3: Package Compilation Errors After Upgrade
**Symptoms**: FILE_ARCHIVER package shows INVALID status
**Solution**:
```sql
-- Check compilation errors
SELECT * FROM USER_ERRORS
WHERE NAME = 'FILE_ARCHIVER'
AND TYPE IN ('PACKAGE', 'PACKAGE BODY')
ORDER BY SEQUENCE;
-- Recompile package
ALTER PACKAGE CT_MRDS.FILE_ARCHIVER COMPILE SPECIFICATION;
ALTER PACKAGE CT_MRDS.FILE_ARCHIVER COMPILE BODY;
-- Verify status
SELECT object_name, object_type, status
FROM user_objects
WHERE object_name = 'FILE_ARCHIVER';
```
## Version History
### v3.1.0 (Current - 2026-02-05)
- **BREAKING CHANGE**: Removed CURRENT_MONTH_ONLY strategy (replaced by MINIMUM_AGE_MONTHS = 0)
- Mathematical equivalence: CURRENT_MONTH_ONLY ≡ MINIMUM_AGE_MONTHS = 0
- Updated trigger validation to allow MINIMUM_AGE_MONTHS >= 0 (previously >= 1)
- Simplified architecture from 4 strategies to 3
- Enhanced error handling
- All 25 Release 01 tables migrated to MINIMUM_AGE_MONTHS (23 with value 0, 2 with value 6)
### v3.0.0 (MARS-828 - 2026-02-04)
- Added ARCHIVAL_STRATEGY configuration column
- Implemented four archival strategies (later reduced to three in v3.1.0):
- THRESHOLD_BASED (backward compatible)
- CURRENT_MONTH_ONLY (deprecated in v3.1.0, use MINIMUM_AGE_MONTHS = 0)
- MINIMUM_AGE_MONTHS
- HYBRID
- Added GET_ARCHIVAL_WHERE_CLAUSE function
- Created validation trigger TRG_BI_A_SRC_FILE_CFG_ARCH_VAL
- Configured 25 Release 01 tables with appropriate strategies
### v2.0.0 (Legacy)
- Initial FILE_ARCHIVER package
- THRESHOLD_BASED archival only
- Fixed DAYS_FOR_ARCHIVE_THRESHOLD configuration
## Related Documentation
- [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md) - File processing and validation
- [Package Deployment Guide](Package_Deployment_Guide.md) - Package deployment standards
- [Universal Package Tracking System](Universal_Package_Tracking_System.md) - Version tracking
- [MARS-828 README](../MARS_Packages/REL01_ADDITIONS/MARS-828/README.md) - Detailed implementation notes
## Dependencies
### Required Packages
- **CT_MRDS.ENV_MANAGER** v3.x - Error handling, logging, version tracking
- **CT_MRDS.FILE_MANAGER** v3.x - Bucket URI resolution, file processing
- **MRDS_LOADER.cloud_wrapper** - DBMS_CLOUD operations wrapper
### Database Objects
- **Table**: CT_MRDS.A_SOURCE_FILE_CONFIG - Configuration storage
- **Table**: CT_MRDS.A_SOURCE_FILE_RECEIVED - File processing tracking
- **Table**: CT_MRDS.A_WORKFLOW_HISTORY - Workflow execution tracking (Airflow + DBT)
- **Trigger**: TRG_BI_A_SRC_FILE_CFG_ARCH_VAL - Configuration validation
- **Credential**: DEF_CRED_ARN - OCI bucket access
### OCI Buckets
- **INBOX**: Incoming file validation (`'INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/'`)
- **ODS/DATA**: Operational data processing (`'ODS/{SOURCE}/{TABLE_NAME}/'`)
- **ARCHIVE**: Historical data storage (`'ARCHIVE/{SOURCE}/{TABLE_NAME}/PARTITION_YEAR=/PARTITION_MONTH=/'`)
## Best Practices
### Strategy Selection Guidelines
1. **Use MINIMUM_AGE_MONTHS when**:
- **MINIMUM_AGE_MONTHS = 0**: Current month only retention
- Data updated frequently (daily/intraday)
- Historical data access is rare
- ODS bucket space is limited
- Example: LM dissemination feeds
- **MINIMUM_AGE_MONTHS = N (N > 0)**: Multi-month retention
- Regulatory compliance requires specific retention period
- Analytical workloads need N-month access
- Data updates are infrequent
- Example: CSDB securities data (MINIMUM_AGE_MONTHS = 6)
2. **Use THRESHOLD_BASED when**:
- Maintaining backward compatibility with legacy behavior
- Simple time-based archival is sufficient
- Migration from FILE_ARCHIVER v2.0.0
3. **Use HYBRID when**:
- Complex retention requirements
- Combining month boundary check with minimum age threshold
- Advanced scenarios not covered by other strategies
### Configuration Best Practices
1. **Test Configuration Changes**:
```sql
-- Test on single table first
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ARCHIVAL_STRATEGY = 'MINIMUM_AGE_MONTHS',
MINIMUM_AGE_MONTHS = 0 -- 0 = current month only
WHERE SOURCE_FILE_ID = 'TEST_FILE'
AND TABLE_ID = 'TEST_TABLE';
-- Monitor archival behavior
-- Expand to other tables after validation
```
2. **Verify Before Bulk Updates**:
```sql
-- Preview changes with SELECT
SELECT
SOURCE_FILE_ID,
TABLE_ID,
'MINIMUM_AGE_MONTHS' AS NEW_STRATEGY,
0 AS NEW_MIN_AGE, -- 0 = current month only
ARCHIVAL_STRATEGY AS OLD_STRATEGY,
MINIMUM_AGE_MONTHS AS OLD_MIN_AGE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID LIKE 'Distribute%';
-- Then execute UPDATE
```
3. **Document Configuration Decisions**:
- Record why specific strategy was chosen
- Note business requirements driving retention policy
- Track configuration changes in version control
4. **Monitor Archival Performance**:
```sql
-- Check archival execution logs
SELECT
PROCESS_NAME,
LOG_MESSAGE,
LOG_TIMESTAMP
FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCESS_NAME LIKE '%ARCHIVE%'
AND LOG_TIMESTAMP > SYSDATE - 7
ORDER BY LOG_TIMESTAMP DESC;
```
5. **Regular Configuration Reviews**:
- Verify strategies still match business requirements
- Check for tables without archival configuration
- Optimize MINIMUM_AGE_MONTHS based on actual usage patterns
## Author
Created by: Grzegorz Michalski
Date: 2026-02-04
Schema: CT_MRDS
Package: FILE_ARCHIVER
Version: 3.1.0

5
confluence/PROCESS_SOURCE_FILE_Guide.md
View File

@@ -15,6 +15,11 @@ This document provides comprehensive documentation for the `FILE_MANAGER.PROCESS
- **Error Resilient**: Comprehensive error handling and logging for validation and file operations
- **Status Tracking**: Updates file processing status throughout validation and preparation workflow
**Migration Context:**
- This procedure is part of the **modern Airflow + DBT system** architecture
- Creates records in `CT_MRDS.A_SOURCE_FILE_RECEIVED` (modern control table)
- For legacy Informatica + WLA data migration, see [System Migration Guide](System_Migration_Informatica_to_Airflow_DBT.md)
## Procedure Signatures
The procedure is available in two variants:

149
confluence/Package_Deployment_Guide.md
View File

@@ -10,6 +10,122 @@ This guide provides step-by-step instructions for developers on how to properly
---
## MANDATORY STANDARDS (NEVER SKIP!)
### 🔴 Critical: Author Field
**ALL scripts MUST have:**
```sql
-- Author: Grzegorz Michalski
```
**❌ WRONG:**
```sql
-- Author: System
-- Author: Developer
-- Author: [your name here]
```
**✅ CORRECT:**
```sql
-- Author: Grzegorz Michalski
```
### 📅 Date Format Standard
**ISO 8601 format only:**
```sql
-- Date: 2026-02-03 (YYYY-MM-DD)
```
### 📝 Script Header Template
**Copy-paste this template for ALL scripts:**
```sql
-- =====================================================================
-- Script: XX_MARS_XXX_description.sql
-- MARS Issue: MARS-XXX
-- Purpose: Brief description of what this script does
-- Author: Grzegorz Michalski
-- Date: YYYY-MM-DD
-- =====================================================================
```
### 🔐 Database User Requirements
**Execute installations as ADMIN:**
```powershell
# ✅ CORRECT
Get-Content "install_marsXXX.sql" | sql "ADMIN/password@service"
# ❌ WRONG
Get-Content "install_marsXXX.sql" | sql "CT_MRDS/password@service"
```
### 🔍 Data Dictionary Views
**Use ALL_* views when installing as ADMIN:**
```sql
-- ✅ CORRECT (ADMIN user)
SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PACKAGE_NAME';
-- ❌ WRONG (shows ADMIN schema, not CT_MRDS)
SELECT * FROM USER_ERRORS WHERE NAME = 'PACKAGE_NAME';
```
### 📂 Log Directory Management
**MANDATORY in all master scripts:**
```sql
-- Create log/ directory before SPOOL (prevents failures)
host mkdir log 2>nul
-- Then configure dynamic SPOOL filename
var filename VARCHAR2(100)
BEGIN
:filename := 'log/INSTALL_MARS_XXX_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
```
### ✋ User Confirmation
**MANDATORY ACCEPT validation in master scripts:**
```sql
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
```
### 🚪 Clean Exit
**End all master scripts with:**
```sql
spool off
quit; -- ← MANDATORY for clean SQLcl/SQL*Plus exit
```
### 📋 Quick Compliance Checklist
Before committing any script, verify:
- [ ] `Author: Grzegorz Michalski` in header
- [ ] Date in ISO 8601 format (YYYY-MM-DD)
- [ ] Master scripts use `host mkdir log 2>nul`
- [ ] Master scripts require ACCEPT validation
- [ ] Master scripts end with `quit;`
- [ ] Verification queries use ALL_* views with OWNER filter
- [ ] Installation instructions specify ADMIN user
**💡 Remember:** These standards prevent common mistakes and ensure consistency across all MARS packages!
---
## Table of Contents
1. [Before You Start](#before-you-start)
@@ -1225,7 +1341,12 @@ SET PAUSE OFF
PROMPT =========================================================================
PROMPT MARS-XXXX: Rollback Package
PROMPT =========================================================================
PROMPT WARNING: This will reverse all changes from MARS-XXXX installation!
PROMPT This will reverse all changes from MARS-XXXX installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Restore previous package version
PROMPT 2. Remove added columns/tables
PROMPT 3. Revert configuration changes
PROMPT =========================================================================
-- Confirm rollback with user
@@ -1259,6 +1380,32 @@ spool off
- Revert configuration changes
- Include verification step after rollback (@@verify_packages_version.sql)
**Rollback Communication Style:**
- ✅ **DO**: Use calm, professional tone describing what will be done
- ✅ **DO**: List specific rollback steps clearly
- ❌ **DON'T**: Use dramatic words like "WARNING", "CRITICAL IMPACT", "DANGER"
- ❌ **DON'T**: Create panic - rollback is a normal part of deployment process
- **Rationale**: Rollbacks happen when installations fail or need to be reversed. This is a standard procedure, not an emergency.
**Example - Professional vs. Dramatic:**
```sql
-- ✅ CORRECT - Professional and Clear
PROMPT This will reverse all changes from MARS-828 installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Remove validation trigger
PROMPT 2. Drop ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns
PROMPT 3. Restore FILE_ARCHIVER package to v2.0.0
-- ❌ WRONG - Unnecessarily Dramatic
PROMPT WARNING: This will restore FILE_ARCHIVER to v2.0.0
PROMPT
PROMPT CRITICAL IMPACT:
PROMPT 1. All archival strategies revert to THRESHOLD_BASED
PROMPT 2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
```
---
### Version Tracking Integration

446
confluence/System_Migration_Informatica_to_Airflow_DBT.md Normal file
View File

@@ -0,0 +1,446 @@
# System Migration: Informatica + WLA → Airflow + DBT
This document describes the migration from the legacy Informatica + WLA data processing system to the modern Airflow + DBT architecture, including control table differences, data export strategies, and known limitations.
## Migration Overview
The MRDS (Market Reference Data System) is undergoing a fundamental technology migration:
**Legacy System (Informatica + WLA):**
- ETL Tool: Informatica PowerCenter
- Workflow Orchestration: WLA (Workflow Automation)
- Control Schema: `CT_ODS` (Operational Data Store Control)
- Primary Control Table: `CT_ODS.A_LOAD_HISTORY`
- Key Column: `A_ETL_LOAD_SET_KEY`
**Modern System (Airflow + DBT):**
- Orchestration: Apache Airflow
- Transformation: DBT (Data Build Tool)
- Control Schema: `CT_MRDS` (MRDS Control)
- Primary Control Tables: `CT_MRDS.A_SOURCE_FILE_RECEIVED`, `CT_MRDS.A_WORKFLOW_HISTORY`
- Key Column: `A_WORKFLOW_HISTORY_KEY`
## Control Table Architecture
### Legacy System: CT_ODS.A_LOAD_HISTORY
**Purpose**: Tracks Informatica PowerCenter workflow executions
**Structure**:
```sql
DESC CT_ODS.A_LOAD_HISTORY;
Name Type
______________________ _____________________
A_ETL_LOAD_SET_KEY NUMBER(38) -- Primary key
WORKFLOW_NAME VARCHAR2(255) -- Informatica workflow name
INFA_RUN_ID NUMBER(38) -- Informatica run ID
LOAD_START TIMESTAMP(6) -- Workflow start time
LOAD_END TIMESTAMP(6) -- Workflow end time
EXDI_APPL_REQ_ID VARCHAR2(255)
EXDI_CORRELATION_ID VARCHAR2(255)
LOAD_SUCCESSFUL CHAR(1) -- Y/N success flag
WLA_RUN_ID NUMBER(28) -- WLA run ID
DQ_FLAG VARCHAR2(5) -- Data quality flag
```
**Usage Pattern**:
- Created by Informatica workflows during ETL execution
- Used for temporal partitioning in DATA_EXPORTER
- Referenced via `A_ETL_LOAD_SET_KEY_FK` foreign key in data tables
### Modern System: CT_MRDS Control Tables
#### 1. A_SOURCE_FILE_RECEIVED
**Purpose**: Tracks individual file processing through the complete lifecycle
**Key Columns**:
```sql
A_SOURCE_FILE_RECEIVED_KEY NUMBER -- Primary key
SOURCE_FILE_NAME VARCHAR2 -- Full OCI path
PROCESSING_STATUS VARCHAR2 -- Status tracking
RECEPTION_DATE DATE -- File arrival timestamp
PARTITION_YEAR VARCHAR2(4) -- Archive partition (year)
PARTITION_MONTH VARCHAR2(2) -- Archive partition (month)
ARCH_FILE_NAME VARCHAR2 -- Parquet archive file path
```
**Status Workflow**:
```
RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED
```
**Usage Pattern**:
- Created by FILE_MANAGER.PROCESS_SOURCE_FILE during file validation
- Updated throughout file lifecycle (validation, ingestion, archival)
- Required by FILE_ARCHIVER for archival operations
- Links to A_WORKFLOW_HISTORY via workflow execution
#### 2. A_WORKFLOW_HISTORY
**Purpose**: Tracks Airflow + DBT workflow executions (similar role to A_LOAD_HISTORY)
**Key Columns**:
```sql
A_WORKFLOW_HISTORY_KEY NUMBER -- Primary key
WORKFLOW_NAME VARCHAR2 -- Airflow DAG name
WORKFLOW_START TIMESTAMP -- Workflow start time
WORKFLOW_END TIMESTAMP -- Workflow end time
WORKFLOW_SUCCESSFUL CHAR(1) -- Y/N success flag
```
**Usage Pattern**:
- Created by Airflow + DBT workflows during data processing
- Used for temporal partitioning decisions in FILE_ARCHIVER
- Referenced via `A_WORKFLOW_HISTORY_KEY_FK` foreign key in data tables
## Data Export Strategies
### Scenario 1: Legacy Data Export (Informatica → ODS)
**Use Case**: One-time migration of historical data loaded by Informatica
**Package**: `DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE`
**Control Table**: Uses `CT_ODS.A_LOAD_HISTORY` for temporal partitioning
**Example**:
```sql
-- Export historical AGGREGATED_ALLOTMENT data (loaded by Informatica)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK', -- Links to A_LOAD_HISTORY
pBucketArea => 'DATA',
pFolderName => 'legacy_migration',
pMinDate => DATE '2020-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
```
**Result**: CSV files in ODS bucket (DATA area), partitioned by LOAD_START from A_LOAD_HISTORY
### Scenario 2: Modern System Data (Airflow + DBT → ODS → ARCHIVE)
**Use Case**: Ongoing processing with new Airflow + DBT system
**Workflow**:
1. **File Arrival**: FILES → INBOX bucket
2. **Validation**: FILE_MANAGER.PROCESS_SOURCE_FILE → creates A_SOURCE_FILE_RECEIVED
3. **Processing**: Airflow + DBT → creates A_WORKFLOW_HISTORY
4. **Export**: DATA_EXPORTER → moves to ODS bucket (DATA area)
5. **Archival**: FILE_ARCHIVER → moves to ARCHIVE bucket (Parquet with Hive partitioning)
**Control Tables**: Uses both A_SOURCE_FILE_RECEIVED and A_WORKFLOW_HISTORY
**Example**:
```sql
-- Archival of data processed by Airflow + DBT
BEGIN
CT_MRDS.FILE_ARCHIVER.ARCHIVE_TABLE_DATA(
pSourceFileConfig => vConfig -- Requires A_SOURCE_FILE_RECEIVED records
);
END;
/
```
## Critical Gap: Legacy Data Archival
### Problem Statement
**Scenario**: Historical data exported using DATA_EXPORTER from Informatica-loaded tables
**Issue**: FILE_ARCHIVER requires records in `A_SOURCE_FILE_RECEIVED`, but legacy exports don't create them
**Impact**: Legacy data exported to ODS/DATA bucket **CANNOT** be archived to ARCHIVE bucket using FILE_ARCHIVER
### Technical Analysis
**DATA_EXPORTER Behavior**:
```sql
-- Uses A_LOAD_HISTORY for partitioning (Informatica workflows)
SELECT DISTINCT TO_CHAR(L.LOAD_START,'YYYY') AS YR,
TO_CHAR(L.LOAD_START,'MM') AS MN
FROM OU_TOP.AGGREGATED_ALLOTMENT T, CT_ODS.A_LOAD_HISTORY L
WHERE T.A_ETL_LOAD_SET_KEY_FK = L.A_ETL_LOAD_SET_KEY
AND L.LOAD_START >= :pMinDate
AND L.LOAD_START < :pMaxDate;
-- Creates CSV files: ODS/legacy_migration/AGGREGATED_ALLOTMENT_YYYYMM.csv
-- Does NOT create A_SOURCE_FILE_RECEIVED records
```
**FILE_ARCHIVER Requirement**:
```sql
-- Joins A_SOURCE_FILE_RECEIVED with A_WORKFLOW_HISTORY
JOIN CT_MRDS.A_SOURCE_FILE_RECEIVED r
ON r.A_SOURCE_FILE_CONFIG_KEY = pSourceFileConfig.A_SOURCE_FILE_CONFIG_KEY
AND r.PROCESSING_STATUS = 'INGESTED';
-- Without A_SOURCE_FILE_RECEIVED records, archival CANNOT proceed
```
### Workaround Strategies
#### Strategy 1: Manual Registration (Recommended for Small Datasets)
Manually create `A_SOURCE_FILE_RECEIVED` records for legacy exported files:
```sql
-- Step 1: Export legacy data to ODS/DATA
BEGIN
DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
pBucketArea => 'DATA',
pFolderName => 'legacy_export',
pMinDate => DATE '2024-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
-- Step 2: List exported CSV files
SELECT object_name, time_created, bytes
FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
credential_name => 'DEF_CRED_ARN',
location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/o/'
)) WHERE object_name LIKE 'ODS/legacy_export/AGGREGATED_ALLOTMENT_%';
-- Step 3: Manually register each file in A_SOURCE_FILE_RECEIVED
-- (Requires source configuration for AGGREGATED_ALLOTMENT to exist)
INSERT INTO CT_MRDS.A_SOURCE_FILE_RECEIVED (
A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
PROCESSING_STATUS,
RECEPTION_DATE,
BYTES,
CHECKSUM,
EXTERNAL_TABLE_NAME
) VALUES (
A_SOURCE_FILE_RECEIVED_KEY_SEQ.NEXTVAL,
(SELECT A_SOURCE_FILE_CONFIG_KEY FROM A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'AGGREGATED_ALLOTMENT' AND SOURCE_FILE_TYPE = 'INPUT'),
'ODS/legacy_export/AGGREGATED_ALLOTMENT_202401.csv',
'INGESTED', -- Skip validation, mark as already ingested
DATE '2024-01-15',
1048576, -- File size in bytes
'manual_registration',
NULL -- No external table needed
);
-- Repeat for all exported CSV files
COMMIT;
-- Step 4: Now FILE_ARCHIVER can process these files
BEGIN
FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfig => vConfig);
END;
/
```
#### Strategy 2: Direct Archive Export (Bypass ODS)
Skip ODS/DATA bucket entirely - export directly to ARCHIVE bucket in Parquet format:
```sql
-- Export legacy data directly to ARCHIVE bucket
BEGIN
DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
pBucketArea => 'ARCHIVE', -- Direct to archive
pFolderName => 'legacy_direct',
pMinDate => DATE '2020-01-01',
pMaxDate => DATE '2024-12-31'
);
END;
/
-- Result: Parquet files with Hive partitioning in ARCHIVE bucket
-- Files: ARCHIVE/legacy_direct/PARTITION_YEAR=2024/PARTITION_MONTH=01/*.parquet
-- No A_SOURCE_FILE_RECEIVED records needed (archival already complete)
```
**Pros**:
- Simple, no manual registration
- Data already in final archive format (Parquet)
- Hive-style partitioning automatically applied
**Cons**:
- No record in A_SOURCE_FILE_RECEIVED (tracking gap)
- Cannot use FILE_ARCHIVER features (archival strategies, status tracking)
- Mixed folder structure (legacy_direct vs. standard source/table paths)
#### Strategy 3: Hybrid Approach (Recommended for Large Datasets)
Use DATA_EXPORTER for initial export, create minimal A_SOURCE_FILE_RECEIVED records programmatically:
```sql
-- Create helper procedure to register legacy exports
CREATE OR REPLACE PROCEDURE REGISTER_LEGACY_EXPORT (
pSourceFileId VARCHAR2,
pBucketArea VARCHAR2,
pFolderName VARCHAR2,
pFilePattern VARCHAR2,
pReceptionDate DATE DEFAULT SYSDATE
) AS
vConfigKey NUMBER;
vBucketUri VARCHAR2(500);
vPrefix VARCHAR2(500);
BEGIN
-- Get source configuration
SELECT A_SOURCE_FILE_CONFIG_KEY
INTO vConfigKey
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = pSourceFileId
AND SOURCE_FILE_TYPE = 'INPUT';
-- Get bucket URI
vBucketUri := CT_MRDS.ENV_MANAGER.GET_BUCKET_URI(pBucketArea);
vPrefix := 'ODS/' || pFolderName || '/';
-- Register all matching files
FOR rec IN (
SELECT object_name, bytes, etag
FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
credential_name => 'DEF_CRED_ARN',
location_uri => vBucketUri
))
WHERE object_name LIKE vPrefix || pFilePattern
) LOOP
INSERT INTO CT_MRDS.A_SOURCE_FILE_RECEIVED (
A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
PROCESSING_STATUS,
RECEPTION_DATE,
BYTES,
CHECKSUM
) VALUES (
A_SOURCE_FILE_RECEIVED_KEY_SEQ.NEXTVAL,
vConfigKey,
rec.object_name,
'INGESTED',
pReceptionDate,
rec.bytes,
rec.etag
);
END LOOP;
COMMIT;
DBMS_OUTPUT.PUT_LINE('Registered ' || SQL%ROWCOUNT || ' legacy files');
END;
/
-- Usage:
BEGIN
-- After DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE completes
REGISTER_LEGACY_EXPORT(
pSourceFileId => 'AGGREGATED_ALLOTMENT',
pBucketArea => 'DATA',
pFolderName => 'legacy_export',
pFilePattern => 'AGGREGATED_ALLOTMENT_2024%.csv',
pReceptionDate => DATE '2024-12-31'
);
-- Now FILE_ARCHIVER can process
FILE_ARCHIVER.ARCHIVE_TABLE_DATA(pSourceFileConfig => vConfig);
END;
/
```
## Migration Timeline and Coexistence
### Phase 1: Legacy System Only (Before Migration)
- All data loaded via Informatica + WLA
- Control table: `CT_ODS.A_LOAD_HISTORY`
- No `A_SOURCE_FILE_RECEIVED` records
### Phase 2: Parallel Operation (During Migration)
- **Old data**: Continue using Informatica + WLA → A_LOAD_HISTORY
- **New data**: Start using Airflow + DBT → A_SOURCE_FILE_RECEIVED + A_WORKFLOW_HISTORY
- **Challenge**: Different control tables for different data vintages
### Phase 3: New System Only (After Migration)
- All new data via Airflow + DBT
- Legacy data archived (one-time export using DATA_EXPORTER)
- Control tables: `CT_MRDS.A_SOURCE_FILE_RECEIVED`, `CT_MRDS.A_WORKFLOW_HISTORY`
## Recommendations
### For New Data (Airflow + DBT)
✅ Use standard workflow:
1. FILE_MANAGER.PROCESS_SOURCE_FILE (creates A_SOURCE_FILE_RECEIVED)
2. Airflow + DBT processing (creates A_WORKFLOW_HISTORY)
3. FILE_ARCHIVER.ARCHIVE_TABLE_DATA (uses both control tables)
### For Legacy Data Migration
✅ **Small Datasets (<1000 files)**: Strategy 1 (Manual Registration)
✅ **Large Datasets (>1000 files)**: Strategy 2 (Direct to ARCHIVE) or Strategy 3 (Hybrid)
❌ **Avoid**: Exporting to ODS/DATA without registration (orphaned files, cannot archive)
### Configuration Requirements
Before archiving legacy data, ensure source configuration exists:
```sql
-- Check if configuration exists
SELECT A_SOURCE_FILE_CONFIG_KEY, SOURCE_FILE_ID, TABLE_ID, ARCHIVAL_STRATEGY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'YOUR_SOURCE_FILE_ID';
-- If missing, create configuration
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'YOUR_SOURCE',
pSourceFileType => 'INPUT',
pSourceFileId => 'YOUR_SOURCE_FILE_ID',
pSourceFileDesc => 'Legacy migrated data',
pSourceFileNamePattern => 'pattern_*.csv',
pTableId => 'YOUR_TABLE_ID',
pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE'
);
```
## Known Limitations
### 1. No Retroactive A_SOURCE_FILE_RECEIVED Creation
DATA_EXPORTER does not automatically create A_SOURCE_FILE_RECEIVED records when exporting legacy data. This is by design - it's a one-time export tool, not a file tracking system.
### 2. FILE_ARCHIVER Requires A_SOURCE_FILE_RECEIVED
FILE_ARCHIVER cannot archive data without corresponding A_SOURCE_FILE_RECEIVED records. This prevents archiving of:
- Legacy Informatica-loaded data exported via DATA_EXPORTER
- Manually uploaded files not processed through FILE_MANAGER.PROCESS_SOURCE_FILE
### 3. Mixed Control Table References
During migration period, some procedures reference A_LOAD_HISTORY (DATA_EXPORTER) while others reference A_WORKFLOW_HISTORY (FILE_ARCHIVER). This is intentional but requires careful understanding of data lineage.
### 4. A_WORKFLOW_HISTORY vs A_LOAD_HISTORY Column Mismatch
The control tables have different schemas:
- **A_LOAD_HISTORY**: `LOAD_START`, `A_ETL_LOAD_SET_KEY`
- **A_WORKFLOW_HISTORY**: `WORKFLOW_START`, `A_WORKFLOW_HISTORY_KEY`
Test scripts must be aware of which table is being used.
## Related Documentation
- [PROCESS_SOURCE_FILE Guide](PROCESS_SOURCE_FILE_Guide.md) - File validation and ingestion workflow
- [FILE_ARCHIVER Guide](FILE_ARCHIVER_Guide.md) - Archival strategies and configuration
- [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md) - System configuration
- [Package Deployment Guide](Package_Deployment_Guide.md) - Deployment procedures
## Summary
The migration from Informatica + WLA to Airflow + DBT introduces new control tables (`A_SOURCE_FILE_RECEIVED`, `A_WORKFLOW_HISTORY`) while maintaining compatibility with legacy control tables (`A_LOAD_HISTORY`). Understanding the relationship between these tables is critical for:
- **Data Lineage**: Tracking which system processed which data
- **Export Operations**: Choosing appropriate DATA_EXPORTER procedures
- **Archival Operations**: Ensuring FILE_ARCHIVER has required metadata
- **Testing**: Using correct control tables in test scenarios
The recommended approach for legacy data migration is **Strategy 2 (Direct to ARCHIVE)** for large datasets, as it avoids the complexity of manual A_SOURCE_FILE_RECEIVED registration while achieving the goal of moving historical data to long-term archival storage.