mars/MARS_Packages/REL01_ADDITIONS/MARS-835/README.md

MARS-835: One-Time CSDB Data Export from Operational Database to External Tables

Overview

This package performs a one-time bulk export of CSDB data from operational database tables (OU_CSDB schema) to new external tables in OCI buckets. The export uses DATA_EXPORTER v2.4.0 with per-column date format handling to move historical data to either DATA bucket (CSV format) or HIST bucket (Parquet format with Hive-style partitioning).

Migration Strategy:

• Split Export (2 tables): DEBT, DEBT_DAILY - Last 6 months → DATA (CSV), Older data → HIST (Parquet)
• HIST Only (4 tables): INSTR_RAT_FULL, INSTR_DESC_FULL, ISSUER_RAT_FULL, ISSUER_DESC_FULL - All data → HIST (Parquet)

Key Transformations:

• Column rename: A_ETL_LOAD_SET_FK → A_WORKFLOW_HISTORY_KEY (all tables)
• Column removal: DEBT (2 columns), DEBT_DAILY (6 columns) not required in new structure

Contents

• install_mars835.sql - Master installation script with SPOOL logging
• rollback_mars835.sql - Master rollback script
• 01_MARS_835_*.sql - Individual installation scripts
• 91_MARS_835_*.sql - Individual rollback scripts
• track_package_versions.sql - Package version tracking
• verify_packages_version.sql - Package verification

Prerequisites

• Oracle Database 23ai
• ADMIN user access (required for all MARS installations)
• ENV_MANAGER v3.1.0+
• Required schema privileges

Installation

Option 1: Master Script (Recommended)

# IMPORTANT: Execute as ADMIN user for proper privilege management
Get-Content "MARS_Packages/REL01_POST_DEACTIVATION/MARS-835/install_mars835.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"

# Log file created: log/INSTALL_MARS_835_<PDB>_<timestamp>.log

Option 2: Individual Scripts

# IMPORTANT: Execute as ADMIN user
Get-Content "01_MARS_835_*.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
Get-Content "02_MARS_835_*.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
# ... etc

Verification

-- Verify package versions
SELECT PACKAGE_NAME.GET_VERSION() FROM DUAL;

-- Check for errors (ADMIN user checks specific schema)
SELECT * FROM ALL_ERRORS 
WHERE OWNER = 'CT_MRDS'  -- Replace with target schema
  AND NAME = 'PACKAGE_NAME';

-- Verify no untracked changes
SELECT ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'PACKAGE_NAME') FROM DUAL;

Rollback

# IMPORTANT: Execute as ADMIN user
Get-Content "MARS_Packages/REL01_POST_DEACTIVATION/MARS-835/rollback_mars835.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"

**NOTE**: Rollback for data exports is **NOT RECOMMENDED** as it would delete exported files from OCI buckets. Only use rollback if export failed and needs to be restarted.

Expected Changes

Data Export Summary

6 CSDB tables exported from OU_CSDB schema:

Group 1: Split DATA + HIST (Time Critical)

1. DEBT - Last 6 months → DATA, Older → HIST
2. DEBT_DAILY - Last 6 months → DATA, Older → HIST

Group 2: HIST Only (Weekend Bulk) 3. INSTR_RAT_FULL - All data → HIST 4. INSTR_DESC_FULL - All data → HIST 5. ISSUER_RAT_FULL - All data → HIST 6. ISSUER_DESC_FULL - All data → HIST

Bucket Destinations (DEV environment)

• DATA Bucket: mrds_data_dev/ODS/CSDB/ (CSV format)
• HIST Bucket: mrds_hist_dev/ARCHIVE/CSDB/ (Parquet with partitioning)

Column Mappings

• All tables: A_ETL_LOAD_SET_FK renamed to A_WORKFLOW_HISTORY_KEY
• DEBT: Removed columns: IDIRDEPOSITORY, VA_BONDDURATION
• DEBT_DAILY: Removed columns: STEPID, PROGRAMNAME, PROGRAMCEILING, PROGRAMSTATUS, ISSUERNACE21SECTOR, INSTRUMENTQUOTATIONBASIS

Testing

Post-Export Verification

1. Verify CSV files in DATA bucket (DEBT, DEBT_DAILY - last 6 months):

-- Check exported files
SELECT object_name, bytes 
FROM TABLE(DBMS_CLOUD.LIST_OBJECTS(
    credential_name => 'DEF_CRED_ARN',
    location_uri => 'https://objectstorage.region.oraclecloud.com/n/namespace/b/mrds_data_dev/o/ODS/CSDB/'
)) WHERE object_name LIKE '%CSDB_DEBT%';

2. Verify Parquet files in HIST bucket (all 6 tables):

-- Check archived files with Hive partitioning
SELECT object_name, bytes 
FROM TABLE(DBMS_CLOUD.LIST_OBJECTS(
    credential_name => 'DEF_CRED_ARN',
    location_uri => 'https://objectstorage.region.oraclecloud.com/n/namespace/b/mrds_hist_dev/o/ARCHIVE/CSDB/'
)) WHERE object_name LIKE '%PARTITION_YEAR=%';

3. Validate row counts match source tables:

-- Compare counts between source and exported data
SELECT COUNT(*) FROM OU_CSDB.DEBT;
SELECT COUNT(*) FROM ODS.CSDB_DEBT_ODS;  -- External table pointing to DATA
SELECT COUNT(*) FROM ODS.CSDB_DEBT_ARCHIVE;  -- External table pointing to HIST

4. Verify column mappings:

-- Check A_WORKFLOW_HISTORY_KEY exists in exported data
SELECT A_WORKFLOW_HISTORY_KEY, COUNT(*) 
FROM ODS.CSDB_DEBT_ARCHIVE 
GROUP BY A_WORKFLOW_HISTORY_KEY;

Known Issues

Timing Constraints

• DATA exports (DEBT, DEBT_DAILY): Must execute during parallel old+new loads phase after Production deployment
• HIST exports (all 6 tables): Can run anytime, recommended for weekend bulk execution to avoid interference

Environment-Specific Configuration

• Bucket names must be adjusted for each environment:
  • DEV: mrds_data_dev, mrds_hist_dev
  • TEST: mrds_data_test, mrds_hist_test
  • PROD: mrds_data_prod, mrds_hist_prod

Data Cutoff Date

• Export scripts use 6-month cutoff date calculated as ADD_MONTHS(SYSDATE, -6)
• Verify cutoff aligns with business requirements before execution

One-Time Execution

• This is a ONE-TIME data migration package
• After successful execution, package should be deactivated (moved to REL01_POST_DEACTIVATION)
• Do not re-run unless explicitly required for data refresh

Related

• JIRA: MARS-835 - CSDB Data Export to External Tables
• Confluence: FILE_MANAGER package - MRDS - Technical Team
• Confluence: Table Setup Guide for FILE PROCESSOR System
• Source Schema: OU_CSDB (Operational Database)
• Target Schema: ODS (External Tables)
• Migration Type: One-time bulk export (deactivated post-execution)

---

Author: Grzegorz Michalski
Date: 2025-12-04
Version: 1.0.0