admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a7d286b1e610a27dc809ae738702e6b506a76b28
mars/MARS_Packages/REL02_POST/MARS-956/README.md
Grzegorz Michalski a7d286b1e6 Refactor MARS-956 package for C2D MPEC data export 
- Updated README.md to enhance clarity on the one-time export process, including detailed prerequisites, data structure, and validation steps.
- Revised install_mars956.sql to streamline installation commands and improve logging mechanisms.
- Modified rollback_mars956.sql to include user confirmation before proceeding with deletions and to clarify rollback steps.
- Removed unnecessary tracking and validation scripts (track_package_versions.sql, validate_export.sql, verify_packages_version.sql) to simplify the package structure.
- Enhanced error handling and validation checks throughout the installation and rollback processes.
2026-02-12 09:56:37 +01:00

11 KiB
Raw Blame History

MARS-956: One-Time C2D MPEC Data Export from Operational Database to External Tables

Overview

This package performs a one-time bulk export of C2D MPEC data from operational database tables (OU_C2D schema) to new external tables in OCI buckets. The export uses DATA_EXPORTER v2.7.5 with pRegisterExport parameter to move historical data to ODS bucket in CSV format with automatic file registration.

Migration Strategy:

  • Source: OU_C2D schema tables (MPEC_ADMIN, MPEC_CONTENT, MPEC_CONTENT_CRITERION)
  • Target: ODS bucket (CSV format)
  • Registration: Files automatically registered in A_SOURCE_FILE_RECEIVED
  • Template Compatibility: Uses CT_ET_TEMPLATES.C2D_MPEC_* for column ordering

Key Features:

  • Automatic file registration with pRegisterExport=TRUE
  • Template table column order compliance
  • Comprehensive verification and rollback capabilities
  • ODS/C2D bucket path structure

Contents

  • install_mars956.sql - Master installation script with SPOOL logging
  • rollback_mars956.sql - Master rollback script
  • 00_MARS_956_*.sql - Prerequisites check
  • 01_MARS_956_*.sql - Data export script
  • 02_MARS_956_*.sql - Export verification
  • 03_MARS_956_*.sql - Data integrity validation
  • 91_MARS_956_*.sql - Rollback scripts
  • 99_MARS_956_*.sql - Rollback verification

Prerequisites

  • Oracle Database 23ai with DBMS_CLOUD
  • ADMIN user access (required for all MARS installations)
  • DATA_EXPORTER package v2.7.5 or higher
  • OU_C2D schema with populated MPEC tables
  • CT_ET_TEMPLATES schema with C2D_MPEC_* template tables
  • OCI bucket access credentials configured

Data Structure

Source Tables (OU_C2D schema)

  • MPEC_ADMIN: 5 records with ETL keys 2001-2005
  • MPEC_CONTENT: 5 records with ETL keys 2006-2008
  • MPEC_CONTENT_CRITERION: 9 records with ETL keys 2009-2010

Template Tables (CT_ET_TEMPLATES schema)

  • C2D_MPEC_ADMIN: 10 columns
  • C2D_MPEC_CONTENT: 87 columns
  • C2D_MPEC_CONTENT_CRITERION: 4 columns

Target Structure (OCI ODS bucket)

mrds_data_dev/ODS/C2D/C2D_MPEC_ADMIN/*.csv
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT/*.csv  
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION/*.csv

Enhanced Validation Framework

Pre-Export Validation (Check 1-5)

  1. Source Table Analysis: Comprehensive source table record counting with business rule validation
  2. Template Table Verification: Ensures external table templates exist and are accessible
  3. ETL Key Validation: Verifies integrity of load history tracking keys (expected: 2001-2010)
  4. Schema Permissions: Validates all required schema access permissions
  5. External Table Readiness: Checks accessibility of target external tables (ODS.C2D_MPEC_*_ODS) with proper handling of empty states

In-Export Validation

  • Pre-Export Record Count: Source and target table counts before export begins
  • Live Progress Monitoring: Real-time feedback during export operations
  • Post-Export Verification: Immediate source vs target COUNT(*) comparison table
  • Error Handling: Comprehensive error capture with detailed diagnostic information

Post-Export Validation

  • Record Count Verification: Detailed source vs target COUNT(*) comparison for all exported tables
  • File Registration Validation: Confirms all exported files are properly registered in A_SOURCE_FILE_RECEIVED
  • Data Integrity Analysis: Validates data structure and completeness using external table access
  • Delta Query Readiness: Confirms external tables ready for operational delta queries mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT/.csv
    mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION/    .csv

## Installation

## Installation

### Prerequisites
**IMPORTANT**: External tables must already exist on the database (created via FILE_MANAGER procedures)

Required external tables in ODS schema:
- `ODS.C2D_MPEC_ADMIN_ODS`
- `ODS.C2D_MPEC_CONTENT_ODS`
- `ODS.C2D_MPEC_CONTENT_CRITERION_ODS`

If external tables don't exist, use `FILE_MANAGER.CREATE_EXTERNAL_TABLE` or `ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` procedures to create them.

### Installation Command
```powershell
# Run as ADMIN user
cd c:\_git\MARS\wkcopy02\MARS_Packages\REL02_POST\MARS-956-NEW
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars956.sql"

# Log file created: log/INSTALL_MARS_956_<PDB>_<timestamp>.log

Alternative: Individual Scripts

# Execute individual scripts for step-by-step validation
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@00_MARS_956_pre_check_prerequisites.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@01_MARS_956_export_c2d_mpec_data.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@02_MARS_956_verify_exports.sql"
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@03_MARS_956_verify_data_integrity.sql"

Expected Results

File Registration

  • 9 CSV files registered in A_SOURCE_FILE_RECEIVED
  • File naming pattern: {ETL_KEY}1{TIMESTAMP}Z.csv
  • Processing status: INGESTED
  • Total size: ~0.33 KB

Export Distribution

  • MPEC_ADMIN: 3 files (ETL keys 2001-2005, 3 distinct keys)
  • MPEC_CONTENT: 3 files (ETL keys 2006-2008, 3 distinct keys)
  • MPEC_CONTENT_CRITERION: 3 files (ETL keys 2009-2010, 3 distinct keys)

Duration

  • Expected: 2-5 minutes (small datasets)
  • Actual: ~16 seconds (based on testing)

Verification

Manual Verification

-- Check export results
SELECT COUNT(*) AS TOTAL_FILES FROM CT_MRDS.A_SOURCE_FILE_RECEIVED 
WHERE RECEPTION_DATE >= SYSDATE - 1/24 AND SOURCE_FILE_NAME LIKE '200%';

-- Check process logs
SELECT PROCESS_NAME, LOG_LEVEL, LOG_MESSAGE FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCESS_NAME = 'MARS-956' ORDER BY LOG_TIMESTAMP DESC;

Cloud Bucket Verification

-- List exported files in OCI bucket
SELECT object_name FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
    credential_name => 'DEF_CRED_ARN',
    location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/'
)) WHERE object_name LIKE 'ODS/C2D/C2D_MPEC_%';

Rollback

Full Rollback (if needed)

cd c:\_git\MARS\wkcopy02\MARS_Packages\REL02_POST\MARS-956-NEW
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars956.sql"

Manual Rollback

-- Remove file registrations
DELETE FROM CT_MRDS.A_SOURCE_FILE_RECEIVED 
WHERE SOURCE_FILE_NAME LIKE '200%' AND RECEPTION_DATE >= SYSDATE - 7;

-- Remove process logs
DELETE FROM CT_MRDS.A_PROCESS_LOG 
WHERE PROCESS_NAME IN ('MARS-956', 'MARS-956-ROLLBACK') AND LOG_TIMESTAMP >= SYSDATE - 7;

Troubleshooting

Common Issues

  1. ORA-17008: Closed connection

    • Use MCP SQLcl server instead of traditional SQLcl
    • Connection: CT_MRDS@ggmichalski_high

  2. No files registered

    • Check DATA_EXPORTER package version (requires v2.7.5+)
    • Verify pRegisterExport=TRUE parameter

  3. Template table errors

    • Ensure CT_ET_TEMPLATES.C2D_MPEC_* tables exist
    • Check column compatibility with source tables

  4. ETL key validation errors

    • Verify A_ETL_LOAD_SET_FK values exist in CT_ODS.A_LOAD_HISTORY
    • Check ETL key range: 2001-2010

Error Resolution

  • Check log/INSTALL_MARS_956_*.log for detailed error information
  • Review A_PROCESS_LOG table for runtime errors
  • Use verification scripts to diagnose specific issues

Common Validation Issues

External Table Access Errors

Problem: "✗ ERROR: Cannot access C2D_MPEC_*_ODS tables"

  • Cause: Missing external table definitions or incorrect bucket paths
  • Solution: Verify external tables exist and point to correct OCI bucket paths
  • Check: SELECT * FROM ODS.C2D_MPEC_ADMIN_ODS WHERE ROWNUM <= 1;
  • Create: Use ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE if tables don't exist

Record Count Mismatches

Problem: Source vs target COUNT(*) differences in post-export validation

  • Cause: Export failure, external table configuration, or bucket synchronization lag
  • Solution: Wait 30-60 seconds and re-run verification script
  • Check: Run @03_MARS_956_verify_data_integrity.sql again

Empty Source Tables

Problem: "⚠ WARNING: No source data found"

  • Cause: Source tables not populated or incorrect schema permissions
  • Solution: Verify OU_C2D schema contains expected MPEC data
  • Check: SELECT COUNT(*) FROM OU_C2D.MPEC_ADMIN; (should return 5)

ETL Key Validation Failures

Problem: ETL keys outside expected range 2001-2010

  • Cause: Data inconsistency or load history corruption
  • Solution: Review source data quality and CT_ODS.A_LOAD_HISTORY entries
  • Check: Verify ETL key distribution in source tables

External Table Empty State Handling

Problem: KUP-13023 or ORA-29913 errors during validation

  • Expected: These errors are normal for empty external tables
  • Handling: Pre-checks properly handle empty states as normal condition
  • Action: No action required if source tables contain data

Dependencies

External Tables Setup (If Not Already Created)

If external tables don't exist, create them using FILE_MANAGER procedures:

-- Option 1: Using ODS.FILE_MANAGER_ODS (recommended - AUTHID DEFINER)
BEGIN
    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
        pTableName => 'C2D_MPEC_ADMIN_ODS',
        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_ADMIN',
        pPrefix => 'ODS/C2D/C2D_MPEC_ADMIN',
        pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
    );
    
    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
        pTableName => 'C2D_MPEC_CONTENT_ODS',
        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT',
        pPrefix => 'ODS/C2D/C2D_MPEC_CONTENT',
        pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
    );
    
    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
        pTableName => 'C2D_MPEC_CONTENT_CRITERION_ODS',
        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT_CRITERION',
        pPrefix => 'ODS/C2D/C2D_MPEC_CONTENT_CRITERION',
        pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
    );
END;
/

-- Option 2: Using CT_MRDS.FILE_MANAGER (AUTHID CURRENT_USER - requires ODS context)
-- Connect as ODS user first
-- Then call CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE with same parameters

Package Dependencies

  • CT_MRDS.DATA_EXPORTER v2.7.5+ (with pRegisterExport support)
  • CT_MRDS.ENV_MANAGER v3.1.0+
  • MRDS_LOADER.cloud_wrapper (for OCI bucket access)

Schema Dependencies

  • OU_C2D: Source data schema
  • CT_ET_TEMPLATES: Template table definitions
  • CT_MRDS: Process logging and configuration
  • CT_ODS: ETL load history validation

Infrastructure Dependencies

  • OCI Object Storage: ODS bucket access
  • Database credentials: DEF_CRED_ARN configured
  • Network connectivity: Frankfurt region access

Related MARS Issues

  • MARS-956: C2D MPEC Historical Data Export (this package)
  • MARS-835: CSDB Data Export (reference implementation)
  • Related: DATA_EXPORTER package enhancements

Support

For technical support or issues:

  1. Check installation log files in log/ directory
  2. Review verification script outputs
  3. Contact Database Team with specific error messages
  4. Include A_PROCESS_LOG entries for debugging

Package Version: 1.0.0
Created: 2026-02-12
Author: System Generated (based on MARS-835 structure)
Status: Ready for deployment

Reference in New Issue View Git Blame Copy Permalink