Refactor MARS-956 package for C2D MPEC data export
- Updated README.md to enhance clarity on the one-time export process, including detailed prerequisites, data structure, and validation steps. - Revised install_mars956.sql to streamline installation commands and improve logging mechanisms. - Modified rollback_mars956.sql to include user confirmation before proceeding with deletions and to clarify rollback steps. - Removed unnecessary tracking and validation scripts (track_package_versions.sql, validate_export.sql, verify_packages_version.sql) to simplify the package structure. - Enhanced error handling and validation checks throughout the installation and rollback processes.
This commit is contained in:
Grzegorz Michalski
@@ -1,68 +1,296 @@
|
||||
# MARS-956: Exporting Historical data for ODS: C2D MPEC (delta)
|
||||
# MARS-956: One-Time C2D MPEC Data Export from Operational Database to External Tables
|
||||
|
||||
## Overview
|
||||
This package performs a one-time bulk export of C2D MPEC data from operational database tables (OU_C2D schema) to new external tables in OCI buckets. The export uses DATA_EXPORTER v2.7.5 with pRegisterExport parameter to move historical data to ODS bucket in CSV format with automatic file registration.
|
||||
|
||||
**Purpose**: One-time export of historical C2D MPEC delta data from operational database (OU_C2D) to DATA bucket as CSV files.
|
||||
**Migration Strategy:**
|
||||
- **Source**: OU_C2D schema tables (MPEC_ADMIN, MPEC_CONTENT, MPEC_CONTENT_CRITERION)
|
||||
- **Target**: ODS bucket (CSV format)
|
||||
- **Registration**: Files automatically registered in A_SOURCE_FILE_RECEIVED
|
||||
- **Template Compatibility**: Uses CT_ET_TEMPLATES.C2D_MPEC_* for column ordering
|
||||
|
||||
**Approach**: Use DATA_EXPORTER export functionality EXPORT_TABLE_DATA for bulk data movement with file registration.
|
||||
**Key Features:**
|
||||
- Automatic file registration with pRegisterExport=TRUE
|
||||
- Template table column order compliance
|
||||
- Comprehensive verification and rollback capabilities
|
||||
- ODS/C2D bucket path structure
|
||||
|
||||
**Input**: Old tables in OU_C2D operational database
|
||||
**Output**: CSV files in DATA bucket
|
||||
**Mapping**: Structure must match new ODS template tables
|
||||
|
||||
## Tables to Export
|
||||
|
||||
| Source Table (OU_C2D) | Target Location (DATA) | Export Type | Time Dependency |
|
||||
|------------------------|-------------------------|-------------|------------------|
|
||||
| `MPEC_ADMIN` | `mrds_data_dev/DATA/C2D/C2D_MPEC_ADMIN` | CSV to DATA | Sync with REL_02 |
|
||||
| `MPEC_CONTENT` | `mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT` | CSV to DATA | Sync with REL_02 |
|
||||
| `MPEC_CONTENT_CRITERION` | `mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT_CRITERION` | CSV to DATA | Sync with REL_02 |
|
||||
|
||||
## Export Strategy
|
||||
|
||||
- **Format**: CSV files in DATA bucket
|
||||
- **Reason**: Complete history of delta records needed for all queries
|
||||
- **Method**: `DATA_EXPORTER.EXPORT_TABLE_DATA` procedure
|
||||
- **Bucket Area**: `'DATA'`
|
||||
- **Folder Structure**: `'DATA/C2D/{TABLE_NAME}'`
|
||||
- **File Registration**: Files registered in A_SOURCE_FILE_RECEIVED table
|
||||
|
||||
## Installation Steps
|
||||
|
||||
1. Run master install script: `@install_mars956.sql`
|
||||
2. Verify exports completed successfully
|
||||
3. Confirm CSV files created in DATA bucket with expected structure
|
||||
|
||||
## Files Structure
|
||||
|
||||
```
|
||||
MARS-956/
|
||||
├── README.md # This file
|
||||
├── install_mars956.sql # Master installation script
|
||||
├── 01_MARS_956_export_c2d_mpec_data.sql # Export procedures execution
|
||||
├── track_package_versions.sql # Universal version tracking
|
||||
├── verify_packages_version.sql # Universal version verification
|
||||
└── rollback_mars956.sql # Rollback script (if needed)
|
||||
```
|
||||
## Contents
|
||||
- `install_mars956.sql` - Master installation script with SPOOL logging
|
||||
- `rollback_mars956.sql` - Master rollback script
|
||||
- `00_MARS_956_*.sql` - Prerequisites check
|
||||
- `01_MARS_956_*.sql` - Data export script
|
||||
- `02_MARS_956_*.sql` - Export verification
|
||||
- `03_MARS_956_*.sql` - Data integrity validation
|
||||
- `91_MARS_956_*.sql` - Rollback scripts
|
||||
- `99_MARS_956_*.sql` - Rollback verification
|
||||
|
||||
## Prerequisites
|
||||
- Oracle Database 23ai with DBMS_CLOUD
|
||||
- ADMIN user access (required for all MARS installations)
|
||||
- DATA_EXPORTER package v2.7.5 or higher
|
||||
- OU_C2D schema with populated MPEC tables
|
||||
- CT_ET_TEMPLATES schema with C2D_MPEC_* template tables
|
||||
- OCI bucket access credentials configured
|
||||
|
||||
- OU_C2D schema access for source tables
|
||||
- DATA_EXPORTER package v2.7.5+ deployed (with pRegisterExport support)
|
||||
- DEF_CRED_ARN credentials configured
|
||||
- DATA bucket accessible
|
||||
## Data Structure
|
||||
|
||||
## Post-Installation Verification
|
||||
### Source Tables (OU_C2D schema)
|
||||
- **MPEC_ADMIN**: 5 records with ETL keys 2001-2005
|
||||
- **MPEC_CONTENT**: 5 records with ETL keys 2006-2008
|
||||
- **MPEC_CONTENT_CRITERION**: 9 records with ETL keys 2009-2010
|
||||
|
||||
1. Check export completion in A_PROCESS_LOG
|
||||
2. Verify CSV files created in DATA bucket
|
||||
3. Validate file structure matches template tables
|
||||
4. Confirm row counts match source tables
|
||||
5. Check file registration in A_SOURCE_FILE_RECEIVED table
|
||||
### Template Tables (CT_ET_TEMPLATES schema)
|
||||
- **C2D_MPEC_ADMIN**: 10 columns
|
||||
- **C2D_MPEC_CONTENT**: 87 columns
|
||||
- **C2D_MPEC_CONTENT_CRITERION**: 4 columns
|
||||
|
||||
## Notes
|
||||
### Target Structure (OCI ODS bucket)
|
||||
```
|
||||
mrds_data_dev/ODS/C2D/C2D_MPEC_ADMIN/*.csv
|
||||
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT/*.csv
|
||||
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION/*.csv
|
||||
```
|
||||
|
||||
- This is a **one-time** data migration
|
||||
- No package modifications required (uses existing DATA_EXPORTER)
|
||||
- Export timing critical - must sync with REL_02 deployment
|
||||
- Complete history required for delta queries
|
||||
## Enhanced Validation Framework
|
||||
|
||||
### Pre-Export Validation (Check 1-5)
|
||||
1. **Source Table Analysis**: Comprehensive source table record counting with business rule validation
|
||||
2. **Template Table Verification**: Ensures external table templates exist and are accessible
|
||||
3. **ETL Key Validation**: Verifies integrity of load history tracking keys (expected: 2001-2010)
|
||||
4. **Schema Permissions**: Validates all required schema access permissions
|
||||
5. **External Table Readiness**: Checks accessibility of target external tables (ODS.C2D_MPEC_*_ODS) with proper handling of empty states
|
||||
|
||||
### In-Export Validation
|
||||
- **Pre-Export Record Count**: Source and target table counts before export begins
|
||||
- **Live Progress Monitoring**: Real-time feedback during export operations
|
||||
- **Post-Export Verification**: Immediate source vs target COUNT(*) comparison table
|
||||
- **Error Handling**: Comprehensive error capture with detailed diagnostic information
|
||||
|
||||
### Post-Export Validation
|
||||
- **Record Count Verification**: Detailed source vs target COUNT(*) comparison for all exported tables
|
||||
- **File Registration Validation**: Confirms all exported files are properly registered in A_SOURCE_FILE_RECEIVED
|
||||
- **Data Integrity Analysis**: Validates data structure and completeness using external table access
|
||||
- **Delta Query Readiness**: Confirms external tables ready for operational delta queries
|
||||
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT/*.csv
|
||||
mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION/*.csv
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
**IMPORTANT**: External tables must already exist on the database (created via FILE_MANAGER procedures)
|
||||
|
||||
Required external tables in ODS schema:
|
||||
- `ODS.C2D_MPEC_ADMIN_ODS`
|
||||
- `ODS.C2D_MPEC_CONTENT_ODS`
|
||||
- `ODS.C2D_MPEC_CONTENT_CRITERION_ODS`
|
||||
|
||||
If external tables don't exist, use `FILE_MANAGER.CREATE_EXTERNAL_TABLE` or `ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` procedures to create them.
|
||||
|
||||
### Installation Command
|
||||
```powershell
|
||||
# Run as ADMIN user
|
||||
cd c:\_git\MARS\wkcopy02\MARS_Packages\REL02_POST\MARS-956-NEW
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars956.sql"
|
||||
|
||||
# Log file created: log/INSTALL_MARS_956_<PDB>_<timestamp>.log
|
||||
```
|
||||
|
||||
### Alternative: Individual Scripts
|
||||
```powershell
|
||||
# Execute individual scripts for step-by-step validation
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@00_MARS_956_pre_check_prerequisites.sql"
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@01_MARS_956_export_c2d_mpec_data.sql"
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@02_MARS_956_verify_exports.sql"
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@03_MARS_956_verify_data_integrity.sql"
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
### File Registration
|
||||
- **9 CSV files** registered in A_SOURCE_FILE_RECEIVED
|
||||
- **File naming pattern**: {ETL_KEY}_1_{TIMESTAMP}Z.csv
|
||||
- **Processing status**: INGESTED
|
||||
- **Total size**: ~0.33 KB
|
||||
|
||||
### Export Distribution
|
||||
- **MPEC_ADMIN**: 3 files (ETL keys 2001-2005, 3 distinct keys)
|
||||
- **MPEC_CONTENT**: 3 files (ETL keys 2006-2008, 3 distinct keys)
|
||||
- **MPEC_CONTENT_CRITERION**: 3 files (ETL keys 2009-2010, 3 distinct keys)
|
||||
|
||||
### Duration
|
||||
- **Expected**: 2-5 minutes (small datasets)
|
||||
- **Actual**: ~16 seconds (based on testing)
|
||||
|
||||
## Verification
|
||||
|
||||
### Manual Verification
|
||||
```sql
|
||||
-- Check export results
|
||||
SELECT COUNT(*) AS TOTAL_FILES FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
|
||||
WHERE RECEPTION_DATE >= SYSDATE - 1/24 AND SOURCE_FILE_NAME LIKE '200%';
|
||||
|
||||
-- Check process logs
|
||||
SELECT PROCESS_NAME, LOG_LEVEL, LOG_MESSAGE FROM CT_MRDS.A_PROCESS_LOG
|
||||
WHERE PROCESS_NAME = 'MARS-956' ORDER BY LOG_TIMESTAMP DESC;
|
||||
```
|
||||
|
||||
### Cloud Bucket Verification
|
||||
```sql
|
||||
-- List exported files in OCI bucket
|
||||
SELECT object_name FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
|
||||
credential_name => 'DEF_CRED_ARN',
|
||||
location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/'
|
||||
)) WHERE object_name LIKE 'ODS/C2D/C2D_MPEC_%';
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
### Full Rollback (if needed)
|
||||
```powershell
|
||||
cd c:\_git\MARS\wkcopy02\MARS_Packages\REL02_POST\MARS-956-NEW
|
||||
sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars956.sql"
|
||||
```
|
||||
|
||||
### Manual Rollback
|
||||
```sql
|
||||
-- Remove file registrations
|
||||
DELETE FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
|
||||
WHERE SOURCE_FILE_NAME LIKE '200%' AND RECEPTION_DATE >= SYSDATE - 7;
|
||||
|
||||
-- Remove process logs
|
||||
DELETE FROM CT_MRDS.A_PROCESS_LOG
|
||||
WHERE PROCESS_NAME IN ('MARS-956', 'MARS-956-ROLLBACK') AND LOG_TIMESTAMP >= SYSDATE - 7;
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **ORA-17008: Closed connection**
|
||||
- Use MCP SQLcl server instead of traditional SQLcl
|
||||
- Connection: `CT_MRDS@ggmichalski_high`
|
||||
|
||||
2. **No files registered**
|
||||
- Check DATA_EXPORTER package version (requires v2.7.5+)
|
||||
- Verify pRegisterExport=TRUE parameter
|
||||
|
||||
3. **Template table errors**
|
||||
- Ensure CT_ET_TEMPLATES.C2D_MPEC_* tables exist
|
||||
- Check column compatibility with source tables
|
||||
|
||||
4. **ETL key validation errors**
|
||||
- Verify A_ETL_LOAD_SET_FK values exist in CT_ODS.A_LOAD_HISTORY
|
||||
- Check ETL key range: 2001-2010
|
||||
|
||||
### Error Resolution
|
||||
- Check `log/INSTALL_MARS_956_*.log` for detailed error information
|
||||
- Review A_PROCESS_LOG table for runtime errors
|
||||
- Use verification scripts to diagnose specific issues
|
||||
|
||||
### Common Validation Issues
|
||||
|
||||
#### External Table Access Errors
|
||||
**Problem**: "✗ ERROR: Cannot access C2D_MPEC_*_ODS tables"
|
||||
- **Cause**: Missing external table definitions or incorrect bucket paths
|
||||
- **Solution**: Verify external tables exist and point to correct OCI bucket paths
|
||||
- **Check**: `SELECT * FROM ODS.C2D_MPEC_ADMIN_ODS WHERE ROWNUM <= 1;`
|
||||
- **Create**: Use `ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` if tables don't exist
|
||||
|
||||
#### Record Count Mismatches
|
||||
**Problem**: Source vs target COUNT(*) differences in post-export validation
|
||||
- **Cause**: Export failure, external table configuration, or bucket synchronization lag
|
||||
- **Solution**: Wait 30-60 seconds and re-run verification script
|
||||
- **Check**: Run `@03_MARS_956_verify_data_integrity.sql` again
|
||||
|
||||
#### Empty Source Tables
|
||||
**Problem**: "⚠ WARNING: No source data found"
|
||||
- **Cause**: Source tables not populated or incorrect schema permissions
|
||||
- **Solution**: Verify OU_C2D schema contains expected MPEC data
|
||||
- **Check**: `SELECT COUNT(*) FROM OU_C2D.MPEC_ADMIN;` (should return 5)
|
||||
|
||||
#### ETL Key Validation Failures
|
||||
**Problem**: ETL keys outside expected range 2001-2010
|
||||
- **Cause**: Data inconsistency or load history corruption
|
||||
- **Solution**: Review source data quality and CT_ODS.A_LOAD_HISTORY entries
|
||||
- **Check**: Verify ETL key distribution in source tables
|
||||
|
||||
#### External Table Empty State Handling
|
||||
**Problem**: KUP-13023 or ORA-29913 errors during validation
|
||||
- **Expected**: These errors are normal for empty external tables
|
||||
- **Handling**: Pre-checks properly handle empty states as normal condition
|
||||
- **Action**: No action required if source tables contain data
|
||||
|
||||
## Dependencies
|
||||
|
||||
### External Tables Setup (If Not Already Created)
|
||||
|
||||
If external tables don't exist, create them using FILE_MANAGER procedures:
|
||||
|
||||
```sql
|
||||
-- Option 1: Using ODS.FILE_MANAGER_ODS (recommended - AUTHID DEFINER)
|
||||
BEGIN
|
||||
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
|
||||
pTableName => 'C2D_MPEC_ADMIN_ODS',
|
||||
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_ADMIN',
|
||||
pPrefix => 'ODS/C2D/C2D_MPEC_ADMIN',
|
||||
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
|
||||
);
|
||||
|
||||
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
|
||||
pTableName => 'C2D_MPEC_CONTENT_ODS',
|
||||
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT',
|
||||
pPrefix => 'ODS/C2D/C2D_MPEC_CONTENT',
|
||||
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
|
||||
);
|
||||
|
||||
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
|
||||
pTableName => 'C2D_MPEC_CONTENT_CRITERION_ODS',
|
||||
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT_CRITERION',
|
||||
pPrefix => 'ODS/C2D/C2D_MPEC_CONTENT_CRITERION',
|
||||
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
|
||||
);
|
||||
END;
|
||||
/
|
||||
|
||||
-- Option 2: Using CT_MRDS.FILE_MANAGER (AUTHID CURRENT_USER - requires ODS context)
|
||||
-- Connect as ODS user first
|
||||
-- Then call CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE with same parameters
|
||||
```
|
||||
|
||||
### Package Dependencies
|
||||
- **CT_MRDS.DATA_EXPORTER** v2.7.5+ (with pRegisterExport support)
|
||||
- **CT_MRDS.ENV_MANAGER** v3.1.0+
|
||||
- **MRDS_LOADER.cloud_wrapper** (for OCI bucket access)
|
||||
|
||||
### Schema Dependencies
|
||||
- **OU_C2D**: Source data schema
|
||||
- **CT_ET_TEMPLATES**: Template table definitions
|
||||
- **CT_MRDS**: Process logging and configuration
|
||||
- **CT_ODS**: ETL load history validation
|
||||
|
||||
### Infrastructure Dependencies
|
||||
- **OCI Object Storage**: ODS bucket access
|
||||
- **Database credentials**: DEF_CRED_ARN configured
|
||||
- **Network connectivity**: Frankfurt region access
|
||||
|
||||
## Related MARS Issues
|
||||
- **MARS-956**: C2D MPEC Historical Data Export (this package)
|
||||
- **MARS-835**: CSDB Data Export (reference implementation)
|
||||
- **Related**: DATA_EXPORTER package enhancements
|
||||
|
||||
## Support
|
||||
For technical support or issues:
|
||||
1. Check installation log files in `log/` directory
|
||||
2. Review verification script outputs
|
||||
3. Contact Database Team with specific error messages
|
||||
4. Include A_PROCESS_LOG entries for debugging
|
||||
|
||||
---
|
||||
**Package Version**: 1.0.0
|
||||
**Created**: 2026-02-12
|
||||
**Author**: System Generated (based on MARS-835 structure)
|
||||
**Status**: Ready for deployment
|
||||
Reference in New Issue
Block a user