admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor export scripts to use DATA_EXPORTER.EXPORT_TABLE_DATA procedure, update folder structure to DATA/C2D, and add file registration validation in README and validation script.

Browse Source
This commit is contained in:
Grzegorz Michalski
2026-02-11 21:22:37 +01:00
parent 641af7415f
commit f8213b76ab
3 changed files with 83 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
MARS_Packages/REL02_POST/MARS-956/01_MARS_956_export_c2d_mpec_data.sql
View File

@@ -3,8 +3,8 @@
-- =================================================================== -- ===================================================================
-- Purpose: One-time export of historical C2D MPEC delta data from -- Purpose: One-time export of historical C2D MPEC delta data from
-- OU_C2D operational database to DATA bucket as CSV files -- OU_C2D operational database to DATA bucket as CSV files
-- Method: Using DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE procedure -- Method: Using DATA_EXPORTER.EXPORT_TABLE_DATA procedure
-- Target: DATA bucket with folder structure ODS/C2D/{TABLE_NAME} -- Target: DATA bucket with folder structure DATA/C2D/{TABLE_NAME}
-- Format: CSV files for complete historical data access -- Format: CSV files for complete historical data access
-- =================================================================== -- ===================================================================
@@ -14,8 +14,9 @@ PROMPT =========================================================================
PROMPT Export Strategy: PROMPT Export Strategy:
PROMPT - Source: OU_C2D schema tables (operational database) PROMPT - Source: OU_C2D schema tables (operational database)
PROMPT - Target: DATA bucket as CSV files PROMPT - Target: DATA bucket as CSV files
PROMPT - Method: DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE PROMPT - Method: DATA_EXPORTER.EXPORT_TABLE_DATA
PROMPT - Structure: Must match ODS template tables PROMPT - Structure: Must match ODS template tables
PROMPT - Registration: Files registered in A_SOURCE_FILE_RECEIVED
PROMPT ========================================================================= PROMPT =========================================================================
-- Log export start -- Log export start
@@ -25,24 +26,21 @@ VALUES ('MARS-956', 'EXPORT_C2D_MPEC_DATA', 'INFO', 'Starting historical C2D MPE
COMMIT; COMMIT;
-- =================================================================== -- ===================================================================
-- TABLE 1: OU_C2D.MPEC_ADMIN -> ODS/C2D/C2D_MPEC_ADMIN -- TABLE 1: OU_C2D.MPEC_ADMIN -> DATA/C2D/C2D_MPEC_ADMIN
-- =================================================================== -- ===================================================================
PROMPT Exporting Table 1/3: OU_C2D.MPEC_ADMIN PROMPT Exporting Table 1/3: OU_C2D.MPEC_ADMIN
PROMPT Target: mrds_data_dev/ODS/C2D/C2D_MPEC_ADMIN PROMPT Target: mrds_data_dev/DATA/C2D/C2D_MPEC_ADMIN
BEGIN BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE( CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA(
pSchemaName => 'OU_C2D', pSchemaName => 'OU_C2D',
pTableName => 'MPEC_ADMIN', pTableName => 'MPEC_ADMIN',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY', -- Adjust if different pKeyColumnName => 'A_ETL_LOAD_SET_FK', -- ETL key for data lookup
pBucketArea => 'DATA', pBucketArea => 'DATA',
pFolderName => 'ODS/C2D/C2D_MPEC_ADMIN', pFolderName => 'DATA/C2D/C2D_MPEC_ADMIN',
pColumnList => NULL, -- Use all columns pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_ADMIN', -- Template for column order
pMinDate => DATE '2020-01-01', -- Full historical range pRegisterExport => TRUE, -- Register files in A_SOURCE_FILE_RECEIVED
pMaxDate => SYSDATE, -- Up to current date
pParallelDegree => 1, -- Sequential for safety
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_ADMIN', -- Template for mapping
pCredentialName => 'DEF_CRED_ARN' pCredentialName => 'DEF_CRED_ARN'
); );
@@ -59,24 +57,21 @@ END;
/ /
-- =================================================================== -- ===================================================================
-- TABLE 2: OU_C2D.MPEC_CONTENT -> ODS/C2D/C2D_MPEC_CONTENT -- TABLE 2: OU_C2D.MPEC_CONTENT -> DATA/C2D/C2D_MPEC_CONTENT
-- =================================================================== -- ===================================================================
PROMPT Exporting Table 2/3: OU_C2D.MPEC_CONTENT PROMPT Exporting Table 2/3: OU_C2D.MPEC_CONTENT
PROMPT Target: mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT PROMPT Target: mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT
BEGIN BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE( CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA(
pSchemaName => 'OU_C2D', pSchemaName => 'OU_C2D',
pTableName => 'MPEC_CONTENT', pTableName => 'MPEC_CONTENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY', -- Adjust if different pKeyColumnName => 'A_ETL_LOAD_SET_FK', -- ETL key for data lookup
pBucketArea => 'DATA', pBucketArea => 'DATA',
pFolderName => 'ODS/C2D/C2D_MPEC_CONTENT', pFolderName => 'DATA/C2D/C2D_MPEC_CONTENT',
pColumnList => NULL, -- Use all columns pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT', -- Template for column order
pMinDate => DATE '2020-01-01', -- Full historical range pRegisterExport => TRUE, -- Register files in A_SOURCE_FILE_RECEIVED
pMaxDate => SYSDATE, -- Up to current date
pParallelDegree => 1, -- Sequential for safety
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT', -- Template for mapping
pCredentialName => 'DEF_CRED_ARN' pCredentialName => 'DEF_CRED_ARN'
); );
@@ -93,24 +88,21 @@ END;
/ /
-- =================================================================== -- ===================================================================
-- TABLE 3: OU_C2D.MPEC_CONTENT_CRITERION -> ODS/C2D/C2D_MPEC_CONTENT_CRITERION -- TABLE 3: OU_C2D.MPEC_CONTENT_CRITERION -> DATA/C2D/C2D_MPEC_CONTENT_CRITERION
-- =================================================================== -- ===================================================================
PROMPT Exporting Table 3/3: OU_C2D.MPEC_CONTENT_CRITERION PROMPT Exporting Table 3/3: OU_C2D.MPEC_CONTENT_CRITERION
PROMPT Target: mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION PROMPT Target: mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT_CRITERION
BEGIN BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE( CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA(
pSchemaName => 'OU_C2D', pSchemaName => 'OU_C2D',
pTableName => 'MPEC_CONTENT_CRITERION', pTableName => 'MPEC_CONTENT_CRITERION',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY', -- Adjust if different pKeyColumnName => 'A_ETL_LOAD_SET_FK', -- ETL key for data lookup
pBucketArea => 'DATA', pBucketArea => 'DATA',
pFolderName => 'ODS/C2D/C2D_MPEC_CONTENT_CRITERION', pFolderName => 'DATA/C2D/C2D_MPEC_CONTENT_CRITERION',
pColumnList => NULL, -- Use all columns pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT_CRITERION', -- Template for column order
pMinDate => DATE '2020-01-01', -- Full historical range pRegisterExport => TRUE, -- Register files in A_SOURCE_FILE_RECEIVED
pMaxDate => SYSDATE, -- Up to current date
pParallelDegree => 1, -- Sequential for safety
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_MPEC_CONTENT_CRITERION', -- Template for mapping
pCredentialName => 'DEF_CRED_ARN' pCredentialName => 'DEF_CRED_ARN'
); );

18
MARS_Packages/REL02_POST/MARS-956/README.md
View File

@@ -4,7 +4,7 @@
**Purpose**: One-time export of historical C2D MPEC delta data from operational database (OU_C2D) to DATA bucket as CSV files. **Purpose**: One-time export of historical C2D MPEC delta data from operational database (OU_C2D) to DATA bucket as CSV files.
**Approach**: Use FILE_MANAGER export functionality EXPORT_TABLE_DATA_BY_DATE for bulk data movement. **Approach**: Use DATA_EXPORTER export functionality EXPORT_TABLE_DATA for bulk data movement with file registration.
**Input**: Old tables in OU_C2D operational database **Input**: Old tables in OU_C2D operational database
**Output**: CSV files in DATA bucket **Output**: CSV files in DATA bucket
@@ -13,18 +13,19 @@
## Tables to Export ## Tables to Export
| Source Table (OU_C2D) | Target Location (DATA) | Export Type | Time Dependency | | Source Table (OU_C2D) | Target Location (DATA) | Export Type | Time Dependency |
|------------------------|-------------------------|-------------|-----------------| |------------------------|-------------------------|-------------|------------------|
| `MPEC_ADMIN` | `mrds_data_dev/ODS/C2D/C2D_MPEC_ADMIN` | CSV to DATA | Sync with REL_02 | | `MPEC_ADMIN` | `mrds_data_dev/DATA/C2D/C2D_MPEC_ADMIN` | CSV to DATA | Sync with REL_02 |
| `MPEC_CONTENT` | `mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT` | CSV to DATA | Sync with REL_02 | | `MPEC_CONTENT` | `mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT` | CSV to DATA | Sync with REL_02 |
| `MPEC_CONTENT_CRITERION` | `mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION` | CSV to DATA | Sync with REL_02 | | `MPEC_CONTENT_CRITERION` | `mrds_data_dev/DATA/C2D/C2D_MPEC_CONTENT_CRITERION` | CSV to DATA | Sync with REL_02 |
## Export Strategy ## Export Strategy
- **Format**: CSV files in DATA bucket - **Format**: CSV files in DATA bucket
- **Reason**: Complete history of delta records needed for all queries - **Reason**: Complete history of delta records needed for all queries
- **Method**: `DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE` procedure - **Method**: `DATA_EXPORTER.EXPORT_TABLE_DATA` procedure
- **Bucket Area**: `'DATA'` - **Bucket Area**: `'DATA'`
- **Folder Structure**: `'ODS/C2D/{TABLE_NAME}'` - **Folder Structure**: `'DATA/C2D/{TABLE_NAME}'`
- **File Registration**: Files registered in A_SOURCE_FILE_RECEIVED table
## Installation Steps ## Installation Steps
@@ -47,7 +48,7 @@ MARS-956/
## Prerequisites ## Prerequisites
- OU_C2D schema access for source tables - OU_C2D schema access for source tables
- DATA_EXPORTER package v2.7.4+ deployed - DATA_EXPORTER package v2.7.5+ deployed (with pRegisterExport support)
- DEF_CRED_ARN credentials configured - DEF_CRED_ARN credentials configured
- DATA bucket accessible - DATA bucket accessible
@@ -57,6 +58,7 @@ MARS-956/
2. Verify CSV files created in DATA bucket 2. Verify CSV files created in DATA bucket
3. Validate file structure matches template tables 3. Validate file structure matches template tables
4. Confirm row counts match source tables 4. Confirm row counts match source tables
5. Check file registration in A_SOURCE_FILE_RECEIVED table
## Notes ## Notes

56
MARS_Packages/REL02_POST/MARS-956/validate_export.sql
View File

@@ -85,24 +85,62 @@ GROUP BY owner, table_name
ORDER BY table_name, owner; ORDER BY table_name, owner;
PROMPT PROMPT
PROMPT 4. Export File Validation Commands PROMPT 4. File Registration Validation
PROMPT =========================================================================
-- Check if exported files were registered in A_SOURCE_FILE_RECEIVED
PROMPT Registered export files (last 24 hours):
SELECT A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
ROUND(BYTES/1024, 2) AS SIZE_KB,
PROCESSING_STATUS,
TO_CHAR(RECEPTION_DATE, 'YYYY-MM-DD HH24:MI:SS') AS REGISTERED_TIME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1 -- Last 24 hours
AND (SOURCE_FILE_NAME LIKE '%MPEC_%' OR A_SOURCE_FILE_CONFIG_KEY IN (
SELECT A_SOURCE_FILE_CONFIG_KEY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'C2D' AND TABLE_ID LIKE '%MPEC%'
))
ORDER BY RECEPTION_DATE DESC;
-- Count registered files per config key
PROMPT File registration summary:
SELECT
CASE WHEN A_SOURCE_FILE_CONFIG_KEY = -1 THEN 'Default (no config)'
ELSE 'Config Key: ' || A_SOURCE_FILE_CONFIG_KEY
END AS CONFIG_INFO,
COUNT(*) AS REGISTERED_FILES
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1 -- Last 24 hours
AND (SOURCE_FILE_NAME LIKE '%MPEC_%' OR A_SOURCE_FILE_CONFIG_KEY IN (
SELECT A_SOURCE_FILE_CONFIG_KEY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'C2D' AND TABLE_ID LIKE '%MPEC%'
))
GROUP BY A_SOURCE_FILE_CONFIG_KEY
ORDER BY A_SOURCE_FILE_CONFIG_KEY;
PROMPT
PROMPT 5. Export File Validation Commands
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT To validate exported CSV files, use these OCI CLI commands: PROMPT To validate exported CSV files, use these OCI CLI commands:
PROMPT PROMPT
PROMPT # List exported files PROMPT # List exported files
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC" PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC"
PROMPT PROMPT
PROMPT # Check file sizes PROMPT # Check file sizes
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_ADMIN" PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_ADMIN"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_CONTENT" PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_CONTENT"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_CONTENT_CRITERION" PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_CONTENT_CRITERION"
PROMPT PROMPT
PROMPT # Download sample file for validation PROMPT # Download sample file for validation
PROMPT oci os object get --bucket-name mrds_data_dev --name "ODS/C2D/C2D_MPEC_ADMIN/filename.csv" --file sample.csv PROMPT oci os object get --bucket-name mrds_data_dev --name "DATA/C2D/C2D_MPEC_ADMIN/filename.csv" --file sample.csv
PROMPT PROMPT
PROMPT 5. Data Quality Checks (Manual) PROMPT 6. Data Quality Checks (Manual)
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT Manual verification steps: PROMPT Manual verification steps:
@@ -112,9 +150,10 @@ PROMPT 3. Check data formats (especially dates) match expectations
PROMPT 4. Confirm row counts approximately match source tables PROMPT 4. Confirm row counts approximately match source tables
PROMPT 5. Validate no empty files were created PROMPT 5. Validate no empty files were created
PROMPT 6. Test loading sample data into external tables PROMPT 6. Test loading sample data into external tables
PROMPT 7. Verify file registration entries in A_SOURCE_FILE_RECEIVED
PROMPT PROMPT
PROMPT 6. Next Steps for ODS Integration PROMPT 7. Next Steps for ODS Integration
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT After validation success: PROMPT After validation success:
@@ -136,6 +175,7 @@ PROMPT Review the output above to confirm:
PROMPT ✓ Export processes completed without errors PROMPT ✓ Export processes completed without errors
PROMPT ✓ Source table row counts are reasonable PROMPT ✓ Source table row counts are reasonable
PROMPT ✓ Template tables exist and have matching structure PROMPT ✓ Template tables exist and have matching structure
PROMPT ✓ Exported files registered in A_SOURCE_FILE_RECEIVED table
PROMPT ✓ Manual file validation steps are understood PROMPT ✓ Manual file validation steps are understood
PROMPT PROMPT
PROMPT If any issues found, check export logs and re-run specific exports if needed. PROMPT If any issues found, check export logs and re-run specific exports if needed.