admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor export scripts to use DATA_EXPORTER.EXPORT_TABLE_DATA procedure, update folder structure to DATA/C2D, and add file registration validation in README and validation script.

Browse Source
This commit is contained in:
Grzegorz Michalski
2026-02-11 21:22:37 +01:00
parent 641af7415f
commit f8213b76ab
3 changed files with 83 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
MARS_Packages/REL02_POST/MARS-956/validate_export.sql
View File

@@ -85,24 +85,62 @@ GROUP BY owner, table_name
ORDER BY table_name, owner;
PROMPT
PROMPT 4. Export File Validation Commands
PROMPT 4. File Registration Validation
PROMPT =========================================================================
-- Check if exported files were registered in A_SOURCE_FILE_RECEIVED
PROMPT Registered export files (last 24 hours):
SELECT A_SOURCE_FILE_RECEIVED_KEY,
A_SOURCE_FILE_CONFIG_KEY,
SOURCE_FILE_NAME,
ROUND(BYTES/1024, 2) AS SIZE_KB,
PROCESSING_STATUS,
TO_CHAR(RECEPTION_DATE, 'YYYY-MM-DD HH24:MI:SS') AS REGISTERED_TIME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1 -- Last 24 hours
AND (SOURCE_FILE_NAME LIKE '%MPEC_%' OR A_SOURCE_FILE_CONFIG_KEY IN (
SELECT A_SOURCE_FILE_CONFIG_KEY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'C2D' AND TABLE_ID LIKE '%MPEC%'
))
ORDER BY RECEPTION_DATE DESC;
-- Count registered files per config key
PROMPT File registration summary:
SELECT
CASE WHEN A_SOURCE_FILE_CONFIG_KEY = -1 THEN 'Default (no config)'
ELSE 'Config Key: ' || A_SOURCE_FILE_CONFIG_KEY
END AS CONFIG_INFO,
COUNT(*) AS REGISTERED_FILES
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1 -- Last 24 hours
AND (SOURCE_FILE_NAME LIKE '%MPEC_%' OR A_SOURCE_FILE_CONFIG_KEY IN (
SELECT A_SOURCE_FILE_CONFIG_KEY
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'C2D' AND TABLE_ID LIKE '%MPEC%'
))
GROUP BY A_SOURCE_FILE_CONFIG_KEY
ORDER BY A_SOURCE_FILE_CONFIG_KEY;
PROMPT
PROMPT 5. Export File Validation Commands
PROMPT =========================================================================
PROMPT To validate exported CSV files, use these OCI CLI commands:
PROMPT
PROMPT # List exported files
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC"
PROMPT
PROMPT # Check file sizes
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_ADMIN"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_CONTENT"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "ODS/C2D/C2D_MPEC_CONTENT_CRITERION"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_ADMIN"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_CONTENT"
PROMPT oci os object list --bucket-name mrds_data_dev --prefix "DATA/C2D/C2D_MPEC_CONTENT_CRITERION"
PROMPT
PROMPT # Download sample file for validation
PROMPT oci os object get --bucket-name mrds_data_dev --name "ODS/C2D/C2D_MPEC_ADMIN/filename.csv" --file sample.csv
PROMPT oci os object get --bucket-name mrds_data_dev --name "DATA/C2D/C2D_MPEC_ADMIN/filename.csv" --file sample.csv
PROMPT
PROMPT 5. Data Quality Checks (Manual)
PROMPT 6. Data Quality Checks (Manual)
PROMPT =========================================================================
PROMPT Manual verification steps:
@@ -112,9 +150,10 @@ PROMPT 3. Check data formats (especially dates) match expectations
PROMPT 4. Confirm row counts approximately match source tables
PROMPT 5. Validate no empty files were created
PROMPT 6. Test loading sample data into external tables
PROMPT 7. Verify file registration entries in A_SOURCE_FILE_RECEIVED
PROMPT
PROMPT 6. Next Steps for ODS Integration
PROMPT 7. Next Steps for ODS Integration
PROMPT =========================================================================
PROMPT After validation success:
@@ -136,6 +175,7 @@ PROMPT Review the output above to confirm:
PROMPT ✓ Export processes completed without errors
PROMPT ✓ Source table row counts are reasonable
PROMPT ✓ Template tables exist and have matching structure
PROMPT ✓ Exported files registered in A_SOURCE_FILE_RECEIVED table
PROMPT ✓ Manual file validation steps are understood
PROMPT
PROMPT If any issues found, check export logs and re-run specific exports if needed.