diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql b/MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql index 31b1432..7f86cc2 100644 --- a/MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql @@ -1,13 +1,12 @@ -- ============================================================================ -- MARS-1005-PREHOOK Installation Script 00: DATA_EXPORTER Package -- ============================================================================ --- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.16.0 --- RFC 4180 FIX: Added REPLACE(col, CHR(34), CHR(34)||CHR(34)) in --- buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns. --- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 doubling: --- escape=true uses backslash (\"), no escape leaves raw quotes. --- Pre-doubling in SELECT query produces compliant CSV readable by --- ODS external tables (ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34)). +-- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.17.0 +-- PARQUET FIX: Added pFormat parameter to buildQueryWithDateFormats. +-- REPLACE(col,CHR(34)) now applied only when pFormat=CSV. +-- EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being +-- corrupted (single " doubled to "") in Parquet binary files. +-- v2.16.0 RFC 4180 FIX remains intact for CSV path. -- Schema: CT_MRDS -- Object: PACKAGE DATA_EXPORTER -- ============================================================================ @@ -19,10 +18,10 @@ PROMPT ========================================================================= PROMPT MARS-1005-PREHOOK: Installing CT_MRDS.DATA_EXPORTER Package PROMPT ============================================================================ PROMPT Package: CT_MRDS.DATA_EXPORTER -PROMPT Version: 2.15.0 -> 2.16.0 -PROMPT Change: RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in -PROMPT buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns. -PROMPT Oracle has no native RFC 4180 doubling; pre-double in SELECT query. +PROMPT Version: 2.16.0 -> 2.17.0 +PROMPT Change: PARQUET FIX - pFormat param added to buildQueryWithDateFormats. +PROMPT REPLACE(col,CHR(34)) applied only when pFormat=CSV. +PROMPT Parquet path no longer corrupts strings containing double quotes. PROMPT ============================================================================ PROMPT @@ -47,7 +46,7 @@ PROMPT PROMPT PROMPT ============================================================================ -PROMPT DATA_EXPORTER Package installation completed (v2.15.0) +PROMPT DATA_EXPORTER Package installation completed (v2.17.0) PROMPT ============================================================================ PROMPT diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/README.md b/MARS_Packages/REL03/MARS-1005-PREHOOK/README.md new file mode 100644 index 0000000..42dc83c --- /dev/null +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/README.md @@ -0,0 +1,115 @@ +# MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance + Parquet Format Support + +## Overview + +Pre-hook for MARS-1005. Deploys an updated `CT_MRDS.DATA_EXPORTER` package (v2.17.0) +that resolves two export format bugs: + +1. **RFC 4180 compliance (v2.15.0 / v2.16.0):** Previous versions used `escape=true` + in `DBMS_CLOUD.EXPORT_DATA`, producing backslash-escaped embedded quotes (`\"`). + ODS external tables (`FIELDS CSV WITHOUT EMBEDDED`) expect RFC 4180 doubling (`""`). + Fix: removed `escape=true`; implemented `REPLACE(col, '"', '""')` in SELECT query. + +2. **Parquet corruption fix (v2.17.0):** The RFC 4180 `REPLACE` was applied to all + export formats, including Parquet — corrupting values that contained double-quotes. + Fix: added `pFormat` parameter to `buildQueryWithDateFormats`; `REPLACE` is now + applied only when `pFormat = 'CSV'`. Parquet exports pass column values unchanged. + +## Contents + +| File | Description | +|------|-------------| +| `install_mars1005_prehook.sql` | Master installation script (SPOOL, ACCEPT, quit) | +| `rollback_mars1005_prehook.sql` | Master rollback script (SPOOL, ACCEPT, quit) | +| `00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql` | Deploy DATA_EXPORTER v2.17.0 | +| `90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql` | Restore DATA_EXPORTER v2.14.0 | +| `track_package_versions.sql` | Universal version tracking script | +| `verify_packages_version.sql` | Universal package verification script | +| `new_version/DATA_EXPORTER.pkg` | Package specification v2.17.0 | +| `new_version/DATA_EXPORTER.pkb` | Package body v2.17.0 | +| `rollback_version/DATA_EXPORTER.pkg` | Package specification v2.14.0 (backup) | +| `rollback_version/DATA_EXPORTER.pkb` | Package body v2.14.0 (backup) | +| `README.md` | This file | + +## Prerequisites + +- Oracle Database 23ai +- `CT_MRDS.ENV_MANAGER` v3.1.0+ +- ADMIN user with EXECUTE privileges on CT_MRDS schema +- Connection service: `ggmichalski_high` + +## Version Change + +| Package | Before | After | +|---------|--------|-------| +| `CT_MRDS.DATA_EXPORTER` | v2.14.0 | v2.17.0 | + +## Installation + +### Option 1: Master Script (Recommended) + +```powershell +# Execute as ADMIN user for proper privilege management +Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high" +``` + +Log file created automatically: `log/INSTALL_MARS_1005_PREHOOK__.log` + +### Option 2: Individual Scripts + +```powershell +# Execute as ADMIN user +Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high" +``` + +## Verification + +```sql +-- Verify package version +SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL; +-- Expected: 2.17.0 + +-- Verify build info +SELECT CT_MRDS.DATA_EXPORTER.GET_BUILD_INFO() FROM DUAL; + +-- Check for compilation errors +SELECT * FROM ALL_ERRORS +WHERE OWNER = 'CT_MRDS' + AND NAME = 'DATA_EXPORTER'; + +-- Verify no untracked changes +SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'DATA_EXPORTER') FROM DUAL; +-- Expected: OK +``` + +## Rollback + +```powershell +# Execute as ADMIN user +Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high" +``` + +Rollback restores `CT_MRDS.DATA_EXPORTER` to v2.14.0 (re-enables `escape=true`). + +## Testing + +After installation, verify the Parquet and CSV export paths: + +**CSV path (ODS):** Export a table, then SELECT from the corresponding ODS external +table. Values with embedded double-quotes should appear as single `"` characters +and not trigger `ORA-30653`. + +**Parquet path (ARCHIVE):** Export a table containing values with embedded +double-quotes to the ARCHIVE bucket area. Download the Parquet file and confirm +the value is stored verbatim (no extra quotes). + +## Expected Changes + +- `CT_MRDS.DATA_EXPORTER`: v2.14.0 → v2.17.0 +- No table structure changes +- No configuration changes + +## Related + +- MARS-1005: Export TOP allotment data (main issue) +- `CT_MRDS.DATA_EXPORTER` source: `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/packages/DATA_EXPORTER.sql` diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql b/MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql index fe5daed..8db75d2 100644 --- a/MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql @@ -1,13 +1,15 @@ -- =================================================================== -- MARS-1005-PREHOOK INSTALL SCRIPT: Fix DATA_EXPORTER RFC 4180 Compliance -- =================================================================== --- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.15.0) --- that exports CSV files in RFC 4180 compliant format. +-- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.17.0) +-- that fixes RFC 4180 CSV compliance and Parquet format corruption. -- Background: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA, -- which produces backslash-escaped embedded quotes (\") -- instead of RFC 4180 doubling (""). --- ODS external tables (FIELDS CSV WITHOUT EMBEDDED) expect RFC 4180. --- Fix: remove escape=true so Oracle uses RFC 4180 doubling. +-- v2.16.0 fixed CSV by applying REPLACE(col, '"', '""') in SELECT, +-- but this corrupted Parquet exports containing double-quote values. +-- v2.17.0 applies REPLACE only when pFormat = 'CSV', leaving +-- Parquet exports unchanged. -- Author: Grzegorz Michalski -- Date: 2026-03-10 @@ -34,14 +36,14 @@ PROMPT ========================================================================= PROMPT MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance PROMPT ========================================================================= PROMPT -PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA -PROMPT which causes backslash-escaped embedded quotes (\") -PROMPT instead of RFC 4180 doubling (""). ODS external tables with -PROMPT FIELDS CSV WITHOUT EMBEDDED reject rows with backslash-escape. +PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true which produces \"-escaped +PROMPT quotes instead of RFC 4180 doubling "". v2.16.0 fixed CSV but +PROMPT applied REPLACE to Parquet exports too, corrupting quote values. PROMPT PROMPT This script will: -PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.15.0 (removes escape=true parameter) -PROMPT - Exported CSV files will use RFC 4180 doubling ("") for embedded quotes +PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.17.0 (RFC 4180 CSV + Parquet fix) +PROMPT - CSV exports use RFC 4180 doubling ("") for embedded quotes +PROMPT - Parquet exports pass column values unchanged (no REPLACE applied) PROMPT - ODS external tables (FIELDS CSV WITHOUT EMBEDDED) are NOT modified PROMPT PROMPT Expected Duration: 1-2 minutes @@ -60,10 +62,22 @@ WHENEVER SQLERROR CONTINUE PROMPT PROMPT ========================================================================= -PROMPT Step 1: Deploy DATA_EXPORTER v2.15.0 (RFC 4180 Fix) +PROMPT Step 1: Deploy DATA_EXPORTER v2.17.0 (Parquet Fix + RFC 4180) PROMPT ========================================================================= @@00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql +PROMPT +PROMPT ========================================================================= +PROMPT Step 2: Track Package Versions +PROMPT ========================================================================= +@@track_package_versions.sql + +PROMPT +PROMPT ========================================================================= +PROMPT Step 3: Verify Package Versions +PROMPT ========================================================================= +@@verify_packages_version.sql + PROMPT PROMPT ========================================================================= PROMPT MARS-1005-PREHOOK Installation - COMPLETED diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkb b/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkb index b4bacbb..93648b0 100644 --- a/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkb +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkb @@ -129,7 +129,8 @@ AS pTableName IN VARCHAR2, pSchemaName IN VARCHAR2, pKeyColumnName IN VARCHAR2, - pTemplateTableName IN VARCHAR2 + pTemplateTableName IN VARCHAR2, + pFormat IN VARCHAR2 DEFAULT 'CSV' ) RETURN VARCHAR2 IS vResult VARCHAR2(32767); vColumns VARCHAR2(32767); @@ -243,14 +244,15 @@ AS vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || 'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol; - -- Other columns: RFC 4180 quote-doubling for character types, as-is for others + -- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others -- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling. -- escape=true -> backslash-escaped (\") - incompatible with ORACLE_LOADER. -- escape=false -> unescaped embedded quotes - also incompatible. - -- Solution: pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote. + -- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote. -- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:. + -- Parquet is binary - no quoting needed; REPLACE would corrupt string data. ELSE - IF vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN + IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || 'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol; ELSE @@ -1091,7 +1093,7 @@ AS VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters); -- Build query with TO_CHAR for date columns (per-column format support) - vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName); + vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET'); ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters); diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkg b/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkg index 1e6a79d..d42270a 100644 --- a/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkg +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkg @@ -9,19 +9,20 @@ AS **/ -- Package Version Information - PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.16.0'; - PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 08:00:00'; + PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0'; + PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 10:00:00'; PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski'; -- Version History (last 3-5 changes) VERSION_HISTORY CONSTANT VARCHAR2(4000) := + 'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) || 'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Oracle DBMS_CLOUD has no native RFC 4180 doubling: escape=true uses backslash, no escape leaves raw quotes. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) || 'v2.15.0 (2026-03-10): INCOMPLETE FIX - Removed escape=true only; embedded quotes still unescaped. Superseded by v2.16.0.' || CHR(10) || - 'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) || - 'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) || - 'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) || - 'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) || - 'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10); + 'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) || + 'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) || + 'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) || + 'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) || + 'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10); cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10); vgMsgTmp VARCHAR2(32000); diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql b/MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql index 5142c1a..d158ead 100644 --- a/MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql @@ -2,7 +2,6 @@ -- MARS-1005-PREHOOK ROLLBACK SCRIPT: Restore DATA_EXPORTER v2.14.0 -- =================================================================== -- Purpose: Rollback for MARS-1005-PREHOOK - Restore DATA_EXPORTER to v2.14.0 --- (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format) -- Author: Grzegorz Michalski -- Date: 2026-03-10 @@ -27,9 +26,11 @@ SET PAUSE OFF PROMPT ========================================================================= PROMPT MARS-1005-PREHOOK: Rollback - Restore DATA_EXPORTER v2.14.0 PROMPT ========================================================================= -PROMPT WARNING: This will reverse all changes from MARS-1005-PREHOOK installation! -PROMPT - Restores CT_MRDS.DATA_EXPORTER to v2.14.0 -PROMPT (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format) +PROMPT This will reverse all changes from MARS-1005-PREHOOK installation. +PROMPT +PROMPT Rollback steps: +PROMPT 1. Restore CT_MRDS.DATA_EXPORTER to v2.14.0 +PROMPT 2. Verify package versions PROMPT ========================================================================= -- Confirm rollback with user @@ -49,6 +50,12 @@ PROMPT Step 1: Restore DATA_EXPORTER v2.14.0 PROMPT ========================================================================= @@90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql +PROMPT +PROMPT ========================================================================= +PROMPT Step 2: Verify Package Versions +PROMPT ========================================================================= +@@verify_packages_version.sql + PROMPT PROMPT ========================================================================= PROMPT MARS-1005-PREHOOK Rollback - COMPLETED diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/track_package_versions.sql b/MARS_Packages/REL03/MARS-1005-PREHOOK/track_package_versions.sql new file mode 100644 index 0000000..c8ae072 --- /dev/null +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/track_package_versions.sql @@ -0,0 +1,86 @@ +-- =================================================================== +-- Simple Package Version Tracking Script +-- =================================================================== +-- Purpose: Track specified Oracle package versions +-- Author: Grzegorz Michalski +-- Date: 2026-03-11 +-- Version: 3.1.0 - List-Based Edition +-- +-- USAGE: +-- 1. Edit package list below (add/remove packages as needed) +-- 2. Include in your install/rollback script: @@track_package_versions.sql +-- =================================================================== + +SET SERVEROUTPUT ON; + +DECLARE + TYPE t_package_rec IS RECORD ( + owner VARCHAR2(50), + package_name VARCHAR2(50), + version VARCHAR2(50) + ); + TYPE t_packages IS TABLE OF t_package_rec; + TYPE t_string_array IS TABLE OF VARCHAR2(100); + + -- =================================================================== + -- PACKAGE LIST - Edit this array to specify packages to track + -- =================================================================== + -- Add or remove entries as needed for your MARS issue + -- Format: 'SCHEMA.PACKAGE_NAME' + -- =================================================================== + vPackageList t_string_array := t_string_array( + 'CT_MRDS.DATA_EXPORTER' + ); + -- =================================================================== + + vPackages t_packages := t_packages(); + vVersion VARCHAR2(50); + vCount NUMBER := 0; + vOwner VARCHAR2(50); + vPackageName VARCHAR2(50); + vDotPos NUMBER; +BEGIN + DBMS_OUTPUT.PUT_LINE('========================================'); + DBMS_OUTPUT.PUT_LINE('Package Version Tracking'); + DBMS_OUTPUT.PUT_LINE('========================================'); + + -- Process each package in the list + FOR i IN 1..vPackageList.COUNT LOOP + vDotPos := INSTR(vPackageList(i), '.'); + IF vDotPos > 0 THEN + vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1); + vPackageName := SUBSTR(vPackageList(i), vDotPos + 1); + ELSE + vOwner := USER; -- Default to current user if no schema specified + vPackageName := vPackageList(i); + END IF; + + BEGIN + -- Get package version + EXECUTE IMMEDIATE + 'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL' + INTO vVersion; + + -- Track the version + CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION( + pPackageOwner => vOwner, + pPackageName => vPackageName, + pPackageVersion => vVersion, + pPackageBuildDate => NULL, -- Will be retrieved from package + pPackageAuthor => NULL -- Will be retrieved from package + ); + + DBMS_OUTPUT.PUT_LINE('SUCCESS: Tracked ' || vOwner || '.' || vPackageName || ' v' || vVersion); + vCount := vCount + 1; + + EXCEPTION + WHEN OTHERS THEN + DBMS_OUTPUT.PUT_LINE('ERROR tracking ' || vOwner || '.' || vPackageName || ': ' || SQLERRM); + END; + END LOOP; + + DBMS_OUTPUT.PUT_LINE('========================================'); + DBMS_OUTPUT.PUT_LINE('Tracked ' || vCount || ' of ' || vPackageList.COUNT || ' packages successfully'); + DBMS_OUTPUT.PUT_LINE('========================================'); +END; +/ diff --git a/MARS_Packages/REL03/MARS-1005-PREHOOK/verify_packages_version.sql b/MARS_Packages/REL03/MARS-1005-PREHOOK/verify_packages_version.sql new file mode 100644 index 0000000..26f2e4d --- /dev/null +++ b/MARS_Packages/REL03/MARS-1005-PREHOOK/verify_packages_version.sql @@ -0,0 +1,62 @@ +-- =================================================================== +-- Universal Package Version Verification Script +-- =================================================================== +-- Purpose: Verify all tracked Oracle packages for code changes +-- Author: Grzegorz Michalski +-- Date: 2026-03-11 +-- Version: 1.0.0 +-- +-- USAGE: +-- Include at the end of install/rollback scripts: @@verify_packages_version.sql +-- +-- OUTPUT: +-- - List of all tracked packages with their current status +-- - OK: Package has not changed since last tracking +-- - WARNING: Package code changed without version update +-- =================================================================== + +SET LINESIZE 200 +SET PAGESIZE 1000 +SET FEEDBACK OFF + +PROMPT +PROMPT ======================================== +PROMPT Package Version Verification +PROMPT ======================================== +PROMPT + +COLUMN PACKAGE_OWNER FORMAT A15 +COLUMN PACKAGE_NAME FORMAT A20 +COLUMN VERSION FORMAT A10 +COLUMN STATUS FORMAT A80 + +SELECT + PACKAGE_OWNER, + PACKAGE_NAME, + PACKAGE_VERSION AS VERSION, + CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS +FROM ( + SELECT + PACKAGE_OWNER, + PACKAGE_NAME, + PACKAGE_VERSION, + ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN + FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING +) +WHERE RN = 1 +ORDER BY PACKAGE_OWNER, PACKAGE_NAME; + +PROMPT +PROMPT ======================================== +PROMPT Verification Complete +PROMPT ======================================== +PROMPT +PROMPT Legend: +PROMPT OK - Package has not changed since last tracking +PROMPT WARNING - Package code changed without version update +PROMPT +PROMPT For detailed hash information, use: +PROMPT SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL; +PROMPT ======================================== + +SET FEEDBACK ON diff --git a/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb b/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb index 6734d50..c474ba2 100644 --- a/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb +++ b/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb @@ -59,7 +59,8 @@ AS pTableName IN VARCHAR2, pSchemaName IN VARCHAR2, pKeyColumnName IN VARCHAR2, - pTemplateTableName IN VARCHAR2 + pTemplateTableName IN VARCHAR2, + pFormat IN VARCHAR2 DEFAULT 'CSV' ) RETURN VARCHAR2 IS vResult VARCHAR2(32767); vColumns VARCHAR2(32767); @@ -173,10 +174,21 @@ AS vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || 'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol; - -- Other columns as-is with T. prefix + -- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others + -- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling. + -- escape=true -> backslash-escaped (\"\) - incompatible with ORACLE_LOADER. + -- escape=false -> unescaped embedded quotes - also incompatible. + -- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote. + -- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:. + -- Parquet is binary - no quoting needed; REPLACE would corrupt string data. ELSE - vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || - 'T.' || vCurrentCol; + IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN + vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || + 'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol; + ELSE + vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || + 'T.' || vCurrentCol; + END IF; END IF; vPos := vNextPos + 1; @@ -808,7 +820,7 @@ AS VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters); -- Build query with TO_CHAR for date columns (per-column format support) - vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName); + vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET'); ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters); diff --git a/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg b/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg index 1209d84..eed375d 100644 --- a/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg +++ b/MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg @@ -9,12 +9,14 @@ AS **/ -- Package Version Information - PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.6.3'; - PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-01-28 19:30:00'; + PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0'; + PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-03-11 10:00:00'; PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team'; -- Version History (last 3-5 changes) VERSION_HISTORY CONSTANT VARCHAR2(4000) := + 'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) || + 'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) || 'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) || 'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) || 'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) ||