admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update DATA_EXPORTER package to v2.17.0: Fix RFC 4180 compliance and Parquet format corruption

Browse Source
This commit is contained in:
Grzegorz Michalski
2026-03-12 08:50:08 +01:00
parent 5ba6c30fda
commit 202b535f9f
10 changed files with 346 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql
View File

@@ -1,13 +1,12 @@
-- ============================================================================ -- ============================================================================
-- MARS-1005-PREHOOK Installation Script 00: DATA_EXPORTER Package -- MARS-1005-PREHOOK Installation Script 00: DATA_EXPORTER Package
-- ============================================================================ -- ============================================================================
-- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.16.0 -- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.17.0
-- RFC 4180 FIX: Added REPLACE(col, CHR(34), CHR(34)||CHR(34)) in -- PARQUET FIX: Added pFormat parameter to buildQueryWithDateFormats.
-- buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns. -- REPLACE(col,CHR(34)) now applied only when pFormat=CSV.
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 doubling: -- EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being
-- escape=true uses backslash (\"), no escape leaves raw quotes. -- corrupted (single " doubled to "") in Parquet binary files.
-- Pre-doubling in SELECT query produces compliant CSV readable by -- v2.16.0 RFC 4180 FIX remains intact for CSV path.
-- ODS external tables (ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34)).
-- Schema: CT_MRDS -- Schema: CT_MRDS
-- Object: PACKAGE DATA_EXPORTER -- Object: PACKAGE DATA_EXPORTER
-- ============================================================================ -- ============================================================================
@@ -19,10 +18,10 @@ PROMPT =========================================================================
PROMPT MARS-1005-PREHOOK: Installing CT_MRDS.DATA_EXPORTER Package PROMPT MARS-1005-PREHOOK: Installing CT_MRDS.DATA_EXPORTER Package
PROMPT ============================================================================ PROMPT ============================================================================
PROMPT Package: CT_MRDS.DATA_EXPORTER PROMPT Package: CT_MRDS.DATA_EXPORTER
PROMPT Version: 2.15.0 -> 2.16.0 PROMPT Version: 2.16.0 -> 2.17.0
PROMPT Change: RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in PROMPT Change: PARQUET FIX - pFormat param added to buildQueryWithDateFormats.
PROMPT buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns. PROMPT REPLACE(col,CHR(34)) applied only when pFormat=CSV.
PROMPT Oracle has no native RFC 4180 doubling; pre-double in SELECT query. PROMPT Parquet path no longer corrupts strings containing double quotes.
PROMPT ============================================================================ PROMPT ============================================================================
PROMPT PROMPT
@@ -47,7 +46,7 @@ PROMPT
PROMPT PROMPT
PROMPT ============================================================================ PROMPT ============================================================================
PROMPT DATA_EXPORTER Package installation completed (v2.15.0) PROMPT DATA_EXPORTER Package installation completed (v2.17.0)
PROMPT ============================================================================ PROMPT ============================================================================
PROMPT PROMPT

115
MARS_Packages/REL03/MARS-1005-PREHOOK/README.md Normal file
View File

@@ -0,0 +1,115 @@
# MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance + Parquet Format Support
## Overview
Pre-hook for MARS-1005. Deploys an updated `CT_MRDS.DATA_EXPORTER` package (v2.17.0)
that resolves two export format bugs:
1. **RFC 4180 compliance (v2.15.0 / v2.16.0):** Previous versions used `escape=true`
in `DBMS_CLOUD.EXPORT_DATA`, producing backslash-escaped embedded quotes (`\"`).
ODS external tables (`FIELDS CSV WITHOUT EMBEDDED`) expect RFC 4180 doubling (`""`).
Fix: removed `escape=true`; implemented `REPLACE(col, '"', '""')` in SELECT query.
2. **Parquet corruption fix (v2.17.0):** The RFC 4180 `REPLACE` was applied to all
export formats, including Parquet — corrupting values that contained double-quotes.
Fix: added `pFormat` parameter to `buildQueryWithDateFormats`; `REPLACE` is now
applied only when `pFormat = 'CSV'`. Parquet exports pass column values unchanged.
## Contents
| File | Description |
|------|-------------|
| `install_mars1005_prehook.sql` | Master installation script (SPOOL, ACCEPT, quit) |
| `rollback_mars1005_prehook.sql` | Master rollback script (SPOOL, ACCEPT, quit) |
| `00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql` | Deploy DATA_EXPORTER v2.17.0 |
| `90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql` | Restore DATA_EXPORTER v2.14.0 |
| `track_package_versions.sql` | Universal version tracking script |
| `verify_packages_version.sql` | Universal package verification script |
| `new_version/DATA_EXPORTER.pkg` | Package specification v2.17.0 |
| `new_version/DATA_EXPORTER.pkb` | Package body v2.17.0 |
| `rollback_version/DATA_EXPORTER.pkg` | Package specification v2.14.0 (backup) |
| `rollback_version/DATA_EXPORTER.pkb` | Package body v2.14.0 (backup) |
| `README.md` | This file |
## Prerequisites
- Oracle Database 23ai
- `CT_MRDS.ENV_MANAGER` v3.1.0+
- ADMIN user with EXECUTE privileges on CT_MRDS schema
- Connection service: `ggmichalski_high`
## Version Change
| Package | Before | After |
|---------|--------|-------|
| `CT_MRDS.DATA_EXPORTER` | v2.14.0 | v2.17.0 |
## Installation
### Option 1: Master Script (Recommended)
```powershell
# Execute as ADMIN user for proper privilege management
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
Log file created automatically: `log/INSTALL_MARS_1005_PREHOOK_<PDB>_<timestamp>.log`
### Option 2: Individual Scripts
```powershell
# Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
## Verification
```sql
-- Verify package version
SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL;
-- Expected: 2.17.0
-- Verify build info
SELECT CT_MRDS.DATA_EXPORTER.GET_BUILD_INFO() FROM DUAL;
-- Check for compilation errors
SELECT * FROM ALL_ERRORS
WHERE OWNER = 'CT_MRDS'
AND NAME = 'DATA_EXPORTER';
-- Verify no untracked changes
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'DATA_EXPORTER') FROM DUAL;
-- Expected: OK
```
## Rollback
```powershell
# Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
Rollback restores `CT_MRDS.DATA_EXPORTER` to v2.14.0 (re-enables `escape=true`).
## Testing
After installation, verify the Parquet and CSV export paths:
**CSV path (ODS):** Export a table, then SELECT from the corresponding ODS external
table. Values with embedded double-quotes should appear as single `"` characters
and not trigger `ORA-30653`.
**Parquet path (ARCHIVE):** Export a table containing values with embedded
double-quotes to the ARCHIVE bucket area. Download the Parquet file and confirm
the value is stored verbatim (no extra quotes).
## Expected Changes
- `CT_MRDS.DATA_EXPORTER`: v2.14.0 → v2.17.0
- No table structure changes
- No configuration changes
## Related
- MARS-1005: Export TOP allotment data (main issue)
- `CT_MRDS.DATA_EXPORTER` source: `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/packages/DATA_EXPORTER.sql`

36
MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql
View File

@@ -1,13 +1,15 @@
-- =================================================================== -- ===================================================================
-- MARS-1005-PREHOOK INSTALL SCRIPT: Fix DATA_EXPORTER RFC 4180 Compliance -- MARS-1005-PREHOOK INSTALL SCRIPT: Fix DATA_EXPORTER RFC 4180 Compliance
-- =================================================================== -- ===================================================================
-- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.15.0) -- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.17.0)
-- that exports CSV files in RFC 4180 compliant format. -- that fixes RFC 4180 CSV compliance and Parquet format corruption.
-- Background: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA, -- Background: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA,
-- which produces backslash-escaped embedded quotes (\") -- which produces backslash-escaped embedded quotes (\")
-- instead of RFC 4180 doubling (""). -- instead of RFC 4180 doubling ("").
-- ODS external tables (FIELDS CSV WITHOUT EMBEDDED) expect RFC 4180. -- v2.16.0 fixed CSV by applying REPLACE(col, '"', '""') in SELECT,
-- Fix: remove escape=true so Oracle uses RFC 4180 doubling. -- but this corrupted Parquet exports containing double-quote values.
-- v2.17.0 applies REPLACE only when pFormat = 'CSV', leaving
-- Parquet exports unchanged.
-- Author: Grzegorz Michalski -- Author: Grzegorz Michalski
-- Date: 2026-03-10 -- Date: 2026-03-10
@@ -34,14 +36,14 @@ PROMPT =========================================================================
PROMPT MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance PROMPT MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT PROMPT
PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true which produces \"-escaped
PROMPT which causes backslash-escaped embedded quotes (\") PROMPT quotes instead of RFC 4180 doubling "". v2.16.0 fixed CSV but
PROMPT instead of RFC 4180 doubling (""). ODS external tables with PROMPT applied REPLACE to Parquet exports too, corrupting quote values.
PROMPT FIELDS CSV WITHOUT EMBEDDED reject rows with backslash-escape.
PROMPT PROMPT
PROMPT This script will: PROMPT This script will:
PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.15.0 (removes escape=true parameter) PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.17.0 (RFC 4180 CSV + Parquet fix)
PROMPT - Exported CSV files will use RFC 4180 doubling ("") for embedded quotes PROMPT - CSV exports use RFC 4180 doubling ("") for embedded quotes
PROMPT - Parquet exports pass column values unchanged (no REPLACE applied)
PROMPT - ODS external tables (FIELDS CSV WITHOUT EMBEDDED) are NOT modified PROMPT - ODS external tables (FIELDS CSV WITHOUT EMBEDDED) are NOT modified
PROMPT PROMPT
PROMPT Expected Duration: 1-2 minutes PROMPT Expected Duration: 1-2 minutes
@@ -60,10 +62,22 @@ WHENEVER SQLERROR CONTINUE
PROMPT PROMPT
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT Step 1: Deploy DATA_EXPORTER v2.15.0 (RFC 4180 Fix) PROMPT Step 1: Deploy DATA_EXPORTER v2.17.0 (Parquet Fix + RFC 4180)
PROMPT ========================================================================= PROMPT =========================================================================
@@00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql @@00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Track Package Versions
PROMPT =========================================================================
@@track_package_versions.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 3: Verify Package Versions
PROMPT =========================================================================
@@verify_packages_version.sql
PROMPT PROMPT
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT MARS-1005-PREHOOK Installation - COMPLETED PROMPT MARS-1005-PREHOOK Installation - COMPLETED

12
MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkb
View File

@@ -129,7 +129,8 @@ AS
pTableName IN VARCHAR2, pTableName IN VARCHAR2,
pSchemaName IN VARCHAR2, pSchemaName IN VARCHAR2,
pKeyColumnName IN VARCHAR2, pKeyColumnName IN VARCHAR2,
pTemplateTableName IN VARCHAR2 pTemplateTableName IN VARCHAR2,
pFormat IN VARCHAR2 DEFAULT 'CSV'
) RETURN VARCHAR2 IS ) RETURN VARCHAR2 IS
vResult VARCHAR2(32767); vResult VARCHAR2(32767);
vColumns VARCHAR2(32767); vColumns VARCHAR2(32767);
@@ -243,14 +244,15 @@ AS
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol; 'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol;
-- Other columns: RFC 4180 quote-doubling for character types, as-is for others -- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling. -- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling.
-- escape=true -> backslash-escaped (\") - incompatible with ORACLE_LOADER. -- escape=true -> backslash-escaped (\") - incompatible with ORACLE_LOADER.
-- escape=false -> unescaped embedded quotes - also incompatible. -- escape=false -> unescaped embedded quotes - also incompatible.
-- Solution: pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote. -- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote.
-- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:. -- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:.
-- Parquet is binary - no quoting needed; REPLACE would corrupt string data.
ELSE ELSE
IF vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol; 'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol;
ELSE ELSE
@@ -1091,7 +1093,7 @@ AS
VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters); VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters);
-- Build query with TO_CHAR for date columns (per-column format support) -- Build query with TO_CHAR for date columns (per-column format support)
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName); vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET');
ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters);
ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters);

15
MARS_Packages/REL03/MARS-1005-PREHOOK/new_version/DATA_EXPORTER.pkg
View File

@@ -9,19 +9,20 @@ AS
**/ **/
-- Package Version Information -- Package Version Information
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.16.0'; PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 08:00:00'; PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 10:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski'; PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (last 3-5 changes) -- Version History (last 3-5 changes)
VERSION_HISTORY CONSTANT VARCHAR2(4000) := VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) ||
'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Oracle DBMS_CLOUD has no native RFC 4180 doubling: escape=true uses backslash, no escape leaves raw quotes. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) || 'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Oracle DBMS_CLOUD has no native RFC 4180 doubling: escape=true uses backslash, no escape leaves raw quotes. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) ||
'v2.15.0 (2026-03-10): INCOMPLETE FIX - Removed escape=true only; embedded quotes still unescaped. Superseded by v2.16.0.' || CHR(10) || 'v2.15.0 (2026-03-10): INCOMPLETE FIX - Removed escape=true only; embedded quotes still unescaped. Superseded by v2.16.0.' || CHR(10) ||
'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) || 'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) ||
'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) || 'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) ||
'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) || 'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) ||
'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) || 'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) ||
'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10); 'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10); cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgMsgTmp VARCHAR2(32000); vgMsgTmp VARCHAR2(32000);

15
MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql
View File

@@ -2,7 +2,6 @@
-- MARS-1005-PREHOOK ROLLBACK SCRIPT: Restore DATA_EXPORTER v2.14.0 -- MARS-1005-PREHOOK ROLLBACK SCRIPT: Restore DATA_EXPORTER v2.14.0
-- =================================================================== -- ===================================================================
-- Purpose: Rollback for MARS-1005-PREHOOK - Restore DATA_EXPORTER to v2.14.0 -- Purpose: Rollback for MARS-1005-PREHOOK - Restore DATA_EXPORTER to v2.14.0
-- (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format)
-- Author: Grzegorz Michalski -- Author: Grzegorz Michalski
-- Date: 2026-03-10 -- Date: 2026-03-10
@@ -27,9 +26,11 @@ SET PAUSE OFF
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT MARS-1005-PREHOOK: Rollback - Restore DATA_EXPORTER v2.14.0 PROMPT MARS-1005-PREHOOK: Rollback - Restore DATA_EXPORTER v2.14.0
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT WARNING: This will reverse all changes from MARS-1005-PREHOOK installation! PROMPT This will reverse all changes from MARS-1005-PREHOOK installation.
PROMPT - Restores CT_MRDS.DATA_EXPORTER to v2.14.0 PROMPT
PROMPT (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format) PROMPT Rollback steps:
PROMPT 1. Restore CT_MRDS.DATA_EXPORTER to v2.14.0
PROMPT 2. Verify package versions
PROMPT ========================================================================= PROMPT =========================================================================
-- Confirm rollback with user -- Confirm rollback with user
@@ -49,6 +50,12 @@ PROMPT Step 1: Restore DATA_EXPORTER v2.14.0
PROMPT ========================================================================= PROMPT =========================================================================
@@90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql @@90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Verify Package Versions
PROMPT =========================================================================
@@verify_packages_version.sql
PROMPT PROMPT
PROMPT ========================================================================= PROMPT =========================================================================
PROMPT MARS-1005-PREHOOK Rollback - COMPLETED PROMPT MARS-1005-PREHOOK Rollback - COMPLETED

86
MARS_Packages/REL03/MARS-1005-PREHOOK/track_package_versions.sql Normal file
View File

@@ -0,0 +1,86 @@
-- ===================================================================
-- Simple Package Version Tracking Script
-- ===================================================================
-- Purpose: Track specified Oracle package versions
-- Author: Grzegorz Michalski
-- Date: 2026-03-11
-- Version: 3.1.0 - List-Based Edition
--
-- USAGE:
-- 1. Edit package list below (add/remove packages as needed)
-- 2. Include in your install/rollback script: @@track_package_versions.sql
-- ===================================================================
SET SERVEROUTPUT ON;
DECLARE
TYPE t_package_rec IS RECORD (
owner VARCHAR2(50),
package_name VARCHAR2(50),
version VARCHAR2(50)
);
TYPE t_packages IS TABLE OF t_package_rec;
TYPE t_string_array IS TABLE OF VARCHAR2(100);
-- ===================================================================
-- PACKAGE LIST - Edit this array to specify packages to track
-- ===================================================================
-- Add or remove entries as needed for your MARS issue
-- Format: 'SCHEMA.PACKAGE_NAME'
-- ===================================================================
vPackageList t_string_array := t_string_array(
'CT_MRDS.DATA_EXPORTER'
);
-- ===================================================================
vPackages t_packages := t_packages();
vVersion VARCHAR2(50);
vCount NUMBER := 0;
vOwner VARCHAR2(50);
vPackageName VARCHAR2(50);
vDotPos NUMBER;
BEGIN
DBMS_OUTPUT.PUT_LINE('========================================');
DBMS_OUTPUT.PUT_LINE('Package Version Tracking');
DBMS_OUTPUT.PUT_LINE('========================================');
-- Process each package in the list
FOR i IN 1..vPackageList.COUNT LOOP
vDotPos := INSTR(vPackageList(i), '.');
IF vDotPos > 0 THEN
vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1);
vPackageName := SUBSTR(vPackageList(i), vDotPos + 1);
ELSE
vOwner := USER; -- Default to current user if no schema specified
vPackageName := vPackageList(i);
END IF;
BEGIN
-- Get package version
EXECUTE IMMEDIATE
'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL'
INTO vVersion;
-- Track the version
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
pPackageOwner => vOwner,
pPackageName => vPackageName,
pPackageVersion => vVersion,
pPackageBuildDate => NULL, -- Will be retrieved from package
pPackageAuthor => NULL -- Will be retrieved from package
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Tracked ' || vOwner || '.' || vPackageName || ' v' || vVersion);
vCount := vCount + 1;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('ERROR tracking ' || vOwner || '.' || vPackageName || ': ' || SQLERRM);
END;
END LOOP;
DBMS_OUTPUT.PUT_LINE('========================================');
DBMS_OUTPUT.PUT_LINE('Tracked ' || vCount || ' of ' || vPackageList.COUNT || ' packages successfully');
DBMS_OUTPUT.PUT_LINE('========================================');
END;
/

62
MARS_Packages/REL03/MARS-1005-PREHOOK/verify_packages_version.sql Normal file
View File

@@ -0,0 +1,62 @@
-- ===================================================================
-- Universal Package Version Verification Script
-- ===================================================================
-- Purpose: Verify all tracked Oracle packages for code changes
-- Author: Grzegorz Michalski
-- Date: 2026-03-11
-- Version: 1.0.0
--
-- USAGE:
-- Include at the end of install/rollback scripts: @@verify_packages_version.sql
--
-- OUTPUT:
-- - List of all tracked packages with their current status
-- - OK: Package has not changed since last tracking
-- - WARNING: Package code changed without version update
-- ===================================================================
SET LINESIZE 200
SET PAGESIZE 1000
SET FEEDBACK OFF
PROMPT
PROMPT ========================================
PROMPT Package Version Verification
PROMPT ========================================
PROMPT
COLUMN PACKAGE_OWNER FORMAT A15
COLUMN PACKAGE_NAME FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN STATUS FORMAT A80
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION AS VERSION,
CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS
FROM (
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION,
ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN
FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING
)
WHERE RN = 1
ORDER BY PACKAGE_OWNER, PACKAGE_NAME;
PROMPT
PROMPT ========================================
PROMPT Verification Complete
PROMPT ========================================
PROMPT
PROMPT Legend:
PROMPT OK - Package has not changed since last tracking
PROMPT WARNING - Package code changed without version update
PROMPT
PROMPT For detailed hash information, use:
PROMPT SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;
PROMPT ========================================
SET FEEDBACK ON

22
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkb
View File

@@ -59,7 +59,8 @@ AS
pTableName IN VARCHAR2, pTableName IN VARCHAR2,
pSchemaName IN VARCHAR2, pSchemaName IN VARCHAR2,
pKeyColumnName IN VARCHAR2, pKeyColumnName IN VARCHAR2,
pTemplateTableName IN VARCHAR2 pTemplateTableName IN VARCHAR2,
pFormat IN VARCHAR2 DEFAULT 'CSV'
) RETURN VARCHAR2 IS ) RETURN VARCHAR2 IS
vResult VARCHAR2(32767); vResult VARCHAR2(32767);
vColumns VARCHAR2(32767); vColumns VARCHAR2(32767);
@@ -173,10 +174,21 @@ AS
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol; 'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol;
-- Other columns as-is with T. prefix -- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling.
-- escape=true -> backslash-escaped (\"\) - incompatible with ORACLE_LOADER.
-- escape=false -> unescaped embedded quotes - also incompatible.
-- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote.
-- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:.
-- Parquet is binary - no quoting needed; REPLACE would corrupt string data.
ELSE ELSE
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END || IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN
'T.' || vCurrentCol; vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol;
ELSE
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
'T.' || vCurrentCol;
END IF;
END IF; END IF;
vPos := vNextPos + 1; vPos := vNextPos + 1;
@@ -808,7 +820,7 @@ AS
VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters); VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters);
-- Build query with TO_CHAR for date columns (per-column format support) -- Build query with TO_CHAR for date columns (per-column format support)
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName); vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET');
ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters);
ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters); ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters);

6
MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/SCHEMA/packages/DATA_EXPORTER.pkg
View File

@@ -9,12 +9,14 @@ AS
**/ **/
-- Package Version Information -- Package Version Information
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.6.3'; PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-01-28 19:30:00'; PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-03-11 10:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team'; PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team';
-- Version History (last 3-5 changes) -- Version History (last 3-5 changes)
VERSION_HISTORY CONSTANT VARCHAR2(4000) := VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) ||
'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) ||
'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) || 'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) ||
'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) || 'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) ||
'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) || 'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) ||