Compare commits
2 Commits
5ba6c30fda
...
7d2fb34ad9
| Author | SHA1 | Date | |
|---|---|---|---|
|
7d2fb34ad9 | ||
|
|
202b535f9f |
@@ -1,13 +1,12 @@
|
||||
-- ============================================================================
|
||||
-- MARS-1005-PREHOOK Installation Script 00: DATA_EXPORTER Package
|
||||
-- ============================================================================
|
||||
-- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.16.0
|
||||
-- RFC 4180 FIX: Added REPLACE(col, CHR(34), CHR(34)||CHR(34)) in
|
||||
-- buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns.
|
||||
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 doubling:
|
||||
-- escape=true uses backslash (\"), no escape leaves raw quotes.
|
||||
-- Pre-doubling in SELECT query produces compliant CSV readable by
|
||||
-- ODS external tables (ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34)).
|
||||
-- Purpose: Deploy updated DATA_EXPORTER package (SPEC + BODY) v2.17.0
|
||||
-- PARQUET FIX: Added pFormat parameter to buildQueryWithDateFormats.
|
||||
-- REPLACE(col,CHR(34)) now applied only when pFormat=CSV.
|
||||
-- EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being
|
||||
-- corrupted (single " doubled to "") in Parquet binary files.
|
||||
-- v2.16.0 RFC 4180 FIX remains intact for CSV path.
|
||||
-- Schema: CT_MRDS
|
||||
-- Object: PACKAGE DATA_EXPORTER
|
||||
-- ============================================================================
|
||||
@@ -19,10 +18,10 @@ PROMPT =========================================================================
|
||||
PROMPT MARS-1005-PREHOOK: Installing CT_MRDS.DATA_EXPORTER Package
|
||||
PROMPT ============================================================================
|
||||
PROMPT Package: CT_MRDS.DATA_EXPORTER
|
||||
PROMPT Version: 2.15.0 -> 2.16.0
|
||||
PROMPT Change: RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in
|
||||
PROMPT buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB columns.
|
||||
PROMPT Oracle has no native RFC 4180 doubling; pre-double in SELECT query.
|
||||
PROMPT Version: 2.16.0 -> 2.17.0
|
||||
PROMPT Change: PARQUET FIX - pFormat param added to buildQueryWithDateFormats.
|
||||
PROMPT REPLACE(col,CHR(34)) applied only when pFormat=CSV.
|
||||
PROMPT Parquet path no longer corrupts strings containing double quotes.
|
||||
PROMPT ============================================================================
|
||||
|
||||
PROMPT
|
||||
@@ -47,7 +46,7 @@ PROMPT
|
||||
|
||||
PROMPT
|
||||
PROMPT ============================================================================
|
||||
PROMPT DATA_EXPORTER Package installation completed (v2.15.0)
|
||||
PROMPT DATA_EXPORTER Package installation completed (v2.17.0)
|
||||
PROMPT ============================================================================
|
||||
PROMPT
|
||||
|
||||
|
||||
115
MARS_Packages/REL03/MARS-1005-PREHOOK/README.md
Normal file
115
MARS_Packages/REL03/MARS-1005-PREHOOK/README.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance + Parquet Format Support
|
||||
|
||||
## Overview
|
||||
|
||||
Pre-hook for MARS-1005. Deploys an updated `CT_MRDS.DATA_EXPORTER` package (v2.17.0)
|
||||
that resolves two export format bugs:
|
||||
|
||||
1. **RFC 4180 compliance (v2.15.0 / v2.16.0):** Previous versions used `escape=true`
|
||||
in `DBMS_CLOUD.EXPORT_DATA`, producing backslash-escaped embedded quotes (`\"`).
|
||||
ODS external tables (`FIELDS CSV WITHOUT EMBEDDED`) expect RFC 4180 doubling (`""`).
|
||||
Fix: removed `escape=true`; implemented `REPLACE(col, '"', '""')` in SELECT query.
|
||||
|
||||
2. **Parquet corruption fix (v2.17.0):** The RFC 4180 `REPLACE` was applied to all
|
||||
export formats, including Parquet — corrupting values that contained double-quotes.
|
||||
Fix: added `pFormat` parameter to `buildQueryWithDateFormats`; `REPLACE` is now
|
||||
applied only when `pFormat = 'CSV'`. Parquet exports pass column values unchanged.
|
||||
|
||||
## Contents
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `install_mars1005_prehook.sql` | Master installation script (SPOOL, ACCEPT, quit) |
|
||||
| `rollback_mars1005_prehook.sql` | Master rollback script (SPOOL, ACCEPT, quit) |
|
||||
| `00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql` | Deploy DATA_EXPORTER v2.17.0 |
|
||||
| `90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql` | Restore DATA_EXPORTER v2.14.0 |
|
||||
| `track_package_versions.sql` | Universal version tracking script |
|
||||
| `verify_packages_version.sql` | Universal package verification script |
|
||||
| `new_version/DATA_EXPORTER.pkg` | Package specification v2.17.0 |
|
||||
| `new_version/DATA_EXPORTER.pkb` | Package body v2.17.0 |
|
||||
| `rollback_version/DATA_EXPORTER.pkg` | Package specification v2.14.0 (backup) |
|
||||
| `rollback_version/DATA_EXPORTER.pkb` | Package body v2.14.0 (backup) |
|
||||
| `README.md` | This file |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Oracle Database 23ai
|
||||
- `CT_MRDS.ENV_MANAGER` v3.1.0+
|
||||
- ADMIN user with EXECUTE privileges on CT_MRDS schema
|
||||
- Connection service: `ggmichalski_high`
|
||||
|
||||
## Version Change
|
||||
|
||||
| Package | Before | After |
|
||||
|---------|--------|-------|
|
||||
| `CT_MRDS.DATA_EXPORTER` | v2.14.0 | v2.17.0 |
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: Master Script (Recommended)
|
||||
|
||||
```powershell
|
||||
# Execute as ADMIN user for proper privilege management
|
||||
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
||||
```
|
||||
|
||||
Log file created automatically: `log/INSTALL_MARS_1005_PREHOOK_<PDB>_<timestamp>.log`
|
||||
|
||||
### Option 2: Individual Scripts
|
||||
|
||||
```powershell
|
||||
# Execute as ADMIN user
|
||||
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```sql
|
||||
-- Verify package version
|
||||
SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL;
|
||||
-- Expected: 2.17.0
|
||||
|
||||
-- Verify build info
|
||||
SELECT CT_MRDS.DATA_EXPORTER.GET_BUILD_INFO() FROM DUAL;
|
||||
|
||||
-- Check for compilation errors
|
||||
SELECT * FROM ALL_ERRORS
|
||||
WHERE OWNER = 'CT_MRDS'
|
||||
AND NAME = 'DATA_EXPORTER';
|
||||
|
||||
-- Verify no untracked changes
|
||||
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'DATA_EXPORTER') FROM DUAL;
|
||||
-- Expected: OK
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
```powershell
|
||||
# Execute as ADMIN user
|
||||
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
||||
```
|
||||
|
||||
Rollback restores `CT_MRDS.DATA_EXPORTER` to v2.14.0 (re-enables `escape=true`).
|
||||
|
||||
## Testing
|
||||
|
||||
After installation, verify the Parquet and CSV export paths:
|
||||
|
||||
**CSV path (ODS):** Export a table, then SELECT from the corresponding ODS external
|
||||
table. Values with embedded double-quotes should appear as single `"` characters
|
||||
and not trigger `ORA-30653`.
|
||||
|
||||
**Parquet path (ARCHIVE):** Export a table containing values with embedded
|
||||
double-quotes to the ARCHIVE bucket area. Download the Parquet file and confirm
|
||||
the value is stored verbatim (no extra quotes).
|
||||
|
||||
## Expected Changes
|
||||
|
||||
- `CT_MRDS.DATA_EXPORTER`: v2.14.0 → v2.17.0
|
||||
- No table structure changes
|
||||
- No configuration changes
|
||||
|
||||
## Related
|
||||
|
||||
- MARS-1005: Export TOP allotment data (main issue)
|
||||
- `CT_MRDS.DATA_EXPORTER` source: `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/packages/DATA_EXPORTER.sql`
|
||||
@@ -1,13 +1,15 @@
|
||||
-- ===================================================================
|
||||
-- MARS-1005-PREHOOK INSTALL SCRIPT: Fix DATA_EXPORTER RFC 4180 Compliance
|
||||
-- ===================================================================
|
||||
-- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.15.0)
|
||||
-- that exports CSV files in RFC 4180 compliant format.
|
||||
-- Purpose: Pre-hook for MARS-1005 - Deploy updated DATA_EXPORTER (v2.17.0)
|
||||
-- that fixes RFC 4180 CSV compliance and Parquet format corruption.
|
||||
-- Background: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA,
|
||||
-- which produces backslash-escaped embedded quotes (\")
|
||||
-- instead of RFC 4180 doubling ("").
|
||||
-- ODS external tables (FIELDS CSV WITHOUT EMBEDDED) expect RFC 4180.
|
||||
-- Fix: remove escape=true so Oracle uses RFC 4180 doubling.
|
||||
-- v2.16.0 fixed CSV by applying REPLACE(col, '"', '""') in SELECT,
|
||||
-- but this corrupted Parquet exports containing double-quote values.
|
||||
-- v2.17.0 applies REPLACE only when pFormat = 'CSV', leaving
|
||||
-- Parquet exports unchanged.
|
||||
-- Author: Grzegorz Michalski
|
||||
-- Date: 2026-03-10
|
||||
|
||||
@@ -34,14 +36,14 @@ PROMPT =========================================================================
|
||||
PROMPT MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance
|
||||
PROMPT =========================================================================
|
||||
PROMPT
|
||||
PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true in DBMS_CLOUD.EXPORT_DATA
|
||||
PROMPT which causes backslash-escaped embedded quotes (\")
|
||||
PROMPT instead of RFC 4180 doubling (""). ODS external tables with
|
||||
PROMPT FIELDS CSV WITHOUT EMBEDDED reject rows with backslash-escape.
|
||||
PROMPT Problem: DATA_EXPORTER v2.14.0 uses escape=true which produces \"-escaped
|
||||
PROMPT quotes instead of RFC 4180 doubling "". v2.16.0 fixed CSV but
|
||||
PROMPT applied REPLACE to Parquet exports too, corrupting quote values.
|
||||
PROMPT
|
||||
PROMPT This script will:
|
||||
PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.15.0 (removes escape=true parameter)
|
||||
PROMPT - Exported CSV files will use RFC 4180 doubling ("") for embedded quotes
|
||||
PROMPT - Deploy CT_MRDS.DATA_EXPORTER v2.17.0 (RFC 4180 CSV + Parquet fix)
|
||||
PROMPT - CSV exports use RFC 4180 doubling ("") for embedded quotes
|
||||
PROMPT - Parquet exports pass column values unchanged (no REPLACE applied)
|
||||
PROMPT - ODS external tables (FIELDS CSV WITHOUT EMBEDDED) are NOT modified
|
||||
PROMPT
|
||||
PROMPT Expected Duration: 1-2 minutes
|
||||
@@ -60,10 +62,22 @@ WHENEVER SQLERROR CONTINUE
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step 1: Deploy DATA_EXPORTER v2.15.0 (RFC 4180 Fix)
|
||||
PROMPT Step 1: Deploy DATA_EXPORTER v2.17.0 (Parquet Fix + RFC 4180)
|
||||
PROMPT =========================================================================
|
||||
@@00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step 2: Track Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@track_package_versions.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step 3: Verify Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@verify_packages_version.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-1005-PREHOOK Installation - COMPLETED
|
||||
|
||||
@@ -129,7 +129,8 @@ AS
|
||||
pTableName IN VARCHAR2,
|
||||
pSchemaName IN VARCHAR2,
|
||||
pKeyColumnName IN VARCHAR2,
|
||||
pTemplateTableName IN VARCHAR2
|
||||
pTemplateTableName IN VARCHAR2,
|
||||
pFormat IN VARCHAR2 DEFAULT 'CSV'
|
||||
) RETURN VARCHAR2 IS
|
||||
vResult VARCHAR2(32767);
|
||||
vColumns VARCHAR2(32767);
|
||||
@@ -243,14 +244,15 @@ AS
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol;
|
||||
|
||||
-- Other columns: RFC 4180 quote-doubling for character types, as-is for others
|
||||
-- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others
|
||||
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling.
|
||||
-- escape=true -> backslash-escaped (\") - incompatible with ORACLE_LOADER.
|
||||
-- escape=false -> unescaped embedded quotes - also incompatible.
|
||||
-- Solution: pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote.
|
||||
-- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote.
|
||||
-- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:.
|
||||
-- Parquet is binary - no quoting needed; REPLACE would corrupt string data.
|
||||
ELSE
|
||||
IF vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN
|
||||
IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol;
|
||||
ELSE
|
||||
@@ -1091,7 +1093,7 @@ AS
|
||||
VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters);
|
||||
|
||||
-- Build query with TO_CHAR for date columns (per-column format support)
|
||||
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName);
|
||||
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET');
|
||||
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters);
|
||||
|
||||
@@ -9,19 +9,20 @@ AS
|
||||
**/
|
||||
|
||||
-- Package Version Information
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.16.0';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 08:00:00';
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2026-03-11 10:00:00';
|
||||
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
|
||||
|
||||
-- Version History (last 3-5 changes)
|
||||
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
|
||||
'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) ||
|
||||
'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Oracle DBMS_CLOUD has no native RFC 4180 doubling: escape=true uses backslash, no escape leaves raw quotes. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) ||
|
||||
'v2.15.0 (2026-03-10): INCOMPLETE FIX - Removed escape=true only; embedded quotes still unescaped. Superseded by v2.16.0.' || CHR(10) ||
|
||||
'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) ||
|
||||
'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) ||
|
||||
'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) ||
|
||||
'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) ||
|
||||
'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10);
|
||||
'v2.14.0 (2026-02-25): OPTIMIZATION - Added pTaskName parameter to EXPORT_PARTITION_PARALLEL for deterministic filtering. Replaced FETCH FIRST 1 ROW ONLY safeguard with precise WHERE CHUNK_ID AND TASK_NAME filter. Eliminates ORDER BY overhead and provides cleaner session isolation.' || CHR(10) ||
|
||||
'v2.13.1 (2026-02-25): CRITICAL FIX - Added START_ID and END_ID aliasses in CREATE_CHUNKS_BY_SQL to avoid ORA-00960 ambiguous column naming error.' || CHR(10) ||
|
||||
'v2.13.0 (2026-02-25): CRITICAL SESSION ISOLATION FIX - Changed CREATE_CHUNKS_BY_NUMBER_COL to CREATE_CHUNKS_BY_SQL with TASK_NAME filter (fixes ORA-01422 in concurrent sessions). Added ORDER BY CREATED_DATE DESC FETCH FIRST 1 ROW safeguard to EXPORT_PARTITION_PARALLEL SELECT. Composite PK (TASK_NAME, CHUNK_ID) now fully functional.' || CHR(10) ||
|
||||
'v2.12.0 (2026-02-24): CRITICAL FIX - Rewritten DELETE_FAILED_EXPORT_FILE to use file-specific pattern matching (prevents deleting parallel CSV chunks in shared folder). Added vQuery logging before DBMS_CLOUD calls. Added CSV maxfilesize logging.' || CHR(10) ||
|
||||
'v2.11.0 (2026-02-18): Added pJobClass parameter to EXPORT_TABLE_DATA_BY_DATE and EXPORT_TABLE_DATA_TO_CSV_BY_DATE for Oracle Scheduler job class support (resource/priority management).' || CHR(10);
|
||||
|
||||
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
|
||||
vgMsgTmp VARCHAR2(32000);
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
-- MARS-1005-PREHOOK ROLLBACK SCRIPT: Restore DATA_EXPORTER v2.14.0
|
||||
-- ===================================================================
|
||||
-- Purpose: Rollback for MARS-1005-PREHOOK - Restore DATA_EXPORTER to v2.14.0
|
||||
-- (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format)
|
||||
-- Author: Grzegorz Michalski
|
||||
-- Date: 2026-03-10
|
||||
|
||||
@@ -27,9 +26,11 @@ SET PAUSE OFF
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-1005-PREHOOK: Rollback - Restore DATA_EXPORTER v2.14.0
|
||||
PROMPT =========================================================================
|
||||
PROMPT WARNING: This will reverse all changes from MARS-1005-PREHOOK installation!
|
||||
PROMPT - Restores CT_MRDS.DATA_EXPORTER to v2.14.0
|
||||
PROMPT (re-enables escape=true in DBMS_CLOUD.EXPORT_DATA CSV format)
|
||||
PROMPT This will reverse all changes from MARS-1005-PREHOOK installation.
|
||||
PROMPT
|
||||
PROMPT Rollback steps:
|
||||
PROMPT 1. Restore CT_MRDS.DATA_EXPORTER to v2.14.0
|
||||
PROMPT 2. Verify package versions
|
||||
PROMPT =========================================================================
|
||||
|
||||
-- Confirm rollback with user
|
||||
@@ -49,6 +50,12 @@ PROMPT Step 1: Restore DATA_EXPORTER v2.14.0
|
||||
PROMPT =========================================================================
|
||||
@@90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step 2: Verify Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@verify_packages_version.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-1005-PREHOOK Rollback - COMPLETED
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
-- ===================================================================
|
||||
-- Simple Package Version Tracking Script
|
||||
-- ===================================================================
|
||||
-- Purpose: Track specified Oracle package versions
|
||||
-- Author: Grzegorz Michalski
|
||||
-- Date: 2026-03-11
|
||||
-- Version: 3.1.0 - List-Based Edition
|
||||
--
|
||||
-- USAGE:
|
||||
-- 1. Edit package list below (add/remove packages as needed)
|
||||
-- 2. Include in your install/rollback script: @@track_package_versions.sql
|
||||
-- ===================================================================
|
||||
|
||||
SET SERVEROUTPUT ON;
|
||||
|
||||
DECLARE
|
||||
TYPE t_package_rec IS RECORD (
|
||||
owner VARCHAR2(50),
|
||||
package_name VARCHAR2(50),
|
||||
version VARCHAR2(50)
|
||||
);
|
||||
TYPE t_packages IS TABLE OF t_package_rec;
|
||||
TYPE t_string_array IS TABLE OF VARCHAR2(100);
|
||||
|
||||
-- ===================================================================
|
||||
-- PACKAGE LIST - Edit this array to specify packages to track
|
||||
-- ===================================================================
|
||||
-- Add or remove entries as needed for your MARS issue
|
||||
-- Format: 'SCHEMA.PACKAGE_NAME'
|
||||
-- ===================================================================
|
||||
vPackageList t_string_array := t_string_array(
|
||||
'CT_MRDS.DATA_EXPORTER'
|
||||
);
|
||||
-- ===================================================================
|
||||
|
||||
vPackages t_packages := t_packages();
|
||||
vVersion VARCHAR2(50);
|
||||
vCount NUMBER := 0;
|
||||
vOwner VARCHAR2(50);
|
||||
vPackageName VARCHAR2(50);
|
||||
vDotPos NUMBER;
|
||||
BEGIN
|
||||
DBMS_OUTPUT.PUT_LINE('========================================');
|
||||
DBMS_OUTPUT.PUT_LINE('Package Version Tracking');
|
||||
DBMS_OUTPUT.PUT_LINE('========================================');
|
||||
|
||||
-- Process each package in the list
|
||||
FOR i IN 1..vPackageList.COUNT LOOP
|
||||
vDotPos := INSTR(vPackageList(i), '.');
|
||||
IF vDotPos > 0 THEN
|
||||
vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1);
|
||||
vPackageName := SUBSTR(vPackageList(i), vDotPos + 1);
|
||||
ELSE
|
||||
vOwner := USER; -- Default to current user if no schema specified
|
||||
vPackageName := vPackageList(i);
|
||||
END IF;
|
||||
|
||||
BEGIN
|
||||
-- Get package version
|
||||
EXECUTE IMMEDIATE
|
||||
'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL'
|
||||
INTO vVersion;
|
||||
|
||||
-- Track the version
|
||||
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
|
||||
pPackageOwner => vOwner,
|
||||
pPackageName => vPackageName,
|
||||
pPackageVersion => vVersion,
|
||||
pPackageBuildDate => NULL, -- Will be retrieved from package
|
||||
pPackageAuthor => NULL -- Will be retrieved from package
|
||||
);
|
||||
|
||||
DBMS_OUTPUT.PUT_LINE('SUCCESS: Tracked ' || vOwner || '.' || vPackageName || ' v' || vVersion);
|
||||
vCount := vCount + 1;
|
||||
|
||||
EXCEPTION
|
||||
WHEN OTHERS THEN
|
||||
DBMS_OUTPUT.PUT_LINE('ERROR tracking ' || vOwner || '.' || vPackageName || ': ' || SQLERRM);
|
||||
END;
|
||||
END LOOP;
|
||||
|
||||
DBMS_OUTPUT.PUT_LINE('========================================');
|
||||
DBMS_OUTPUT.PUT_LINE('Tracked ' || vCount || ' of ' || vPackageList.COUNT || ' packages successfully');
|
||||
DBMS_OUTPUT.PUT_LINE('========================================');
|
||||
END;
|
||||
/
|
||||
@@ -0,0 +1,62 @@
|
||||
-- ===================================================================
|
||||
-- Universal Package Version Verification Script
|
||||
-- ===================================================================
|
||||
-- Purpose: Verify all tracked Oracle packages for code changes
|
||||
-- Author: Grzegorz Michalski
|
||||
-- Date: 2026-03-11
|
||||
-- Version: 1.0.0
|
||||
--
|
||||
-- USAGE:
|
||||
-- Include at the end of install/rollback scripts: @@verify_packages_version.sql
|
||||
--
|
||||
-- OUTPUT:
|
||||
-- - List of all tracked packages with their current status
|
||||
-- - OK: Package has not changed since last tracking
|
||||
-- - WARNING: Package code changed without version update
|
||||
-- ===================================================================
|
||||
|
||||
SET LINESIZE 200
|
||||
SET PAGESIZE 1000
|
||||
SET FEEDBACK OFF
|
||||
|
||||
PROMPT
|
||||
PROMPT ========================================
|
||||
PROMPT Package Version Verification
|
||||
PROMPT ========================================
|
||||
PROMPT
|
||||
|
||||
COLUMN PACKAGE_OWNER FORMAT A15
|
||||
COLUMN PACKAGE_NAME FORMAT A20
|
||||
COLUMN VERSION FORMAT A10
|
||||
COLUMN STATUS FORMAT A80
|
||||
|
||||
SELECT
|
||||
PACKAGE_OWNER,
|
||||
PACKAGE_NAME,
|
||||
PACKAGE_VERSION AS VERSION,
|
||||
CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS
|
||||
FROM (
|
||||
SELECT
|
||||
PACKAGE_OWNER,
|
||||
PACKAGE_NAME,
|
||||
PACKAGE_VERSION,
|
||||
ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN
|
||||
FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING
|
||||
)
|
||||
WHERE RN = 1
|
||||
ORDER BY PACKAGE_OWNER, PACKAGE_NAME;
|
||||
|
||||
PROMPT
|
||||
PROMPT ========================================
|
||||
PROMPT Verification Complete
|
||||
PROMPT ========================================
|
||||
PROMPT
|
||||
PROMPT Legend:
|
||||
PROMPT OK - Package has not changed since last tracking
|
||||
PROMPT WARNING - Package code changed without version update
|
||||
PROMPT
|
||||
PROMPT For detailed hash information, use:
|
||||
PROMPT SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;
|
||||
PROMPT ========================================
|
||||
|
||||
SET FEEDBACK ON
|
||||
@@ -59,7 +59,8 @@ AS
|
||||
pTableName IN VARCHAR2,
|
||||
pSchemaName IN VARCHAR2,
|
||||
pKeyColumnName IN VARCHAR2,
|
||||
pTemplateTableName IN VARCHAR2
|
||||
pTemplateTableName IN VARCHAR2,
|
||||
pFormat IN VARCHAR2 DEFAULT 'CSV'
|
||||
) RETURN VARCHAR2 IS
|
||||
vResult VARCHAR2(32767);
|
||||
vColumns VARCHAR2(32767);
|
||||
@@ -173,10 +174,21 @@ AS
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'TO_CHAR(T.' || vCurrentCol || ', ''' || vDateFormat || ''') AS ' || vCurrentCol;
|
||||
|
||||
-- Other columns as-is with T. prefix
|
||||
-- Other columns: RFC 4180 quote-doubling for CSV character types, as-is for Parquet/others
|
||||
-- Oracle DBMS_CLOUD.EXPORT_DATA has no native RFC 4180 "" doubling.
|
||||
-- escape=true -> backslash-escaped (\"\) - incompatible with ORACLE_LOADER.
|
||||
-- escape=false -> unescaped embedded quotes - also incompatible.
|
||||
-- Solution (CSV only): pre-double any " in VARCHAR2/CHAR/CLOB before Oracle wraps in quote.
|
||||
-- Oracle then writes: "BIDDER-'""=:" which ORACLE_LOADER reads as BIDDER-'"=:.
|
||||
-- Parquet is binary - no quoting needed; REPLACE would corrupt string data.
|
||||
ELSE
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'T.' || vCurrentCol;
|
||||
IF pFormat = 'CSV' AND vDataType IN ('VARCHAR2', 'NVARCHAR2', 'CHAR', 'NCHAR', 'CLOB', 'NCLOB') THEN
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'REPLACE(T.' || vCurrentCol || ', CHR(34), CHR(34)||CHR(34)) AS ' || vCurrentCol;
|
||||
ELSE
|
||||
vResult := vResult || CASE WHEN vResult IS NOT NULL THEN ', ' ELSE '' END ||
|
||||
'T.' || vCurrentCol;
|
||||
END IF;
|
||||
END IF;
|
||||
|
||||
vPos := vNextPos + 1;
|
||||
@@ -808,7 +820,7 @@ AS
|
||||
VALIDATE_TABLE_AND_COLUMNS(vSchemaName, vTableName, vKeyColumnName, pColumnList, vParameters);
|
||||
|
||||
-- Build query with TO_CHAR for date columns (per-column format support)
|
||||
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName);
|
||||
vProcessedColumnList := buildQueryWithDateFormats(pColumnList, vTableName, vSchemaName, vKeyColumnName, pTemplateTableName, 'PARQUET');
|
||||
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Input column list: ' || NVL(pColumnList, 'NULL (building dynamic list from table metadata)'), 'DEBUG', vParameters);
|
||||
ENV_MANAGER.LOG_PROCESS_EVENT('Processed column list with TO_CHAR for date columns: ' || vProcessedColumnList, 'DEBUG', vParameters);
|
||||
|
||||
@@ -9,12 +9,14 @@ AS
|
||||
**/
|
||||
|
||||
-- Package Version Information
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.6.3';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-01-28 19:30:00';
|
||||
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.17.0';
|
||||
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(19) := '2026-03-11 10:00:00';
|
||||
PACKAGE_AUTHOR CONSTANT VARCHAR2(50) := 'MRDS Development Team';
|
||||
|
||||
-- Version History (last 3-5 changes)
|
||||
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
|
||||
'v2.17.0 (2026-03-11): PARQUET FIX - Added pFormat parameter to buildQueryWithDateFormats. REPLACE(col,CHR(34)) now applied only when pFormat=CSV. EXPORT_TABLE_DATA_BY_DATE passes PARQUET - string data was being corrupted (single " doubled to ""). Parquet is binary and needs no quote escaping.' || CHR(10) ||
|
||||
'v2.16.0 (2026-03-11): RFC 4180 FIX - Added REPLACE(col,CHR(34),CHR(34)||CHR(34)) in buildQueryWithDateFormats for VARCHAR2/CHAR/CLOB. Pre-doubled values produce compliant CSV for ORACLE_LOADER OPTIONALLY ENCLOSED BY chr(34).' || CHR(10) ||
|
||||
'v2.6.3 (2026-01-28): COMPILATION FIX - Resolved ORA-00904 error in EXPORT_PARTITION_PARALLEL. SQLERRM and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE cannot be used directly in SQL UPDATE statements. Now properly assigned to vgMsgTmp variable before UPDATE.' || CHR(10) ||
|
||||
'v2.6.2 (2026-01-28): CRITICAL FIX - Race condition when multiple exports run simultaneously. Changed DELETE to filter by age (>24h) instead of deleting all COMPLETED chunks. Prevents concurrent sessions from deleting each other chunks. Session-safe cleanup with TASK_NAME filtering. Enables true parallel execution of multiple export jobs.' || CHR(10) ||
|
||||
'v2.6.1 (2026-01-28): Added DELETE_FAILED_EXPORT_FILE procedure to clean up partial/corrupted files before retry. When partition fails mid-export, partial file is deleted before retry to prevent Oracle from creating _1 suffixed duplicates. Ensures clean retry without orphaned files in OCI bucket.' || CHR(10) ||
|
||||
|
||||
@@ -97,3 +97,9 @@ cd .\MARS_Packages\REL03\MARS-1005
|
||||
echo 'yes' | sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars1005.sql"
|
||||
echo 'yes' | sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars1005.sql"
|
||||
7z a -pMojeSuperHaslo#123 -mhe=on M1005_arch.7z MARS-1005
|
||||
|
||||
cd .\MARS_Packages\REL03\MARS-1005-PREHOOK
|
||||
echo 'yes' | sql "ADMIN/Cloudpass#34@ggmichalski_high" "@install_mars1005_prehook.sql"
|
||||
echo 'yes' | sql "ADMIN/Cloudpass#34@ggmichalski_high" "@rollback_mars1005_prehook.sql"
|
||||
7z a -pMojeSuperHaslo#123 -mhe=on M1005PH_arch.7z MARS-1005-PREHOOK
|
||||
|
||||
|
||||
Reference in New Issue
Block a user