admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update DATA_EXPORTER package to v2.17.0: Fix RFC 4180 compliance and Parquet format corruption

Browse Source
This commit is contained in:
Grzegorz Michalski
2026-03-12 08:50:08 +01:00
parent 5ba6c30fda
commit 202b535f9f
10 changed files with 346 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

115
MARS_Packages/REL03/MARS-1005-PREHOOK/README.md Normal file
View File

@@ -0,0 +1,115 @@
# MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance + Parquet Format Support
## Overview
Pre-hook for MARS-1005. Deploys an updated `CT_MRDS.DATA_EXPORTER` package (v2.17.0)
that resolves two export format bugs:
1. **RFC 4180 compliance (v2.15.0 / v2.16.0):** Previous versions used `escape=true`
in `DBMS_CLOUD.EXPORT_DATA`, producing backslash-escaped embedded quotes (`\"`).
ODS external tables (`FIELDS CSV WITHOUT EMBEDDED`) expect RFC 4180 doubling (`""`).
Fix: removed `escape=true`; implemented `REPLACE(col, '"', '""')` in SELECT query.
2. **Parquet corruption fix (v2.17.0):** The RFC 4180 `REPLACE` was applied to all
export formats, including Parquet — corrupting values that contained double-quotes.
Fix: added `pFormat` parameter to `buildQueryWithDateFormats`; `REPLACE` is now
applied only when `pFormat = 'CSV'`. Parquet exports pass column values unchanged.
## Contents
| File | Description |
|------|-------------|
| `install_mars1005_prehook.sql` | Master installation script (SPOOL, ACCEPT, quit) |
| `rollback_mars1005_prehook.sql` | Master rollback script (SPOOL, ACCEPT, quit) |
| `00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql` | Deploy DATA_EXPORTER v2.17.0 |
| `90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql` | Restore DATA_EXPORTER v2.14.0 |
| `track_package_versions.sql` | Universal version tracking script |
| `verify_packages_version.sql` | Universal package verification script |
| `new_version/DATA_EXPORTER.pkg` | Package specification v2.17.0 |
| `new_version/DATA_EXPORTER.pkb` | Package body v2.17.0 |
| `rollback_version/DATA_EXPORTER.pkg` | Package specification v2.14.0 (backup) |
| `rollback_version/DATA_EXPORTER.pkb` | Package body v2.14.0 (backup) |
| `README.md` | This file |
## Prerequisites
- Oracle Database 23ai
- `CT_MRDS.ENV_MANAGER` v3.1.0+
- ADMIN user with EXECUTE privileges on CT_MRDS schema
- Connection service: `ggmichalski_high`
## Version Change
| Package | Before | After |
|---------|--------|-------|
| `CT_MRDS.DATA_EXPORTER` | v2.14.0 | v2.17.0 |
## Installation
### Option 1: Master Script (Recommended)
```powershell
# Execute as ADMIN user for proper privilege management
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
Log file created automatically: `log/INSTALL_MARS_1005_PREHOOK_<PDB>_<timestamp>.log`
### Option 2: Individual Scripts
```powershell
# Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
## Verification
```sql
-- Verify package version
SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL;
-- Expected: 2.17.0
-- Verify build info
SELECT CT_MRDS.DATA_EXPORTER.GET_BUILD_INFO() FROM DUAL;
-- Check for compilation errors
SELECT * FROM ALL_ERRORS
WHERE OWNER = 'CT_MRDS'
AND NAME = 'DATA_EXPORTER';
-- Verify no untracked changes
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'DATA_EXPORTER') FROM DUAL;
-- Expected: OK
```
## Rollback
```powershell
# Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```
Rollback restores `CT_MRDS.DATA_EXPORTER` to v2.14.0 (re-enables `escape=true`).
## Testing
After installation, verify the Parquet and CSV export paths:
**CSV path (ODS):** Export a table, then SELECT from the corresponding ODS external
table. Values with embedded double-quotes should appear as single `"` characters
and not trigger `ORA-30653`.
**Parquet path (ARCHIVE):** Export a table containing values with embedded
double-quotes to the ARCHIVE bucket area. Download the Parquet file and confirm
the value is stored verbatim (no extra quotes).
## Expected Changes
- `CT_MRDS.DATA_EXPORTER`: v2.14.0 → v2.17.0
- No table structure changes
- No configuration changes
## Related
- MARS-1005: Export TOP allotment data (main issue)
- `CT_MRDS.DATA_EXPORTER` source: `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/packages/DATA_EXPORTER.sql`