116 lines
4.0 KiB
Markdown
116 lines
4.0 KiB
Markdown
# MARS-1005-PREHOOK: Fix DATA_EXPORTER RFC 4180 Compliance + Parquet Format Support
|
|
|
|
## Overview
|
|
|
|
Pre-hook for MARS-1005. Deploys an updated `CT_MRDS.DATA_EXPORTER` package (v2.17.0)
|
|
that resolves two export format bugs:
|
|
|
|
1. **RFC 4180 compliance (v2.15.0 / v2.16.0):** Previous versions used `escape=true`
|
|
in `DBMS_CLOUD.EXPORT_DATA`, producing backslash-escaped embedded quotes (`\"`).
|
|
ODS external tables (`FIELDS CSV WITHOUT EMBEDDED`) expect RFC 4180 doubling (`""`).
|
|
Fix: removed `escape=true`; implemented `REPLACE(col, '"', '""')` in SELECT query.
|
|
|
|
2. **Parquet corruption fix (v2.17.0):** The RFC 4180 `REPLACE` was applied to all
|
|
export formats, including Parquet — corrupting values that contained double-quotes.
|
|
Fix: added `pFormat` parameter to `buildQueryWithDateFormats`; `REPLACE` is now
|
|
applied only when `pFormat = 'CSV'`. Parquet exports pass column values unchanged.
|
|
|
|
## Contents
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| `install_mars1005_prehook.sql` | Master installation script (SPOOL, ACCEPT, quit) |
|
|
| `rollback_mars1005_prehook.sql` | Master rollback script (SPOOL, ACCEPT, quit) |
|
|
| `00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql` | Deploy DATA_EXPORTER v2.17.0 |
|
|
| `90_MARS_1005_PREHOOK_rollback_DATA_EXPORTER.sql` | Restore DATA_EXPORTER v2.14.0 |
|
|
| `track_package_versions.sql` | Universal version tracking script |
|
|
| `verify_packages_version.sql` | Universal package verification script |
|
|
| `new_version/DATA_EXPORTER.pkg` | Package specification v2.17.0 |
|
|
| `new_version/DATA_EXPORTER.pkb` | Package body v2.17.0 |
|
|
| `rollback_version/DATA_EXPORTER.pkg` | Package specification v2.14.0 (backup) |
|
|
| `rollback_version/DATA_EXPORTER.pkb` | Package body v2.14.0 (backup) |
|
|
| `README.md` | This file |
|
|
|
|
## Prerequisites
|
|
|
|
- Oracle Database 23ai
|
|
- `CT_MRDS.ENV_MANAGER` v3.1.0+
|
|
- ADMIN user with EXECUTE privileges on CT_MRDS schema
|
|
- Connection service: `ggmichalski_high`
|
|
|
|
## Version Change
|
|
|
|
| Package | Before | After |
|
|
|---------|--------|-------|
|
|
| `CT_MRDS.DATA_EXPORTER` | v2.14.0 | v2.17.0 |
|
|
|
|
## Installation
|
|
|
|
### Option 1: Master Script (Recommended)
|
|
|
|
```powershell
|
|
# Execute as ADMIN user for proper privilege management
|
|
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/install_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
|
```
|
|
|
|
Log file created automatically: `log/INSTALL_MARS_1005_PREHOOK_<PDB>_<timestamp>.log`
|
|
|
|
### Option 2: Individual Scripts
|
|
|
|
```powershell
|
|
# Execute as ADMIN user
|
|
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/00_MARS_1005_PREHOOK_install_DATA_EXPORTER.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
|
```
|
|
|
|
## Verification
|
|
|
|
```sql
|
|
-- Verify package version
|
|
SELECT CT_MRDS.DATA_EXPORTER.GET_VERSION() FROM DUAL;
|
|
-- Expected: 2.17.0
|
|
|
|
-- Verify build info
|
|
SELECT CT_MRDS.DATA_EXPORTER.GET_BUILD_INFO() FROM DUAL;
|
|
|
|
-- Check for compilation errors
|
|
SELECT * FROM ALL_ERRORS
|
|
WHERE OWNER = 'CT_MRDS'
|
|
AND NAME = 'DATA_EXPORTER';
|
|
|
|
-- Verify no untracked changes
|
|
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'DATA_EXPORTER') FROM DUAL;
|
|
-- Expected: OK
|
|
```
|
|
|
|
## Rollback
|
|
|
|
```powershell
|
|
# Execute as ADMIN user
|
|
Get-Content "MARS_Packages/REL03/MARS-1005-PREHOOK/rollback_mars1005_prehook.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
|
|
```
|
|
|
|
Rollback restores `CT_MRDS.DATA_EXPORTER` to v2.14.0 (re-enables `escape=true`).
|
|
|
|
## Testing
|
|
|
|
After installation, verify the Parquet and CSV export paths:
|
|
|
|
**CSV path (ODS):** Export a table, then SELECT from the corresponding ODS external
|
|
table. Values with embedded double-quotes should appear as single `"` characters
|
|
and not trigger `ORA-30653`.
|
|
|
|
**Parquet path (ARCHIVE):** Export a table containing values with embedded
|
|
double-quotes to the ARCHIVE bucket area. Download the Parquet file and confirm
|
|
the value is stored verbatim (no extra quotes).
|
|
|
|
## Expected Changes
|
|
|
|
- `CT_MRDS.DATA_EXPORTER`: v2.14.0 → v2.17.0
|
|
- No table structure changes
|
|
- No configuration changes
|
|
|
|
## Related
|
|
|
|
- MARS-1005: Export TOP allotment data (main issue)
|
|
- `CT_MRDS.DATA_EXPORTER` source: `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/packages/DATA_EXPORTER.sql`
|