Develop
This commit is contained in:
Grzegorz Michalski
@@ -84,7 +84,7 @@ c:\_git\_local_rep\
|
||||
│ ├── MARS_Packages\
|
||||
│ │ └── REL01\
|
||||
│ │ └── MARS-1057\ # New package folder
|
||||
│ │ ├── rollback_version\
|
||||
│ │ ├── current_version\
|
||||
│ │ ├── new_version\
|
||||
│ │ └── *.sql files
|
||||
│ ├── database\
|
||||
@@ -169,6 +169,157 @@ The deployment system uses these key components:
|
||||
|
||||
---
|
||||
|
||||
## Universal Tracking and Verification Scripts
|
||||
|
||||
### MANDATORY: Use Standard Template Files
|
||||
|
||||
**CRITICAL:** Every MARS package MUST include two universal scripts for tracking and verification. These scripts are reusable across all MARS issues and provide consistent version management.
|
||||
|
||||
#### Required Files Structure
|
||||
|
||||
```
|
||||
MARS_Packages/
|
||||
└── REL0X/
|
||||
└── MARS-XXXX/
|
||||
├── 01_MARS_XXXX_install_*.sql
|
||||
├── 02_MARS_XXXX_install_*.sql
|
||||
├── track_package_versions.sql # ✅ MANDATORY - Universal tracking
|
||||
├── verify_packages_version.sql # ✅ MANDATORY - Universal verification
|
||||
├── install_marsXXXX.sql # Master script
|
||||
└── rollback_marsXXXX.sql
|
||||
```
|
||||
|
||||
### track_package_versions.sql
|
||||
|
||||
**Purpose:** Universal script for tracking multiple package versions with editable package list.
|
||||
|
||||
**Key Features:**
|
||||
- ✅ **Reusable** - Same file for all MARS issues
|
||||
- ✅ **Configurable** - Edit package list array to specify which packages to track
|
||||
- ✅ **Automatic** - Calls ENV_MANAGER.TRACK_PACKAGE_VERSION for each package
|
||||
- ✅ **Summary Output** - Shows tracked packages count and versions
|
||||
|
||||
**Template:**
|
||||
```sql
|
||||
-- ===================================================================
|
||||
-- PACKAGE LIST - Edit this array to specify packages to track
|
||||
-- ===================================================================
|
||||
vPackageList t_string_array := t_string_array(
|
||||
'CT_MRDS.DATA_EXPORTER',
|
||||
'CT_MRDS.FILE_MANAGER',
|
||||
'ODS.FILE_MANAGER_ODS'
|
||||
);
|
||||
-- ===================================================================
|
||||
```
|
||||
|
||||
**Usage in Install Script:**
|
||||
```sql
|
||||
PROMPT Step 4: Track Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@track_package_versions.sql
|
||||
```
|
||||
|
||||
**Benefits over custom tracking scripts:**
|
||||
- No hardcoded package names in multiple places
|
||||
- Single point of configuration
|
||||
- Consistent error handling
|
||||
- Standardized output format
|
||||
- Copy-paste ready for new MARS issues
|
||||
|
||||
### verify_packages_version.sql
|
||||
|
||||
**Purpose:** Universal script for verifying ALL tracked packages for code changes.
|
||||
|
||||
**Key Features:**
|
||||
- ✅ **Automatic Discovery** - Queries A_PACKAGE_VERSION_TRACKING to find all tracked packages
|
||||
- ✅ **Hash Validation** - Calls CHECK_PACKAGE_CHANGES for each package
|
||||
- ✅ **Status Report** - Shows OK or WARNING for each package
|
||||
- ✅ **Zero Configuration** - No editing needed, works automatically
|
||||
|
||||
**Usage in Install Script:**
|
||||
```sql
|
||||
PROMPT Final Step: Verify All Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@verify_packages_version.sql
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
```
|
||||
PACKAGE_OWNER PACKAGE_NAME VERSION STATUS
|
||||
--------------- ------------------- ---------- -------------------------------
|
||||
CT_MRDS DATA_EXPORTER 2.2.0 OK: Package has not changed
|
||||
CT_MRDS FILE_MANAGER 3.3.0 OK: Package has not changed
|
||||
ODS FILE_MANAGER_ODS 2.1.0 OK: Package has not changed
|
||||
```
|
||||
|
||||
### Master Install Script Structure (MANDATORY)
|
||||
|
||||
Every `install_marsXXXX.sql` MUST follow this exact structure:
|
||||
|
||||
```sql
|
||||
-- ===================================================================
|
||||
-- MARS-XXXX INSTALL SCRIPT
|
||||
-- ===================================================================
|
||||
|
||||
-- Dynamic spool file generation
|
||||
host mkdir log 2>nul
|
||||
-- ... spool setup ...
|
||||
|
||||
PROMPT Step 1: Deploy Package Specification
|
||||
@@01_MARS_XXXX_install_PACKAGE_SPEC.sql
|
||||
|
||||
PROMPT Step 2: Deploy Package Body
|
||||
@@02_MARS_XXXX_install_PACKAGE_BODY.sql
|
||||
|
||||
PROMPT Step 3: Track Package Versions
|
||||
@@track_package_versions.sql -- ✅ MANDATORY universal script
|
||||
|
||||
PROMPT Step 4: Verify Package Versions
|
||||
@@verify_packages_version.sql -- ✅ MANDATORY universal script
|
||||
|
||||
spool off
|
||||
quit;
|
||||
```
|
||||
|
||||
### DO NOT Create Custom Track/Verify Scripts
|
||||
|
||||
**❌ WRONG (Custom tracking per MARS issue):**
|
||||
```sql
|
||||
@@04_MARS_XXXX_track_version.sql -- DON'T DO THIS
|
||||
@@05_MARS_XXXX_verify_installation.sql -- DON'T DO THIS
|
||||
```
|
||||
|
||||
**✅ CORRECT (Universal scripts):**
|
||||
```sql
|
||||
@@track_package_versions.sql -- Always use this
|
||||
@@verify_packages_version.sql -- Always use this
|
||||
```
|
||||
|
||||
**Why universal scripts are mandatory:**
|
||||
- **Consistency** - Same behavior across all MARS packages
|
||||
- **Maintainability** - Bug fixes propagate to all packages
|
||||
- **Simplicity** - No need to create custom scripts for each issue
|
||||
- **Compliance** - Ensures all packages follow same standard
|
||||
- **Code Review** - Easier to review when structure is identical
|
||||
|
||||
### Copying Template Files
|
||||
|
||||
**For each new MARS package, copy from reference implementation:**
|
||||
|
||||
```powershell
|
||||
# Copy universal scripts from MARS-826-PREHOOK (reference implementation)
|
||||
Copy-Item "MARS_Packages\REL01_POST_DEACTIVATION\MARS-826-PREHOOK\track_package_versions.sql" `
|
||||
"MARS_Packages\RELXX\MARS-YYYY\track_package_versions.sql"
|
||||
|
||||
Copy-Item "MARS_Packages\REL01_POST_DEACTIVATION\MARS-826-PREHOOK\verify_packages_version.sql" `
|
||||
"MARS_Packages\RELXX\MARS-YYYY\verify_packages_version.sql"
|
||||
|
||||
# Edit package list in track_package_versions.sql
|
||||
# No changes needed to verify_packages_version.sql (works automatically)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standard Deployment Workflow
|
||||
|
||||
### Step 1: Check Current Package Status
|
||||
@@ -592,8 +743,6 @@ WHERE OWNER = 'CT_MRDS' -- Replace with target schema
|
||||
|
||||
MARS packages (e.g., MARS-1049, MARS-1011) are deployment packages that bundle database changes for specific features or fixes. They follow a standardized structure with logging, version tracking, and rollback capabilities.
|
||||
|
||||
**PREREQUISITE:** Before creating any MARS package, ensure you have completed **[Step 0: Prepare Dedicated Working Directory and Git Branch](#step-0-prepare-dedicated-working-directory-and-git-branch)**. Each MARS issue MUST have its own isolated working directory and feature branch.
|
||||
|
||||
**CRITICAL:** All MARS package installations MUST be executed as **ADMIN user** to ensure proper schema creation, privilege management, and cross-schema operations.
|
||||
|
||||
**Installation User Requirements:**
|
||||
@@ -632,7 +781,7 @@ mock_data/
|
||||
- track_package_versions.sql and verify_packages_version.sql
|
||||
- README.md
|
||||
- .gitignore (with standardized exclusions)
|
||||
- rollback_version/ and new_version/ folders (if package modifications)
|
||||
- current_version/ and new_version/ folders (if package modifications)
|
||||
- Core deployment files only
|
||||
|
||||
Standard MARS package directory structure:
|
||||
@@ -651,7 +800,7 @@ MARS_Packages/REL01/MARS-XXXX/
|
||||
├── verify_packages_version.sql # Package verification script
|
||||
├── ...
|
||||
├── README.md # Package documentation
|
||||
├── rollback_version/ # Backup of objects BEFORE changes (for rollback)
|
||||
├── current_version/ # Backup of objects BEFORE changes (for rollback)
|
||||
│ ├── PACKAGE_NAME.pkg # Previous package specification
|
||||
│ └── PACKAGE_NAME.pkb # Previous package body
|
||||
├── new_version/ # Updated objects AFTER changes (for installation)
|
||||
@@ -677,12 +826,12 @@ MARS_Packages/REL01/MARS-XXXX/
|
||||
- **Master scripts**: `install_marsXXXX.sql` and `rollback_marsXXXX.sql`
|
||||
|
||||
**Version Management Folders (for Database Object Modifications):**
|
||||
- **`rollback_version/`** - Contains backup of database objects BEFORE changes (for rollback)
|
||||
- **`current_version/`** - Contains backup of database objects BEFORE changes (for rollback)
|
||||
- **`new_version/`** - Contains updated database objects AFTER changes (for installation)
|
||||
|
||||
**When to use version folders:**
|
||||
- Required when MARS package modifies ANY existing database objects (packages, tables, views, indexes, triggers, etc.)
|
||||
- Copy original object definitions to `rollback_version/` before making changes
|
||||
- Copy original object definitions to `current_version/` before making changes
|
||||
- Place modified object definitions in `new_version/` after changes
|
||||
- Examples: FILE_MANAGER.pkg/pkb, A_SOURCE_FILE_CONFIG.sql (table), V_STATUS.sql (view), IDX_CONFIG.sql (index)
|
||||
|
||||
@@ -767,7 +916,19 @@ PROMPT Step 2: Second Installation Step
|
||||
PROMPT =========================================================================
|
||||
@@02_MARS_XXXX_install_step2.sql
|
||||
|
||||
-- ... more steps ...
|
||||
-- ... more deployment steps ...
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step N: Track Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@track_package_versions.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Final Step: Verify Package Versions
|
||||
PROMPT =========================================================================
|
||||
@@verify_packages_version.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
@@ -791,7 +952,9 @@ quit;
|
||||
7. **ACCEPT Validation**: User confirmation required before execution (safety feature)
|
||||
8. **ALTER SESSION SET CURRENT_SCHEMA** (optional): Sets working schema for package operations
|
||||
9. **@@ Includes**: All sub-scripts executed via `@@` command
|
||||
10. **quit;**: Exits SQLcl/SQL*Plus after completion (important for automated deployments)
|
||||
10. **track_package_versions.sql**: MANDATORY universal tracking script (must be included before verify)
|
||||
11. **verify_packages_version.sql**: MANDATORY universal verification script (must be last step before completion)
|
||||
12. **quit;**: Exits SQLcl/SQL*Plus after completion (important for automated deployments)
|
||||
|
||||
**SET ECHO OFF Benefits:**
|
||||
- **Clean Output**: PROMPT messages appear only once in console and log files
|
||||
@@ -841,6 +1004,114 @@ quit;
|
||||
|
||||
---
|
||||
|
||||
### Complete MARS Package Example
|
||||
|
||||
**MARS-835-PREHOOK** - Real-world example following all standards:
|
||||
|
||||
**File Structure:**
|
||||
```
|
||||
MARS_Packages/REL01_POST_DEACTIVATION/MARS-835-PREHOOK/
|
||||
├── .gitignore
|
||||
├── install_mars835_prehook.sql # Master install (uses universal scripts)
|
||||
├── rollback_mars835_prehook.sql # Master rollback
|
||||
├── 01_MARS_835_install_DATA_EXPORTER_SPEC.sql
|
||||
├── 02_MARS_835_install_DATA_EXPORTER_BODY.sql
|
||||
├── 91_MARS_835_rollback_DATA_EXPORTER_BODY.sql
|
||||
├── 92_MARS_835_rollback_DATA_EXPORTER_SPEC.sql
|
||||
├── track_package_versions.sql # ✅ Universal tracking
|
||||
├── verify_packages_version.sql # ✅ Universal verification
|
||||
├── README.md
|
||||
├── current_version/
|
||||
│ ├── DATA_EXPORTER.pkg # v2.1.1 (before changes)
|
||||
│ └── DATA_EXPORTER.pkb
|
||||
├── new_version/
|
||||
│ ├── DATA_EXPORTER.pkg # v2.2.0 (after changes)
|
||||
│ └── DATA_EXPORTER.pkb
|
||||
└── log/ # Auto-created by install script
|
||||
```
|
||||
|
||||
**track_package_versions.sql** (edited for this MARS issue):
|
||||
```sql
|
||||
-- ===================================================================
|
||||
-- PACKAGE LIST - Edit this array to specify packages to track
|
||||
-- ===================================================================
|
||||
vPackageList t_string_array := t_string_array(
|
||||
'CT_MRDS.DATA_EXPORTER' -- Only package modified in MARS-835
|
||||
);
|
||||
-- ===================================================================
|
||||
```
|
||||
|
||||
**install_mars835_prehook.sql** (master script):
|
||||
```sql
|
||||
-- ===================================================================
|
||||
-- MARS-835-PREHOOK INSTALL SCRIPT
|
||||
-- ===================================================================
|
||||
|
||||
host mkdir log 2>nul
|
||||
|
||||
var filename VARCHAR2(100)
|
||||
BEGIN
|
||||
:filename := 'log/INSTALL_MARS_835_PREHOOK_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
|
||||
END;
|
||||
/
|
||||
column filename new_value _filename
|
||||
select :filename filename from dual;
|
||||
spool &_filename
|
||||
|
||||
SET ECHO OFF
|
||||
SET TIMING ON
|
||||
SET SERVEROUTPUT ON SIZE UNLIMITED
|
||||
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-835-PREHOOK: DRY Refactoring for DATA_EXPORTER
|
||||
PROMPT =========================================================================
|
||||
|
||||
ACCEPT continue CHAR PROMPT 'Type YES to continue: '
|
||||
WHENEVER SQLERROR EXIT SQL.SQLCODE
|
||||
BEGIN
|
||||
IF UPPER(TRIM('&continue')) != 'YES' THEN
|
||||
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted');
|
||||
END IF;
|
||||
END;
|
||||
/
|
||||
WHENEVER SQLERROR CONTINUE
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 1: Deploy DATA_EXPORTER Package Specification (v2.2.0)
|
||||
@@01_MARS_835_install_DATA_EXPORTER_SPEC.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 2: Deploy DATA_EXPORTER Package Body (v2.2.0)
|
||||
@@02_MARS_835_install_DATA_EXPORTER_BODY.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 3: Track Package Versions
|
||||
@@track_package_versions.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 4: Verify Package Versions
|
||||
@@verify_packages_version.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-835-PREHOOK Installation - COMPLETED
|
||||
PROMPT =========================================================================
|
||||
|
||||
spool off
|
||||
quit;
|
||||
```
|
||||
|
||||
**Why this is the correct pattern:**
|
||||
- ✅ Uses universal `track_package_versions.sql` (no custom 04_MARS_835_track_version.sql)
|
||||
- ✅ Uses universal `verify_packages_version.sql` (no custom 03_MARS_835_verify_installation.sql)
|
||||
- ✅ Tracking happens AFTER deployment (Step 3)
|
||||
- ✅ Verification happens LAST (Step 4) to confirm everything is correct
|
||||
- ✅ Single configuration point (edit package list in track_package_versions.sql)
|
||||
- ✅ Consistent with ALL other MARS packages
|
||||
- ✅ Easy to copy/paste for new MARS issues
|
||||
|
||||
---
|
||||
|
||||
### SPOOL Logging Benefits
|
||||
|
||||
**Why SPOOL is mandatory:**
|
||||
@@ -1292,7 +1563,7 @@ Before creating a MARS package, ensure:
|
||||
- [ ] README.md documents installation process
|
||||
- [ ] Rollback scripts reverse all changes
|
||||
- [ ] Rollback includes verification step (@@test/verify_packages_version.sql)
|
||||
- [ ] **rollback_version/ folder**: Contains backup of objects before changes (if modifying packages)
|
||||
- [ ] **current_version/ folder**: Contains backup of objects before changes (if modifying packages)
|
||||
- [ ] **new_version/ folder**: Contains updated objects after changes (if modifying packages)
|
||||
- [ ] **test/ folder**: All test files and verification scripts organized in subfolder
|
||||
- [ ] **log/ folder**: SPOOL log files automatically created by master scripts
|
||||
@@ -1320,7 +1591,7 @@ MARS_Packages/REL01/MARS-1046/
|
||||
├── track_package_versions.sql # Version tracking
|
||||
├── verify_packages_version.sql # Package verification
|
||||
├── README.md # Comprehensive documentation
|
||||
├── rollback_version/ # v3.3.0 backup for rollback
|
||||
├── current_version/ # v3.3.0 backup for rollback
|
||||
│ ├── FILE_MANAGER.pkg
|
||||
│ └── FILE_MANAGER.pkb
|
||||
├── new_version/ # v3.3.1 updated packages
|
||||
@@ -1342,7 +1613,7 @@ MARS_Packages/REL01/MARS-1046/
|
||||
- SPOOL logging: `INSTALL_MARS_1046_<PDB>_<timestamp>.log`
|
||||
- ACCEPT validation: Requires explicit "YES" to proceed (both install & rollback)
|
||||
- quit; command: Clean SQLcl exit after completion
|
||||
- Version folders: rollback_version/ (v3.3.0) and new_version/ (v3.3.1)
|
||||
- Version folders: current_version/ (v3.3.0) and new_version/ (v3.3.1)
|
||||
- Test subfolder: All test files organized separately
|
||||
- 4-step workflow: Install packages → Track version → Verify packages → quit
|
||||
- Full rollback: Restore to previous version with verification
|
||||
@@ -1409,7 +1680,7 @@ Move-Item -Path "*.log" -Destination "log" -Force
|
||||
**Step 4: Verify clean structure**
|
||||
```powershell
|
||||
# Check root directory - should only have deployment scripts and README
|
||||
Get-ChildItem | Where-Object { $_.Name -notmatch "^(01|02|91|92|install|rollback|README|rollback_version|new_version|test|log)" }
|
||||
Get-ChildItem | Where-Object { $_.Name -notmatch "^(01|02|91|92|install|rollback|README|current_version|new_version|test|log)" }
|
||||
# Expected: No output (all non-deployment files moved)
|
||||
```
|
||||
|
||||
@@ -1425,7 +1696,7 @@ MARS_Packages/REL01/MARS-XXXX/
|
||||
├── track_package_versions.sql
|
||||
├── verify_packages_version.sql
|
||||
├── README.md # Documentation
|
||||
├── rollback_version/ # Backup packages
|
||||
├── current_version/ # Backup packages
|
||||
├── new_version/ # Updated packages
|
||||
└── test/ # ALL test artifacts
|
||||
├── test_marsXXXX.sql
|
||||
|
||||
@@ -0,0 +1,338 @@
|
||||
# Oracle External Tables - Column Order Mapping Issue
|
||||
|
||||
## Document Information
|
||||
- **Date**: 2026-01-08
|
||||
- **Issue**: CSV External Tables map columns by position, not by name
|
||||
- **Impact**: HIGH - NULL values in A_WORKFLOW_HISTORY_KEY column for CSV exports
|
||||
- **Status**: IDENTIFIED - Solution required
|
||||
|
||||
---
|
||||
|
||||
## Problem Summary
|
||||
|
||||
Oracle External Tables using **CSV format** map columns **by position** (ignoring header names), while **Parquet format** uses **schema-based mapping** (by column name). This causes critical data loss when exporting to CSV files that are read by external tables with different column ordering than the source table.
|
||||
|
||||
### Affected Operations
|
||||
- ✅ **Parquet exports to ARCHIVE** - Working correctly (uses column names)
|
||||
- ❌ **CSV exports to ODS/DATA** - Returns NULL for A_WORKFLOW_HISTORY_KEY (positional mismatch)
|
||||
|
||||
---
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Source Table Structure (OU_CSDB.LEGACY_DEBT_DAILY)
|
||||
```
|
||||
Column Position | Column Name | Data Type
|
||||
----------------|----------------------|----------
|
||||
1 | A_KEY | NUMBER
|
||||
2 | NEWUPDATED | DATE
|
||||
3 | IDLOADDATE_DIM | DATE
|
||||
... | ... | ...
|
||||
72 | A_ETL_LOAD_SET_FK | NUMBER ← Key column
|
||||
... | ... | ...
|
||||
129 | PLACEHOLDER50 | VARCHAR2
|
||||
```
|
||||
|
||||
### External Table Structure (ODS.CSDB_DEBT_DAILY_ODS)
|
||||
```
|
||||
Column Position | Column Name | Data Type
|
||||
----------------|----------------------------|----------
|
||||
1 | A_KEY | NUMBER
|
||||
2 | A_WORKFLOW_HISTORY_KEY | NUMBER ← Expected here!
|
||||
3 | NEWUPDATED | DATE
|
||||
4 | IDLOADDATE_DIM | DATE
|
||||
... | ... | ...
|
||||
129 | PLACEHOLDER50 | VARCHAR2
|
||||
```
|
||||
|
||||
### Export Query Generated
|
||||
```sql
|
||||
SELECT
|
||||
T.A_KEY, -- Position 1
|
||||
T.NEWUPDATED, -- Position 2
|
||||
T.IDLOADDATE_DIM, -- Position 3
|
||||
...,
|
||||
T.A_ETL_LOAD_SET_FK AS A_WORKFLOW_HISTORY_KEY, -- Position 72
|
||||
...
|
||||
FROM OU_CSDB.LEGACY_DEBT_DAILY T
|
||||
```
|
||||
|
||||
### CSV File Structure
|
||||
```csv
|
||||
A_KEY,NEWUPDATED,IDLOADDATE_DIM,...,A_WORKFLOW_HISTORY_KEY,...
|
||||
1,2026-01-08,2025-12-01,...,999,...
|
||||
```
|
||||
|
||||
### What External Table Reads
|
||||
```
|
||||
Position 1 → A_KEY (NUMBER) ✅ Reads: 1 (correct)
|
||||
Position 2 → A_WORKFLOW_HISTORY_KEY ❌ Reads: 2026-01-08 (NEWUPDATED as DATE)
|
||||
Converts DATE → NUMBER → NULL
|
||||
Position 3 → NEWUPDATED (DATE) ❌ Reads: 2025-12-01 (IDLOADDATE_DIM)
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Experimental Verification
|
||||
|
||||
### Test 1: CSV with Mismatched Column Order
|
||||
|
||||
**CSV File Content:**
|
||||
```csv
|
||||
A_KEY,NEWUPDATED,A_WORKFLOW_HISTORY_KEY
|
||||
1,2026-01-08,999
|
||||
```
|
||||
|
||||
**External Table Definition (Wrong Order):**
|
||||
```sql
|
||||
CREATE EXTERNAL TABLE TEST_COLUMN_ORDER_WRONG (
|
||||
A_KEY NUMBER,
|
||||
A_WORKFLOW_HISTORY_KEY NUMBER, -- Position 2
|
||||
NEWUPDATED DATE -- Position 3
|
||||
)
|
||||
```
|
||||
|
||||
**Result:**
|
||||
```
|
||||
❌ ORA-30653: reject limit reached
|
||||
```
|
||||
**Reason**: Oracle tried to convert NEWUPDATED (DATE) to A_WORKFLOW_HISTORY_KEY (NUMBER) → Conversion failed
|
||||
|
||||
---
|
||||
|
||||
### Test 2: CSV with Matching Column Order
|
||||
|
||||
**CSV File Content:**
|
||||
```csv
|
||||
A_KEY,NEWUPDATED,A_WORKFLOW_HISTORY_KEY
|
||||
1,2026-01-08,999
|
||||
```
|
||||
|
||||
**External Table Definition (Correct Order):**
|
||||
```sql
|
||||
CREATE EXTERNAL TABLE TEST_COLUMN_ORDER_MATCHING (
|
||||
A_KEY NUMBER,
|
||||
NEWUPDATED DATE, -- Position 2
|
||||
A_WORKFLOW_HISTORY_KEY NUMBER -- Position 3
|
||||
)
|
||||
```
|
||||
|
||||
**Result:**
|
||||
```
|
||||
✅ SUCCESS
|
||||
A_KEY=1, NEWUPDATED=2026-01-08, A_WORKFLOW_HISTORY_KEY=999
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Test 3: Parquet with Mismatched Column Order
|
||||
|
||||
**Parquet File Columns:**
|
||||
```
|
||||
A_KEY (NUMBER), NEWUPDATED (DATE), A_WORKFLOW_HISTORY_KEY (NUMBER)
|
||||
```
|
||||
|
||||
**External Table Definition (Different Order):**
|
||||
```sql
|
||||
CREATE EXTERNAL TABLE TEST_PARQUET_ORDER_WRONG (
|
||||
A_KEY NUMBER,
|
||||
A_WORKFLOW_HISTORY_KEY NUMBER, -- Different position than Parquet!
|
||||
NEWUPDATED DATE
|
||||
) ... FORMAT parquet
|
||||
```
|
||||
|
||||
**Result:**
|
||||
```
|
||||
✅ SUCCESS
|
||||
A_KEY=1, A_WORKFLOW_HISTORY_KEY=999, NEWUPDATED=2026-01-08
|
||||
```
|
||||
**Reason**: Parquet uses **schema-based mapping** - matches by column name, not position
|
||||
|
||||
---
|
||||
|
||||
## Format Comparison
|
||||
|
||||
| Aspect | CSV | Parquet |
|
||||
|--------|-----|---------|
|
||||
| **Column Mapping** | By position (1, 2, 3, ...) | By column name (schema) |
|
||||
| **Header Usage** | Ignored (cosmetic only) | Used for mapping |
|
||||
| **Order Sensitivity** | ❌ HIGH - Must match exactly | ✅ LOW - Order irrelevant |
|
||||
| **Type Mismatch** | Reject/NULL conversion | Proper validation |
|
||||
| **Current Status** | ❌ BROKEN for ODS exports | ✅ WORKING for ARCHIVE |
|
||||
|
||||
---
|
||||
|
||||
## Current Implementation Issue
|
||||
|
||||
### processColumnList Function (DATA_EXPORTER.pkb)
|
||||
```sql
|
||||
FUNCTION processColumnList(...) RETURN VARCHAR2 IS
|
||||
BEGIN
|
||||
IF pColumnList IS NULL THEN
|
||||
-- Build list of all columns from SOURCE table
|
||||
SELECT LISTAGG(column_name, ', ') WITHIN GROUP (ORDER BY column_id)
|
||||
INTO vAllCols
|
||||
FROM all_tab_columns
|
||||
WHERE table_name = pTableName -- Source: LEGACY_DEBT_DAILY
|
||||
AND owner = pSchemaName;
|
||||
|
||||
-- Columns ordered by SOURCE table structure (1, 2, ..., 72, ...)
|
||||
vResult := 'T.' || REPLACE(vAllCols, ', ', ', T.');
|
||||
|
||||
-- Add alias at original position (72)
|
||||
vResult := REPLACE(vResult, 'T.' || pKeyColumnName,
|
||||
'T.' || pKeyColumnName || ' AS A_WORKFLOW_HISTORY_KEY');
|
||||
|
||||
RETURN vResult;
|
||||
END IF;
|
||||
-- ...
|
||||
END;
|
||||
```
|
||||
|
||||
**Problem**: Generates columns in **LEGACY_DEBT_DAILY** order, not **CSDB_DEBT_DAILY_ODS** order!
|
||||
|
||||
---
|
||||
|
||||
## Real-World Impact Example
|
||||
|
||||
### Export Command
|
||||
```sql
|
||||
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
|
||||
pSchemaName => 'OU_CSDB',
|
||||
pTableName => 'LEGACY_DEBT_DAILY',
|
||||
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
|
||||
pBucketArea => 'DATA',
|
||||
pFolderName => 'ODS/CSDB/CSDB_DEBT_DAILY',
|
||||
pMinDate => DATE '2025-01-01',
|
||||
pMaxDate => SYSDATE,
|
||||
pParallelDegree => 8
|
||||
);
|
||||
```
|
||||
|
||||
### Result
|
||||
- ✅ **Export completed successfully** - 6 CSV files created
|
||||
- ✅ **Parquet export to ARCHIVE** - A_WORKFLOW_HISTORY_KEY populated correctly
|
||||
- ❌ **CSV read via ODS.CSDB_DEBT_DAILY_ODS** - Returns NULL for A_WORKFLOW_HISTORY_KEY
|
||||
|
||||
### Verification Query
|
||||
```sql
|
||||
-- Returns 0 non-NULL values
|
||||
SELECT COUNT(*)
|
||||
FROM ODS.CSDB_DEBT_DAILY_ODS
|
||||
WHERE A_WORKFLOW_HISTORY_KEY IS NOT NULL;
|
||||
|
||||
-- Returns: 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Solution Options
|
||||
|
||||
### Option 1: Rebuild External Tables (RECOMMENDED for existing system)
|
||||
**Action**: Update all ODS external table definitions to match source table column order
|
||||
|
||||
**Pros:**
|
||||
- No code changes in DATA_EXPORTER
|
||||
- Maintains backward compatibility
|
||||
- One-time migration effort
|
||||
|
||||
**Cons:**
|
||||
- Requires access to all external table definitions
|
||||
- Must coordinate with downstream consumers
|
||||
|
||||
---
|
||||
|
||||
### Option 2: Modify Export Logic (RECOMMENDED for new exports)
|
||||
**Action**: Update `processColumnList` to query external table structure and match column order
|
||||
|
||||
**Implementation:**
|
||||
```sql
|
||||
FUNCTION processColumnList(..., pExternalTableOwner VARCHAR2, pExternalTableName VARCHAR2) IS
|
||||
BEGIN
|
||||
IF pExternalTableName IS NOT NULL THEN
|
||||
-- Get column order from EXTERNAL table, not source table
|
||||
SELECT LISTAGG(column_name, ', ') WITHIN GROUP (ORDER BY column_id)
|
||||
INTO vAllCols
|
||||
FROM all_tab_columns
|
||||
WHERE table_name = pExternalTableName
|
||||
AND owner = pExternalTableOwner;
|
||||
|
||||
-- Build SELECT mapping source columns to external table order
|
||||
-- ...
|
||||
ELSE
|
||||
-- Existing logic (source table order)
|
||||
-- ...
|
||||
END IF;
|
||||
END;
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Future-proof for new tables
|
||||
- Explicit mapping control
|
||||
- Self-documenting exports
|
||||
|
||||
**Cons:**
|
||||
- Requires API change (new parameters)
|
||||
- More complex logic
|
||||
- Performance overhead (additional metadata query)
|
||||
|
||||
---
|
||||
|
||||
### Option 3: Use Parquet Format for ODS
|
||||
**Action**: Switch ODS exports from CSV to Parquet
|
||||
|
||||
**Pros:**
|
||||
- Eliminates column order issue
|
||||
- Better compression
|
||||
- Stronger typing
|
||||
|
||||
**Cons:**
|
||||
- May break downstream Airflow/DBT pipelines expecting CSV
|
||||
- Larger migration effort
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate Action (Hot Fix)
|
||||
1. ✅ Document issue (this document)
|
||||
2. Identify all affected CSV exports to ODS
|
||||
3. Choose between Option 1 (rebuild tables) or Option 2 (modify export)
|
||||
|
||||
### Long-term Strategy
|
||||
1. Migrate ODS exports to **Parquet format** where possible
|
||||
2. Maintain CSV only for systems requiring it
|
||||
3. Add validation tests comparing source vs external table column order
|
||||
4. Update documentation to warn about CSV positional mapping
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
- [x] Verify CSV positional mapping behavior
|
||||
- [x] Verify Parquet schema-based mapping behavior
|
||||
- [x] Confirm NULL values in ODS.CSDB_DEBT_DAILY_ODS
|
||||
- [x] Confirm correct values in CSDB_DEBT_DAILY_ARCHIVE (Parquet)
|
||||
- [ ] Test proposed solution
|
||||
- [ ] Validate all affected external tables
|
||||
- [ ] Update deployment procedures
|
||||
|
||||
---
|
||||
|
||||
## Related Issues
|
||||
- MARS-835-PREHOOK: Parallel processing implementation (exposed this issue)
|
||||
- A_WORKFLOW_HISTORY_KEY aliasing fix (partial - doesn't solve column order)
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
- Oracle Documentation: External Tables - Data Access Parameters
|
||||
- DBMS_CLOUD.EXPORT_DATA format options
|
||||
- Oracle External Tables Column Mapping Behavior (CSV vs Parquet)
|
||||
|
||||
---
|
||||
|
||||
**Document Status**: DRAFT - Awaiting solution decision
|
||||
**Last Updated**: 2026-01-08
|
||||
**Author**: GitHub Copilot (Analysis Session)
|
||||
BIN
confluence/additions/package_version_history.xlsx
Normal file
BIN
confluence/additions/package_version_history.xlsx
Normal file
Binary file not shown.
Reference in New Issue
Block a user