Init

2026-02-02 10:59:29 +01:00
commit ecd833f682
679 changed files with 122717 additions and 0 deletions
--- a/MARS_Packages/REL02/MARS-1046/README.md
+++ b/MARS_Packages/REL02/MARS-1046/README.md
@@ -0,0 +1,255 @@
+# MARS-1046: ISO 8601 Date Format Fix for FILE_MANAGER
+
+## Overview
+
+This package fixes parsing of ISO 8601 datetime formats with milliseconds and timezone in Oracle FILE_MANAGER external tables. The issue occurs when processing dates like `2012-03-02T14:16:23.798+01:00` where the literal 'T' separator between date and time components must be quoted in Oracle date format strings.
+
+## Problem Description
+
+**Issue**: When configuring date formats in `CT_MRDS.A_COLUMN_DATE_FORMAT` table with ISO 8601 format containing the 'T' separator, data is not correctly interpreted by external tables.
+
+**Example**:
+- **CSV Data**: `2012-03-02T14:16:23.798+01:00`
+- **Configured Format**: `YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM` (FAILS)
+- **Required Format**: `YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM` (WORKS)
+
+**Root Cause**: Oracle external table FIELD_LIST requires literal characters (like 'T') to be enclosed in double quotes. The configured format in `A_COLUMN_DATE_FORMAT` table does not include these quotes, causing parsing failures.
+
+## Solution
+
+MARS-1046 introduces a private function `NORMALIZE_DATE_FORMAT` that automatically normalizes date format strings before they are used in external table definitions. This function:
+
+1. Detects unquoted 'T' separators in ISO 8601 formats
+2. Wraps them in double quotes for Oracle compatibility
+3. Maintains backward compatibility with existing date formats
+4. Returns original format if normalization fails (safety feature)
+
+**Function Logic:**
+```sql
+-- Input:  'YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM'
+-- Output: 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'
+```
+
+The function is applied automatically in `GENERATE_EXTERNAL_TABLE_PARAMS` when building FIELD_LIST for DATE/TIMESTAMP columns.
+
+## Contents
+
+- `install_mars1046.sql` - Master installation script with SPOOL logging
+- `rollback_mars1046.sql` - Master rollback script
+- `01_MARS_1046_install_CT_MRDS_FILE_MANAGER_SPEC.sql` - Update package specification
+- `02_MARS_1046_install_CT_MRDS_FILE_MANAGER_BODY.sql` - Update package body
+- `91_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_BODY.sql` - Rollback package body
+- `92_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_SPEC.sql` - Rollback package specification
+- `track_package_versions.sql` - Package version tracking script
+- `verify_packages_version.sql` - Universal package verification script
+- `FILE_MANAGER.pkg` - Updated package specification (v3.3.1)
+- `FILE_MANAGER.pkb` - Updated package body (v3.3.1)
+
+## Prerequisites
+
+- Oracle Database 23ai
+- ENV_MANAGER package v3.1.0+
+- FILE_MANAGER package v3.3.0 (MARS-1056)
+- ADMIN user privileges
+
+## Installation
+
+### Option 1: Master Script (Recommended)
+
+**IMPORTANT**: Execute as ADMIN user for proper privilege management.
+
+```powershell
+# Using PowerShell
+Get-Content "MARS_Packages/REL02/MARS-1046/install_mars1046.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+
+# Log file created: INSTALL_MARS_1046_<PDB>_<timestamp>.log
+```
+
+### Option 2: Individual Scripts
+
+```powershell
+# IMPORTANT: Execute as ADMIN user
+Get-Content "01_MARS_1046_install_CT_MRDS_FILE_MANAGER_SPEC.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+Get-Content "02_MARS_1046_install_CT_MRDS_FILE_MANAGER_BODY.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+Get-Content "track_package_versions.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+Get-Content "verify_packages_version.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+```
+
+## Verification
+
+After installation, verify the changes:
+
+```sql
+-- Check package version
+SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
+-- Expected: 3.3.1
+
+-- Check build info
+SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
+-- Expected: Package: FILE_MANAGER
+--           Version: 3.3.1
+--           Build Date: 2025-11-27 14:00:00
+--           Author: Grzegorz Michalski
+
+-- Verify no untracked changes
+SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;
+-- Expected: OK: Package CT_MRDS.FILE_MANAGER has not changed.
+
+-- Check for compilation errors
+SELECT * FROM ALL_ERRORS 
+WHERE OWNER = 'CT_MRDS'
+  AND NAME = 'FILE_MANAGER'
+ORDER BY SEQUENCE;
+-- Expected: No rows returned
+```
+
+## Rollback
+
+If rollback is needed, use the master rollback script:
+
+```powershell
+# IMPORTANT: Execute as ADMIN user
+Get-Content "MARS_Packages/REL02/MARS-1046/rollback_mars1046.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
+
+# Log file created: ROLLBACK_MARS_1046_<PDB>_<timestamp>.log
+```
+
+This restores FILE_MANAGER to version 3.3.0 (MARS-1056).
+
+## Expected Changes
+
+**FILE_MANAGER Package:**
+- Version: 3.3.0 → 3.3.1 (PATCH - Bug fix)
+- New private function: `NORMALIZE_DATE_FORMAT`
+- Modified procedure: `GENERATE_EXTERNAL_TABLE_PARAMS`
+- Version history updated with MARS-1046 entry
+
+**No schema changes** - This is a code-only patch with no database structure modifications.
+
+## Testing
+
+### Test Scenario: ISO 8601 Date Format with T Separator
+
+**1. Configure date format with T separator:**
+```sql
+CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
+    pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE',
+    pColumnName => 'YOUR_DATE_COLUMN',
+    pDateFormat => 'YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM'
+);
+```
+
+**2. Create external table:**
+```sql
+BEGIN
+    CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+        pTableName => 'YOUR_EXTERNAL_TABLE',
+        pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE',
+        pPrefix => 'INBOX/SOURCE/FILE_ID/TABLE/',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
+        pFileName => NULL
+    );
+END;
+/
+```
+
+**3. Verify external table definition:**
+```sql
+-- Check generated FIELD_LIST for proper date format
+SELECT DBMS_METADATA.GET_DDL('TABLE', 'YOUR_EXTERNAL_TABLE', 'ODS') FROM DUAL;
+```
+
+**Expected Result**: Date column should have format: `DATE 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'` (with quoted T)
+
+**4. Test data parsing:**
+```sql
+-- Sample data in CSV: 2012-03-02T14:16:23.798+01:00
+SELECT YOUR_DATE_COLUMN FROM ODS.YOUR_EXTERNAL_TABLE WHERE ROWNUM <= 10;
+```
+
+**Expected Result**: Dates are parsed correctly without ORA-01858 errors.
+
+### Additional Test Cases
+
+**Test Case 1: Standard formats (no change)**
+- Format: `DD/MM/YYYY HH24:MI:SS`
+- Expected: No normalization applied (backward compatible)
+
+**Test Case 2: Other ISO 8601 variants**
+- Format: `YYYY-MM-DD HH24:MI:SS` (space separator)
+- Expected: Works as before (no T to normalize)
+
+**Test Case 3: NULL format**
+- Format: NULL
+- Expected: Falls back to ENV_MANAGER.gvDefaultDateFormat
+
+## Known Issues
+
+None currently identified.
+
+## Related
+
+- **MARS-1056**: VARCHAR2 CHAR/BYTE semantics fix
+- **MARS-1049**: CSV encoding support
+- **Package Deployment Guide**: `confluence/Package_Deployment_Guide.md`
+- **Copilot Instructions**: `.github/copilot-instructions.md`
+
+## Technical Details
+
+### NORMALIZE_DATE_FORMAT Function Signature
+
+```sql
+FUNCTION NORMALIZE_DATE_FORMAT(pDateFormat IN VARCHAR2)
+RETURN VARCHAR2
+IS
+   vNormalizedFormat VARCHAR2(200);
+BEGIN
+   IF pDateFormat IS NULL THEN
+      RETURN NULL;
+   END IF;
+
+   vNormalizedFormat := pDateFormat;
+
+   -- Fix ISO 8601 'T' separator
+   IF INSTR(vNormalizedFormat, '"T"') = 0 AND REGEXP_LIKE(vNormalizedFormat, '[YMD]T[HM]') THEN
+      vNormalizedFormat := REGEXP_REPLACE(vNormalizedFormat, '([YMD])T([HM])', '\\1"T"\\2');
+   END IF;
+
+   RETURN vNormalizedFormat;
+EXCEPTION
+   WHEN OTHERS THEN
+      RETURN pDateFormat; -- Safety: return original on error
+END NORMALIZE_DATE_FORMAT;
+```
+
+### Integration Point
+
+Modified in `GENERATE_EXTERNAL_TABLE_PARAMS`:
+
+```sql
+-- Before (MARS-1056):
+rec.quoted_column_name || ' DATE ' || CHR(39) || GET_DATE_FORMAT(...) || CHR(39)
+
+-- After (MARS-1046):
+rec.quoted_column_name || ' DATE ' || CHR(39) || NORMALIZE_DATE_FORMAT(GET_DATE_FORMAT(...)) || CHR(39)
+```
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.0.0 | 2025-11-27 | Initial release - ISO 8601 T separator fix |
+
+## Support
+
+For questions or issues:
+1. Check verification queries above
+2. Review process logs: `SELECT * FROM CT_MRDS.A_PROCESS_LOG WHERE LOG_TIMESTAMP > SYSDATE - 1 ORDER BY LOG_TIMESTAMP DESC;`
+3. Check package documentation: `SELECT GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;`
+4. Contact: Grzegorz Michalski
+
+---
+
+**Last Updated**: 2025-11-27  
+**Package Version**: FILE_MANAGER 3.3.1  
+**MARS Issue**: MARS-1046