mars/MARS_Packages/REL02/MARS-1046/README.md

# MARS-1046: ISO 8601 Date Format Fix for FILE_MANAGER

## Overview

This package fixes parsing of ISO 8601 datetime formats with milliseconds and timezone in Oracle FILE_MANAGER external tables. The issue occurs when processing dates like `2012-03-02T14:16:23.798+01:00` where the literal 'T' separator between date and time components must be quoted in Oracle date format strings.

## Problem Description

**Issue**: When configuring date formats in `CT_MRDS.A_COLUMN_DATE_FORMAT` table with ISO 8601 format containing the 'T' separator, data is not correctly interpreted by external tables.

**Example**:
- **CSV Data**: `2012-03-02T14:16:23.798+01:00`
- **Configured Format**: `YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM` ❌ (fails)
- **Required Format**: `YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM` ✅ (works)

**Root Cause**: Oracle external table FIELD_LIST requires literal characters (like 'T') to be enclosed in double quotes. The configured format in `A_COLUMN_DATE_FORMAT` table does not include these quotes, causing parsing failures.

## Solution

MARS-1046 introduces a private function `NORMALIZE_DATE_FORMAT` that automatically normalizes date format strings before they are used in external table definitions. This function:

1. Detects unquoted 'T' separators in ISO 8601 formats
2. Wraps them in double quotes for Oracle compatibility
3. Maintains backward compatibility with existing date formats
4. Returns original format if normalization fails (safety feature)

**Function Logic:**
```sql
-- Input:  'YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM'
-- Output: 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'
```

The function is applied automatically in `GENERATE_EXTERNAL_TABLE_PARAMS` when building FIELD_LIST for DATE/TIMESTAMP columns.

## Contents

- `install_mars1046.sql` - Master installation script with SPOOL logging
- `rollback_mars1046.sql` - Master rollback script
- `01_MARS_1046_install_CT_MRDS_FILE_MANAGER_SPEC.sql` - Update package specification
- `02_MARS_1046_install_CT_MRDS_FILE_MANAGER_BODY.sql` - Update package body
- `91_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_BODY.sql` - Rollback package body
- `92_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_SPEC.sql` - Rollback package specification
- `track_package_versions.sql` - Package version tracking script
- `verify_packages_version.sql` - Universal package verification script
- `FILE_MANAGER.pkg` - Updated package specification (v3.3.1)
- `FILE_MANAGER.pkb` - Updated package body (v3.3.1)

## Prerequisites

- Oracle Database 23ai
- ENV_MANAGER package v3.1.0+
- FILE_MANAGER package v3.3.0 (MARS-1056)
- ADMIN user privileges

## Installation

### Option 1: Master Script (Recommended)

**IMPORTANT**: Execute as ADMIN user for proper privilege management.

```powershell
# Using PowerShell
Get-Content "MARS_Packages/REL02/MARS-1046/install_mars1046.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"

# Log file created: INSTALL_MARS_1046_<PDB>_<timestamp>.log
```

### Option 2: Individual Scripts

```powershell
# IMPORTANT: Execute as ADMIN user
Get-Content "01_MARS_1046_install_CT_MRDS_FILE_MANAGER_SPEC.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
Get-Content "02_MARS_1046_install_CT_MRDS_FILE_MANAGER_BODY.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
Get-Content "track_package_versions.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
Get-Content "verify_packages_version.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"
```

## Verification

After installation, verify the changes:

```sql
-- Check package version
SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
-- Expected: 3.3.1

-- Check build info
SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
-- Expected: Package: FILE_MANAGER
--           Version: 3.3.1
--           Build Date: 2025-11-27 14:00:00
--           Author: Grzegorz Michalski

-- Verify no untracked changes
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;
-- Expected: OK: Package CT_MRDS.FILE_MANAGER has not changed.

-- Check for compilation errors
SELECT * FROM ALL_ERRORS 
WHERE OWNER = 'CT_MRDS'
  AND NAME = 'FILE_MANAGER'
ORDER BY SEQUENCE;
-- Expected: No rows returned
```

## Rollback

If rollback is needed, use the master rollback script:

```powershell
# IMPORTANT: Execute as ADMIN user
Get-Content "MARS_Packages/REL02/MARS-1046/rollback_mars1046.sql" | sql "ADMIN/Cloudpass#34@ggmichalski_high"

# Log file created: ROLLBACK_MARS_1046_<PDB>_<timestamp>.log
```

This restores FILE_MANAGER to version 3.3.0 (MARS-1056).

## Expected Changes

**FILE_MANAGER Package:**
- Version: 3.3.0 → 3.3.1 (PATCH - Bug fix)
- New private function: `NORMALIZE_DATE_FORMAT`
- Modified procedure: `GENERATE_EXTERNAL_TABLE_PARAMS`
- Version history updated with MARS-1046 entry

**No schema changes** - This is a code-only patch with no database structure modifications.

## Testing

### Test Scenario: ISO 8601 Date Format with T Separator

**1. Configure date format with T separator:**
```sql
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE',
    pColumnName => 'YOUR_DATE_COLUMN',
    pDateFormat => 'YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM'
);
```

**2. Create external table:**
```sql
BEGIN
    CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
        pTableName => 'YOUR_EXTERNAL_TABLE',
        pTemplateTableName => 'CT_ET_TEMPLATES.YOUR_TEMPLATE',
        pPrefix => 'INBOX/SOURCE/FILE_ID/TABLE/',
        pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
        pFileName => NULL
    );
END;
/
```

**3. Verify external table definition:**
```sql
-- Check generated FIELD_LIST for proper date format
SELECT DBMS_METADATA.GET_DDL('TABLE', 'YOUR_EXTERNAL_TABLE', 'ODS') FROM DUAL;
```

**Expected Result**: Date column should have format: `DATE 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'` (with quoted T)

**4. Test data parsing:**
```sql
-- Sample data in CSV: 2012-03-02T14:16:23.798+01:00
SELECT YOUR_DATE_COLUMN FROM ODS.YOUR_EXTERNAL_TABLE WHERE ROWNUM <= 10;
```

**Expected Result**: Dates are parsed correctly without ORA-01858 errors.

### Additional Test Cases

**Test Case 1: Standard formats (no change)**
- Format: `DD/MM/YYYY HH24:MI:SS`
- Expected: No normalization applied (backward compatible)

**Test Case 2: Other ISO 8601 variants**
- Format: `YYYY-MM-DD HH24:MI:SS` (space separator)
- Expected: Works as before (no T to normalize)

**Test Case 3: NULL format**
- Format: NULL
- Expected: Falls back to ENV_MANAGER.gvDefaultDateFormat

## Known Issues

None currently identified.

## Related

- **MARS-1056**: VARCHAR2 CHAR/BYTE semantics fix
- **MARS-1049**: CSV encoding support
- **Package Deployment Guide**: `confluence/Package_Deployment_Guide.md`
- **Copilot Instructions**: `.github/copilot-instructions.md`

## Technical Details

### NORMALIZE_DATE_FORMAT Function Signature

```sql
FUNCTION NORMALIZE_DATE_FORMAT(pDateFormat IN VARCHAR2)
RETURN VARCHAR2
IS
   vNormalizedFormat VARCHAR2(200);
BEGIN
   IF pDateFormat IS NULL THEN
      RETURN NULL;
   END IF;

   vNormalizedFormat := pDateFormat;

   -- Fix ISO 8601 'T' separator
   IF INSTR(vNormalizedFormat, '"T"') = 0 AND REGEXP_LIKE(vNormalizedFormat, '[YMD]T[HM]') THEN
      vNormalizedFormat := REGEXP_REPLACE(vNormalizedFormat, '([YMD])T([HM])', '\\1"T"\\2');
   END IF;

   RETURN vNormalizedFormat;
EXCEPTION
   WHEN OTHERS THEN
      RETURN pDateFormat; -- Safety: return original on error
END NORMALIZE_DATE_FORMAT;
```

### Integration Point

Modified in `GENERATE_EXTERNAL_TABLE_PARAMS`:

```sql
-- Before (MARS-1056):
rec.quoted_column_name || ' DATE ' || CHR(39) || GET_DATE_FORMAT(...) || CHR(39)

-- After (MARS-1046):
rec.quoted_column_name || ' DATE ' || CHR(39) || NORMALIZE_DATE_FORMAT(GET_DATE_FORMAT(...)) || CHR(39)
```

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2025-11-27 | Initial release - ISO 8601 T separator fix |

## Support

For questions or issues:
1. Check verification queries above
2. Review process logs: `SELECT * FROM CT_MRDS.A_PROCESS_LOG WHERE LOG_TIMESTAMP > SYSDATE - 1 ORDER BY LOG_TIMESTAMP DESC;`
3. Check package documentation: `SELECT GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;`
4. Contact: Grzegorz Michalski

---

**Last Updated**: 2025-11-27  
**Package Version**: FILE_MANAGER 3.3.1  
**MARS Issue**: MARS-1046