Mearge recent development
This commit is contained in:
Grzegorz Michalski
@@ -0,0 +1,205 @@
|
||||
# MARS-1046: External Table Direct Test Results
|
||||
|
||||
**Test Date:** 2026-01-13
|
||||
**Tester:** Grzegorz Michalski
|
||||
**Purpose:** Verify FILE_MANAGER.CREATE_EXTERNAL_TABLE with ISO 8601 date format
|
||||
|
||||
---
|
||||
|
||||
## Test Execution Summary
|
||||
|
||||
### Test Scenario
|
||||
Direct creation of external table using `ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` procedure to read CSV file with ISO 8601 timestamp format containing:
|
||||
- Date separator: `T` (literal character)
|
||||
- Milliseconds: `.FF3`
|
||||
- Timezone offset: `TZH:TZM`
|
||||
|
||||
### Test Steps Performed
|
||||
|
||||
#### Step 1: Template Table Creation
|
||||
- Template: `CT_ET_TEMPLATES.ISO8601_TEST_DATA`
|
||||
- Column: `EVENT_TIMESTAMP TIMESTAMP(3) WITH TIME ZONE`
|
||||
- Status: CREATED
|
||||
|
||||
#### Step 2: FILE_MANAGER Configuration
|
||||
- Source: `TEST`
|
||||
- File Config: `ISO8601_TEST_DATA`
|
||||
- Date Format: `YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM`
|
||||
- Status: CONFIGURED
|
||||
|
||||
#### Step 3: CSV File Upload
|
||||
- Bucket: `inbox`
|
||||
- Path: `INBOX/TEST/ISO8601/ISO8601_TEST_DATA/iso8601_test_data.csv`
|
||||
- Records: 5
|
||||
- Format: Full ISO 8601 with milliseconds and timezone
|
||||
- Status: UPLOADED
|
||||
|
||||
#### Step 4: External Table Creation
|
||||
```sql
|
||||
CALL ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
|
||||
pTableName => 'ISO8601_TEST_DATA_INBOX',
|
||||
pTemplateTableName => 'CT_ET_TEMPLATES.ISO8601_TEST_DATA',
|
||||
pPrefix => 'INBOX/TEST/ISO8601/ISO8601_TEST_DATA',
|
||||
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
|
||||
);
|
||||
```
|
||||
- External Table: `ODS.ISO8601_TEST_DATA_INBOX`
|
||||
- Type: `ORACLE_LOADER`
|
||||
- Status: CREATED
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL VERIFICATION: ACCESS PARAMETERS
|
||||
|
||||
### Expected Format
|
||||
```
|
||||
DATE_FORMAT TIMESTAMP WITH TIME ZONE MASK 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'
|
||||
```
|
||||
|
||||
### Actual Format (from all_external_tables)
|
||||
```
|
||||
RECORDS IGNORE_HEADER=1 DELIMITED BY DETECTED NEWLINE NOLOGFILE NOBADFILE NODISCARDFILE READSIZE=10000000 CREDENTIAL '"OCI$RESOURCE_PRINCIPAL"'
|
||||
FIELDS CSV WITHOUT EMBEDDED MISSING FIELD VALUES ARE NULL NOTRIM (
|
||||
"ID"
|
||||
,"EVENT_NAME" CHAR(100)
|
||||
,"EVENT_TIMESTAMP" CHAR(50) DATE_FORMAT TIMESTAMP WITH TIME ZONE MASK 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'
|
||||
,"DESCRIPTION" CHAR(500)
|
||||
)
|
||||
```
|
||||
|
||||
**RESULT: PASS**
|
||||
Separator `T` is correctly quoted as `"T"` in the MASK clause.
|
||||
|
||||
---
|
||||
|
||||
## Data Reading Test Results
|
||||
|
||||
### Test Query
|
||||
```sql
|
||||
SELECT
|
||||
ID,
|
||||
EVENT_NAME,
|
||||
TO_CHAR(EVENT_TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS.FF3 TZH:TZM') AS PARSED_TIMESTAMP,
|
||||
DESCRIPTION
|
||||
FROM ODS.ISO8601_TEST_DATA_INBOX
|
||||
ORDER BY ID;
|
||||
```
|
||||
|
||||
### Results
|
||||
| ID | EVENT_NAME | PARSED_TIMESTAMP | DESCRIPTION |
|
||||
|----|------------|------------------|-------------|
|
||||
| 1 | System Start | 2012-03-02 14:16:23.798 +01:00 | Initial system startup with timezone offset |
|
||||
| 2 | Data Processing | 2024-11-27 09:30:45.123 +01:00 | Batch processing job executed |
|
||||
| 3 | User Login | 2025-01-15 18:22:10.456 +02:00 | User authentication successful |
|
||||
| 4 | Error Event | 2023-07-20 03:45:00.789 +00:00 | Critical error detected in module X |
|
||||
| 5 | Backup Complete | 2024-12-31 23:59:59.999 +01:00 | Year-end backup finished successfully |
|
||||
|
||||
**RESULT: PASS**
|
||||
All 5 records successfully parsed with complete timestamp data.
|
||||
|
||||
---
|
||||
|
||||
## Verification Summary
|
||||
|
||||
### Completeness Check
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) AS TOTAL_RECORDS,
|
||||
COUNT(EVENT_TIMESTAMP) AS PARSED_TIMESTAMPS,
|
||||
CASE
|
||||
WHEN COUNT(*) = COUNT(EVENT_TIMESTAMP) THEN 'PASS - All timestamps parsed correctly'
|
||||
ELSE 'FAIL - Some timestamps NULL'
|
||||
END AS TEST_RESULT
|
||||
FROM ODS.ISO8601_TEST_DATA_INBOX;
|
||||
```
|
||||
|
||||
| TOTAL_RECORDS | PARSED_TIMESTAMPS | TEST_RESULT |
|
||||
|---------------|-------------------|-------------|
|
||||
| 5 | 5 | PASS - All timestamps parsed correctly |
|
||||
|
||||
---
|
||||
|
||||
## NORMALIZE_DATE_FORMAT Function Verification
|
||||
|
||||
### Function Behavior
|
||||
The `NORMALIZE_DATE_FORMAT` private function in `FILE_MANAGER` package body:
|
||||
1. Detects unquoted `T` separator in ISO 8601 format strings
|
||||
2. Automatically wraps `T` in double quotes for Oracle External Tables compatibility
|
||||
3. Maintains idempotency - does not re-quote already quoted separators
|
||||
|
||||
### Input/Output Examples
|
||||
| Input Format | Output Format | Status |
|
||||
|--------------|---------------|--------|
|
||||
| `YYYY-MM-DDTHH24:MI:SS.FF3TZH:TZM` | `YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM` | NORMALIZED |
|
||||
| `YYYY-MM-DD"T"HH24:MI:SS` | `YYYY-MM-DD"T"HH24:MI:SS` | UNCHANGED (idempotent) |
|
||||
| `DD/MM/YYYY HH24:MI:SS` | `DD/MM/YYYY HH24:MI:SS` | UNCHANGED (no T separator) |
|
||||
|
||||
---
|
||||
|
||||
## Test Results Summary
|
||||
|
||||
| Test Category | Test Case | Result | Notes |
|
||||
|---------------|-----------|--------|-------|
|
||||
| **Package Compilation** | FILE_MANAGER SPEC | PASS | VALID |
|
||||
| **Package Compilation** | FILE_MANAGER BODY | PASS | VALID |
|
||||
| **Version Tracking** | ENV_MANAGER.TRACK_PACKAGE | PASS | No untracked changes |
|
||||
| **External Table Creation** | CREATE_EXTERNAL_TABLE call | PASS | Table created successfully |
|
||||
| **Date Format Normalization** | T separator quoting | PASS | `"T"` correctly inserted |
|
||||
| **Access Parameters** | DDL format verification | PASS | MASK contains `YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM` |
|
||||
| **Data Reading** | CSV file parsing | PASS | All 5 records read |
|
||||
| **Timestamp Parsing** | Milliseconds preservation | PASS | .FF3 values preserved (.798, .123, .456, .789, .999) |
|
||||
| **Timestamp Parsing** | Timezone preservation | PASS | TZH:TZM values preserved (+01:00, +02:00, +00:00) |
|
||||
| **Data Completeness** | NULL check | PASS | 5/5 timestamps successfully parsed |
|
||||
|
||||
---
|
||||
|
||||
## FINAL VERDICT
|
||||
|
||||
**STATUS: ALL TESTS PASSED**
|
||||
|
||||
### Key Achievements
|
||||
1. External table successfully created using `FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE`
|
||||
2. ISO 8601 format with literal `T` separator correctly handled
|
||||
3. Date format automatically normalized with quoted `"T"`
|
||||
4. All CSV records successfully parsed from INBOX bucket
|
||||
5. Milliseconds and timezone information preserved in timestamps
|
||||
6. No NULL values in timestamp columns
|
||||
|
||||
### Production Readiness
|
||||
- Package FILE_MANAGER v3.3.1 is **PRODUCTION READY**
|
||||
- NORMALIZE_DATE_FORMAT function operates correctly
|
||||
- External table creation workflow validated end-to-end
|
||||
- ISO 8601 date parsing fully functional
|
||||
|
||||
---
|
||||
|
||||
## Test Artifacts
|
||||
|
||||
### Files Used
|
||||
- Template Table: `create_iso8601_test_template.sql`
|
||||
- Configuration: `configure_iso8601_test.sql`
|
||||
- Test Script: `test_create_external_table.sql`
|
||||
- Test Data: `iso8601_test_data.csv` (5 records with full ISO 8601 format)
|
||||
|
||||
### Database Objects Created
|
||||
- Template: `CT_ET_TEMPLATES.ISO8601_TEST_DATA`
|
||||
- External Table: `ODS.ISO8601_TEST_DATA_INBOX`
|
||||
- External Table: `ODS.ISO8601_TEST_DATA_ODS`
|
||||
|
||||
### Cleanup (Optional)
|
||||
```sql
|
||||
-- Drop external tables after testing
|
||||
DROP TABLE ODS.ISO8601_TEST_DATA_INBOX;
|
||||
DROP TABLE ODS.ISO8601_TEST_DATA_ODS;
|
||||
|
||||
-- Remove template table
|
||||
DROP TABLE CT_ET_TEMPLATES.ISO8601_TEST_DATA;
|
||||
|
||||
-- Remove configuration
|
||||
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(pSourceKey => 'TEST');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Test Completed:** 2026-01-13 13:24:48
|
||||
**Approved for Production Deployment:** YES
|
||||
Reference in New Issue
Block a user