Init

2026-02-02 10:59:29 +01:00
commit ecd833f682
679 changed files with 122717 additions and 0 deletions
--- a/confluence/Tables_setup.md
+++ b/confluence/Tables_setup.md
@@ -0,0 +1,692 @@
+# Table Setup Guide for FILE PROCESSOR System
+
+This document describes the process of setting up tables for the FILE PROCESSOR system using the example of `C2D_A_UC_DISSEM_METADATA_LOADS` table setup.
+
+## Overview
+
+The table setup process involves migrating existing tables from operational schemas (like `OU_C2D`) to the FILE PROCESSOR framework, which includes:
+- Creating template tables for external table definitions
+- Setting up external tables for different storage locations (INBOX, ODS, ARCHIVE)
+- Exporting existing data to cloud storage
+- Creating legacy backup tables
+- Setting up views to maintain compatibility
+
+## Step-by-Step Setup Process
+
+**Recommended Approach:** Use the `ODS.FILE_MANAGER_ODS` package for creating external tables. The `FILE_MANAGER_ODS` package is a wrapper for the FILE_MANAGER package that uses `AUTHID DEFINER` instead of `AUTHID CURRENT_USER`, which means it will always create objects in the ODS schema regardless of which user executes the procedures.
+
+**Alternative Approach:** If you prefer using the original package, you can use `CT_MRDS.FILE_MANAGER`, but the CREATE_EXTERNAL_TABLE procedure must be run as the ODS user due to the `AUTHID CURRENT_USER` clause. Export procedures (EXPORT_TABLE_DATA_TO_CSV_BY_DATE, EXPORT_TABLE_DATA_BY_DATE) have been moved to the `CT_MRDS.DATA_EXPORTER` package.
+
+Example usage:
+```sql
+-- Recommended: Use DEFINER package (works from any user context)
+ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(...)
+
+-- Alternative: Use CURRENT_USER package (requires ODS user context)
+-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(...)
+```
+
+### Step 1: Create Template Table
+
+Create a template table in the `CT_ET_TEMPLATES` schema based on the existing operational table structure:
+
+```sql
+CREATE TABLE CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS
+   AS SELECT * FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS WHERE 1=2;
+```
+
+**Purpose:**
+- The template table defines the structure for external tables
+- Uses `WHERE 1=2` to copy only the structure, not the data
+- Located in `CT_ET_TEMPLATES` schema for centralized template management
+
+### Step 2: Configure FILE_MANAGER System
+
+Before creating external tables, configure the FILE_MANAGER system with source information and file processing rules, as the `CREATE_EXTERNAL_TABLE` procedure uses data from configuration tables. For detailed information on configuring the FILE_MANAGER package procedures including:
+
+- **ADD_SOURCE**: Registering new source systems
+- **ADD_SOURCE_FILE_CONFIG**: Configuring file processing rules and naming patterns (**MARS-1049**: now includes `pEncoding` parameter for CSV character set support)
+- **ADD_COLUMN_DATE_FORMAT**: Setting up date format handling for specific columns
+
+See the comprehensive [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md).
+
+**MARS-1049 Enhancement Note**: The `ADD_SOURCE_FILE_CONFIG` procedure now supports the `pEncoding` parameter to specify character encodings (e.g., UTF8, WE8MSWIN1252) for proper international character handling in CSV files.
+
+This configuration enables automatic file processing workflows where files uploaded to the INBOX are automatically processed and moved to ODS, with historical data archived in partitioned PARQUET format.
+
+### Step 3: Create External Tables
+
+Create external tables for different storage locations using the `FILE_MANAGER.CREATE_EXTERNAL_TABLE` procedure:
+
+```sql
+-- External table for INBOX (incoming files)
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
+    );
+END;
+/
+
+-- Alternative using CURRENT_USER package (requires ODS user context)
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
+--     );
+-- END;
+-- /
+
+-- External table for ODS (operational data store)
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
+    );
+END;
+/
+
+-- Alternative using CURRENT_USER package (requires ODS user context)
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
+--     );
+-- END;
+-- /
+
+-- External table for ARCHIVE (historical data)
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
+    );
+END;
+/
+
+-- Alternative using CURRENT_USER package (requires ODS user context)
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
+--     );
+-- END;
+-- /
+```
+
+**Parameters:**
+- `pTableName`: Name of the external table to create
+- `pTemplateTableName`: Template table defining the structure
+- `pPrefix`: Storage path prefix in Oracle Cloud Storage
+- `pBucketUri`: URI of the target bucket (uses ENV_MANAGER global variables for different storage types)
+
+**Storage Locations:**
+- **INBOX**: For incoming files awaiting processing (uses `gvInboxBucketUri`)
+- **ODS**: For processed files in operational data store (uses `gvDataBucketUri`)
+- **ARCHIVE**: For historical/archived files (uses `gvArchiveBucketUri`)
+
+### Step 4: Export Data to ODS Bucket (CSV Format)
+
+Export existing operational data to ODS bucket in CSV format using the `DATA_EXPORTER` package:
+
+```sql
+-- Export all data to CSV format (using default pMinDate = 1900-01-01 and pMaxDate = SYSDATE)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
+        pSchemaName => 'OU_C2D',                                    -- Source schema
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',                -- Source table
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',                     -- Key column for partitioning
+        pBucketName => 'mrds_data_poc',                             -- Target bucket (ODS)
+        pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS' -- Target folder
+    );
+END;
+/
+
+-- Export data within a specific date range to CSV
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketName => 'mrds_data_poc',
+        pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pMinDate => DATE '2024-01-01',                              -- Export data from 2024-01-01
+        pMaxDate => DATE '2024-12-31'                               -- Export data up to 2024-12-31
+    );
+END;
+/
+
+-- Export only recent data to CSV (last 30 days)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketName => 'mrds_data_poc',
+        pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pMinDate => SYSDATE - 30                                   -- Export data from last 30 days
+    );
+END;
+/
+```
+
+**Purpose:**
+- Exports existing data to ODS bucket in CSV format for immediate operational use
+- Creates CSV files partitioned by date (YYYYMM format)
+- Files are ready for consumption by external tables and processing workflows
+- Uses `mrds_data_poc` bucket which corresponds to `gvDataBucketUri`
+
+### Step 5: Export Historical Data to Archive Bucket (PARQUET Format)
+
+Export existing operational data to cloud storage using partitioning by date with the `DATA_EXPORTER` package:
+
+```sql
+-- Export all data (using default pMinDate = 1900-01-01 and pMaxDate = SYSDATE)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
+        pSchemaName => 'OU_C2D',                                    -- Source schema
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',               -- Source table
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',                        -- Key column for partitioning
+        pBucketName => 'mrds_history_poc',                           -- Target bucket
+        pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS'   -- Target folder
+    );
+END;
+/
+
+-- Export data within a specific date range
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketName => 'mrds_history_poc',
+        pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS',
+        pMinDate => DATE '2024-01-01',                              -- Export data from 2024-01-01
+        pMaxDate => DATE '2024-12-31'                               -- Export data up to 2024-12-31
+    );
+END;
+/
+
+-- Export only recent data (last 30 days)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketName => 'mrds_history_poc',
+        pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS',
+        pMinDate => SYSDATE - 30                                   -- Export data from last 30 days
+    );
+END;
+/
+```
+
+**Purpose:**
+- Exports existing data to cloud storage before switching to FILE_MANAGER
+- Creates PARQUET files partitioned by date (YEAR_MONTH)
+- Ensures data preservation during migration
+
+### Step 6: Create Legacy Backup Table
+
+Rename the original table to preserve existing data:
+
+```sql
+ALTER TABLE OU_C2D.A_UC_DISSEM_METADATA_LOADS 
+    RENAME TO A_UC_DISSEM_METADATA_LOADS_LEGACY;
+```
+
+**Purpose:**
+- Preserves original table as backup
+- Allows rollback if needed
+- Maintains data integrity during migration
+
+### Step 7: Create Compatibility View
+
+Create a view that points to the new ODS external table:
+
+```sql
+-- Grant access to the ODS external table
+GRANT SELECT ON ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS TO OU_C2D;
+
+-- Create view with original table name
+CREATE OR REPLACE VIEW OU_C2D.A_UC_DISSEM_METADATA_LOADS AS 
+    SELECT * FROM ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS;
+```
+
+**Purpose:**
+- Maintains compatibility with existing applications
+- Redirects queries to FILE PROCESSOR external tables
+- Seamless transition for dependent systems
+
+## Package Procedures Used
+
+### FILE_MANAGER Package (External Tables)
+**Recommended Approach:** Use the **ODS.FILE_MANAGER_ODS** package which uses `AUTHID DEFINER` and will create objects in the ODS schema regardless of the executing user. This is the most reliable approach for creating external tables.
+
+**Alternative Approach:** You can use the `CT_MRDS.FILE_MANAGER` package, but the `CREATE_EXTERNAL_TABLE` procedure must be executed as the **ODS user** due to the `AUTHID CURRENT_USER` clause.
+
+### DATA_EXPORTER Package (Data Export)
+Data export procedures have been moved to the `CT_MRDS.DATA_EXPORTER` package, which provides enhanced functionality and better separation of concerns.
+
+### CREATE_EXTERNAL_TABLE
+
+Creates external tables that can read data from Oracle Cloud Storage.
+
+**Signature:**
+```sql
+PROCEDURE CREATE_EXTERNAL_TABLE (
+    pTableName          IN VARCHAR2,
+    pTemplateTableName  IN VARCHAR2,
+    pPrefix             IN VARCHAR2,
+    pBucketUri          IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
+    pFileName           IN VARCHAR2 DEFAULT NULL,
+    pDelimiter          IN VARCHAR2 DEFAULT ','
+);
+```
+
+### DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE
+
+Exports table data to cloud storage in CSV format with date-based partitioning.
+
+**Signature:**
+```sql
+PROCEDURE EXPORT_TABLE_DATA_TO_CSV_BY_DATE (
+    pSchemaName      IN VARCHAR2,
+    pTableName       IN VARCHAR2,
+    pKeyColumnName   IN VARCHAR2,
+    pBucketArea      IN VARCHAR2,        -- Updated: now uses pBucketArea instead of pBucketName
+    pFolderName      IN VARCHAR2,
+    pFileName        IN VARCHAR2 DEFAULT NULL,
+    pColumnList      IN VARCHAR2 DEFAULT NULL,  -- New: allows specifying custom columns
+    pMinDate         IN DATE DEFAULT DATE '1900-01-01',
+    pMaxDate         IN DATE DEFAULT SYSDATE,
+    pCredentialName  IN VARCHAR2 DEFAULT ENV_MANAGER.gvCredentialName
+);
+```
+
+**Parameters:**
+- `pSchemaName`: Schema containing the source table
+- `pTableName`: Name of the table to export
+- `pKeyColumnName`: Column name used for date-based filtering (typically LOAD_START or similar date column)
+- `pBucketName`: Oracle Cloud Storage bucket name for export
+- `pFolderName`: Folder path within the bucket
+- `pFileName`: Custom filename prefix (optional). If NULL, uses table name as prefix
+- `pMinDate`: Minimum date for filtering records (defaults to DATE '1900-01-01'). Only records where the key column is >= pMinDate will be exported
+- `pMaxDate`: Maximum date for filtering records (defaults to SYSDATE). Only records where the key column is <= pMaxDate will be exported
+- `pNamespace`: OCI namespace (defaults to environment configuration)
+- `pRegion`: OCI region (defaults to environment configuration)  
+- `pCredentialName`: OCI credentials (defaults to environment configuration)
+
+**Purpose:**
+- Creates separate CSV files partitioned by year and month (YYYYMM format)
+- Files are immediately ready for consumption by external tables
+- Ideal for operational data that needs to be processed quickly
+- File naming pattern: `{pFileName}_YYYYMM.csv` or `{TABLE_NAME}_YYYYMM.csv` (if pFileName is NULL)
+
+### DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE
+
+Exports table data to cloud storage in PARQUET format with date-based partitioning.
+
+**Signature:**
+```sql
+PROCEDURE EXPORT_TABLE_DATA_BY_DATE (
+    pSchemaName      IN VARCHAR2,
+    pTableName       IN VARCHAR2,
+    pKeyColumnName   IN VARCHAR2,
+    pBucketArea      IN VARCHAR2,        -- Updated: now uses pBucketArea instead of pBucketName
+    pFolderName      IN VARCHAR2,
+    pColumnList      IN VARCHAR2 DEFAULT NULL,  -- New: allows specifying custom columns
+    pMinDate         IN DATE DEFAULT DATE '1900-01-01',
+    pMaxDate         IN DATE DEFAULT SYSDATE,
+    pCredentialName  IN VARCHAR2 DEFAULT ENV_MANAGER.gvCredentialName
+);
+```
+
+**Parameters:**
+- `pSchemaName`: Schema containing the source table
+- `pTableName`: Name of the table to export
+- `pKeyColumnName`: Column name used for date-based filtering (typically LOAD_START or similar date column)
+- `pBucketArea`: Bucket area identifier ('INBOX', 'ODS', 'DATA', 'ARCHIVE') - automatically maps to correct bucket URI
+- `pFolderName`: Folder path within the bucket
+- `pColumnList`: Optional comma-separated list of columns to export (uses T.* if NULL)
+- `pMinDate`: Minimum date for filtering records (defaults to DATE '1900-01-01'). Only records where the key column is >= pMinDate will be exported
+- `pMaxDate`: Maximum date for filtering records (defaults to SYSDATE). Only records where the key column is <= pMaxDate will be exported
+- `pCredentialName`: OCI credentials (defaults to environment configuration)
+
+## Best Practices
+
+### 1. Naming Conventions
+
+- **Template tables**: `CT_ET_TEMPLATES.{SOURCE}_{TABLE_NAME}`
+- **External tables**: `{SOURCE}_{TABLE_NAME}_{LOCATION}` (e.g., `_INBOX`, `_ODS`, `_ARCHIVE`)
+- **Legacy tables**: `{ORIGINAL_NAME}_LEGACY`
+
+### 2. Schema Organization
+
+- **CT_ET_TEMPLATES**: Template table definitions
+- **ODS**: External tables for processed data
+- **OU_{SOURCE}**: Compatibility views and legacy tables
+
+### 3. Storage Structure
+
+```
+Oracle Cloud Storage Buckets Structure:
+
+INBOX Bucket:
+├── INBOX/
+│   └── {SOURCE}/
+│       └── {SOURCE_FILE_ID}/
+│           └── {TABLE_NAME}/
+
+DATA Bucket:
+├── ODS/
+│   └── {SOURCE}/
+│       └── {TABLE_NAME}/
+
+ARCHIVE Bucket:
+└── ARCHIVE/
+    └── {SOURCE}/
+        └── {TABLE_NAME}/
+            └── PARTITION_YEAR=*/
+                └── PARTITION_MONTH=*/
+                    └── *.parquet
+```
+
+### 4. Migration Checklist
+
+- [ ] Create template table
+- [ ] Configure FILE_MANAGER system (ADD_SOURCE, ADD_SOURCE_FILE_CONFIG, ADD_COLUMN_DATE_FORMAT)
+- [ ] Create external tables (INBOX, ODS, ARCHIVE)
+- [ ] Export data to ODS bucket (CSV format) (optional) (data baudries)
+- [ ] Export historical data to archive bucket (PARQUET format)
+- [ ] Rename original table to legacy
+- [ ] Create compatibility view
+- [ ] Test data access through view
+- [ ] Verify error handling
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Execution Context Issues**
+   - **Problem:** External tables created in wrong schema
+   - **Solution 1 (Recommended):** Use the DEFINER package that works from any user context
+   ```sql
+   -- Use ODS.FILE_MANAGER_ODS instead of CT_MRDS.FILE_MANAGER
+   BEGIN
+       ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(...);
+   END;
+   /
+   ```
+   - **Solution 2 (Alternative):** Ensure CREATE_EXTERNAL_TABLE procedure is executed as ODS user
+   ```sql
+   -- Check current user context
+   SELECT USER FROM DUAL;
+   -- Should return: ODS
+   ```
+
+2. **Permission Errors**
+   ```sql
+   -- Grant necessary permissions
+   GRANT SELECT ON CT_ET_TEMPLATES.{TEMPLATE_TABLE} TO CT_MRDS;
+   GRANT SELECT ON ods.{EXTERNAL_TABLE} TO {TARGET_SCHEMA};
+   ```
+
+3. **External Table Creation Failures**
+   - Verify bucket and folder paths exist
+   - Check credential configuration
+   - Ensure template table structure is correct
+
+4. **Data Export Issues**
+   - Verify source table has data
+   - Check key column exists and has appropriate data type
+   - Ensure sufficient storage space in target bucket
+
+5. **File Validation Failures (ORA-20011)**
+   - **Problem:** CSV file contains more columns than template table allows
+   - **Error Message:** "EXCESS COLUMNS DETECTED! CSV file has X columns but template expects only Y"
+   - **Solutions:**
+     - Remove excess columns from CSV file before processing
+     - Add missing columns to template table
+     - Use `ANALYZE_VALIDATION_ERRORS()` function for detailed analysis
+   - **Prevention:** Ensure CSV structure matches template table before upload
+
+6. **External Table Query Failures (ORA-29913/KUP-05002)**
+   - **Problem:** "ORA-29913: error while processing ODCIEXTTABLEOPEN routine" with "KUP-05002: There are no matching files for any file specification in the LOCATION clause"
+   - **Cause:** Oracle cannot find files in the specified external table location
+   - **Common Scenarios:**
+     - Empty bucket/folder - no files uploaded yet
+     - Incorrect file path in LOCATION clause
+     - Wrong file name pattern (e.g., looking for *.csv but files are *.txt)
+     - Case sensitivity issues in file names
+   - **Solutions:**
+     ```sql
+     -- 1. Check external table location
+     SELECT table_name, location 
+     FROM user_external_locations 
+     WHERE table_name = 'YOUR_EXTERNAL_TABLE';
+     
+     -- 2. List files in bucket to verify they exist
+     SELECT object_name FROM DBMS_CLOUD.LIST_OBJECTS(
+         credential_name => 'YOUR_CREDENTIAL',
+         location_uri => 'YOUR_BUCKET_URI',
+         prefix => 'YOUR_FOLDER_PREFIX'
+     );
+     
+     -- 3. Test with a simple file upload
+     -- Upload a test CSV file to the exact location specified in external table
+     ```
+   - **Prevention:** Always verify file location matches external table LOCATION clause exactly
+
+### Verification Queries
+
+```sql
+-- Check template table structure
+SELECT column_name, data_type, data_length 
+FROM user_tab_columns 
+WHERE table_name = '{TEMPLATE_TABLE_NAME}'
+ORDER BY column_id;
+
+-- Verify external table creation
+SELECT table_name, table_type 
+FROM user_tables 
+WHERE table_name LIKE '%{SOURCE}%{TABLE}%';
+
+-- Test view access
+SELECT COUNT(*) FROM OU_{SOURCE}.{TABLE_NAME};
+
+-- Check exported files
+SELECT object_name 
+FROM dbms_cloud.list_objects('{BUCKET_NAME}', '{FOLDER_PREFIX}');
+```
+
+## Example: Complete Setup Script
+
+Here's a complete example based on the C2D UC_DISSEM metadata loads table:
+
+```sql
+-- Step 1: Create template table
+CREATE TABLE CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS
+   AS SELECT * FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS WHERE 1=2;
+
+-- Step 2: Configure FILE_MANAGER system
+CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
+    pSourceKey => 'C2D', 
+    pSourceName => 'Central Bank Data System'
+);
+
+CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
+    pSourceKey => 'C2D',
+    pSourceFileType => 'INPUT',
+    pSourceFileId => 'UC_DISSEM',
+    pSourceFileDesc => 'UC Dissemination Metadata Loads',
+    pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
+    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
+    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+    pContainerFileKey => NULL
+);
+
+CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
+    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+    pColumnName => 'A_ETL_LOAD_SET_FK',
+    pDateFormat => 'YYYY-MM-DD'
+);
+
+-- Step 3: Create external tables
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
+    );
+END;
+/
+
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
+    );
+END;
+/
+
+BEGIN
+    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
+        pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
+        pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+        pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
+        pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
+    );
+END;
+/
+
+-- Alternative using CURRENT_USER package (requires connection as ODS user):
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
+--     );
+-- END;
+-- /
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
+--     );
+-- END;
+-- /
+-- BEGIN
+--     CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
+--         pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
+--         pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
+--         pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
+--         pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
+--     );
+-- END;
+-- /
+
+-- Step 4: Export data to ODS bucket (CSV format)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketArea => 'ODS',  -- Updated: uses bucket area instead of bucket name
+        pFolderName => 'C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS'
+        -- pMinDate defaults to DATE '1900-01-01' (exports all historical data)
+        -- pMaxDate defaults to SYSDATE (exports all data up to current date)
+    );
+END;
+/
+
+-- Step 5: Export historical data to archive bucket (PARQUET format)
+BEGIN
+    CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
+        pSchemaName => 'OU_C2D',
+        pTableName => 'A_UC_DISSEM_METADATA_LOADS',
+        pKeyColumnName => 'A_ETL_LOAD_SET_FK',
+        pBucketArea => 'ARCHIVE',  -- Updated: uses bucket area instead of bucket name
+        pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS'
+        -- pMinDate defaults to DATE '1900-01-01' (exports all historical data)
+        -- pMaxDate defaults to SYSDATE (exports all data up to current date)
+    );
+END;
+/
+
+-- Step 6: Create legacy backup
+ALTER TABLE OU_C2D.A_UC_DISSEM_METADATA_LOADS RENAME TO A_UC_DISSEM_METADATA_LOADS_LEGACY;
+
+-- Step 7: Create compatibility view
+GRANT SELECT ON ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS TO OU_C2D;
+CREATE OR REPLACE VIEW OU_C2D.A_UC_DISSEM_METADATA_LOADS AS 
+    SELECT * FROM ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS;
+
+-- Verify setup
+SHOW ERRORS;
+SELECT COUNT(*) FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS;
+```
+
+## ENV_MANAGER Global Variables
+
+The system uses `ENV_MANAGER` package global variables to manage different bucket URIs:
+
+- `gvInboxBucketUri`: URI for incoming files bucket
+- `gvDataBucketUri`: URI for operational data store bucket  
+- `gvArchiveBucketUri`: URI for historical/archive data bucket
+
+This ensures consistent bucket configuration across all external table creation calls.
+
+### Checking ENV_MANAGER Variable Values
+
+To verify the current values of ENV_MANAGER global variables, use this script:
+
+```sql
+SET SERVEROUTPUT ON;
+DECLARE
+  v_value1 VARCHAR2(4000);
+  v_value2 VARCHAR2(4000);
+  v_value3 VARCHAR2(4000);
+BEGIN
+  v_value1 := CT_MRDS.ENV_MANAGER.gvInboxBucketUri;
+  v_value2 := CT_MRDS.ENV_MANAGER.gvDataBucketUri;
+  v_value3 := CT_MRDS.ENV_MANAGER.gvArchiveBucketUri;
+  DBMS_OUTPUT.PUT_LINE('------>>>> gvInboxBucketUri: ' || v_value1);
+  DBMS_OUTPUT.PUT_LINE('------>>>> gvDataBucketUri: ' || v_value2);
+  DBMS_OUTPUT.PUT_LINE('------>>>> gvArchiveBucketUri: ' || v_value3);
+END;
+/
+```
+
+This script helps troubleshoot external table creation issues by verifying that bucket URIs are properly configured.
+
+## Summary
+
+This process successfully migrates traditional Oracle tables to the FILE PROCESSOR framework, enabling cloud-based file processing while maintaining application compatibility. The migration includes:
+
+1. **Template Creation** - Defining table structures in CT_ET_TEMPLATES schema
+2. **FILE_MANAGER Configuration** - Setting up source systems, file processing rules, and date formats
+3. **External Tables Setup** - Creating INBOX, ODS, and ARCHIVE external tables
+4. **Data Export (CSV)** - Moving existing data to ODS bucket in CSV format for immediate processing
+5. **Data Export (PARQUET)** - Moving historical data to archive bucket in PARQUET format for long-term storage
+6. **Legacy Preservation** - Backing up original tables and creating compatibility views
+
+After completion, your system will be ready for automated file processing with cloud-based storage and external table access patterns.