admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a35e28042becd202292f23a8e2e83a17b78d9060
mars/confluence/Tables_setup.md

698 lines
27 KiB
Markdown
Raw Blame History

# Table Setup Guide for FILE PROCESSOR System
This document describes the process of setting up tables for the FILE PROCESSOR system using the example of `C2D_A_UC_DISSEM_METADATA_LOADS` table setup.
## Overview
The table setup process involves migrating existing tables from operational schemas (like `OU_C2D`) to the FILE PROCESSOR framework, which includes:
- Creating template tables for external table definitions
- Setting up external tables for different storage locations (INBOX, ODS, ARCHIVE)
- Exporting existing data to cloud storage
- Creating legacy backup tables
- Setting up views to maintain compatibility
## Step-by-Step Setup Process
**Recommended Approach:** Use the `ODS.FILE_MANAGER_ODS` package for creating external tables. The `FILE_MANAGER_ODS` package is a wrapper for the FILE_MANAGER package that uses `AUTHID DEFINER` instead of `AUTHID CURRENT_USER`, which means it will always create objects in the ODS schema regardless of which user executes the procedures.
**Alternative Approach:** If you prefer using the original package, you can use `CT_MRDS.FILE_MANAGER`, but the CREATE_EXTERNAL_TABLE procedure must be run as the ODS user due to the `AUTHID CURRENT_USER` clause. Export procedures (EXPORT_TABLE_DATA_TO_CSV_BY_DATE, EXPORT_TABLE_DATA_BY_DATE) have been moved to the `CT_MRDS.DATA_EXPORTER` package.
Example usage:
```sql
-- Recommended: Use DEFINER package (works from any user context)
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(...)
-- Alternative: Use CURRENT_USER package (requires ODS user context)
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(...)
```
### Step 1: Create Template Table
Create a template table in the `CT_ET_TEMPLATES` schema based on the existing operational table structure:
```sql
CREATE TABLE CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS
AS SELECT * FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS WHERE 1=2;
```
**Purpose:**
- The template table defines the structure for external tables
- Uses `WHERE 1=2` to copy only the structure, not the data
- Located in `CT_ET_TEMPLATES` schema for centralized template management
### Step 2: Configure FILE_MANAGER System
Before creating external tables, configure the FILE_MANAGER system with source information and file processing rules, as the `CREATE_EXTERNAL_TABLE` procedure uses data from configuration tables. For detailed information on configuring the FILE_MANAGER package procedures including:
- **ADD_SOURCE**: Registering new source systems
- **ADD_SOURCE_FILE_CONFIG**: Configuring file processing rules and naming patterns (**MARS-1049**: now includes `pEncoding` parameter for CSV character set support)
- **ADD_COLUMN_DATE_FORMAT**: Setting up date format handling for specific columns
See the comprehensive [FILE_MANAGER Configuration Guide](FILE_MANAGER_Configuration_Guide.md).
**MARS-1049 Enhancement Note**: The `ADD_SOURCE_FILE_CONFIG` procedure now supports the `pEncoding` parameter to specify character encodings (e.g., UTF8, WE8MSWIN1252) for proper international character handling in CSV files.
This configuration enables automatic file processing workflows where files uploaded to the INBOX are automatically processed and moved to ODS, with historical data archived in partitioned PARQUET format.
### Step 3: Create External Tables
Create external tables for different storage locations using the `FILE_MANAGER.CREATE_EXTERNAL_TABLE` procedure:
```sql
-- External table for INBOX (incoming files)
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
-- Alternative using CURRENT_USER package (requires ODS user context)
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
-- );
-- END;
-- /
-- External table for ODS (operational data store)
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
-- Alternative using CURRENT_USER package (requires ODS user context)
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
-- );
-- END;
-- /
-- External table for ARCHIVE (historical data)
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
-- Alternative using CURRENT_USER package (requires ODS user context)
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
-- );
-- END;
-- /
```
**Parameters:**
- `pTableName`: Name of the external table to create
- `pTemplateTableName`: Template table defining the structure
- `pPrefix`: Storage path prefix in Oracle Cloud Storage
- `pBucketUri`: URI of the target bucket (uses ENV_MANAGER global variables for different storage types)
**Storage Locations:**
- **INBOX**: For incoming files awaiting processing (uses `gvInboxBucketUri`)
- **ODS**: For processed files in operational data store (uses `gvDataBucketUri`)
- **ARCHIVE**: For historical/archived files (uses `gvArchiveBucketUri`)
### Step 4: Export Data to ODS Bucket (CSV Format)
Export existing operational data to ODS bucket in CSV format using the `DATA_EXPORTER` package:
```sql
-- Export all data to CSV format (using default pMinDate = 1900-01-01 and pMaxDate = SYSDATE)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_C2D', -- Source schema
pTableName => 'A_UC_DISSEM_METADATA_LOADS', -- Source table
pKeyColumnName => 'A_ETL_LOAD_SET_FK', -- Key column for partitioning
pBucketName => 'mrds_data_poc', -- Target bucket (ODS)
pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS' -- Target folder
);
END;
/
-- Export data within a specific date range to CSV
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketName => 'mrds_data_poc',
pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
pMinDate => DATE '2024-01-01', -- Export data from 2024-01-01
pMaxDate => DATE '2024-12-31' -- Export data up to 2024-12-31
);
END;
/
-- Export only recent data to CSV (last 30 days)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketName => 'mrds_data_poc',
pFolderName => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
pMinDate => SYSDATE - 30 -- Export data from last 30 days
);
END;
/
```
**Purpose:**
- Exports existing data to ODS bucket in CSV format for immediate operational use
- Creates CSV files partitioned by date (YYYYMM format)
- Files are ready for consumption by external tables and processing workflows
- Uses `mrds_data_poc` bucket which corresponds to `gvDataBucketUri`
### Step 5: Export Historical Data to Archive Bucket (PARQUET Format)
Export existing operational data to cloud storage using partitioning by date with the `DATA_EXPORTER` package:
```sql
-- Export all data (using default pMinDate = 1900-01-01 and pMaxDate = SYSDATE)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_C2D', -- Source schema
pTableName => 'A_UC_DISSEM_METADATA_LOADS', -- Source table
pKeyColumnName => 'A_ETL_LOAD_SET_FK', -- Key column for partitioning
pBucketName => 'mrds_history_poc', -- Target bucket
pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS' -- Target folder
);
END;
/
-- Export data within a specific date range
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketName => 'mrds_history_poc',
pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS',
pMinDate => DATE '2024-01-01', -- Export data from 2024-01-01
pMaxDate => DATE '2024-12-31' -- Export data up to 2024-12-31
);
END;
/
-- Export only recent data (last 30 days)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketName => 'mrds_history_poc',
pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS',
pMinDate => SYSDATE - 30 -- Export data from last 30 days
);
END;
/
```
**Purpose:**
- Exports existing data to cloud storage before switching to FILE_MANAGER
- Creates PARQUET files partitioned by date (YEAR_MONTH)
- Ensures data preservation during migration
### Step 6: Create Legacy Backup Table
Rename the original table to preserve existing data:
```sql
ALTER TABLE OU_C2D.A_UC_DISSEM_METADATA_LOADS
RENAME TO A_UC_DISSEM_METADATA_LOADS_LEGACY;
```
**Purpose:**
- Preserves original table as backup
- Allows rollback if needed
- Maintains data integrity during migration
### Step 7: Create Compatibility View
Create a view that points to the new ODS external table:
```sql
-- Grant access to the ODS external table
GRANT SELECT ON ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS TO OU_C2D;
-- Create view with original table name
CREATE OR REPLACE VIEW OU_C2D.A_UC_DISSEM_METADATA_LOADS AS
SELECT * FROM ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS;
```
**Purpose:**
- Maintains compatibility with existing applications
- Redirects queries to FILE PROCESSOR external tables
- Seamless transition for dependent systems
## Package Procedures Used
### FILE_MANAGER Package (External Tables)
**Recommended Approach:** Use the **ODS.FILE_MANAGER_ODS** package which uses `AUTHID DEFINER` and will create objects in the ODS schema regardless of the executing user. This is the most reliable approach for creating external tables.
**Alternative Approach:** You can use the `CT_MRDS.FILE_MANAGER` package, but the `CREATE_EXTERNAL_TABLE` procedure must be executed as the **ODS user** due to the `AUTHID CURRENT_USER` clause.
### DATA_EXPORTER Package (Data Export)
Data export procedures have been moved to the `CT_MRDS.DATA_EXPORTER` package, which provides enhanced functionality and better separation of concerns.
### CREATE_EXTERNAL_TABLE
Creates external tables that can read data from Oracle Cloud Storage.
**Signature:**
```sql
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ','
);
```
### DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE
Exports table data to cloud storage in CSV format with date-based partitioning.
**Signature:**
```sql
PROCEDURE EXPORT_TABLE_DATA_TO_CSV_BY_DATE (
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2, -- Updated: now uses pBucketArea instead of pBucketName
pFolderName IN VARCHAR2,
pFileName IN VARCHAR2 DEFAULT NULL,
pColumnList IN VARCHAR2 DEFAULT NULL, -- New: allows specifying custom columns
pMinDate IN DATE DEFAULT DATE '1900-01-01',
pMaxDate IN DATE DEFAULT SYSDATE,
pCredentialName IN VARCHAR2 DEFAULT ENV_MANAGER.gvCredentialName
);
```
**Parameters:**
- `pSchemaName`: Schema containing the source table
- `pTableName`: Name of the table to export
- `pKeyColumnName`: Column name used for date-based filtering (typically LOAD_START or similar date column)
- `pBucketName`: Oracle Cloud Storage bucket name for export
- `pFolderName`: Folder path within the bucket
- `pFileName`: Custom filename prefix (optional). If NULL, uses table name as prefix
- `pMinDate`: Minimum date for filtering records (defaults to DATE '1900-01-01'). Only records where the key column is >= pMinDate will be exported
- `pMaxDate`: Maximum date for filtering records (defaults to SYSDATE). Only records where the key column is <= pMaxDate will be exported
- `pNamespace`: OCI namespace (defaults to environment configuration)
- `pRegion`: OCI region (defaults to environment configuration)
- `pCredentialName`: OCI credentials (defaults to environment configuration)
**Purpose:**
- Creates separate CSV files partitioned by year and month (YYYYMM format)
- Files are immediately ready for consumption by external tables
- Ideal for operational data that needs to be processed quickly
- File naming pattern: `{pFileName}_YYYYMM.csv` or `{TABLE_NAME}_YYYYMM.csv` (if pFileName is NULL)
### DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE
Exports table data to cloud storage in PARQUET format with date-based partitioning.
**Signature:**
```sql
PROCEDURE EXPORT_TABLE_DATA_BY_DATE (
pSchemaName IN VARCHAR2,
pTableName IN VARCHAR2,
pKeyColumnName IN VARCHAR2,
pBucketArea IN VARCHAR2, -- Updated: now uses pBucketArea instead of pBucketName
pFolderName IN VARCHAR2,
pColumnList IN VARCHAR2 DEFAULT NULL, -- New: allows specifying custom columns
pMinDate IN DATE DEFAULT DATE '1900-01-01',
pMaxDate IN DATE DEFAULT SYSDATE,
pCredentialName IN VARCHAR2 DEFAULT ENV_MANAGER.gvCredentialName
);
```
**Parameters:**
- `pSchemaName`: Schema containing the source table
- `pTableName`: Name of the table to export
- `pKeyColumnName`: Column name used for date-based filtering (typically LOAD_START or similar date column)
- `pBucketArea`: Bucket area identifier ('INBOX', 'ODS', 'DATA', 'ARCHIVE') - automatically maps to correct bucket URI
- `pFolderName`: Folder path within the bucket
- `pColumnList`: Optional comma-separated list of columns to export (uses T.* if NULL)
- `pMinDate`: Minimum date for filtering records (defaults to DATE '1900-01-01'). Only records where the key column is >= pMinDate will be exported
- `pMaxDate`: Maximum date for filtering records (defaults to SYSDATE). Only records where the key column is <= pMaxDate will be exported
- `pCredentialName`: OCI credentials (defaults to environment configuration)
## Best Practices
### 1. Naming Conventions
- **Template tables**: `CT_ET_TEMPLATES.{SOURCE}_{TABLE_NAME}`
- **External tables**: `{SOURCE}_{TABLE_NAME}_{LOCATION}` (e.g., `_INBOX`, `_ODS`, `_ARCHIVE`)
- **Legacy tables**: `{ORIGINAL_NAME}_LEGACY`
### 2. Schema Organization
- **CT_ET_TEMPLATES**: Template table definitions
- **ODS**: External tables for processed data
- **OU_{SOURCE}**: Compatibility views and legacy tables
### 3. Storage Structure
```
Oracle Cloud Storage Buckets Structure:
INBOX Bucket:
├── INBOX/
│ └── {SOURCE}/
│ └── {SOURCE_FILE_ID}/
│ └── {TABLE_NAME}/
DATA Bucket:
├── ODS/
│ └── {SOURCE}/
│ └── {TABLE_NAME}/
└── TRASH/ -- File retention subfolder (not a separate bucket)
└── {SOURCE}/
└── {TABLE_NAME}/ -- CSV files after archival (ARCHIVED_AND_TRASHED status)
ARCHIVE Bucket:
└── ARCHIVE/
└── {SOURCE}/
└── {TABLE_NAME}/
└── PARTITION_YEAR=*/
└── PARTITION_MONTH=*/
└── *.parquet
Note: TRASH is a subfolder within the DATA bucket for file retention and rollback capability.
```
### 4. Migration Checklist
- [ ] Create template table
- [ ] Configure FILE_MANAGER system (ADD_SOURCE, ADD_SOURCE_FILE_CONFIG, ADD_COLUMN_DATE_FORMAT)
- [ ] Create external tables (INBOX, ODS, ARCHIVE)
- [ ] Export data to ODS bucket (CSV format) (optional) (data baudries)
- [ ] Export historical data to archive bucket (PARQUET format)
- [ ] Rename original table to legacy
- [ ] Create compatibility view
- [ ] Test data access through view
- [ ] Verify error handling
## Troubleshooting
### Common Issues
1. **Execution Context Issues**
- **Problem:** External tables created in wrong schema
- **Solution 1 (Recommended):** Use the DEFINER package that works from any user context
```sql
-- Use ODS.FILE_MANAGER_ODS instead of CT_MRDS.FILE_MANAGER
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(...);
END;
/
```
- **Solution 2 (Alternative):** Ensure CREATE_EXTERNAL_TABLE procedure is executed as ODS user
```sql
-- Check current user context
SELECT USER FROM DUAL;
-- Should return: ODS
```
2. **Permission Errors**
```sql
-- Grant necessary permissions
GRANT SELECT ON CT_ET_TEMPLATES.{TEMPLATE_TABLE} TO CT_MRDS;
GRANT SELECT ON ods.{EXTERNAL_TABLE} TO {TARGET_SCHEMA};
```
3. **External Table Creation Failures**
- Verify bucket and folder paths exist
- Check credential configuration
- Ensure template table structure is correct
4. **Data Export Issues**
- Verify source table has data
- Check key column exists and has appropriate data type
- Ensure sufficient storage space in target bucket
5. **File Validation Failures (ORA-20011)**
- **Problem:** CSV file contains more columns than template table allows
- **Error Message:** "EXCESS COLUMNS DETECTED! CSV file has X columns but template expects only Y"
- **Solutions:**
- Remove excess columns from CSV file before processing
- Add missing columns to template table
- Use `ANALYZE_VALIDATION_ERRORS()` function for detailed analysis
- **Prevention:** Ensure CSV structure matches template table before upload
6. **External Table Query Failures (ORA-29913/KUP-05002)**
- **Problem:** "ORA-29913: error while processing ODCIEXTTABLEOPEN routine" with "KUP-05002: There are no matching files for any file specification in the LOCATION clause"
- **Cause:** Oracle cannot find files in the specified external table location
- **Common Scenarios:**
- Empty bucket/folder - no files uploaded yet
- Incorrect file path in LOCATION clause
- Wrong file name pattern (e.g., looking for *.csv but files are *.txt)
- Case sensitivity issues in file names
- **Solutions:**
```sql
-- 1. Check external table location
SELECT table_name, location
FROM user_external_locations
WHERE table_name = 'YOUR_EXTERNAL_TABLE';
-- 2. List files in bucket to verify they exist
SELECT object_name FROM DBMS_CLOUD.LIST_OBJECTS(
credential_name => 'YOUR_CREDENTIAL',
location_uri => 'YOUR_BUCKET_URI',
prefix => 'YOUR_FOLDER_PREFIX'
);
-- 3. Test with a simple file upload
-- Upload a test CSV file to the exact location specified in external table
```
- **Prevention:** Always verify file location matches external table LOCATION clause exactly
### Verification Queries
```sql
-- Check template table structure
SELECT column_name, data_type, data_length
FROM user_tab_columns
WHERE table_name = '{TEMPLATE_TABLE_NAME}'
ORDER BY column_id;
-- Verify external table creation
SELECT table_name, table_type
FROM user_tables
WHERE table_name LIKE '%{SOURCE}%{TABLE}%';
-- Test view access
SELECT COUNT(*) FROM OU_{SOURCE}.{TABLE_NAME};
-- Check exported files
SELECT object_name
FROM dbms_cloud.list_objects('{BUCKET_NAME}', '{FOLDER_PREFIX}');
```
## Example: Complete Setup Script
Here's a complete example based on the C2D UC_DISSEM metadata loads table:
```sql
-- Step 1: Create template table
CREATE TABLE CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS
AS SELECT * FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS WHERE 1=2;
-- Step 2: Configure FILE_MANAGER system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
pSourceKey => 'C2D',
pSourceName => 'Central Bank Data System'
);
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'C2D',
pSourceFileType => 'INPUT',
pSourceFileId => 'UC_DISSEM',
pSourceFileDesc => 'UC Dissemination Metadata Loads',
pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
pTableId => 'A_UC_DISSEM_METADATA_LOADS',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pContainerFileKey => NULL
);
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pColumnName => 'A_ETL_LOAD_SET_FK',
pDateFormat => 'YYYY-MM-DD'
);
-- Step 3: Create external tables
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
-- Alternative using CURRENT_USER package (requires connection as ODS user):
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
-- );
-- END;
-- /
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ODS',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'ODS/C2D/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
-- );
-- END;
-- /
-- BEGIN
-- CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
-- pTableName => 'C2D_A_UC_DISSEM_METADATA_LOADS_ARCHIVE',
-- pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
-- pPrefix => 'ARCHIVE/C2D/A_UC_DISSEM_METADATA_LOADS',
-- pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
-- );
-- END;
-- /
-- Step 4: Export data to ODS bucket (CSV format)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketArea => 'ODS', -- Updated: uses bucket area instead of bucket name
pFolderName => 'C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS'
-- pMinDate defaults to DATE '1900-01-01' (exports all historical data)
-- pMaxDate defaults to SYSDATE (exports all data up to current date)
);
END;
/
-- Step 5: Export historical data to archive bucket (PARQUET format)
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_C2D',
pTableName => 'A_UC_DISSEM_METADATA_LOADS',
pKeyColumnName => 'A_ETL_LOAD_SET_FK',
pBucketArea => 'ARCHIVE', -- Updated: uses bucket area instead of bucket name
pFolderName => 'C2D/A_UC_DISSEM_METADATA_LOADS'
-- pMinDate defaults to DATE '1900-01-01' (exports all historical data)
-- pMaxDate defaults to SYSDATE (exports all data up to current date)
);
END;
/
-- Step 6: Create legacy backup
ALTER TABLE OU_C2D.A_UC_DISSEM_METADATA_LOADS RENAME TO A_UC_DISSEM_METADATA_LOADS_LEGACY;
-- Step 7: Create compatibility view
GRANT SELECT ON ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS TO OU_C2D;
CREATE OR REPLACE VIEW OU_C2D.A_UC_DISSEM_METADATA_LOADS AS
SELECT * FROM ods.C2D_A_UC_DISSEM_METADATA_LOADS_ODS;
-- Verify setup
SHOW ERRORS;
SELECT COUNT(*) FROM OU_C2D.A_UC_DISSEM_METADATA_LOADS;
```
## ENV_MANAGER Global Variables
The system uses `ENV_MANAGER` package global variables to manage different bucket URIs:
- `gvInboxBucketUri`: URI for incoming files bucket
- `gvDataBucketUri`: URI for operational data store bucket
- `gvArchiveBucketUri`: URI for historical/archive data bucket
This ensures consistent bucket configuration across all external table creation calls.
### Checking ENV_MANAGER Variable Values
To verify the current values of ENV_MANAGER global variables, use this script:
```sql
SET SERVEROUTPUT ON;
DECLARE
v_value1 VARCHAR2(4000);
v_value2 VARCHAR2(4000);
v_value3 VARCHAR2(4000);
BEGIN
v_value1 := CT_MRDS.ENV_MANAGER.gvInboxBucketUri;
v_value2 := CT_MRDS.ENV_MANAGER.gvDataBucketUri;
v_value3 := CT_MRDS.ENV_MANAGER.gvArchiveBucketUri;
DBMS_OUTPUT.PUT_LINE('------>>>> gvInboxBucketUri: ' || v_value1);
DBMS_OUTPUT.PUT_LINE('------>>>> gvDataBucketUri: ' || v_value2);
DBMS_OUTPUT.PUT_LINE('------>>>> gvArchiveBucketUri: ' || v_value3);
END;
/
```
This script helps troubleshoot external table creation issues by verifying that bucket URIs are properly configured.
## Summary
This process successfully migrates traditional Oracle tables to the FILE PROCESSOR framework, enabling cloud-based file processing while maintaining application compatibility. The migration includes:
1. **Template Creation** - Defining table structures in CT_ET_TEMPLATES schema
2. **FILE_MANAGER Configuration** - Setting up source systems, file processing rules, and date formats
3. **External Tables Setup** - Creating INBOX, ODS, and ARCHIVE external tables
4. **Data Export (CSV)** - Moving existing data to ODS bucket in CSV format for immediate processing
5. **Data Export (PARQUET)** - Moving historical data to archive bucket in PARQUET format for long-term storage
6. **Legacy Preservation** - Backing up original tables and creating compatibility views
After completion, your system will be ready for automated file processing with cloud-based storage and external table access patterns.
Reference in New Issue View Git Blame Copy Permalink