377 lines
11 KiB
Markdown
377 lines
11 KiB
Markdown
# MARS-1057: Batch External Table Creation
|
|
|
|
## Overview
|
|
This MARS package adds batch external table creation capabilities to the FILE_MANAGER and FILE_MANAGER_ODS packages, enabling automatic creation of external table sets (INBOX, ODS, ARCHIVE) based on A_SOURCE_FILE_CONFIG metadata.
|
|
|
|
**Jira Issue:** MARS-1057
|
|
**Package Version:** FILE_MANAGER 3.4.0, FILE_MANAGER_ODS 2.2.0
|
|
**Author:** Grzegorz Michalski
|
|
**Date:** 2025-11-27
|
|
|
|
## Contents
|
|
- `install_mars1057.sql` - Master installation script with SPOOL logging
|
|
- `rollback_mars1057.sql` - Master rollback script
|
|
- `01_MARS_1057_install_CT_MRDS_FILE_MANAGER_SPEC.sql` - Install FILE_MANAGER package specification
|
|
- `02_MARS_1057_install_CT_MRDS_FILE_MANAGER_BODY.sql` - Install FILE_MANAGER package body
|
|
- `03_MARS_1057_install_ODS_FILE_MANAGER_ODS_SPEC.sql` - Install FILE_MANAGER_ODS package specification
|
|
- `04_MARS_1057_install_ODS_FILE_MANAGER_ODS_BODY.sql` - Install FILE_MANAGER_ODS package body
|
|
- `91_MARS_1057_rollback_CT_MRDS_FILE_MANAGER_SPEC.sql` - Rollback FILE_MANAGER package specification
|
|
- `92_MARS_1057_rollback_CT_MRDS_FILE_MANAGER_BODY.sql` - Rollback FILE_MANAGER package body
|
|
- `93_MARS_1057_rollback_ODS_FILE_MANAGER_ODS_BODY.sql` - Rollback FILE_MANAGER_ODS package body
|
|
- `94_MARS_1057_rollback_ODS_FILE_MANAGER_ODS_SPEC.sql` - Rollback FILE_MANAGER_ODS package specification
|
|
- `track_package_versions.sql` - Universal package version tracking
|
|
- `verify_packages_version.sql` - Universal package verification
|
|
- `rollback_version/` - FILE_MANAGER v3.3.0, FILE_MANAGER_ODS v2.1.0 (before MARS-1057)
|
|
- `new_version/` - FILE_MANAGER v3.4.0, FILE_MANAGER_ODS v2.2.0 (after MARS-1057)
|
|
- `.gitignore` - Git exclusions for temporary files
|
|
|
|
## Prerequisites
|
|
- Oracle Database 23ai
|
|
- FILE_MANAGER package v3.3.0 installed
|
|
- FILE_MANAGER_ODS package v2.1.0 installed
|
|
- ENV_MANAGER package v3.1.0+ with version tracking
|
|
- ADMIN user access for deployment
|
|
- ODS schema with necessary privileges
|
|
|
|
## New Features
|
|
|
|
### 1. CREATE_EXTERNAL_TABLES_SET
|
|
Creates a complete set of 3 external tables (INBOX, ODS, ARCHIVE) for a single configuration from A_SOURCE_FILE_CONFIG.
|
|
|
|
**Signature:**
|
|
```sql
|
|
PROCEDURE CREATE_EXTERNAL_TABLES_SET (
|
|
pSourceFileConfigKey IN NUMBER,
|
|
pRecreate IN BOOLEAN DEFAULT FALSE
|
|
);
|
|
```
|
|
|
|
**Example:**
|
|
```sql
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123,
|
|
pRecreate => FALSE
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
**Features:**
|
|
- Automatic table naming: `{TABLE_ID}_{INBOX|ODS|ARCHIVE}`
|
|
- Official path patterns compliance
|
|
- Optional drop and recreate
|
|
- Full ENV_MANAGER logging
|
|
- Error handling with detailed messages
|
|
|
|
### 2. CREATE_EXTERNAL_TABLES_BATCH
|
|
Creates external table sets for multiple configurations based on filter criteria.
|
|
|
|
**Signature:**
|
|
```sql
|
|
PROCEDURE CREATE_EXTERNAL_TABLES_BATCH (
|
|
pSourceKey IN VARCHAR2 DEFAULT NULL,
|
|
pSourceFileId IN VARCHAR2 DEFAULT NULL,
|
|
pTableId IN VARCHAR2 DEFAULT NULL,
|
|
pRecreate IN BOOLEAN DEFAULT FALSE
|
|
);
|
|
```
|
|
|
|
**Examples:**
|
|
```sql
|
|
-- All external tables for C2D source
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pSourceKey => 'C2D'
|
|
);
|
|
END;
|
|
/
|
|
|
|
-- Recreate all external tables
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pRecreate => TRUE
|
|
);
|
|
END;
|
|
/
|
|
|
|
-- Specific table across all sources
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pTableId => 'A_UC_DISSEM_METADATA_LOADS'
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
**Features:**
|
|
- Filters only INPUT type files
|
|
- Continues processing on errors
|
|
- Returns summary (Total/Processed/Failed)
|
|
- Comprehensive logging
|
|
|
|
### 3. FILE_MANAGER_ODS Wrapper Procedures
|
|
The ODS.FILE_MANAGER_ODS package now includes DEFINER rights wrappers for the new batch procedures.
|
|
|
|
**New Wrappers:**
|
|
```sql
|
|
-- Create single external table set via ODS wrapper
|
|
EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123,
|
|
pRecreate => FALSE
|
|
);
|
|
|
|
-- Create batch external tables via ODS wrapper
|
|
EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pSourceKey => 'C2D',
|
|
pRecreate => FALSE
|
|
);
|
|
```
|
|
|
|
**Benefits:**
|
|
- Objects created in ODS schema regardless of execution user
|
|
- Same functionality as CT_MRDS.FILE_MANAGER with DEFINER rights
|
|
- Full logging and error handling
|
|
- Consistent API with original package
|
|
|
|
## Installation
|
|
|
|
### Option 1: Master Script (Recommended)
|
|
```powershell
|
|
# IMPORTANT: Execute as ADMIN user
|
|
Get-Content "MARS_Packages/REL03/MARS-1057/install_mars1057.sql" | sql "ADMIN/password@service"
|
|
|
|
# Log file created: log/INSTALL_MARS_1057_<PDB>_<timestamp>.log
|
|
```
|
|
|
|
**Installation Steps:**
|
|
1. Install FILE_MANAGER package specification v3.4.0
|
|
2. Install FILE_MANAGER package body v3.4.0
|
|
3. Install FILE_MANAGER_ODS package specification v2.2.0
|
|
4. Install FILE_MANAGER_ODS package body v2.2.0
|
|
5. Track versions in A_PACKAGE_VERSION_TRACKING
|
|
6. Verify all tracked packages for untracked changes
|
|
|
|
### Option 2: Individual Scripts
|
|
```powershell
|
|
# IMPORTANT: Execute as ADMIN user
|
|
Get-Content "01_MARS_1057_install_CT_MRDS_FILE_MANAGER_SPEC.sql" | sql "ADMIN/password@service"
|
|
Get-Content "02_MARS_1057_install_CT_MRDS_FILE_MANAGER_BODY.sql" | sql "ADMIN/password@service"
|
|
Get-Content "03_MARS_1057_install_ODS_FILE_MANAGER_ODS_SPEC.sql" | sql "ADMIN/password@service"
|
|
Get-Content "04_MARS_1057_install_ODS_FILE_MANAGER_ODS_BODY.sql" | sql "ADMIN/password@service"
|
|
Get-Content "track_package_versions.sql" | sql "ADMIN/password@service"
|
|
Get-Content "verify_packages_version.sql" | sql "ADMIN/password@service"
|
|
```
|
|
|
|
## Verification
|
|
```sql
|
|
-- Check FILE_MANAGER version
|
|
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL;
|
|
-- Expected: 3.4.0
|
|
|
|
-- Check FILE_MANAGER_ODS version
|
|
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
|
|
-- Expected: 2.2.0
|
|
|
|
-- Check for errors (ADMIN user checks specific schema)
|
|
SELECT * FROM ALL_ERRORS
|
|
WHERE OWNER IN ('CT_MRDS', 'ODS')
|
|
AND NAME IN ('FILE_MANAGER', 'FILE_MANAGER_ODS');
|
|
-- Expected: No rows
|
|
|
|
-- Verify new FILE_MANAGER procedures exist
|
|
SELECT procedure_name
|
|
FROM ALL_PROCEDURES
|
|
WHERE OWNER = 'CT_MRDS'
|
|
AND object_name = 'FILE_MANAGER'
|
|
AND procedure_name IN ('CREATE_EXTERNAL_TABLES_SET', 'CREATE_EXTERNAL_TABLES_BATCH');
|
|
-- Expected: 2 rows
|
|
|
|
-- Verify new FILE_MANAGER_ODS procedures exist
|
|
SELECT procedure_name
|
|
FROM ALL_PROCEDURES
|
|
WHERE OWNER = 'ODS'
|
|
AND object_name = 'FILE_MANAGER_ODS'
|
|
AND procedure_name IN ('CREATE_EXTERNAL_TABLES_SET', 'CREATE_EXTERNAL_TABLES_BATCH');
|
|
-- Expected: 2 rows
|
|
|
|
-- Check for untracked changes
|
|
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;
|
|
-- Expected: OK: Package CT_MRDS.FILE_MANAGER has not changed.
|
|
|
|
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('ODS', 'FILE_MANAGER_ODS') FROM DUAL;
|
|
-- Expected: OK: Package ODS.FILE_MANAGER_ODS has not changed.
|
|
```
|
|
|
|
## Rollback
|
|
```powershell
|
|
# IMPORTANT: Execute as ADMIN user
|
|
Get-Content "MARS_Packages/REL03/MARS-1057/rollback_mars1057.sql" | sql "ADMIN/password@service"
|
|
```
|
|
|
|
**Rollback restores:**
|
|
- FILE_MANAGER package specification v3.3.0
|
|
- FILE_MANAGER package body v3.3.0
|
|
- FILE_MANAGER_ODS package specification v2.1.0
|
|
- FILE_MANAGER_ODS package body v2.1.0
|
|
- Removes CREATE_EXTERNAL_TABLES_SET procedures from both packages
|
|
- Removes CREATE_EXTERNAL_TABLES_BATCH procedures from both packages
|
|
|
|
## Expected Changes
|
|
- **FILE_MANAGER package**: v3.3.0 → v3.4.0
|
|
- **FILE_MANAGER_ODS package**: v2.1.0 → v2.2.0
|
|
- **New procedures in FILE_MANAGER**: CREATE_EXTERNAL_TABLES_SET, CREATE_EXTERNAL_TABLES_BATCH
|
|
- **New procedures in FILE_MANAGER_ODS**: CREATE_EXTERNAL_TABLES_SET, CREATE_EXTERNAL_TABLES_BATCH
|
|
- **FILE_MANAGER SPEC size**: +3.3 KB (declaration and documentation)
|
|
- **FILE_MANAGER BODY size**: +8.6 KB (implementation with logging)
|
|
- **FILE_MANAGER_ODS SPEC size**: +2.8 KB (wrapper declarations)
|
|
- **FILE_MANAGER_ODS BODY size**: +4.5 KB (wrapper implementations)
|
|
|
|
## Testing
|
|
|
|
### Test 1: Create Single Set (CT_MRDS.FILE_MANAGER)
|
|
```sql
|
|
BEGIN
|
|
CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123
|
|
);
|
|
END;
|
|
/
|
|
|
|
-- Verify tables created
|
|
SELECT table_name
|
|
FROM ALL_TABLES
|
|
WHERE OWNER = 'ODS'
|
|
AND (table_name LIKE '%_INBOX'
|
|
OR table_name LIKE '%_ODS'
|
|
OR table_name LIKE '%_ARCHIVE');
|
|
```
|
|
|
|
### Test 2: Create Single Set (ODS.FILE_MANAGER_ODS)
|
|
```sql
|
|
BEGIN
|
|
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123,
|
|
pRecreate => TRUE
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
### Test 3: Batch Creation (CT_MRDS.FILE_MANAGER)
|
|
```sql
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123
|
|
);
|
|
END;
|
|
/
|
|
|
|
-- Verify tables created
|
|
SELECT table_name
|
|
FROM ALL_TABLES
|
|
WHERE OWNER = 'ODS'
|
|
AND table_name LIKE '%_INBOX'
|
|
OR table_name LIKE '%_ODS'
|
|
OR table_name LIKE '%_ARCHIVE';
|
|
```
|
|
|
|
### Test 2: Batch Creation
|
|
```sql
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pSourceKey => 'C2D'
|
|
);
|
|
END;
|
|
/
|
|
|
|
-- Check process log for results
|
|
SELECT *
|
|
FROM CT_MRDS.A_PROCESS_LOG
|
|
WHERE LOG_TIMESTAMP > SYSDATE - INTERVAL '1' HOUR
|
|
AND PROCEDURE_NAME LIKE '%CREATE_EXTERNAL_TABLES%'
|
|
ORDER BY LOG_TIMESTAMP DESC;
|
|
```
|
|
|
|
### Test 3: Recreate Existing Tables
|
|
```sql
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
|
|
pSourceFileConfigKey => 123,
|
|
pRecreate => TRUE
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
## Usage Examples
|
|
|
|
### Example 1: Setup All External Tables for New Source
|
|
```sql
|
|
-- 1. Add source configuration
|
|
CALL FILE_MANAGER.ADD_SOURCE('LM', 'Liquidity Management');
|
|
|
|
-- 2. Add file configurations
|
|
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
|
|
pSourceKey => 'LM',
|
|
pSourceFileType => 'INPUT',
|
|
pSourceFileId => 'STANDING_FACILITIES',
|
|
pSourceFileDesc => 'Standing Facilities Data',
|
|
pSourceFileNamePattern => 'SF_*.csv',
|
|
pTableId => 'STANDING_FACILITIES',
|
|
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
|
|
pEncoding => 'UTF8'
|
|
);
|
|
|
|
-- 3. Create all external tables for this source
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pSourceKey => 'LM'
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
### Example 2: Recreate All External Tables
|
|
```sql
|
|
-- Useful after bucket URI changes or template table modifications
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pRecreate => TRUE
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
### Example 3: Create Tables for Specific File Type
|
|
```sql
|
|
BEGIN
|
|
FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
|
|
pSourceFileId => 'UC_DISSEM'
|
|
);
|
|
END;
|
|
/
|
|
```
|
|
|
|
## Known Issues
|
|
None
|
|
|
|
## Dependencies
|
|
- **CT_MRDS.ENV_MANAGER** - Logging and error handling
|
|
- **ODS.FILE_MANAGER_ODS** - AUTHID DEFINER wrapper for external table creation
|
|
- **CT_MRDS.A_SOURCE_FILE_CONFIG** - Source file configuration metadata
|
|
- **CT_ET_TEMPLATES schema** - Template table definitions
|
|
|
|
## Related
|
|
- **MARS-1056** - VARCHAR2 CHAR/BYTE semantics fix
|
|
- **MARS-1049** - CSV encoding support
|
|
- **Package Deployment Guide** - Standard deployment procedures
|
|
- **Tables Setup Guide** - External table configuration guide
|
|
|
|
## Notes
|
|
- All installations must be executed as ADMIN user
|
|
- Use `ALL_*` views instead of `USER_*` views for verification
|
|
- Master scripts include SPOOL logging for audit trail
|
|
- ACCEPT validation prevents accidental execution
|
|
- Follows official path patterns: INBOX (3-level), ODS (2-level), ARCHIVE (2-level)
|