admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
64dc830a2b5b1e4c7b9d754823b738cfffceb283
mars/MARS_Packages/REL03/MARS-1057/README.md

11 KiB
Raw Blame History

MARS-1057: Batch External Table Creation

Overview

This MARS package adds batch external table creation capabilities to the FILE_MANAGER and FILE_MANAGER_ODS packages, enabling automatic creation of external table sets (INBOX, ODS, ARCHIVE) based on A_SOURCE_FILE_CONFIG metadata.

Jira Issue: MARS-1057
Package Version: FILE_MANAGER 3.4.0, FILE_MANAGER_ODS 2.2.0
Author: Grzegorz Michalski
Date: 2025-11-27

Contents

  • install_mars1057.sql - Master installation script with SPOOL logging
  • rollback_mars1057.sql - Master rollback script
  • 01_MARS_1057_install_CT_MRDS_FILE_MANAGER_SPEC.sql - Install FILE_MANAGER package specification
  • 02_MARS_1057_install_CT_MRDS_FILE_MANAGER_BODY.sql - Install FILE_MANAGER package body
  • 03_MARS_1057_install_ODS_FILE_MANAGER_ODS_SPEC.sql - Install FILE_MANAGER_ODS package specification
  • 04_MARS_1057_install_ODS_FILE_MANAGER_ODS_BODY.sql - Install FILE_MANAGER_ODS package body
  • 91_MARS_1057_rollback_CT_MRDS_FILE_MANAGER_SPEC.sql - Rollback FILE_MANAGER package specification
  • 92_MARS_1057_rollback_CT_MRDS_FILE_MANAGER_BODY.sql - Rollback FILE_MANAGER package body
  • 93_MARS_1057_rollback_ODS_FILE_MANAGER_ODS_BODY.sql - Rollback FILE_MANAGER_ODS package body
  • 94_MARS_1057_rollback_ODS_FILE_MANAGER_ODS_SPEC.sql - Rollback FILE_MANAGER_ODS package specification
  • track_package_versions.sql - Universal package version tracking
  • verify_packages_version.sql - Universal package verification
  • rollback_version/ - FILE_MANAGER v3.3.0, FILE_MANAGER_ODS v2.1.0 (before MARS-1057)
  • new_version/ - FILE_MANAGER v3.4.0, FILE_MANAGER_ODS v2.2.0 (after MARS-1057)
  • .gitignore - Git exclusions for temporary files

Prerequisites

  • Oracle Database 23ai
  • FILE_MANAGER package v3.3.0 installed
  • FILE_MANAGER_ODS package v2.1.0 installed
  • ENV_MANAGER package v3.1.0+ with version tracking
  • ADMIN user access for deployment
  • ODS schema with necessary privileges

New Features

1. CREATE_EXTERNAL_TABLES_SET

Creates a complete set of 3 external tables (INBOX, ODS, ARCHIVE) for a single configuration from A_SOURCE_FILE_CONFIG.

Signature:

PROCEDURE CREATE_EXTERNAL_TABLES_SET (
    pSourceFileConfigKey IN NUMBER,
    pRecreate            IN BOOLEAN DEFAULT FALSE
);

Example:

BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
        pSourceFileConfigKey => 123,
        pRecreate => FALSE
    );
END;
/

Features:

  • Automatic table naming: {TABLE_ID}_{INBOX|ODS|ARCHIVE}
  • Official path patterns compliance
  • Optional drop and recreate
  • Full ENV_MANAGER logging
  • Error handling with detailed messages

2. CREATE_EXTERNAL_TABLES_BATCH

Creates external table sets for multiple configurations based on filter criteria.

Signature:

PROCEDURE CREATE_EXTERNAL_TABLES_BATCH (
    pSourceKey      IN VARCHAR2 DEFAULT NULL,
    pSourceFileId   IN VARCHAR2 DEFAULT NULL,
    pTableId        IN VARCHAR2 DEFAULT NULL,
    pRecreate       IN BOOLEAN DEFAULT FALSE
);

Examples:

-- All external tables for C2D source
BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pSourceKey => 'C2D'
    );
END;
/

-- Recreate all external tables
BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pRecreate => TRUE
    );
END;
/

-- Specific table across all sources
BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pTableId => 'A_UC_DISSEM_METADATA_LOADS'
    );
END;
/

Features:

  • Filters only INPUT type files
  • Continues processing on errors
  • Returns summary (Total/Processed/Failed)
  • Comprehensive logging

3. FILE_MANAGER_ODS Wrapper Procedures

The ODS.FILE_MANAGER_ODS package now includes DEFINER rights wrappers for the new batch procedures.

New Wrappers:

-- Create single external table set via ODS wrapper
EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_SET(
    pSourceFileConfigKey => 123,
    pRecreate => FALSE
);

-- Create batch external tables via ODS wrapper
EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_BATCH(
    pSourceKey => 'C2D',
    pRecreate => FALSE
);

Benefits:

  • Objects created in ODS schema regardless of execution user
  • Same functionality as CT_MRDS.FILE_MANAGER with DEFINER rights
  • Full logging and error handling
  • Consistent API with original package

Installation

Option 1: Master Script (Recommended)

# IMPORTANT: Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1057/install_mars1057.sql" | sql "ADMIN/password@service"

# Log file created: log/INSTALL_MARS_1057_<PDB>_<timestamp>.log

Installation Steps:

  1. Install FILE_MANAGER package specification v3.4.0
  2. Install FILE_MANAGER package body v3.4.0
  3. Install FILE_MANAGER_ODS package specification v2.2.0
  4. Install FILE_MANAGER_ODS package body v2.2.0
  5. Track versions in A_PACKAGE_VERSION_TRACKING
  6. Verify all tracked packages for untracked changes

Option 2: Individual Scripts

# IMPORTANT: Execute as ADMIN user
Get-Content "01_MARS_1057_install_CT_MRDS_FILE_MANAGER_SPEC.sql" | sql "ADMIN/password@service"
Get-Content "02_MARS_1057_install_CT_MRDS_FILE_MANAGER_BODY.sql" | sql "ADMIN/password@service"
Get-Content "03_MARS_1057_install_ODS_FILE_MANAGER_ODS_SPEC.sql" | sql "ADMIN/password@service"
Get-Content "04_MARS_1057_install_ODS_FILE_MANAGER_ODS_BODY.sql" | sql "ADMIN/password@service"
Get-Content "track_package_versions.sql" | sql "ADMIN/password@service"
Get-Content "verify_packages_version.sql" | sql "ADMIN/password@service"

Verification

-- Check FILE_MANAGER version
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL;
-- Expected: 3.4.0

-- Check FILE_MANAGER_ODS version
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
-- Expected: 2.2.0

-- Check for errors (ADMIN user checks specific schema)
SELECT * FROM ALL_ERRORS 
WHERE OWNER IN ('CT_MRDS', 'ODS') 
  AND NAME IN ('FILE_MANAGER', 'FILE_MANAGER_ODS');
-- Expected: No rows

-- Verify new FILE_MANAGER procedures exist
SELECT procedure_name
FROM ALL_PROCEDURES
WHERE OWNER = 'CT_MRDS'
  AND object_name = 'FILE_MANAGER'
  AND procedure_name IN ('CREATE_EXTERNAL_TABLES_SET', 'CREATE_EXTERNAL_TABLES_BATCH');
-- Expected: 2 rows

-- Verify new FILE_MANAGER_ODS procedures exist
SELECT procedure_name
FROM ALL_PROCEDURES
WHERE OWNER = 'ODS'
  AND object_name = 'FILE_MANAGER_ODS'
  AND procedure_name IN ('CREATE_EXTERNAL_TABLES_SET', 'CREATE_EXTERNAL_TABLES_BATCH');
-- Expected: 2 rows

-- Check for untracked changes
SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;
-- Expected: OK: Package CT_MRDS.FILE_MANAGER has not changed.

SELECT CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES('ODS', 'FILE_MANAGER_ODS') FROM DUAL;
-- Expected: OK: Package ODS.FILE_MANAGER_ODS has not changed.

Rollback

# IMPORTANT: Execute as ADMIN user
Get-Content "MARS_Packages/REL03/MARS-1057/rollback_mars1057.sql" | sql "ADMIN/password@service"

Rollback restores:

  • FILE_MANAGER package specification v3.3.0
  • FILE_MANAGER package body v3.3.0
  • FILE_MANAGER_ODS package specification v2.1.0
  • FILE_MANAGER_ODS package body v2.1.0
  • Removes CREATE_EXTERNAL_TABLES_SET procedures from both packages
  • Removes CREATE_EXTERNAL_TABLES_BATCH procedures from both packages

Expected Changes

  • FILE_MANAGER package: v3.3.0 → v3.4.0
  • FILE_MANAGER_ODS package: v2.1.0 → v2.2.0
  • New procedures in FILE_MANAGER: CREATE_EXTERNAL_TABLES_SET, CREATE_EXTERNAL_TABLES_BATCH
  • New procedures in FILE_MANAGER_ODS: CREATE_EXTERNAL_TABLES_SET, CREATE_EXTERNAL_TABLES_BATCH
  • FILE_MANAGER SPEC size: +3.3 KB (declaration and documentation)
  • FILE_MANAGER BODY size: +8.6 KB (implementation with logging)
  • FILE_MANAGER_ODS SPEC size: +2.8 KB (wrapper declarations)
  • FILE_MANAGER_ODS BODY size: +4.5 KB (wrapper implementations)

Testing

Test 1: Create Single Set (CT_MRDS.FILE_MANAGER)

BEGIN
    CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
        pSourceFileConfigKey => 123
    );
END;
/

-- Verify tables created
SELECT table_name 
FROM ALL_TABLES 
WHERE OWNER = 'ODS' 
  AND (table_name LIKE '%_INBOX' 
    OR table_name LIKE '%_ODS' 
    OR table_name LIKE '%_ARCHIVE');

Test 2: Create Single Set (ODS.FILE_MANAGER_ODS)

BEGIN
    ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLES_SET(
        pSourceFileConfigKey => 123,
        pRecreate => TRUE
    );
END;
/

Test 3: Batch Creation (CT_MRDS.FILE_MANAGER)

BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
        pSourceFileConfigKey => 123
    );
END;
/

-- Verify tables created
SELECT table_name 
FROM ALL_TABLES 
WHERE OWNER = 'ODS' 
  AND table_name LIKE '%_INBOX' 
   OR table_name LIKE '%_ODS' 
   OR table_name LIKE '%_ARCHIVE';

Test 2: Batch Creation

BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pSourceKey => 'C2D'
    );
END;
/

-- Check process log for results
SELECT * 
FROM CT_MRDS.A_PROCESS_LOG 
WHERE LOG_TIMESTAMP > SYSDATE - INTERVAL '1' HOUR
  AND PROCEDURE_NAME LIKE '%CREATE_EXTERNAL_TABLES%'
ORDER BY LOG_TIMESTAMP DESC;

Test 3: Recreate Existing Tables

BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_SET(
        pSourceFileConfigKey => 123,
        pRecreate => TRUE
    );
END;
/

Usage Examples

Example 1: Setup All External Tables for New Source

-- 1. Add source configuration
CALL FILE_MANAGER.ADD_SOURCE('LM', 'Liquidity Management');

-- 2. Add file configurations
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'Standing Facilities Data',
    pSourceFileNamePattern => 'SF_*.csv',
    pTableId => 'STANDING_FACILITIES',
    pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
    pEncoding => 'UTF8'
);

-- 3. Create all external tables for this source
BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pSourceKey => 'LM'
    );
END;
/

Example 2: Recreate All External Tables

-- Useful after bucket URI changes or template table modifications
BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pRecreate => TRUE
    );
END;
/

Example 3: Create Tables for Specific File Type

BEGIN
    FILE_MANAGER.CREATE_EXTERNAL_TABLES_BATCH(
        pSourceFileId => 'UC_DISSEM'
    );
END;
/

Known Issues

None

Dependencies

  • CT_MRDS.ENV_MANAGER - Logging and error handling
  • ODS.FILE_MANAGER_ODS - AUTHID DEFINER wrapper for external table creation
  • CT_MRDS.A_SOURCE_FILE_CONFIG - Source file configuration metadata
  • CT_ET_TEMPLATES schema - Template table definitions

Related

  • MARS-1056 - VARCHAR2 CHAR/BYTE semantics fix
  • MARS-1049 - CSV encoding support
  • Package Deployment Guide - Standard deployment procedures
  • Tables Setup Guide - External table configuration guide

Notes

  • All installations must be executed as ADMIN user
  • Use ALL_* views instead of USER_* views for verification
  • Master scripts include SPOOL logging for audit trail
  • ACCEPT validation prevents accidental execution
  • Follows official path patterns: INBOX (3-level), ODS (2-level), ARCHIVE (2-level)
Reference in New Issue View Git Blame Copy Permalink