admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ca11debd935b1319adcc12e889faa4d3b0db2d35
mars/confluence/FILE_MANAGER_Configuration_Guide.md

43 KiB
Raw Blame History

FILE_MANAGER Configuration Guide

This document describes the configuration procedures available in the FILE_MANAGER package for setting up file processing rules, defining file naming patterns, and configuring date format handling.

Overview

The FILE_MANAGER package provides several configuration procedures that work together to create a complete file processing workflow:

  • ADD_SOURCE: Registers new source systems
  • ADD_SOURCE_FILE_CONFIG: Configures file processing rules for specific file types
  • ADD_COLUMN_DATE_FORMAT: Sets up date format handling for specific columns

These procedures are typically used after setting up external tables to enable automatic file processing workflows.

Note: Data export functionality (CSV and Parquet export procedures) has been moved to the DATA_EXPORTER package. This separation provides better modularity - FILE_MANAGER focuses on file processing and validation, while DATA_EXPORTER handles data export operations.

Note: Data export functionality (CSV and Parquet export procedures) has been moved to the DATA_EXPORTER package. This separation provides better modularity - FILE_MANAGER focuses on file processing and validation, while DATA_EXPORTER handles data export operations.

File Validation Features

Automatic CSV Column Validation

The FILE_MANAGER package includes automatic validation for CSV files to ensure they match the expected template structure:

  • Excess Column Detection: Automatically detects when CSV files contain more columns than defined in the template table
  • Pre-Processing Validation: Validation occurs before external table creation to prevent processing errors
  • Detailed Error Messages: Provides specific information about column count mismatches and suggests solutions

Error Code: -20011 (ERR_EXCESS_COLUMNS_DETECTED)

EXCESS COLUMNS DETECTED!
CSV file has 9 columns but template expects only 6
Excess columns: 3
CSV header: ID,NAME,DESCRIPTION,STATUS,AMOUNT,CREATED_DATE,PRIORITY,EXTRA_COLUMN,ANOTHER_EXTRA
POSSIBLE SOLUTIONS:
  1. Remove excess columns from CSV file before processing
  2. Add excess columns to template table: CT_ET_TEMPLATES.TEMPLATE_NAME

ANALYZE_VALIDATION_ERRORS Function

Wrapper function that simplifies validation error analysis by automatically deriving required parameters from file metadata:

Signature:

FUNCTION ANALYZE_VALIDATION_ERRORS(
    pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;

Parameters:

  • pSourceFileReceivedKey: Key from A_SOURCE_FILE_RECEIVED table for the file to analyze

Automatic Parameter Derivation:

  • Template Schema/Table: Extracted from TEMPLATE_TABLE_NAME in configuration
  • CSV File URI: Built from bucket configuration and file paths
  • Validation Log Table: Automatically finds most recent validation log

Purpose:

  • Provides detailed analysis of file validation failures with simplified interface
  • Eliminates need to manually specify 4 complex parameters
  • Returns structured error information for troubleshooting
  • Integrates with enhanced error logging system

Examples:

-- Find file key for problematic file
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, PROCESSING_STATUS
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED 
WHERE SOURCE_FILE_NAME LIKE '%problematic%';

-- Analyze validation errors (simplified interface)
SELECT CT_MRDS.FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;

Comparison with Direct Function:

-- Wrapper (1 parameter) - RECOMMENDED
SELECT CT_MRDS.FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;

-- Direct function (4 parameters) - Advanced usage
SELECT CT_MRDS.ENV_MANAGER.ANALYZE_VALIDATION_ERRORS(
    'VALIDATE$253_LOG', 'CT_ET_TEMPLATES', 
    'MOCK_PROC_TABLE', 'https://bucket-uri/file.csv'
) FROM DUAL;

Configuration Procedures

ADD_SOURCE

Adds a new source system to the configuration.

Signature:

PROCEDURE ADD_SOURCE (
    pSourceKey   IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
    pSourceName  IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);

Parameters:

  • pSourceKey: Unique identifier for the source system (e.g., 'C2D', 'LM', 'TOP')
  • pSourceName: Human-readable name for the source system

Example:

CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'Central Bank Data System'
);

Purpose:

  • Registers a new data source in the system
  • Required before configuring file types for the source
  • Creates master record for source system identification

ADD_SOURCE_FILE_CONFIG

Configures file processing rules for a specific file type within a source system.

Signature:

PROCEDURE ADD_SOURCE_FILE_CONFIG (
    pSourceKey             IN   CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE,
    pSourceFileType        IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE,
    pSourceFileId          IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE,
    pSourceFileDesc        IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE,
    pSourceFileNamePattern IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE,
    pTableId               IN   CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE                  DEFAULT NULL,
    pTemplateTableName     IN   CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE       DEFAULT NULL,
    pContainerFileKey      IN   CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE        DEFAULT NULL,
    pEncoding              IN   CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING%TYPE                  DEFAULT NULL  -- MARS-1049: CSV character set encoding
);

Parameters:

  • pSourceKey: Source system identifier (must exist in A_SOURCE table)
  • pSourceFileType: Type of file ('INPUT', 'OUTPUT', 'REFERENCE', etc.)
  • pSourceFileId: Unique identifier for the file type within the source
  • pSourceFileDesc: Human-readable description of the file type
  • pSourceFileNamePattern: Pattern to match incoming files (supports wildcards like *)
  • pTableId: Table identifier used in storage paths (optional)
  • pTemplateTableName: Template table defining the structure (optional)
  • pContainerFileKey: Reference to container file for nested structures (optional)
  • pEncoding: MARS-1049 - Oracle character set encoding for CSV files (optional) - Examples: 'UTF8', 'WE8MSWIN1252', 'EE8ISO8859P2', 'CL8MSWIN1251'. Used by CREATE_EXTERNAL_TABLE for CHARACTERSET specification.

Example:

CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC DISSEM Metadata Loads',
    pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pContainerFileKey => NULL,
    pEncoding => 'UTF8'  -- MARS-1049: CSV character set encoding
);

MARS-1049 CSV Encoding Support: The pEncoding parameter enables proper handling of international character sets in CSV files:

Common Character Encodings:

  • 'UTF8' - Unicode UTF-8 (recommended for international data)
  • 'WE8MSWIN1252' - Western European Windows-1252 (CSDB data)
  • 'EE8ISO8859P2' - Eastern European ISO-8859-2
  • 'CL8MSWIN1251' - Cyrillic Windows-1251
  • 'JA16SJIS' - Japanese Shift-JIS

Encoding Examples:

-- UTF-8 for international customer data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'INTL_SYS',
    pSourceFileId => 'CUSTOMER_DATA',
    pEncoding => 'UTF8'
);

-- Windows-1252 for CSDB European data  
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'CSDB',
    pSourceFileId => 'DEBT_DAILY',
    pEncoding => 'WE8MSWIN1252'
);

-- Backward compatibility - no encoding parameter
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LEGACY_SYS',
    pSourceFileId => 'OLD_DATA'
    -- pEncoding omitted - uses database default
);

Purpose:

  • Defines how files should be processed for a specific file type
  • Establishes file naming patterns for automatic recognition
  • Links file types to template tables and storage paths
  • Enables automatic file processing workflows

ADD_COLUMN_DATE_FORMAT

Configures date format handling for specific columns in template tables.

Signature:

PROCEDURE ADD_COLUMN_DATE_FORMAT (
    pTemplateTableName     IN   CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE,
    pColumnName            IN   CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE,
    pDateFormat            IN   CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);

Parameters:

  • pTemplateTableName: Name of the template table (e.g., 'CT_ET_TEMPLATES.C2D_TABLE_NAME')
  • pColumnName: Name of the column containing date data
  • pDateFormat: Oracle date format mask (e.g., 'YYYY-MM-DD', 'DD/MM/YYYY HH24:MI:SS')

Examples:

-- Configure date column with ISO format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);

-- Configure date column with European format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'LOAD_START',
    pDateFormat => 'DD/MM/YYYY'
);

-- Configure timestamp column
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'PROCESS_TIMESTAMP',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);

Purpose:

  • Defines how date columns should be parsed from CSV files
  • Ensures correct date format conversion during external table queries
  • Prevents date format errors and data type mismatches
  • Supports various international date formats

Complete Configuration Workflow

Here's a complete example showing the typical configuration sequence:

-- Step 1: Add source system (if not already exists)
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'Central Bank Data System'
);

-- Step 2: Configure file type for processing
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC DISSEM Metadata Loads',
    pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pContainerFileKey => NULL,
    pEncoding => 'UTF8'  -- MARS-1049: Character set encoding
);

-- Step 3: Configure date formats for specific columns
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'LOAD_START', 
    pDateFormat => 'DD/MM/YYYY'
);

-- Step 4: Add additional file types as needed
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'AGGREGATED_ALLOTMENT',
    pSourceFileDesc => 'Aggregated Allotment Data',
    pSourceFileNamePattern => 'AGGREGATED_ALLOTMENT_*.csv',
    pTableId => 'AGGREGATED_ALLOTMENT',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_AGGREGATED_ALLOTMENT',
    pContainerFileKey => NULL,
    pEncoding => 'WE8MSWIN1252'  -- MARS-1049: Windows-1252 encoding for European data
);

Configuration Verification

Use these queries to verify your configuration setup:

Check Source Systems

SELECT A_SOURCE_KEY, SOURCE_NAME, CREATED_ON
FROM CT_MRDS.A_SOURCE 
ORDER BY A_SOURCE_KEY;

Check File Configurations

SELECT sfc.A_SOURCE_KEY, 
       sfc.SOURCE_FILE_ID, 
       sfc.SOURCE_FILE_DESC, 
       sfc.SOURCE_FILE_NAME_PATTERN, 
       sfc.TABLE_ID, 
       sfc.TEMPLATE_TABLE_NAME,
       sfc.SOURCE_FILE_TYPE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
JOIN CT_MRDS.A_SOURCE s ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY = 'C2D'  -- Replace with your source key
ORDER BY sfc.SOURCE_FILE_ID;

Check Date Format Configurations

SELECT TEMPLATE_TABLE_NAME, 
       COLUMN_NAME, 
       DATE_FORMAT,
       CREATED_ON
FROM CT_MRDS.A_COLUMN_DATE_FORMAT
WHERE TEMPLATE_TABLE_NAME LIKE '%C2D%'  -- Replace with your template pattern
ORDER BY TEMPLATE_TABLE_NAME, COLUMN_NAME;

Complete Configuration Overview

-- Complete view of configuration for a specific source
SELECT 
    s.A_SOURCE_KEY,
    s.SOURCE_NAME,
    sfc.SOURCE_FILE_ID,
    sfc.SOURCE_FILE_DESC,
    sfc.SOURCE_FILE_NAME_PATTERN,
    sfc.TABLE_ID,
    sfc.TEMPLATE_TABLE_NAME,
    cdf.COLUMN_NAME,
    cdf.DATE_FORMAT
FROM CT_MRDS.A_SOURCE s
LEFT JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
LEFT JOIN CT_MRDS.A_COLUMN_DATE_FORMAT cdf ON sfc.TEMPLATE_TABLE_NAME = cdf.TEMPLATE_TABLE_NAME
WHERE s.A_SOURCE_KEY = 'C2D'  -- Replace with your source key
ORDER BY sfc.SOURCE_FILE_ID, cdf.COLUMN_NAME;

Storage Path Structure Based on Configuration

The configuration parameters directly affect how files are organized in Oracle Cloud Storage according to the official path patterns:

Oracle Cloud Storage Structure (OFFICIAL PATTERNS):

INBOX Bucket - Pattern: 'INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/'
└── INBOX/
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pSourceFileId}/    -- e.g., "UC_DISSEM", "STANDING_FACILITIES"  
            └── {pTableId}/     -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
                └── files matching {pSourceFileNamePattern}

DATA Bucket - Patterns: 'ODS/{SOURCE}/{TABLE_NAME}/' and 'TRASH/{SOURCE}/{TABLE_NAME}/'
├── ODS/
│   └── {pSourceKey}/           -- e.g., "C2D", "LM"
│       └── {pTableId}/         -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
│           └── processed files
└── TRASH/                      -- File retention subfolder (not a separate bucket)
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pTableId}/         -- CSV files after archival (ARCHIVED_AND_TRASHED status)

ARCHIVE Bucket - Pattern: 'ARCHIVE/{SOURCE}/{TABLE_NAME}/'
└── ARCHIVE/
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pTableId}/         -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
            └── PARTITION_YEAR=*/
                └── PARTITION_MONTH=*/
                    └── *.parquet files

Critical Path Pattern Requirements:

  • INBOX requires full 3-level path: INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/
  • ODS uses simplified 2-level path: ODS/{SOURCE}/{TABLE_NAME}/ (no SOURCE_FILE_ID)
  • TRASH uses simplified 2-level path: TRASH/{SOURCE}/{TABLE_NAME}/ (subfolder in DATA bucket)
  • ARCHIVE uses simplified 2-level path: ARCHIVE/{SOURCE}/{TABLE_NAME}/ (no SOURCE_FILE_ID)
  • All patterns are mandatory - no simplified versions allowed
  • File names must match pSourceFileNamePattern for automatic processing
  • Note: TRASH is NOT a separate bucket - it's a subfolder within the DATA bucket

Configuration Management Best Practices

1. Naming Conventions

Source Keys:

  • Use short, descriptive codes (e.g., 'C2D', 'LM', 'TOP')
  • Consistent with existing system abbreviations
  • Uppercase for standardization

Source File IDs:

  • Descriptive but concise (e.g., 'UC_DISSEM', 'AGGREGATED_ALLOTMENT')
  • Use underscores for readability
  • Reflect the business purpose

Table IDs:

  • Match the actual table name patterns
  • Consistent with template table naming
  • Use underscores for compound names

2. File Name Patterns

Wildcards:

  • Use * for variable parts (dates, sequence numbers)
  • Be specific enough to avoid conflicts
  • Consider file extensions (.csv, .txt, .json)

Examples:

-- Good patterns:
'UC_NMA_DISSEM-*.csv'           -- Clear prefix with wildcard
'AGGREGATED_ALLOTMENT_*.csv'    -- Descriptive with extension
'DATA_EXPORT_????-??-??.txt'   -- Date-specific pattern

-- Avoid overly broad patterns:
'*.csv'                         -- Too generic
'DATA*'                         -- Missing extension

3. Date Format Configuration

Common Formats:

-- ISO standard formats (recommended)
'YYYY-MM-DD'                    -- Date only
'YYYY-MM-DD HH24:MI:SS'        -- Date with time
'YYYY-MM-DD"T"HH24:MI:SS'      -- ISO timestamp

-- Regional formats
'DD/MM/YYYY'                    -- European format
'MM/DD/YYYY'                    -- US format
'DD-MON-YYYY'                   -- Oracle default format

4. Configuration Sequence

Always follow this order:

  1. ADD_SOURCE (if new source system)
  2. ADD_SOURCE_FILE_CONFIG (for each file type)
  3. ADD_COLUMN_DATE_FORMAT (for date columns in each template)

Troubleshooting Configuration Issues

Common Problems

  1. Source Key Not Found

    ORA-02291: integrity constraint violated - parent key not found

    Solution: Ensure ADD_SOURCE is called before ADD_SOURCE_FILE_CONFIG

  2. Template Table Not Found

    ORA-00942: table or view does not exist

    Solution: Verify template table exists and name is correct (including schema)

  3. Date Format Errors

    ORA-01821: date format not recognized

    Solution: Check date format mask matches actual data format in files

  4. Excess Column Errors (Code: -20011)

    ORA-20011: CSV file contains more columns than template allows
EXCESS COLUMNS DETECTED!
CSV file has 9 columns but template expects only 6

    Solutions:

    • Remove excess columns from CSV file before processing
    • Add missing columns to template table
    • Use ANALYZE_VALIDATION_ERRORS() for detailed analysis

  5. File Pattern Not Matching

    • Files uploaded but not processed Solution: Verify file names match the configured pattern exactly

Enhanced Error Logging

The system includes enhanced error logging through the LOG_PROCESS_ERROR function:

  • Structured Logging: All errors logged with full context and stack traces
  • Error Analysis: Automatic categorization and analysis of validation failures
  • Troubleshooting Support: Detailed error information for faster problem resolution

Viewing Error Logs:

SELECT LOG_LEVEL, LOG_MESSAGE, LOG_TIMESTAMP 
FROM CT_MRDS.A_PROCESS_LOG 
WHERE LOG_LEVEL = 'ERROR' 
AND LOG_MESSAGE LIKE '%EXCESS COLUMNS%'
ORDER BY LOG_TIMESTAMP DESC;

Error Log Content Example:

Error Message:
ORA-20011: CSV file contains more columns than template allows
EXCESS COLUMNS DETECTED!
CSV file has 9 columns but template expects only 6
Excess columns: 3
CSV header: ID,NAME,DESCRIPTION,STATUS,AMOUNT,CREATED_DATE,PRIORITY,EXTRA_COLUMN,ANOTHER_EXTRA
POSSIBLE SOLUTIONS:
  1. Remove excess columns from CSV file before processing
  2. Add excess columns to template table: CT_ET_TEMPLATES.MOCK_PROC_TABLE
-------------------------------------------------------
Error Stack: [Full Oracle error stack trace]
-------------------------------------------------------
Error Backtrace: [Full Oracle error backtrace]

Diagnostic Queries

-- Check for orphaned configurations
SELECT sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
LEFT JOIN CT_MRDS.A_SOURCE s ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY IS NULL;

-- Check for missing template tables
SELECT DISTINCT sfc.TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
WHERE sfc.TEMPLATE_TABLE_NAME IS NOT NULL
  AND NOT EXISTS (
    SELECT 1 FROM all_tables 
    WHERE owner||'.'||table_name = sfc.TEMPLATE_TABLE_NAME
  );

-- Check date format configurations without corresponding columns
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE NOT EXISTS (
    SELECT 1 FROM all_tab_columns
    WHERE owner||'.'||table_name = cdf.TEMPLATE_TABLE_NAME
      AND column_name = cdf.COLUMN_NAME
);

This configuration enables automatic file processing workflows where files uploaded to the INBOX are automatically processed and moved to ODS, with historical data archived in partitioned PARQUET format.

Register your source system using the FILE_MANAGER package procedure:

-- Use the FILE_MANAGER procedure to add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'YOUR_SYS',
    pSourceName => 'Your System Name Description'
);

Parameters:

  • pSourceKey: Short identifier for your source system (e.g., 'C2D', 'PAYROLL', 'ISD', 'LM')
  • pSourceName: Descriptive name of the source system

Step 2: Configure File Type in A_SOURCE_FILE_CONFIG

Add your file configuration to define how the system should handle your file type using the FILE_MANAGER package procedure:

-- Use the FILE_MANAGER procedure to add source file configuration
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'YOUR_SYS',                    -- Your source system key
    pSourceFileType => 'INPUT',                  -- File type: INPUT/CONTAINER/LOAD_CONFIG
    pSourceFileId => 'YOUR_FILE_ID',             -- Unique identifier for this file type
    pSourceFileDesc => 'Description of your file type',  -- Human-readable description
    pSourceFileNamePattern => 'your_file_pattern_*.csv', -- File name pattern with wildcards
    pTableId => 'YOUR_TABLE_ID',                 -- Target table identifier
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',  -- Template table for External Table creation
    pContainerFileKey => NULL                    -- Container key (for INPUT files, use NULL)
);

-- Note: Additional configuration parameters like thresholds, schema names, etc.
-- need to be set separately by updating the record after creation if needed:
/*
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET DAYS_FOR_ARCHIVE_THRESHOLD = 30,
    FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 100,
    BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824,
    ODS_SCHEMA_NAME = 'YOUR_TARGET_SCHEMA',
    ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 1000000,
    HOURS_TO_EXPIRE_STATISTICS = 24
WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
  AND SOURCE_FILE_TYPE = 'INPUT';
*/

Additional Configuration Parameters:

Parameter Purpose Example Value Description
DAYS_FOR_ARCHIVE_THRESHOLD Archival threshold 30 Number of days after which files are considered old and eligible for archival. Used by FILE_ARCHIVER to identify files that should be moved to archive bucket.
FILES_COUNT_OVER_ARCHIVE_THRESHOLD Archival trigger 100 Maximum number of old files (older than DAYS_FOR_ARCHIVE_THRESHOLD) before automatic archival is triggered. When exceeded, FILE_ARCHIVER exports old data to PARQUET and removes CSV files.
BYTES_SUM_OVER_ARCHIVE_THRESHOLD Archival trigger 1073741824 (1GB) Maximum total size in bytes of old files before archival is triggered. Helps manage storage usage by automatically archiving large amounts of historical data.
ODS_SCHEMA_NAME Target schema 'ODS' Name of the schema where final operational data tables are located. Used by FILE_ARCHIVER to access tables for statistics gathering and archival operations. Format: {ODS_SCHEMA_NAME}.{SOURCE_KEY}_{TABLE_ID}_ODS
ROWS_COUNT_OVER_ARCHIVE_THRESHOLD Archival trigger 1000000 Maximum number of rows in old files before archival is triggered. Prevents tables from growing too large by archiving historical data when row count threshold is exceeded.
HOURS_TO_EXPIRE_STATISTICS Statistics refresh 24 Number of hours after which table statistics expire and need to be recalculated. FILE_ARCHIVER refreshes statistics if they are older than this threshold before making archival decisions. 

**File Type Descriptions:**

| File Type | Purpose | Example Use Cases |
|-----------|---------|-------------------|
| `INPUT` | Data files for processing | CSV files, Excel files, data extracts |
| `CONTAINER` | Archive/zip files containing multiple files | XML archives, ZIP files with multiple CSVs |
| `LOAD_CONFIG` | Configuration files for complex loading scenarios | Files mapped to multiple tables, routing configs |

**File Naming Patterns:**
- Use `*` for any characters: `employee_*.csv` matches `employee_2024.csv`, `employee_january.csv`
- Use `.*.csv` for regex patterns: `.*.csv` matches any CSV file
- Be specific to avoid conflicts: `payroll_[0-9]*.xlsx` for numbered payroll files

### Step 3: Configure Date Formats (Optional)

If your file contains date columns that need special formatting, add them using the FILE_MANAGER package procedure:

```sql
-- Use the FILE_MANAGER procedure to add date format configuration
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',  -- Same as in Step 2
    pColumnName => 'BIRTH_DATE',                         -- Column name containing dates
    pDateFormat => 'DD/MM/YYYY'                          -- Expected date format in your files
);

-- Add more columns if needed
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',
    pColumnName => 'HIRE_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);

Alternative approach using parameters from seed2.sql: Note: Some implementations may use an extended signature (verify with your FILE_MANAGER package version):

-- Alternative signature that might be available in some versions
/*
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pSourceFileId => 'YOUR_FILE_ID',
    pTableId => 'YOUR_TABLE_ID',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);
*/

Step 4: Test Your Configuration

Use the FILE_MANAGER package functions to verify your configuration:

-- Test file configuration retrieval
SELECT FILE_MANAGER.GET_SOURCE_FILE_CONFIG(
    pFileUri => 'your_test_file.csv'
) FROM DUAL;

-- Get detailed configuration information
SELECT FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO(
    pSourceFileConfigKey => YOUR_CONFIG_KEY,
    pIncludeContainerInfo => 1,
    pIncludeColumnFormatInfo => 1
) FROM DUAL;

Step 5: Process Your First File

When your file arrives in Oracle Cloud Storage, process it using:

-- Process a file by name
SELECT FILE_MANAGER.PROCESS_SOURCE_FILE(
    pSourceFileReceivedName => 'INBOX/your_file.csv'
) FROM DUAL;

File Processing Workflow

  1. File Arrival: File is uploaded to Oracle Cloud Storage bucket
  2. Registration: FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED() creates record
  3. Status: RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED_AND_TRASHED → ARCHIVED_AND_PURGED (optional)
    • Legacy ARCHIVED status maintained for backward compatibility
    • ARCHIVED_AND_TRASHED: Files archived to Parquet and kept in TRASH folder (default)
    • ARCHIVED_AND_PURGED: Files archived to Parquet and deleted from TRASH folder
  4. External Table: Created automatically based on template table
  5. Data Loading: Data is loaded into target ODS schema
  6. Archival: File is moved to archive bucket after processing

Status Tracking

Monitor your file processing status:

-- Check current file status
SELECT
    SOURCE_FILE_NAME,
    PROCESSING_STATUS,
    RECEPTION_DATE,
    EXTERNAL_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE SOURCE_FILE_NAME LIKE '%your_file%'
ORDER BY RECEPTION_DATE DESC;

Troubleshooting

Common Configuration Issues:

  1. Cannot Delete Configuration Entry - ORA-02292

    ORA-02292: integrity constraint (CT_MRDS.ASFR_A_SOURCE_FILE_CONFIG_KEY_FK) violated - child record found

    This error occurs when trying to delete a record from CT_MRDS.A_SOURCE_FILE_CONFIG that has related files processed in CT_MRDS.A_SOURCE_FILE_RECEIVED.

    Solutions:

    -- Option 1: Check which files are linked to this configuration
SELECT sfr.SOURCE_FILE_NAME, sfr.PROCESSING_STATUS, sfr.RECEPTION_DATE
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
WHERE sfc.SOURCE_FILE_ID = 'YOUR_FILE_ID'
AND sfc.SOURCE_FILE_TYPE = 'INPUT';

-- Option 2: Delete child records first (Keep in mind - This removes processing history)
DELETE FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE A_SOURCE_FILE_CONFIG_KEY = (
    SELECT A_SOURCE_FILE_CONFIG_KEY 
    FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
    WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID' 
    AND SOURCE_FILE_TYPE = 'INPUT'
);

-- Option 3: Update the configuration instead of deleting
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET SOURCE_FILE_NAME_PATTERN = 'new_pattern_*.csv',
    SOURCE_FILE_DESC = 'Updated description'
WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
AND SOURCE_FILE_TYPE = 'INPUT';

  2. Template Table Not Found

    -- Check if template table exists
SELECT * FROM ALL_TABLES
WHERE OWNER = 'CT_ET_TEMPLATES'
AND TABLE_NAME = 'YOUR_TEMPLATE_TABLE';

-- Grant access if needed
GRANT SELECT ON CT_ET_TEMPLATES.YOUR_TEMPLATE_TABLE TO CT_MRDS;

  3. File Pattern Not Matching

    -- Test your pattern against actual filenames
SELECT 'your_test_file.csv' AS filename,
       CASE WHEN 'your_test_file.csv' LIKE 'your_file_*.csv'
            THEN 'MATCHES'
            ELSE 'NO MATCH'
       END AS pattern_test
FROM DUAL;

  4. Date Format Errors

    -- Test date format parsing
SELECT TO_DATE('2024-12-25', 'YYYY-MM-DD') AS parsed_date FROM DUAL;
SELECT TO_DATE('25/12/2024 14:30:00', 'DD/MM/YYYY HH24:MI:SS') AS parsed_datetime FROM DUAL;

  5. Multiple Configurations for Same File

    -- Check for conflicting configurations
SELECT SOURCE_FILE_TYPE, SOURCE_FILE_ID, TABLE_ID, SOURCE_FILE_NAME_PATTERN
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
ORDER BY SOURCE_FILE_TYPE;

  6. Missing A_COLUMN_DATE_FORMAT Table

    -- Create the table if it doesn't exist (run create_column_date_format.sql)
SELECT * FROM USER_TABLES WHERE TABLE_NAME = 'A_COLUMN_DATE_FORMAT';

Verification Queries:

-- Check your complete configuration
SELECT s.A_SOURCE_KEY, s.SOURCE_NAME,
       sfc.SOURCE_FILE_TYPE, sfc.SOURCE_FILE_ID, sfc.TABLE_ID,
       sfc.SOURCE_FILE_NAME_PATTERN, sfc.TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE s
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY sfc.SOURCE_FILE_TYPE, sfc.SOURCE_FILE_ID;

-- Check date format configurations
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME, cdf.DATE_FORMAT
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME LIKE '%YOUR_TEMPLATE%'
ORDER BY cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME;

-- Check system configuration
SELECT ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE
FROM CT_MRDS.A_FILE_MANAGER_CONFIG
ORDER BY ENVIRONMENT_ID, CONFIG_VARIABLE;

Performance Monitoring:

-- Monitor file processing status
SELECT SOURCE_FILE_NAME, PROCESSING_STATUS, RECEPTION_DATE,
       EXTERNAL_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1
ORDER BY RECEPTION_DATE DESC;

-- Check processing logs
SELECT LOG_TIMESTAMP, PROCEDURE_NAME, PROCEDURE_PARAMETERS,
       LOG_LEVEL, LOG_MESSAGE
FROM CT_MRDS.A_PROCESS_LOG
WHERE LOG_TIMESTAMP >= SYSDATE - 1
AND PROCEDURE_NAME LIKE '%FILE_MANAGER%'
ORDER BY LOG_TIMESTAMP DESC;

Real-World Examples

The following examples are based on actual configurations:

Example 1: Simple Data File (ISD - Integrated Surface Data)

-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'ISD',
    pSourceName => 'Integrated Surface Data'
);

-- Step 2: Configure file type
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'ISD',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'IDS_DATA',
    pSourceFileDesc => 'Yearly figures by station',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'IDS_DATA',
    pTemplateTableName => 'CT_ET_TEMPLATES.IDS_DATA'
);

Example 2: Container File with Multiple Inputs (LM - Standing Facilities)

-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'LM',
    pSourceName => 'LM'
);

-- Step 2: Configure container file (archive containing multiple files)
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'CONTAINER',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities',
    pSourceFileNamePattern => '*.xml',
    pTableId => NULL,
    pTemplateTableName => NULL,
    pContainerFileKey => NULL
);

-- Step 3: Configure files extracted from container - Header data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities header',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'STANDING_FACILITIES_HEADER',
    pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
);

-- Step 4: Configure files extracted from container - Detail data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'STANDING_FACILITIES',
    pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES'
);

Example 3: Complex Configuration with Date Formats (C2D - UC_DISSEM)

-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'C2D'
);

-- Step 2: Configure container file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'CONTAINER',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC Dissemination',
    pSourceFileNamePattern => '*.xml',
    pTableId => NULL,
    pTemplateTableName => NULL,
    pContainerFileKey => NULL
);

-- Step 3: Configure metadata file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC Dissemination Metadata',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS'
);

-- Step 4: Configure date formats for metadata file
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'FILE_CREATION_DATE',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);

-- Step 5: Configure marketable assets file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC MarketableAssets Dissemination',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'UC_MA_DISSEM',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM'
);

-- Step 6: Configure date formats for marketable assets file
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM',
    pColumnName => 'FILE_CREATION_DATE',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);

Example 4: Load Configuration File (MDP)

-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'MDP', 
    pSourceName => 'MDP'
);

-- Step 2: Configure regular input file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'MDP',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'BBG_EQUITY_OT',
    pSourceFileDesc => 'Bloomberg Debt',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'BBG_EQUITY_OT',
    pTemplateTableName => 'CT_ET_TEMPLATES.ODS_MDP_BBG_EQUITY_OT'
);

-- Step 3: Configure load configuration file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'MDP',
    pSourceFileType => 'LOAD_CONFIG',
    pSourceFileId => NULL,
    pSourceFileDesc => 'Reuters list ODS Cloud',
    pSourceFileNamePattern => '.*.txt',
    pTableId => NULL,
    pTemplateTableName => NULL
);

Configuration Management

Removing Complete Source System Configuration

When you need to completely remove a source system and all its related configurations, use the DELETE_SOURCE_CASCADE procedure. This procedure performs a cascading delete that removes all related data in the correct order to maintain referential integrity.

DELETE_SOURCE_CASCADE Procedure

Purpose: Safely removes a source system and all its related configurations, including:

  • All file configurations (A_SOURCE_FILE_CONFIG)
  • All processed file records (A_SOURCE_FILE_RECEIVED)
  • All column date format configurations (A_COLUMN_DATE_FORMAT)
  • The source system record (A_SOURCE)

Syntax:

CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'YOUR_SOURCE_KEY'
);

Parameters:

  • pSourceKey: The source system key (PRIMARY KEY from A_SOURCE table)

Usage Examples

Example 1: Remove Test Configuration

-- Remove a test source system and all its configurations
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'TEST_SYS'
);

Example 2: Remove Deprecated System

-- Remove an old system that is no longer used
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'C2D_OLD'
);

Example 3: Clean Up Before Reconfiguration

-- Remove existing configuration before setting up new one
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'C2D'
);

-- Now add new configuration
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D',
    pSourceName => 'C2D - Reconfigured'
);

Example 4: Safe Deletion with Shared Template Protection

-- This example shows how the procedure protects shared date formats
-- Assume 'CT_ET_TEMPLATES.COMMON_DATES' is used by multiple source systems

-- Remove source system 'OLD_SYS' - date formats will be preserved 
-- because template table is shared
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'OLD_SYS'
);

-- The procedure will automatically:
-- 1. Delete all A_SOURCE_FILE_RECEIVED records for OLD_SYS
-- 2. Skip deleting A_COLUMN_DATE_FORMAT records (template is shared)
-- 3. Delete all A_SOURCE_FILE_CONFIG records for OLD_SYS  
-- 4. Delete the A_SOURCE record for OLD_SYS
-- 5. Log warning about skipped date format deletion

What Gets Deleted

The procedure deletes data in this order to respect foreign key constraints:

  1. A_SOURCE_FILE_RECEIVED - All processed file records for each file configuration
  2. A_COLUMN_DATE_FORMAT - Date format configurations for template tables used by the source (only if template table is not shared with other source systems)
  3. A_SOURCE_FILE_CONFIG - All file configurations for the source system
  4. A_SOURCE - The source system record itself

Smart Date Format Protection: The procedure automatically checks if template tables are shared between multiple source systems. If a template table is used by other sources, its date format configurations are preserved to avoid breaking other systems.

Pre-deletion Verification

Before using DELETE_SOURCE_CASCADE, you can verify what will be deleted:

-- Check source system existence
SELECT A_SOURCE_KEY, SOURCE_NAME 
FROM CT_MRDS.A_SOURCE 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Check file configurations that will be deleted
SELECT SOURCE_FILE_TYPE, SOURCE_FILE_ID, TABLE_ID, 
       SOURCE_FILE_NAME_PATTERN, TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY SOURCE_FILE_TYPE, SOURCE_FILE_ID;

-- Check processed files that will be deleted (processing history)
SELECT sfc.SOURCE_FILE_ID, sfc.TABLE_ID, 
       sfr.SOURCE_FILE_NAME, sfr.PROCESSING_STATUS, sfr.RECEPTION_DATE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
JOIN CT_MRDS.A_SOURCE_FILE_RECEIVED sfr ON sfc.A_SOURCE_FILE_CONFIG_KEY = sfr.A_SOURCE_FILE_CONFIG_KEY
WHERE sfc.A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY sfr.RECEPTION_DATE DESC;

-- Check date format configurations that will be deleted
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME, cdf.DATE_FORMAT
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME IN (
    SELECT DISTINCT TEMPLATE_TABLE_NAME 
    FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
    WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
    AND TEMPLATE_TABLE_NAME IS NOT NULL
);

Important Information

✅ Safe Deletion: The procedure now automatically protects shared date format configurations:

  • All processing history for files from this source system will be deleted
  • All configuration settings for the source system will be deleted
  • Date format configurations will be preserved if template tables are shared with other source systems

⚠️ Data Loss Warning: This operation is IRREVERSIBLE for:

  • All processing history for files from this source system
  • All configuration settings for the source system
  • Date format configurations for template tables that are NOT shared with other sources

ℹ️ Shared Template Tables Protection: The procedure automatically detects if template tables are shared between multiple source systems and preserves their date format configurations. You can verify shared usage with:

-- Check if template tables are used by other source systems
SELECT TEMPLATE_TABLE_NAME, COUNT(DISTINCT A_SOURCE_KEY) as source_count,
       LISTAGG(A_SOURCE_KEY, ', ') WITHIN GROUP (ORDER BY A_SOURCE_KEY) as source_keys
FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE TEMPLATE_TABLE_NAME IS NOT NULL
GROUP BY TEMPLATE_TABLE_NAME
HAVING COUNT(DISTINCT A_SOURCE_KEY) > 1
ORDER BY TEMPLATE_TABLE_NAME;

Recovery Options

If you need to preserve some data before deletion:

-- Backup processing history before deletion
CREATE TABLE BACKUP_SOURCE_FILE_RECEIVED AS
SELECT sfr.*
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
WHERE sfc.A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Backup configuration before deletion
CREATE TABLE BACKUP_SOURCE_FILE_CONFIG AS
SELECT * FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Backup date formats before deletion
CREATE TABLE BACKUP_COLUMN_DATE_FORMAT AS
SELECT cdf.*
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME IN (
    SELECT DISTINCT TEMPLATE_TABLE_NAME 
    FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
    WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
    AND TEMPLATE_TABLE_NAME IS NOT NULL
);

Support

-- Get complete FILE_MANAGER package documentation
SELECT CT_MRDS.GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;

Note: For complete formatted output, use proper SQL*Plus settings:

SET PAGESIZE 0
SET LINESIZE 32000
SET LONG 1000000
SELECT CT_MRDS.GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;
Reference in New Issue View Git Blame Copy Permalink