mars/confluence/FILE_MANAGER_Configuration_Guide.md

# FILE_MANAGER Configuration Guide

This document describes the configuration procedures available in the FILE_MANAGER package for setting up file processing rules, defining file naming patterns, and configuring date format handling.

## Overview

The FILE_MANAGER package provides several configuration procedures that work together to create a complete file processing workflow:

- **ADD_SOURCE**: Registers new source systems
- **ADD_SOURCE_FILE_CONFIG**: Configures file processing rules for specific file types
- **ADD_COLUMN_DATE_FORMAT**: Sets up date format handling for specific columns

These procedures are typically used after setting up external tables to enable automatic file processing workflows.

**Note:** Data export functionality (CSV and Parquet export procedures) has been moved to the `DATA_EXPORTER` package. This separation provides better modularity - `FILE_MANAGER` focuses on file processing and validation, while `DATA_EXPORTER` handles data export operations.

**Note:** Data export functionality (CSV and Parquet export procedures) has been moved to the `DATA_EXPORTER` package. This separation provides better modularity - `FILE_MANAGER` focuses on file processing and validation, while `DATA_EXPORTER` handles data export operations.

## File Validation Features

### Automatic CSV Column Validation

The FILE_MANAGER package includes automatic validation for CSV files to ensure they match the expected template structure:

- **Excess Column Detection**: Automatically detects when CSV files contain more columns than defined in the template table
- **Pre-Processing Validation**: Validation occurs before external table creation to prevent processing errors
- **Detailed Error Messages**: Provides specific information about column count mismatches and suggests solutions

**Error Code: -20011 (ERR_EXCESS_COLUMNS_DETECTED)**
```
EXCESS COLUMNS DETECTED!
CSV file has 9 columns but template expects only 6
Excess columns: 3
CSV header: ID,NAME,DESCRIPTION,STATUS,AMOUNT,CREATED_DATE,PRIORITY,EXTRA_COLUMN,ANOTHER_EXTRA
POSSIBLE SOLUTIONS:
  1. Remove excess columns from CSV file before processing
  2. Add excess columns to template table: CT_ET_TEMPLATES.TEMPLATE_NAME
```

### ANALYZE_VALIDATION_ERRORS Function

Wrapper function that simplifies validation error analysis by automatically deriving required parameters from file metadata:

**Signature:**
```sql
FUNCTION ANALYZE_VALIDATION_ERRORS(
    pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;
```

**Parameters:**
- `pSourceFileReceivedKey`: Key from A_SOURCE_FILE_RECEIVED table for the file to analyze

**Automatic Parameter Derivation:**
- **Template Schema/Table**: Extracted from TEMPLATE_TABLE_NAME in configuration
- **CSV File URI**: Built from bucket configuration and file paths
- **Validation Log Table**: Automatically finds most recent validation log

**Purpose:**
- Provides detailed analysis of file validation failures with simplified interface
- Eliminates need to manually specify 4 complex parameters
- Returns structured error information for troubleshooting
- Integrates with enhanced error logging system

**Examples:**
```sql
-- Find file key for problematic file
SELECT A_SOURCE_FILE_RECEIVED_KEY, SOURCE_FILE_NAME, PROCESSING_STATUS
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED 
WHERE SOURCE_FILE_NAME LIKE '%problematic%';

-- Analyze validation errors (simplified interface)
SELECT CT_MRDS.FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;
```

**Comparison with Direct Function:**
```sql
-- Wrapper (1 parameter) - RECOMMENDED
SELECT CT_MRDS.FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;

-- Direct function (4 parameters) - Advanced usage
SELECT CT_MRDS.ENV_MANAGER.ANALYZE_VALIDATION_ERRORS(
    'VALIDATE$253_LOG', 'CT_ET_TEMPLATES', 
    'MOCK_PROC_TABLE', 'https://bucket-uri/file.csv'
) FROM DUAL;
```

## Configuration Procedures

### ADD_SOURCE

Adds a new source system to the configuration.

**Signature:**
```sql
PROCEDURE ADD_SOURCE (
    pSourceKey   IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
    pSourceName  IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);
```

**Parameters:**
- `pSourceKey`: Unique identifier for the source system (e.g., 'C2D', 'LM', 'TOP')
- `pSourceName`: Human-readable name for the source system

**Example:**
```sql
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'Central Bank Data System'
);
```

**Purpose:**
- Registers a new data source in the system
- Required before configuring file types for the source
- Creates master record for source system identification

### ADD_SOURCE_FILE_CONFIG

Configures file processing rules for a specific file type within a source system.

**Signature:**
```sql
PROCEDURE ADD_SOURCE_FILE_CONFIG (
    pSourceKey             IN   CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE,
    pSourceFileType        IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE,
    pSourceFileId          IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE,
    pSourceFileDesc        IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE,
    pSourceFileNamePattern IN   CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE,
    pTableId               IN   CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE                  DEFAULT NULL,
    pTemplateTableName     IN   CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE       DEFAULT NULL,
    pContainerFileKey      IN   CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE        DEFAULT NULL,
    pEncoding              IN   CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING%TYPE                  DEFAULT NULL  -- MARS-1049: CSV character set encoding
);
```

**Parameters:**
- `pSourceKey`: Source system identifier (must exist in A_SOURCE table)
- `pSourceFileType`: Type of file ('INPUT', 'OUTPUT', 'REFERENCE', etc.)
- `pSourceFileId`: Unique identifier for the file type within the source
- `pSourceFileDesc`: Human-readable description of the file type
- `pSourceFileNamePattern`: Pattern to match incoming files (supports wildcards like *)
- `pTableId`: Table identifier used in storage paths (optional)
- `pTemplateTableName`: Template table defining the structure (optional)
- `pContainerFileKey`: Reference to container file for nested structures (optional)
- `pEncoding`: **MARS-1049** - Oracle character set encoding for CSV files (optional) - Examples: 'UTF8', 'WE8MSWIN1252', 'EE8ISO8859P2', 'CL8MSWIN1251'. Used by CREATE_EXTERNAL_TABLE for CHARACTERSET specification.

**Example:**
```sql
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC DISSEM Metadata Loads',
    pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pContainerFileKey => NULL,
    pEncoding => 'UTF8'  -- MARS-1049: CSV character set encoding
);
```

**MARS-1049 CSV Encoding Support:**
The `pEncoding` parameter enables proper handling of international character sets in CSV files:

**Common Character Encodings:**
- `'UTF8'` - Unicode UTF-8 (recommended for international data)
- `'WE8MSWIN1252'` - Western European Windows-1252 (CSDB data)
- `'EE8ISO8859P2'` - Eastern European ISO-8859-2
- `'CL8MSWIN1251'` - Cyrillic Windows-1251
- `'JA16SJIS'` - Japanese Shift-JIS

**Encoding Examples:**
```sql
-- UTF-8 for international customer data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'INTL_SYS',
    pSourceFileId => 'CUSTOMER_DATA',
    pEncoding => 'UTF8'
);

-- Windows-1252 for CSDB European data  
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'CSDB',
    pSourceFileId => 'DEBT_DAILY',
    pEncoding => 'WE8MSWIN1252'
);

-- Backward compatibility - no encoding parameter
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LEGACY_SYS',
    pSourceFileId => 'OLD_DATA'
    -- pEncoding omitted - uses database default
);
```

**Purpose:**
- Defines how files should be processed for a specific file type
- Establishes file naming patterns for automatic recognition
- Links file types to template tables and storage paths
- Enables automatic file processing workflows

### ADD_COLUMN_DATE_FORMAT

Configures date format handling for specific columns in template tables.

**Signature:**
```sql
PROCEDURE ADD_COLUMN_DATE_FORMAT (
    pTemplateTableName     IN   CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE,
    pColumnName            IN   CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE,
    pDateFormat            IN   CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);
```

**Parameters:**
- `pTemplateTableName`: Name of the template table (e.g., 'CT_ET_TEMPLATES.C2D_TABLE_NAME')
- `pColumnName`: Name of the column containing date data
- `pDateFormat`: Oracle date format mask (e.g., 'YYYY-MM-DD', 'DD/MM/YYYY HH24:MI:SS')

**Examples:**
```sql
-- Configure date column with ISO format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);

-- Configure date column with European format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'LOAD_START',
    pDateFormat => 'DD/MM/YYYY'
);

-- Configure timestamp column
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'PROCESS_TIMESTAMP',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);
```

**Purpose:**
- Defines how date columns should be parsed from CSV files
- Ensures correct date format conversion during external table queries
- Prevents date format errors and data type mismatches
- Supports various international date formats

## Complete Configuration Workflow

Here's a complete example showing the typical configuration sequence:

```sql
-- Step 1: Add source system (if not already exists)
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'Central Bank Data System'
);

-- Step 2: Configure file type for processing
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC DISSEM Metadata Loads',
    pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pContainerFileKey => NULL,
    pEncoding => 'UTF8'  -- MARS-1049: Character set encoding
);

-- Step 3: Configure date formats for specific columns
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'LOAD_START', 
    pDateFormat => 'DD/MM/YYYY'
);

-- Step 4: Add additional file types as needed
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'AGGREGATED_ALLOTMENT',
    pSourceFileDesc => 'Aggregated Allotment Data',
    pSourceFileNamePattern => 'AGGREGATED_ALLOTMENT_*.csv',
    pTableId => 'AGGREGATED_ALLOTMENT',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_AGGREGATED_ALLOTMENT',
    pContainerFileKey => NULL,
    pEncoding => 'WE8MSWIN1252'  -- MARS-1049: Windows-1252 encoding for European data
);
```

## Configuration Verification

Use these queries to verify your configuration setup:

### Check Source Systems
```sql
SELECT A_SOURCE_KEY, SOURCE_NAME, CREATED_ON
FROM CT_MRDS.A_SOURCE 
ORDER BY A_SOURCE_KEY;
```

### Check File Configurations
```sql
SELECT sfc.A_SOURCE_KEY, 
       sfc.SOURCE_FILE_ID, 
       sfc.SOURCE_FILE_DESC, 
       sfc.SOURCE_FILE_NAME_PATTERN, 
       sfc.TABLE_ID, 
       sfc.TEMPLATE_TABLE_NAME,
       sfc.SOURCE_FILE_TYPE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
JOIN CT_MRDS.A_SOURCE s ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY = 'C2D'  -- Replace with your source key
ORDER BY sfc.SOURCE_FILE_ID;
```

### Check Date Format Configurations
```sql
SELECT TEMPLATE_TABLE_NAME, 
       COLUMN_NAME, 
       DATE_FORMAT,
       CREATED_ON
FROM CT_MRDS.A_COLUMN_DATE_FORMAT
WHERE TEMPLATE_TABLE_NAME LIKE '%C2D%'  -- Replace with your template pattern
ORDER BY TEMPLATE_TABLE_NAME, COLUMN_NAME;
```

### Complete Configuration Overview
```sql
-- Complete view of configuration for a specific source
SELECT 
    s.A_SOURCE_KEY,
    s.SOURCE_NAME,
    sfc.SOURCE_FILE_ID,
    sfc.SOURCE_FILE_DESC,
    sfc.SOURCE_FILE_NAME_PATTERN,
    sfc.TABLE_ID,
    sfc.TEMPLATE_TABLE_NAME,
    cdf.COLUMN_NAME,
    cdf.DATE_FORMAT
FROM CT_MRDS.A_SOURCE s
LEFT JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
LEFT JOIN CT_MRDS.A_COLUMN_DATE_FORMAT cdf ON sfc.TEMPLATE_TABLE_NAME = cdf.TEMPLATE_TABLE_NAME
WHERE s.A_SOURCE_KEY = 'C2D'  -- Replace with your source key
ORDER BY sfc.SOURCE_FILE_ID, cdf.COLUMN_NAME;
```

## Storage Path Structure Based on Configuration

The configuration parameters directly affect how files are organized in Oracle Cloud Storage according to the **official path patterns**:

```
Oracle Cloud Storage Structure (OFFICIAL PATTERNS):

INBOX Bucket - Pattern: 'INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/'
└── INBOX/
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pSourceFileId}/    -- e.g., "UC_DISSEM", "STANDING_FACILITIES"  
            └── {pTableId}/     -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
                └── files matching {pSourceFileNamePattern}

DATA Bucket - Patterns: 'ODS/{SOURCE}/{TABLE_NAME}/' and 'TRASH/{SOURCE}/{TABLE_NAME}/'
├── ODS/
│   └── {pSourceKey}/           -- e.g., "C2D", "LM"
│       └── {pTableId}/         -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
│           └── processed files
└── TRASH/                      -- File retention subfolder (not a separate bucket)
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pTableId}/         -- CSV files after archival (ARCHIVED_AND_TRASHED status)

ARCHIVE Bucket - Pattern: 'ARCHIVE/{SOURCE}/{TABLE_NAME}/'
└── ARCHIVE/
    └── {pSourceKey}/           -- e.g., "C2D", "LM"
        └── {pTableId}/         -- e.g., "A_UC_DISSEM_METADATA_LOADS", "STANDING_FACILITIES"
            └── PARTITION_YEAR=*/
                └── PARTITION_MONTH=*/
                    └── *.parquet files
```

**Critical Path Pattern Requirements:**
- **INBOX** requires full 3-level path: `INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/`
- **ODS** uses simplified 2-level path: `ODS/{SOURCE}/{TABLE_NAME}/` (no SOURCE_FILE_ID)
- **TRASH** uses simplified 2-level path: `TRASH/{SOURCE}/{TABLE_NAME}/` (subfolder in DATA bucket)
- **ARCHIVE** uses simplified 2-level path: `ARCHIVE/{SOURCE}/{TABLE_NAME}/` (no SOURCE_FILE_ID)
- **All patterns are mandatory** - no simplified versions allowed
- File names must match `pSourceFileNamePattern` for automatic processing
- **Note**: TRASH is NOT a separate bucket - it's a subfolder within the DATA bucket

## Configuration Management Best Practices

### 1. Naming Conventions

**Source Keys:**
- Use short, descriptive codes (e.g., 'C2D', 'LM', 'TOP')
- Consistent with existing system abbreviations
- Uppercase for standardization

**Source File IDs:**
- Descriptive but concise (e.g., 'UC_DISSEM', 'AGGREGATED_ALLOTMENT')
- Use underscores for readability
- Reflect the business purpose

**Table IDs:**
- Match the actual table name patterns
- Consistent with template table naming
- Use underscores for compound names

### 2. File Name Patterns

**Wildcards:**
- Use `*` for variable parts (dates, sequence numbers)
- Be specific enough to avoid conflicts
- Consider file extensions (.csv, .txt, .json)

**Examples:**
```sql
-- Good patterns:
'UC_NMA_DISSEM-*.csv'           -- Clear prefix with wildcard
'AGGREGATED_ALLOTMENT_*.csv'    -- Descriptive with extension
'DATA_EXPORT_????-??-??.txt'   -- Date-specific pattern

-- Avoid overly broad patterns:
'*.csv'                         -- Too generic
'DATA*'                         -- Missing extension
```

### 3. Date Format Configuration

**Common Formats:**
```sql
-- ISO standard formats (recommended)
'YYYY-MM-DD'                    -- Date only
'YYYY-MM-DD HH24:MI:SS'        -- Date with time
'YYYY-MM-DD"T"HH24:MI:SS'      -- ISO timestamp

-- Regional formats
'DD/MM/YYYY'                    -- European format
'MM/DD/YYYY'                    -- US format
'DD-MON-YYYY'                   -- Oracle default format
```

### 4. Configuration Sequence

Always follow this order:
1. **ADD_SOURCE** (if new source system)
2. **ADD_SOURCE_FILE_CONFIG** (for each file type)
3. **ADD_COLUMN_DATE_FORMAT** (for date columns in each template)

## Troubleshooting Configuration Issues

### Common Problems

1. **Source Key Not Found**
   ```
   ORA-02291: integrity constraint violated - parent key not found
   ```
   **Solution:** Ensure ADD_SOURCE is called before ADD_SOURCE_FILE_CONFIG

2. **Template Table Not Found**
   ```
   ORA-00942: table or view does not exist
   ```
   **Solution:** Verify template table exists and name is correct (including schema)

3. **Date Format Errors**
   ```
   ORA-01821: date format not recognized
   ```
   **Solution:** Check date format mask matches actual data format in files

4. **Excess Column Errors (Code: -20011)**
   ```
   ORA-20011: CSV file contains more columns than template allows
   EXCESS COLUMNS DETECTED!
   CSV file has 9 columns but template expects only 6
   ```
   **Solutions:**
   - Remove excess columns from CSV file before processing
   - Add missing columns to template table
   - Use `ANALYZE_VALIDATION_ERRORS()` for detailed analysis

5. **File Pattern Not Matching**
   - Files uploaded but not processed
   **Solution:** Verify file names match the configured pattern exactly

### Enhanced Error Logging

The system includes enhanced error logging through the `LOG_PROCESS_ERROR` function:

- **Structured Logging**: All errors logged with full context and stack traces
- **Error Analysis**: Automatic categorization and analysis of validation failures
- **Troubleshooting Support**: Detailed error information for faster problem resolution

**Viewing Error Logs:**
```sql
SELECT LOG_LEVEL, LOG_MESSAGE, LOG_TIMESTAMP 
FROM CT_MRDS.A_PROCESS_LOG 
WHERE LOG_LEVEL = 'ERROR' 
AND LOG_MESSAGE LIKE '%EXCESS COLUMNS%'
ORDER BY LOG_TIMESTAMP DESC;
```

**Error Log Content Example:**
```
Error Message:
ORA-20011: CSV file contains more columns than template allows
EXCESS COLUMNS DETECTED!
CSV file has 9 columns but template expects only 6
Excess columns: 3
CSV header: ID,NAME,DESCRIPTION,STATUS,AMOUNT,CREATED_DATE,PRIORITY,EXTRA_COLUMN,ANOTHER_EXTRA
POSSIBLE SOLUTIONS:
  1. Remove excess columns from CSV file before processing
  2. Add excess columns to template table: CT_ET_TEMPLATES.MOCK_PROC_TABLE
-------------------------------------------------------
Error Stack: [Full Oracle error stack trace]
-------------------------------------------------------
Error Backtrace: [Full Oracle error backtrace]
```

### Diagnostic Queries

```sql
-- Check for orphaned configurations
SELECT sfc.A_SOURCE_KEY, sfc.SOURCE_FILE_ID
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
LEFT JOIN CT_MRDS.A_SOURCE s ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY IS NULL;

-- Check for missing template tables
SELECT DISTINCT sfc.TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
WHERE sfc.TEMPLATE_TABLE_NAME IS NOT NULL
  AND NOT EXISTS (
    SELECT 1 FROM all_tables 
    WHERE owner||'.'||table_name = sfc.TEMPLATE_TABLE_NAME
  );

-- Check date format configurations without corresponding columns
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE NOT EXISTS (
    SELECT 1 FROM all_tab_columns
    WHERE owner||'.'||table_name = cdf.TEMPLATE_TABLE_NAME
      AND column_name = cdf.COLUMN_NAME
);
```

This configuration enables automatic file processing workflows where files uploaded to the INBOX are automatically processed and moved to ODS, with historical data archived in partitioned PARQUET format.

Register your source system using the FILE_MANAGER package procedure:

```sql
-- Use the FILE_MANAGER procedure to add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'YOUR_SYS',
    pSourceName => 'Your System Name Description'
);
```

**Parameters:**
- `pSourceKey`: Short identifier for your source system (e.g., 'C2D', 'PAYROLL', 'ISD', 'LM')
- `pSourceName`: Descriptive name of the source system

### Step 2: Configure File Type in A_SOURCE_FILE_CONFIG

Add your file configuration to define how the system should handle your file type using the FILE_MANAGER package procedure:

```sql
-- Use the FILE_MANAGER procedure to add source file configuration
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'YOUR_SYS',                    -- Your source system key
    pSourceFileType => 'INPUT',                  -- File type: INPUT/CONTAINER/LOAD_CONFIG
    pSourceFileId => 'YOUR_FILE_ID',             -- Unique identifier for this file type
    pSourceFileDesc => 'Description of your file type',  -- Human-readable description
    pSourceFileNamePattern => 'your_file_pattern_*.csv', -- File name pattern with wildcards
    pTableId => 'YOUR_TABLE_ID',                 -- Target table identifier
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',  -- Template table for External Table creation
    pContainerFileKey => NULL                    -- Container key (for INPUT files, use NULL)
);

-- Note: Additional configuration parameters like thresholds, schema names, etc.
-- need to be set separately by updating the record after creation if needed:
/*
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET DAYS_FOR_ARCHIVE_THRESHOLD = 30,
    FILES_COUNT_OVER_ARCHIVE_THRESHOLD = 100,
    BYTES_SUM_OVER_ARCHIVE_THRESHOLD = 1073741824,
    ODS_SCHEMA_NAME = 'YOUR_TARGET_SCHEMA',
    ROWS_COUNT_OVER_ARCHIVE_THRESHOLD = 1000000,
    HOURS_TO_EXPIRE_STATISTICS = 24
WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
  AND SOURCE_FILE_TYPE = 'INPUT';
*/
```

**Additional Configuration Parameters:**

| Parameter | Purpose | Example Value | Description |
|-----------|---------|---------------|-------------|
| `DAYS_FOR_ARCHIVE_THRESHOLD` | Archival threshold | `30` | Number of days after which files are considered old and eligible for archival. Used by FILE_ARCHIVER to identify files that should be moved to archive bucket. |
| `FILES_COUNT_OVER_ARCHIVE_THRESHOLD` | Archival trigger | `100` | Maximum number of old files (older than DAYS_FOR_ARCHIVE_THRESHOLD) before automatic archival is triggered. When exceeded, FILE_ARCHIVER exports old data to PARQUET and removes CSV files. |
| `BYTES_SUM_OVER_ARCHIVE_THRESHOLD` | Archival trigger | `1073741824` (1GB) | Maximum total size in bytes of old files before archival is triggered. Helps manage storage usage by automatically archiving large amounts of historical data. |
| `ODS_SCHEMA_NAME` | Target schema | `'ODS'` | Name of the schema where final operational data tables are located. Used by FILE_ARCHIVER to access tables for statistics gathering and archival operations. Format: `{ODS_SCHEMA_NAME}.{SOURCE_KEY}_{TABLE_ID}_ODS` |
| `ROWS_COUNT_OVER_ARCHIVE_THRESHOLD` | Archival trigger | `1000000` | Maximum number of rows in old files before archival is triggered. Prevents tables from growing too large by archiving historical data when row count threshold is exceeded. |
| `HOURS_TO_EXPIRE_STATISTICS` | Statistics refresh | `24` | Number of hours after which table statistics expire and need to be recalculated. FILE_ARCHIVER refreshes statistics if they are older than this threshold before making archival decisions. |
```

**File Type Descriptions:**

| File Type | Purpose | Example Use Cases |
|-----------|---------|-------------------|
| `INPUT` | Data files for processing | CSV files, Excel files, data extracts |
| `CONTAINER` | Archive/zip files containing multiple files | XML archives, ZIP files with multiple CSVs |
| `LOAD_CONFIG` | Configuration files for complex loading scenarios | Files mapped to multiple tables, routing configs |

**File Naming Patterns:**
- Use `*` for any characters: `employee_*.csv` matches `employee_2024.csv`, `employee_january.csv`
- Use `.*.csv` for regex patterns: `.*.csv` matches any CSV file
- Be specific to avoid conflicts: `payroll_[0-9]*.xlsx` for numbered payroll files

### Step 3: Configure Date Formats (Optional)

If your file contains date columns that need special formatting, add them using the FILE_MANAGER package procedure:

```sql
-- Use the FILE_MANAGER procedure to add date format configuration
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',  -- Same as in Step 2
    pColumnName => 'BIRTH_DATE',                         -- Column name containing dates
    pDateFormat => 'DD/MM/YYYY'                          -- Expected date format in your files
);

-- Add more columns if needed
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'SCHEMA.YOUR_TEMPLATE_TABLE',
    pColumnName => 'HIRE_DATE',
    pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);
```

**Alternative approach using parameters from seed2.sql:**
Note: Some implementations may use an extended signature (verify with your FILE_MANAGER package version):

```sql
-- Alternative signature that might be available in some versions
/*
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pSourceFileId => 'YOUR_FILE_ID',
    pTableId => 'YOUR_TABLE_ID',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);
*/
```

### Step 4: Test Your Configuration

Use the FILE_MANAGER package functions to verify your configuration:

```sql
-- Test file configuration retrieval
SELECT FILE_MANAGER.GET_SOURCE_FILE_CONFIG(
    pFileUri => 'your_test_file.csv'
) FROM DUAL;

-- Get detailed configuration information
SELECT FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO(
    pSourceFileConfigKey => YOUR_CONFIG_KEY,
    pIncludeContainerInfo => 1,
    pIncludeColumnFormatInfo => 1
) FROM DUAL;
```

### Step 5: Process Your First File

When your file arrives in Oracle Cloud Storage, process it using:

```sql
-- Process a file by name
SELECT FILE_MANAGER.PROCESS_SOURCE_FILE(
    pSourceFileReceivedName => 'INBOX/your_file.csv'
) FROM DUAL;
```

## File Processing Workflow

1. **File Arrival**: File is uploaded to Oracle Cloud Storage bucket
2. **Registration**: FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED() creates record
3. **Status**: RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED_AND_TRASHED → ARCHIVED_AND_PURGED (optional)
   - Legacy ARCHIVED status maintained for backward compatibility
   - ARCHIVED_AND_TRASHED: Files archived to Parquet and kept in TRASH folder (default)
   - ARCHIVED_AND_PURGED: Files archived to Parquet and deleted from TRASH folder
4. **External Table**: Created automatically based on template table
5. **Data Loading**: Data is loaded into target ODS schema
6. **Archival**: File is moved to archive bucket after processing

## Status Tracking

Monitor your file processing status:

```sql
-- Check current file status
SELECT
    SOURCE_FILE_NAME,
    PROCESSING_STATUS,
    RECEPTION_DATE,
    EXTERNAL_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE SOURCE_FILE_NAME LIKE '%your_file%'
ORDER BY RECEPTION_DATE DESC;
```

## Troubleshooting

### Common Configuration Issues:

1. **Cannot Delete Configuration Entry - ORA-02292**
   ```
   ORA-02292: integrity constraint (CT_MRDS.ASFR_A_SOURCE_FILE_CONFIG_KEY_FK) violated - child record found
   ```
   This error occurs when trying to delete a record from `CT_MRDS.A_SOURCE_FILE_CONFIG` that has related files processed in `CT_MRDS.A_SOURCE_FILE_RECEIVED`.
   
   **Solutions:**
   ```sql
   -- Option 1: Check which files are linked to this configuration
   SELECT sfr.SOURCE_FILE_NAME, sfr.PROCESSING_STATUS, sfr.RECEPTION_DATE
   FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
   JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
   WHERE sfc.SOURCE_FILE_ID = 'YOUR_FILE_ID'
   AND sfc.SOURCE_FILE_TYPE = 'INPUT';
   
   -- Option 2: Delete child records first (Keep in mind - This removes processing history)
   DELETE FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
   WHERE A_SOURCE_FILE_CONFIG_KEY = (
       SELECT A_SOURCE_FILE_CONFIG_KEY 
       FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
       WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID' 
       AND SOURCE_FILE_TYPE = 'INPUT'
   );
   
   -- Option 3: Update the configuration instead of deleting
   UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
   SET SOURCE_FILE_NAME_PATTERN = 'new_pattern_*.csv',
       SOURCE_FILE_DESC = 'Updated description'
   WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
   AND SOURCE_FILE_TYPE = 'INPUT';
   ```

2. **Template Table Not Found**
   ```sql
   -- Check if template table exists
   SELECT * FROM ALL_TABLES
   WHERE OWNER = 'CT_ET_TEMPLATES'
   AND TABLE_NAME = 'YOUR_TEMPLATE_TABLE';
   
   -- Grant access if needed
   GRANT SELECT ON CT_ET_TEMPLATES.YOUR_TEMPLATE_TABLE TO CT_MRDS;
   ```

2. **File Pattern Not Matching**
   ```sql
   -- Test your pattern against actual filenames
   SELECT 'your_test_file.csv' AS filename,
          CASE WHEN 'your_test_file.csv' LIKE 'your_file_*.csv'
               THEN 'MATCHES'
               ELSE 'NO MATCH'
          END AS pattern_test
   FROM DUAL;
   ```

3. **Date Format Errors**
   ```sql
   -- Test date format parsing
   SELECT TO_DATE('2024-12-25', 'YYYY-MM-DD') AS parsed_date FROM DUAL;
   SELECT TO_DATE('25/12/2024 14:30:00', 'DD/MM/YYYY HH24:MI:SS') AS parsed_datetime FROM DUAL;
   ```

4. **Multiple Configurations for Same File**
   ```sql
   -- Check for conflicting configurations
   SELECT SOURCE_FILE_TYPE, SOURCE_FILE_ID, TABLE_ID, SOURCE_FILE_NAME_PATTERN
   FROM CT_MRDS.A_SOURCE_FILE_CONFIG
   WHERE SOURCE_FILE_ID = 'YOUR_FILE_ID'
   ORDER BY SOURCE_FILE_TYPE;
   ```

5. **Missing A_COLUMN_DATE_FORMAT Table**
   ```sql
   -- Create the table if it doesn't exist (run create_column_date_format.sql)
   SELECT * FROM USER_TABLES WHERE TABLE_NAME = 'A_COLUMN_DATE_FORMAT';
   ```

### Verification Queries:

```sql
-- Check your complete configuration
SELECT s.A_SOURCE_KEY, s.SOURCE_NAME,
       sfc.SOURCE_FILE_TYPE, sfc.SOURCE_FILE_ID, sfc.TABLE_ID,
       sfc.SOURCE_FILE_NAME_PATTERN, sfc.TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE s
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON s.A_SOURCE_KEY = sfc.A_SOURCE_KEY
WHERE s.A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY sfc.SOURCE_FILE_TYPE, sfc.SOURCE_FILE_ID;

-- Check date format configurations
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME, cdf.DATE_FORMAT
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME LIKE '%YOUR_TEMPLATE%'
ORDER BY cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME;

-- Check system configuration
SELECT ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE
FROM CT_MRDS.A_FILE_MANAGER_CONFIG
ORDER BY ENVIRONMENT_ID, CONFIG_VARIABLE;
```

### Performance Monitoring:

```sql
-- Monitor file processing status
SELECT SOURCE_FILE_NAME, PROCESSING_STATUS, RECEPTION_DATE,
       EXTERNAL_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED
WHERE RECEPTION_DATE >= SYSDATE - 1
ORDER BY RECEPTION_DATE DESC;

-- Check processing logs
SELECT LOG_TIMESTAMP, PROCEDURE_NAME, PROCEDURE_PARAMETERS,
       LOG_LEVEL, LOG_MESSAGE
FROM CT_MRDS.A_PROCESS_LOG
WHERE LOG_TIMESTAMP >= SYSDATE - 1
AND PROCEDURE_NAME LIKE '%FILE_MANAGER%'
ORDER BY LOG_TIMESTAMP DESC;
```

## Real-World Examples

The following examples are based on actual configurations:

### Example 1: Simple Data File (ISD - Integrated Surface Data)

```sql
-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'ISD',
    pSourceName => 'Integrated Surface Data'
);

-- Step 2: Configure file type
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'ISD',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'IDS_DATA',
    pSourceFileDesc => 'Yearly figures by station',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'IDS_DATA',
    pTemplateTableName => 'CT_ET_TEMPLATES.IDS_DATA'
);
```

### Example 2: Container File with Multiple Inputs (LM - Standing Facilities)

```sql
-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'LM',
    pSourceName => 'LM'
);

-- Step 2: Configure container file (archive containing multiple files)
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'CONTAINER',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities',
    pSourceFileNamePattern => '*.xml',
    pTableId => NULL,
    pTemplateTableName => NULL,
    pContainerFileKey => NULL
);

-- Step 3: Configure files extracted from container - Header data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities header',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'STANDING_FACILITIES_HEADER',
    pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
);

-- Step 4: Configure files extracted from container - Detail data
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'LM',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'STANDING_FACILITIES',
    pSourceFileDesc => 'LM Standing facilities',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'STANDING_FACILITIES',
    pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES'
);
```

### Example 3: Complex Configuration with Date Formats (C2D - UC_DISSEM)

```sql
-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D', 
    pSourceName => 'C2D'
);

-- Step 2: Configure container file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'CONTAINER',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC Dissemination',
    pSourceFileNamePattern => '*.xml',
    pTableId => NULL,
    pTemplateTableName => NULL,
    pContainerFileKey => NULL
);

-- Step 3: Configure metadata file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC Dissemination Metadata',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'A_UC_DISSEM_METADATA_LOADS',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS'
);

-- Step 4: Configure date formats for metadata file
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
    pColumnName => 'FILE_CREATION_DATE',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);

-- Step 5: Configure marketable assets file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'C2D',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'UC_DISSEM',
    pSourceFileDesc => 'UC MarketableAssets Dissemination',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'UC_MA_DISSEM',
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM'
);

-- Step 6: Configure date formats for marketable assets file
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM',
    pColumnName => 'SNAPSHOT_DATE',
    pDateFormat => 'YYYY-MM-DD'
);

CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
    pTemplateTableName => 'CT_ET_TEMPLATES.C2D_UC_MA_DISSEM',
    pColumnName => 'FILE_CREATION_DATE',
    pDateFormat => 'YYYY-MM-DD"T"HH24:MI:SS'
);
```

### Example 4: Load Configuration File (MDP)

```sql
-- Step 1: Add source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'MDP', 
    pSourceName => 'MDP'
);

-- Step 2: Configure regular input file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'MDP',
    pSourceFileType => 'INPUT',
    pSourceFileId => 'BBG_EQUITY_OT',
    pSourceFileDesc => 'Bloomberg Debt',
    pSourceFileNamePattern => '.*.csv',
    pTableId => 'BBG_EQUITY_OT',
    pTemplateTableName => 'CT_ET_TEMPLATES.ODS_MDP_BBG_EQUITY_OT'
);

-- Step 3: Configure load configuration file
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
    pSourceKey => 'MDP',
    pSourceFileType => 'LOAD_CONFIG',
    pSourceFileId => NULL,
    pSourceFileDesc => 'Reuters list ODS Cloud',
    pSourceFileNamePattern => '.*.txt',
    pTableId => NULL,
    pTemplateTableName => NULL
);
```

## Configuration Management

### Removing Complete Source System Configuration

When you need to completely remove a source system and all its related configurations, use the `DELETE_SOURCE_CASCADE` procedure. This procedure performs a **cascading delete** that removes all related data in the correct order to maintain referential integrity.

#### DELETE_SOURCE_CASCADE Procedure

**Purpose**: Safely removes a source system and all its related configurations, including:
- All file configurations (`A_SOURCE_FILE_CONFIG`)
- All processed file records (`A_SOURCE_FILE_RECEIVED`) 
- All column date format configurations (`A_COLUMN_DATE_FORMAT`)
- The source system record (`A_SOURCE`)

**Syntax**:
```sql
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'YOUR_SOURCE_KEY'
);
```

**Parameters**:
- `pSourceKey`: The source system key (PRIMARY KEY from `A_SOURCE` table)

#### Usage Examples

**Example 1: Remove Test Configuration**
```sql
-- Remove a test source system and all its configurations
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'TEST_SYS'
);
```

**Example 2: Remove Deprecated System**
```sql
-- Remove an old system that is no longer used
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'C2D_OLD'
);
```

**Example 3: Clean Up Before Reconfiguration**
```sql
-- Remove existing configuration before setting up new one
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'C2D'
);

-- Now add new configuration
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
    pSourceKey => 'C2D',
    pSourceName => 'C2D - Reconfigured'
);
```

**Example 4: Safe Deletion with Shared Template Protection**
```sql
-- This example shows how the procedure protects shared date formats
-- Assume 'CT_ET_TEMPLATES.COMMON_DATES' is used by multiple source systems

-- Remove source system 'OLD_SYS' - date formats will be preserved 
-- because template table is shared
CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(
    pSourceKey => 'OLD_SYS'
);

-- The procedure will automatically:
-- 1. Delete all A_SOURCE_FILE_RECEIVED records for OLD_SYS
-- 2. Skip deleting A_COLUMN_DATE_FORMAT records (template is shared)
-- 3. Delete all A_SOURCE_FILE_CONFIG records for OLD_SYS  
-- 4. Delete the A_SOURCE record for OLD_SYS
-- 5. Log warning about skipped date format deletion
```

#### What Gets Deleted

The procedure deletes data in this order to respect foreign key constraints:

1. **A_SOURCE_FILE_RECEIVED** - All processed file records for each file configuration
2. **A_COLUMN_DATE_FORMAT** - Date format configurations for template tables used by the source (only if template table is not shared with other source systems)
3. **A_SOURCE_FILE_CONFIG** - All file configurations for the source system
4. **A_SOURCE** - The source system record itself

**Smart Date Format Protection**: The procedure automatically checks if template tables are shared between multiple source systems. If a template table is used by other sources, its date format configurations are preserved to avoid breaking other systems.

#### Pre-deletion Verification

Before using `DELETE_SOURCE_CASCADE`, you can verify what will be deleted:

```sql
-- Check source system existence
SELECT A_SOURCE_KEY, SOURCE_NAME 
FROM CT_MRDS.A_SOURCE 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Check file configurations that will be deleted
SELECT SOURCE_FILE_TYPE, SOURCE_FILE_ID, TABLE_ID, 
       SOURCE_FILE_NAME_PATTERN, TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY SOURCE_FILE_TYPE, SOURCE_FILE_ID;

-- Check processed files that will be deleted (processing history)
SELECT sfc.SOURCE_FILE_ID, sfc.TABLE_ID, 
       sfr.SOURCE_FILE_NAME, sfr.PROCESSING_STATUS, sfr.RECEPTION_DATE
FROM CT_MRDS.A_SOURCE_FILE_CONFIG sfc
JOIN CT_MRDS.A_SOURCE_FILE_RECEIVED sfr ON sfc.A_SOURCE_FILE_CONFIG_KEY = sfr.A_SOURCE_FILE_CONFIG_KEY
WHERE sfc.A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
ORDER BY sfr.RECEPTION_DATE DESC;

-- Check date format configurations that will be deleted
SELECT cdf.TEMPLATE_TABLE_NAME, cdf.COLUMN_NAME, cdf.DATE_FORMAT
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME IN (
    SELECT DISTINCT TEMPLATE_TABLE_NAME 
    FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
    WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
    AND TEMPLATE_TABLE_NAME IS NOT NULL
);
```

#### Important Information

✅ **Safe Deletion**: The procedure now automatically protects shared date format configurations:
- **All processing history** for files from this source system will be deleted
- **All configuration settings** for the source system will be deleted  
- **Date format configurations** will be preserved if template tables are shared with other source systems

⚠️ **Data Loss Warning**: This operation is **IRREVERSIBLE** for:
- **All processing history** for files from this source system
- **All configuration settings** for the source system
- **Date format configurations** for template tables that are NOT shared with other sources

ℹ️ **Shared Template Tables Protection**: The procedure automatically detects if template tables are shared between multiple source systems and preserves their date format configurations. You can verify shared usage with:

```sql
-- Check if template tables are used by other source systems
SELECT TEMPLATE_TABLE_NAME, COUNT(DISTINCT A_SOURCE_KEY) as source_count,
       LISTAGG(A_SOURCE_KEY, ', ') WITHIN GROUP (ORDER BY A_SOURCE_KEY) as source_keys
FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE TEMPLATE_TABLE_NAME IS NOT NULL
GROUP BY TEMPLATE_TABLE_NAME
HAVING COUNT(DISTINCT A_SOURCE_KEY) > 1
ORDER BY TEMPLATE_TABLE_NAME;
```

#### Recovery Options

If you need to preserve some data before deletion:

```sql
-- Backup processing history before deletion
CREATE TABLE BACKUP_SOURCE_FILE_RECEIVED AS
SELECT sfr.*
FROM CT_MRDS.A_SOURCE_FILE_RECEIVED sfr
JOIN CT_MRDS.A_SOURCE_FILE_CONFIG sfc ON sfr.A_SOURCE_FILE_CONFIG_KEY = sfc.A_SOURCE_FILE_CONFIG_KEY
WHERE sfc.A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Backup configuration before deletion
CREATE TABLE BACKUP_SOURCE_FILE_CONFIG AS
SELECT * FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY';

-- Backup date formats before deletion
CREATE TABLE BACKUP_COLUMN_DATE_FORMAT AS
SELECT cdf.*
FROM CT_MRDS.A_COLUMN_DATE_FORMAT cdf
WHERE cdf.TEMPLATE_TABLE_NAME IN (
    SELECT DISTINCT TEMPLATE_TABLE_NAME 
    FROM CT_MRDS.A_SOURCE_FILE_CONFIG 
    WHERE A_SOURCE_KEY = 'YOUR_SOURCE_KEY'
    AND TEMPLATE_TABLE_NAME IS NOT NULL
);
```


## Support


```sql
-- Get complete FILE_MANAGER package documentation
SELECT CT_MRDS.GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;
```

**Note**: For complete formatted output, use proper SQL*Plus settings:
```sql
SET PAGESIZE 0
SET LINESIZE 32000
SET LONG 1000000
SELECT CT_MRDS.GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS') FROM DUAL;
```