This commit is contained in:
Grzegorz Michalski
2026-02-02 10:59:29 +01:00
commit ecd833f682
679 changed files with 122717 additions and 0 deletions

974
.github/copilot-instructions.md vendored Normal file
View File

@@ -0,0 +1,974 @@
# GitHub Copilot Instructions - Oracle Database File Management System
## Critical Development Workflow
### **CODE-FIRST DEVELOPMENT PRINCIPLE**
**MANDATORY**: Always prepare code changes in files BEFORE deploying to database.
**Correct Workflow:**
1. **ANALYZE** current code structure and identify required changes
2. **UPDATE STRUCTURE DEFINITIONS** in source files (.sql table/view/index definitions)
3. **PREPARE MIGRATION SCRIPTS** (ALTER TABLE, CREATE INDEX, etc.) for existing databases
4. **MODIFY PACKAGE CODE** (.pkb, .pks) to utilize new structures
5. **VALIDATE** syntax and logic in all code files
6. **TEST** changes through file deployment scripts
7. **DEPLOY** migration scripts first, then updated packages to database
**NEVER:**
- Execute DDL/DML directly on database without corresponding file updates
- Create database objects without updating source control files
- Modify database structures ad-hoc through SQL commands
- Skip creation of migration scripts for existing environments
**Two-Tier File Management:**
- **Structure Definitions** - Complete CREATE statements in `database/{schema}/tables/`
- **Migration Scripts** - Incremental ALTER statements in `database/migrations/`
**File-First Approach Benefits:**
- Version control integrity maintained
- Reproducible deployments across environments
- Code review capability for all changes
- Rollback capability through source control
- Team collaboration on structured changes
- Clear separation between baseline and incremental changes
---
## Project Architecture Overview
This Oracle database system provides enterprise-grade file processing and data export capabilities for Oracle Cloud Infrastructure (OCI). The architecture centers around multi-schema data pipelines with bucket-based storage management and secure cloud wrapper functionality.
### Dual System Architecture
- **File Management System** (Primary) - Enterprise data export and processing capabilities
- **Cloud Wrapper System** (Foundation) - Secure DBMS_CLOUD access wrapper for controlled cloud operations
### Core Components
**Primary Packages:**
- `CT_MRDS.FILE_MANAGER` - Main file processing and validation orchestration package (AUTHID CURRENT_USER)
- `CT_MRDS.DATA_EXPORTER` - Data export operations with Hive-style partitioning support (separated from FILE_MANAGER)
- `CT_MRDS.ENV_MANAGER` - Environment configuration and error management
- `CT_MRDS.FILE_ARCHIVER` - Data archiving and archival operations
- `ODS.FILE_MANAGER_ODS` - Wrapper package with DEFINER rights for external table creation (RECOMMENDED)
- `MRDS_LOADER.cloud_wrapper` - Secure wrapper around DBMS_CLOUD functions
- `MRDS_LOADER.DATA_REPLICATOR` - Cloud data replication and export operations
**Package Architecture Recommendation:**
- **RECOMMENDED**: Use `ODS.FILE_MANAGER_ODS` (AUTHID DEFINER) - works from any user context
- **ALTERNATIVE**: Use `CT_MRDS.FILE_MANAGER` (AUTHID CURRENT_USER) - requires ODS user context for CREATE_EXTERNAL_TABLE
- **DATA_EXPORTER**: Export procedures moved from FILE_MANAGER to dedicated DATA_EXPORTER package for better modularity
**Schema Architecture:**
- **CT_MRDS** - Core file management and processing schema
- **CT_ODS** - Operational Data Store with load history tracking
- **ODS** - External table creation schema with DEFINER privileges
- **OU_TOP** - Business data schema (e.g., AGGREGATED_ALLOTMENT table)
- **MRDS_LOADER** - Cloud operations schema with DBMS_CLOUD access and package ownership
- **CT_ET_TEMPLATES** - Template table definitions
- **ADMIN** - Database administrator with full privileges (REQUIRED for schema creation)
- **MICHALZ** - Standard development user with execute privileges on cloud_wrapper
### Critical Architecture Pattern
**Privilege Separation Model**: MRDS_LOADER owns cloud_wrapper package with DBMS_CLOUD access, other users get EXECUTE grants for controlled cloud operations.
### File Processing Configuration
**Core Configuration Procedures:**
- `FILE_MANAGER.ADD_SOURCE` - Register new source systems (e.g., 'C2D', 'LM', 'TOP')
- `FILE_MANAGER.ADD_SOURCE_FILE_CONFIG` - Configure file processing rules and naming patterns
- `FILE_MANAGER.ADD_COLUMN_DATE_FORMAT` - Set up date format handling for specific columns
- `FILE_MANAGER.DELETE_SOURCE_CASCADE` - Safe removal of source systems with cascade delete
- `FILE_MANAGER.ANALYZE_VALIDATION_ERRORS` - Analyze file validation failures with detailed reporting
**Official Path Patterns (MANDATORY):**
- **INBOX**: `'INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/'` (3-level path)
- **ODS**: `'ODS/{SOURCE}/{TABLE_NAME}/'` (2-level path, no SOURCE_FILE_ID)
- **ARCHIVE**: `'ARCHIVE/{SOURCE}/{TABLE_NAME}/'` (2-level path, no SOURCE_FILE_ID)
**File Processing Workflow:**
- `FILE_MANAGER.PROCESS_SOURCE_FILE` - Main umbrella procedure (6-step validation workflow)
- Status tracking: RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED
- Automatic file movement: INBOX → ODS/DATA → ARCHIVE (with Hive-style partitioning)
## Data Export System
### Daily Data Flow Workflow
**1. Data Ingestion (INBOX)**
Daily data files arrive in the INBOX bucket area where they undergo validation:
```sql
-- File validation and processing (umbrella procedure with 6 steps)
FILE_MANAGER.PROCESS_SOURCE_FILE(
pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS/data_file.csv'
);
```
**PROCESS_SOURCE_FILE Workflow (6 Steps):**
1. **REGISTER_SOURCE_FILE_RECEIVED** - Register file and extract metadata
2. **CREATE_EXTERNAL_TABLE** - Create temporary external table for validation
3. **VALIDATE_SOURCE_FILE_RECEIVED** - Comprehensive data validation (column count, data types, business rules)
4. **DROP_EXTERNAL_TABLE** - Clean up temporary external table
5. **MOVE_FILE** - Relocate from INBOX to ODS bucket
6. **SET_SOURCE_FILE_RECEIVED_STATUS** - Update status to 'READY_FOR_INGESTION'
**Status Progression:** RECEIVED → VALIDATED → READY_FOR_INGESTION → INGESTED → ARCHIVED
- Files are validated for structure, format, and business rules using template tables
- Successfully validated files are moved to ODS/DATA bucket area and prepared for Airflow+DBT processing
- Failed validation results in error logging with detailed analysis via `ANALYZE_VALIDATION_ERRORS()` function
- Comprehensive error handling for column mismatches, file access issues, and configuration problems
**2. Data Processing (ODS/DATA)**
Validated files are processed and stored in the operational data area for further operations.
**3. Legacy Data Migration (DATA_EXPORTER)**
The DATA_EXPORTER package handles migration of data from legacy systems with three export scenarios:
**Scenario 1: Complete DATA Export (CSV)**
- All data exported to DATA bucket area in CSV format
- No implicit partitioning applied
- Used for simple data migrations
**Scenario 2: Split Export (DATA + ARCHIVE)**
- Partial data to DATA bucket area (CSV format)
- Partial data to ARCHIVE bucket area (Parquet with Hive-style partitioning)
- Requires two separate export operations
**Scenario 3: Complete ARCHIVE Export (Parquet)**
- All data exported to ARCHIVE bucket area
- Parquet format with Hive-style partitioning (PARTITION_YEAR=/PARTITION_MONTH=)
- Used for long-term storage and analytics
### Bucket Area Mapping
The system uses logical bucket areas that map to physical OCI buckets via `GET_BUCKET_URI()`:
- `'INBOX'` → gvInboxBucketUri (incoming data validation)
- `'ODS'` → gvDataBucketUri (operational data processing)
- `'DATA'` → gvDataBucketUri (processed data - CSV format)
- `'ARCHIVE'` → gvArchiveBucketUri (archived data - Parquet with partitioning)
### Export Procedures
**Scenario 1: Complete DATA Export (CSV Format)**
```sql
-- Export all data to DATA bucket area in CSV format without partitioning
FILE_MANAGER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY',
pBucketArea => 'DATA',
pFolderName => 'legacy_migration',
pMinDate => DATE '2024-01-01',
pMaxDate => SYSDATE
);
```
**Scenario 2: Split Export (DATA CSV + ARCHIVE Parquet)**
```sql
-- Part 1: Export recent data to DATA bucket (CSV format)
FILE_MANAGER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY',
pBucketArea => 'DATA',
pFolderName => 'current_data',
pMinDate => DATE '2024-01-01',
pMaxDate => SYSDATE
);
-- Part 2: Export historical data to ARCHIVE bucket (Parquet with partitioning)
DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY',
pBucketArea => 'ARCHIVE',
pFolderName => 'historical_data',
pMinDate => DATE '2020-01-01',
pMaxDate => DATE '2023-12-31'
);
```
**Scenario 3: Complete ARCHIVE Export (Parquet with Partitioning)**
```sql
-- Export all data to ARCHIVE bucket with Hive-style partitioning
DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY',
pBucketArea => 'ARCHIVE',
pFolderName => 'complete_export',
pMinDate => DATE '2020-01-01',
pMaxDate => SYSDATE
);
-- Creates: complete_export/PARTITION_YEAR=2024/PARTITION_MONTH=01/*.parquet
```
**Daily Processing Workflow**
```sql
-- Standard daily file processing procedure
FILE_MANAGER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'CT_MRDS',
pTableName => 'MY_TABLE',
pBucketArea => 'DATA',
pFolderName => 'daily_processing'
);
```
## Critical Implementation Patterns
### Error Handling Architecture
The system uses centralized error management through `ENV_MANAGER`:
**CRITICAL**: All error handling modifications must be implemented in source files first.
```sql
-- CORRECT: Use constant concatenation to avoid duplication
-- UPDATE FILE: database/CT_MRDS/packages/ENV_MANAGER.sql
FUNCTION GET_BUCKET_URI(pBucketName VARCHAR2) RETURN VARCHAR2 IS
BEGIN
CASE pBucketName
WHEN 'INBOX' THEN RETURN ENV_MANAGER.gvInboxBucketUri;
-- ... other cases
ELSE
RAISE_APPLICATION_ERROR(ENV_MANAGER.CODE_INVALID_BUCKET_AREA,
ENV_MANAGER.MSG_INVALID_BUCKET_AREA || ' Provided: ''' || pBucketName || '''');
END CASE;
END;
-- WRONG: Never modify global MSG_ variables (causes duplication in same session)
-- ENV_MANAGER.MSG_INVALID_BUCKET_AREA := ENV_MANAGER.MSG_INVALID_BUCKET_AREA || '...';
-- Structured exception handling with local variables
EXCEPTION
WHEN ENV_MANAGER.ERR_TABLE_NOT_EXISTS THEN
vgMsgTmp := ENV_MANAGER.MSG_TABLE_NOT_EXISTS || ': ' || vTableName;
ENV_MANAGER.LOG_PROCESS_EVENT(vgMsgTmp, 'ERROR', vParameters);
RAISE_APPLICATION_ERROR(ENV_MANAGER.CODE_TABLE_NOT_EXISTS, vgMsgTmp);
```
**Error Message Pattern**: Always use `MSG_*` constants for fresh messages, never modify global variables.
### AUTHID Patterns
- `AUTHID CURRENT_USER` - Functions execute with caller's privileges (FILE_MANAGER)
- `AUTHID DEFINER` - Objects created in package owner schema (FILE_MANAGER_ODS)
### Date Format Configuration System
**Three-Tier Hierarchical Date Format Resolution** (Confirmed through GET_DATE_FORMAT function analysis):
1. **Column-Specific Configuration** - First priority lookup in `A_COLUMN_DATE_FORMAT` table:
```sql
SELECT DATE_FORMAT FROM A_COLUMN_DATE_FORMAT
WHERE TEMPLATE_TABLE_NAME = 'CT_ET_TEMPLATES.TABLE_NAME'
AND COLUMN_NAME = 'SPECIFIC_COLUMN'
```
2. **DEFAULT Column Fallback** - Second priority using special DEFAULT entry:
```sql
SELECT DATE_FORMAT FROM A_COLUMN_DATE_FORMAT
WHERE TEMPLATE_TABLE_NAME = 'CT_ET_TEMPLATES.TABLE_NAME'
AND COLUMN_NAME = 'DEFAULT'
```
3. **Global Fallback** - Final fallback to ENV_MANAGER global variable:
```sql
-- Falls back to ENV_MANAGER.gvDefaultDateFormat ('DD/MM/YYYY HH24:MI:SS')
-- Configured via A_FILE_MANAGER_CONFIG table with CONFIG_VARIABLE = 'DefaultDateFormat'
```
**Configuration Examples:**
```sql
-- Column-specific date format
CALL FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pColumnName => 'SNAPSHOT_DATE',
pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);
-- DEFAULT fallback for entire template table
CALL FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pColumnName => 'DEFAULT',
pDateFormat => 'MM/DD/YYYY'
);
-- Global configuration (via A_FILE_MANAGER_CONFIG)
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'DefaultDateFormat', 'DD-MON-YYYY HH24:MI:SS');
```
### Date Filtering Integration
Export procedures integrate with `CT_ODS.A_LOAD_HISTORY` for temporal partitioning:
```sql
-- Date-based partitioning query pattern
SELECT DISTINCT TO_CHAR(L.LOAD_START,'YYYY') AS YR, TO_CHAR(L.LOAD_START,'MM') AS MN
FROM target_table T, CT_ODS.A_LOAD_HISTORY L
WHERE T.A_WORKFLOW_HISTORY_KEY = L.A_WORKFLOW_HISTORY_KEY
AND L.LOAD_START >= :pMinDate AND L.LOAD_START < :pMaxDate
```
### Hive-Style Partitioning Support
**CONFIRMED**: DBMS_CLOUD.EXPORT_DATA **FULLY SUPPORTS** Hive-style partitioning with key=value format:
```
folder/PARTITION_YEAR=2025/PARTITION_MONTH=09/file.parquet
```
Export creates directory structure compatible with big data tools (Spark, Hive, etc.).
### Database Triggers
**A_SOURCE_FILE_CONFIG_KEY Auto-Generation**:
- `TRG_BI_A_SOURCE_FILE_CONFIG_CHECK` trigger automatically generates primary keys
- Uses `A_SOURCE_FILE_CONFIG_KEY_SEQ.NEXTVAL` sequence
- Validates CONTAINER relationships and business rules
### External Table Management
**Template Tables and External Tables Setup:**
- **Template Tables**: Located in `CT_ET_TEMPLATES` schema, define structure for external tables
- **CRITICAL**: Template tables can ONLY be created in `CT_ET_TEMPLATES` schema by ADMIN or CT_ET_TEMPLATES user
- **MRDS_LOADER Limitation**: Cannot create template tables - must use existing ones or request ADMIN to create them
- **External Table Creation**: Use `ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` (recommended) or `CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE`
- **Storage Locations**: INBOX (incoming), ODS (operational), ARCHIVE (historical with partitioning)
- **Validation Features**: Automatic CSV column validation, excess column detection, detailed error reporting
**MRDS_LOADER External Table Creation Pattern:**
- **RECOMMENDED APPROACH**: Use MRDS_LOADER connection with `ODS.FILE_MANAGER_ODS` package to create external tables in ODS schema
- **Required Prerequisites**:
- Template table must exist in CT_ET_TEMPLATES (created by ADMIN)
- MRDS_LOADER must have comprehensive privileges to CT_MRDS schema objects
- ODS schema must have access to CT_MRDS.ENV_MANAGER for logging operations
- **Privilege Requirements for MRDS_LOADER**:
```sql
-- CT_MRDS schema access (essential for FILE_MANAGER_ODS operations)
GRANT SELECT, INSERT, UPDATE, DELETE ON CT_MRDS.A_PROCESS_LOG TO MRDS_LOADER;
GRANT EXECUTE ON CT_MRDS.ENV_MANAGER TO MRDS_LOADER;
GRANT EXECUTE ON ODS.FILE_MANAGER_ODS TO MRDS_LOADER;
```
- **Privilege Requirements for ODS Schema**:
```sql
-- Required for FILE_MANAGER_ODS to access logging and configuration
GRANT SELECT, INSERT, UPDATE, DELETE ON CT_MRDS.A_PROCESS_LOG TO ODS;
GRANT EXECUTE ON CT_MRDS.ENV_MANAGER TO ODS;
```
**Configuration Examples:**
```sql
-- Register source system
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(pSourceKey => 'C2D', pSourceName => 'Central Bank Data');
-- Configure file processing
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'C2D', pSourceFileType => 'INPUT', pSourceFileId => 'UC_DISSEM',
pSourceFileDesc => 'UC DISSEM Metadata', pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
pTableId => 'A_UC_DISSEM_METADATA_LOADS', pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS'
);
-- Configure date formats
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pColumnName => 'SNAPSHOT_DATE', pDateFormat => 'YYYY-MM-DD HH24:MI:SS'
);
```
**Complete External Table Creation Example (STANDING_FACILITIES):**
```sql
-- Step 1: Create template table (as ADMIN)
CREATE TABLE CT_ET_TEMPLATES.LM_STANDING_FACILITIES (
A_KEY NUMBER NOT NULL,
REV_NUMBER NUMBER,
REF_DATE DATE,
FREE_TEXT VARCHAR2(1000),
MLF_BS_TOTAL NUMBER,
DF_BS_TOTAL NUMBER,
MLF_SF_TOTAL NUMBER,
DF_SF_TOTAL NUMBER
);
-- Step 2: Configure date format (as CT_MRDS or via MRDS_LOADER with privileges)
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pColumnName => 'REF_DATE',
pDateFormat => 'DD/MM/YYYY'
);
-- Step 3: Create external tables using MRDS_LOADER connection
-- (Requires all prerequisite privileges for MRDS_LOADER and ODS schemas)
-- Following OFFICIAL PATH PATTERNS
-- INBOX table (CSV format for incoming files) - 3-level path
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'INBOX/LM/STANDING_FACILITIES/STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
-- ODS table (CSV format for operational data) - 2-level path
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'ODS/LM/STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
-- ARCHIVE table (Parquet format for historical data) - 2-level path
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'ARCHIVE/LM/STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
-- Result: Three external tables created in ODS schema:
-- ODS.LM_STANDING_FACILITIES_INBOX (CSV)
-- ODS.LM_STANDING_FACILITIES_ODS (CSV)
-- ODS.LM_STANDING_FACILITIES_ARCHIVE (Parquet)
```
## Development Workflows
### Database Operations Approaches
#### **MCP Server Approach (Primary - Recommended)**
Use MCP SQLcl server for most database operations:
```sql
-- Connect to specific schema
mcp_sqlcl_connect("ADMIN@ggmichalski_high")
mcp_sqlcl_connect("CT_MRDS@ggmichalski_high")
-- Execute SQL directly
mcp_sqlcl_run-sql("SELECT USER FROM DUAL")
mcp_sqlcl_run-sql("CREATE OR REPLACE PACKAGE...")
-- List available connections
mcp_sqlcl_list-connections()
```
#### **Traditional SQLcl Approach (Secondary)**
Use direct SQLcl commands for complex scripts:
```powershell
# Single script execution pattern
Get-Content "path/to/script.sql" | sql "USERNAME/PASSWORD@SERVICE"
```
### Package Deployment
```powershell
# Core packages deployment sequence using source DDL files
Get-Content "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\packages\ENV_MANAGER.sql" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high"
Get-Content "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\packages\FILE_MANAGER.sql" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high"
Get-Content "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\ODS\packages\FILE_MANAGER_ODS.sql" | sql "ODS/Cloudpass#34@ggmichalski_high"
# MARS issue deployment using release packages
Get-Content "MARS_Packages\REL01\MARS-846\*.sql" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high"
```
### Archive Creation
```powershell
# Create encrypted archive of core packages from source DDL
7z a -pMojeSuperHaslo -mhe=on FM_arch.7z MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\packages\FILE_MANAGER.sql MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\packages\FILE_ARCHIVER.sql MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\packages\ENV_MANAGER.sql
# Create archive of specific MARS release
7z a -pMojeSuperHaslo -mhe=on MARS_846_arch.7z MARS_Packages\REL01\MARS-846\*.sql
```
### PowerShell Integration
Use the `OracleHelper` module for batch operations:
```powershell
Import-Module ".\powershell_modules\OracleHelper\OracleHelper.psm1" -Force
Invoke-OracleScript -ScriptPath "database\CT_MRDS\packages\FILE_MANAGER.sql" -User "CT_MRDS"
```
### Development File Structure Guidelines
**Source Code Organization:**
```
MARS_Packages/
├── mrds_elt-dev-database/
│ └── mrds_elt-dev-database/
│ └── database/ # SOURCE DDL FILES (baseline definitions)
│ ├── CT_MRDS/
│ │ ├── packages/
│ │ │ ├── ENV_MANAGER.sql # Environment management
│ │ │ ├── FILE_MANAGER.sql # Main file processing
│ │ │ ├── DATA_EXPORTER.sql # Data export operations
│ │ │ └── FILE_ARCHIVER.sql # Archival operations
│ │ ├── tables/
│ │ │ ├── A_SOURCE_FILE_CONFIG.sql # Source configuration (CREATE TABLE)
│ │ │ ├── A_PROCESS_LOG.sql # Process logging (CREATE TABLE)
│ │ │ └── A_FILE_MANAGER_CONFIG.sql # Environment config (CREATE TABLE)
│ │ ├── views/
│ │ │ └── V_SOURCE_FILE_STATUS.sql # View definitions (CREATE VIEW)
│ │ ├── indexes/
│ │ │ └── IDX_SOURCE_FILE_CONFIG.sql # Index definitions (CREATE INDEX)
│ │ └── triggers/
│ │ └── TRG_BI_A_SOURCE_FILE_CONFIG.sql
│ ├── ODS/
│ │ ├── FILE_MANAGER_ODS.pkb # DEFINER wrapper body
│ │ ├── FILE_MANAGER_ODS.pkg # DEFINER wrapper spec
│ │ ├── CREATE_USER.sql # User creation scripts
│ │ └── CREATE_ROLE.sql # Role creation scripts
│ └── CT_ET_TEMPLATES/
│ └── tables/
│ ├── CSDB_DEBT.sql # Template tables
│ ├── CSDB_DEBT_DAILY.sql
│ └── [other_templates].sql
├── REL01/ # RELEASE PACKAGES (deployment scripts)
│ ├── MARS-846/
│ │ ├── 01_MARS_846_install_*.sql
│ │ ├── 02_MARS_846_*.sql
│ │ ├── 91_MARS_846_rollback_*.sql
│ │ └── README.md
│ ├── MARS-1011/
│ └── [other_issues]/
├── REL02/ # NEXT RELEASE
│ ├── MARS-949/
│ ├── MARS-954/
│ └── [other_issues]/
└── migrations/ # MIGRATION SCRIPTS (for existing environments)
├── 001_add_characterset_columns.sql # ALTER TABLE scripts
├── 002_create_config_indexes.sql # New indexes
├── 003_update_default_values.sql # Data updates
└── rollback/
├── 001_rollback_characterset.sql # Rollback scripts
└── 002_rollback_indexes.sql
```
**Mandatory File Updates for Structural Changes:**
**1. Structure Definition Updates:**
- **New Tables**: Create complete `CREATE TABLE` in `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/{schema}/tables/{table_name}.sql`
- **Table Modifications**: Update existing table definition files with new structure
- **New Indexes**: Add `CREATE INDEX` statements in `database/{schema}/indexes/`
- **View Changes**: Update view definitions in `database/{schema}/views/`
**2. Migration Script Creation:**
- **Existing Environment Changes**: Create `ALTER TABLE` scripts in `MARS_Packages/migrations/`
- **Data Migration**: Prepare `INSERT/UPDATE` scripts for configuration changes
- **Rollback Scripts**: Create corresponding rollback scripts in `MARS_Packages/migrations/rollback/`
- **Deployment Order**: Number migration scripts sequentially (001_, 002_, etc.)
**3. Release Package Creation:**
- **MARS Issue Packages**: Create deployment scripts in `MARS_Packages/{RELEASE}/{MARS_ISSUE}/`
- **Installation Scripts**: Numbered install scripts (01_, 02_, etc.)
- **Rollback Scripts**: Numbered rollback scripts (91_, 92_, etc.)
- **Documentation**: Include README.md with deployment instructions
**4. Package Code Updates:**
- **Package Modifications**: Update .pks (spec) and .pkb (body) files in source `database/{schema}/packages/`
- **Wrapper Packages**: Update related wrapper packages if applicable
- **Documentation**: Update confluence/ directory with changes
**Change Management Process:**
```sql
-- 1. PREPARE STRUCTURE DEFINITIONS: Update baseline table definitions
-- File: MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/CT_MRDS/tables/A_SOURCE_FILE_CONFIG.sql
CREATE TABLE A_SOURCE_FILE_CONFIG (
A_SOURCE_FILE_CONFIG_KEY NUMBER NOT NULL,
-- ... existing columns ...
CHARACTERSET VARCHAR2(50) DEFAULT 'WE8MSWIN1252', -- NEW COLUMN
DELIMITER VARCHAR2(10) DEFAULT ',', -- NEW COLUMN
CONSTRAINT PK_A_SOURCE_FILE_CONFIG PRIMARY KEY (A_SOURCE_FILE_CONFIG_KEY)
);
-- 2. PREPARE MIGRATION SCRIPT: For existing environments
-- File: MARS_Packages/migrations/001_add_characterset_columns.sql
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
CHARACTERSET VARCHAR2(50) DEFAULT 'WE8MSWIN1252',
DELIMITER VARCHAR2(10) DEFAULT ','
);
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.CHARACTERSET IS 'File encoding';
-- 3. CREATE RELEASE PACKAGE: For MARS issue deployment
-- File: MARS_Packages/REL02/MARS-XXXX/01_MARS_XXXX_add_characterset_columns.sql
-- (Copy migration script content with proper header and logging)
-- 4. PREPARE ROLLBACK SCRIPT: For emergency rollback
-- File: MARS_Packages/REL02/MARS-XXXX/91_MARS_XXXX_rollback_characterset.sql
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP (CHARACTERSET, DELIMITER);
-- 5. DEPLOY: Execute MARS package scripts
Get-Content "MARS_Packages/REL02/MARS-XXXX/01_MARS_XXXX_add_characterset_columns.sql" | sql "CT_MRDS/password@service"
Get-Content "MARS_Packages/REL02/MARS-XXXX/02_MARS_XXXX_update_packages.sql" | sql "CT_MRDS/password@service"
```
### Cloud Storage Integration
#### Available Object Storage Buckets
- **obucket** - Primary bucket for data operations (contains test files and parquet exports)
- **data** - Data storage bucket
- **archive** - Archived data bucket
- **inbox** - Incoming data bucket
#### OCI CLI Operations
```powershell
# Object Storage operations
oci os object put --bucket-name obucket --file "data.csv" --name "import/data.csv"
oci os object get --bucket-name obucket --name "export/results.parquet" --file "results.parquet"
oci os object list --bucket-name obucket --prefix "folder/"
# Namespace: frtgjxu7zl7c
# Region: eu-frankfurt-1 (Frankfurt)
```
#### Credential Management Pattern
Cloud operations use `DEF_CRED_ARN` credential pointing to Oracle Cloud Storage bucket: `https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/obucket/`
## File Organization Conventions
### Storage Hierarchy
```
OCI Bucket Structure (OFFICIAL PATTERNS):
INBOX/{SOURCE}/{SOURCE_FILE_ID}/{TABLE_NAME}/ -- 3-level (incoming validation)
ODS/{SOURCE}/{TABLE_NAME}/ -- 2-level (operational data)
ARCHIVE/{SOURCE}/{TABLE_NAME}/PARTITION_YEAR={YYYY}/PARTITION_MONTH={MM}/ -- 2-level + partitioning
```
### Documentation Location
- Technical guides → `confluence/` directory
- Package documentation → Auto-generated to `confluence/`
- Setup procedures → `confluence/Tables_setup.md`
#### Documentation Generation Workflow
```powershell
# Generate package documentation using GET_PACKAGE_DOCUMENTATION function
echo "SET PAGESIZE 0`nSET LINESIZE 32000`nSET LONG 1000000`nSELECT GET_PACKAGE_DOCUMENTATION('PACKAGE_NAME', 'SCHEMA_NAME') FROM DUAL;" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high" > confluence/package_name_documentation.md
```
**GET_PACKAGE_DOCUMENTATION Function:**
- **Purpose**: Automatically generates comprehensive markdown documentation from Oracle packages
- **Features**: Extracts procedural metadata, function signatures, parameter information, embedded comments
- **Usage**: `GET_PACKAGE_DOCUMENTATION('FILE_MANAGER', 'CT_MRDS')` returns CLOB with formatted markdown
- **Output**: Complete package documentation including usage examples and return types
## Testing and Validation
### SQL Package Testing Infrastructure
**Standard Package Testing Pattern with @install:**
```sql
-- Master install script pattern (executes sub-scripts using @)
-- Example: install_mars1049.sql
@@01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql
@@02_MARS_1049_install_CT_MRDS_FILE_MANAGER_SPEC.sql
@@03_MARS_1049_install_CT_MRDS_FILE_MANAGER_BODY.sql
@@04_MARS_1049_install_ODS_FILE_MANAGER_ODS_SPEC.sql
@@05_MARS_1049_install_ODS_FILE_MANAGER_ODS_BODY.sql
@@06_MARS_1049_verify_encoding_functionality.sql
@@07_MARS_1049_track_version.sql
-- Usage with PowerShell/SQLcl:
sql 'CT_MRDS/Cloudpass#34@ggmichalski_high' '@install_mars1049.sql'
```
**Package Verification Standards:**
```sql
-- Standard verification script pattern (XX_verify_*.sql)
-- Check compilation status
SELECT object_name, object_type, status
FROM user_objects
WHERE object_name = 'PACKAGE_NAME'
AND object_type IN ('PACKAGE', 'PACKAGE BODY');
-- Verify package version
SELECT SCHEMA.PACKAGE_NAME.GET_VERSION() AS VERSION FROM DUAL;
-- Check build information
SELECT SCHEMA.PACKAGE_NAME.GET_BUILD_INFO() AS BUILD_INFO FROM DUAL;
-- Verify functionality (package-specific tests)
-- Example: Test new encoding functionality
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(..., pEncoding => 'WE8MSWIN1252');
```
**Testing Workflow Integration:**
- **Phase 1**: Structure Changes (`01_*_install_*_TABLE.sql`)
- **Phase 2**: Package Specification (`02_*_install_*_SPEC.sql`)
- **Phase 3**: Package Body (`03_*_install_*_BODY.sql`)
- **Phase 4**: Verification (`04_*_verify_*.sql`)
- **Phase 5**: Version Tracking (`05_*_track_*.sql`)
### Key Test Files
- `MARS_Packages/REL*/MARS-*/install_mars*.sql` - Master installation scripts with @include pattern
- `MARS_Packages/REL*/MARS-*/rollback_mars*.sql` - Complete rollback testing
### Common Operations
```sql
-- Test bucket area validation
SELECT GET_BUCKET_URI('DATA') FROM DUAL; -- Returns: https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/o/
-- Validate export procedures with Hive-style partitioning
BEGIN
DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_WORKFLOW_HISTORY_KEY',
pBucketArea => 'DATA',
pFolderName => 'test_export',
pMinDate => DATE '2025-09-01',
pMaxDate => DATE '2025-09-30'
);
END;
-- Creates files in: test_export/PARTITION_YEAR=2025/PARTITION_MONTH=09/*.parquet
-- List cloud storage files using cloud_wrapper
SELECT object_name FROM TABLE(MRDS_LOADER.cloud_wrapper.list_objects(
credential_name => 'DEF_CRED_ARN',
location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/'
)) WHERE object_name LIKE 'test_export%';
```
## Connection Management
### User Credentials
- **Password**: All users use `Cloudpass#34`
- **Service**: All connect via `ggmichalski_high` service
- **Available Users**: ADMIN, MRDS_LOADER, MICHALZ, CT_MRDS, CT_ODS, ODS, CT_ET_TEMPLATES, OU_TOP
### SQLcl Connection Commands
```sql
-- Connect to database as specific user
connect ADMIN/Cloudpass#34@ggmichalski_high
connect CT_MRDS/Cloudpass#34@ggmichalski_high
connect MRDS_LOADER/Cloudpass#34@ggmichalski_high
-- Connection management
connmgr list -- List all saved connections
connmgr test ODS@ggmichalski_high -- Test specific connection
```
## Critical Implementation Patterns
### MCP Server Integration
The system supports Model Context Protocol (MCP) servers for enhanced database operations:
```sql
-- Use MCP SQLcl tools for database operations
mcp_sqlcl_connect("CT_MRDS@ggmichalski_high")
mcp_sqlcl_run-sql("EXEC FILE_MANAGER.EXPORT_TABLE_DATA(...)")
mcp_pylance_mcp_s_pylanceRunCodeSnippet() -- For Python validation
```
### Configuration Management Pattern
Environment variables are centrally managed via `A_FILE_MANAGER_CONFIG` table.
**MANDATORY**: Update configuration table definition files before database changes.
```sql
-- UPDATE FILE: database/CT_MRDS/tables/A_FILE_MANAGER_CONFIG.sql
-- INCLUDE: Initial data setup with correct camelCase format
CREATE TABLE A_FILE_MANAGER_CONFIG (
ENVIRONMENT_ID VARCHAR2(50) NOT NULL,
CONFIG_VARIABLE VARCHAR2(100) NOT NULL,
CONFIG_VARIABLE_VALUE VARCHAR2(4000),
CONSTRAINT PK_A_FILE_MANAGER_CONFIG PRIMARY KEY (ENVIRONMENT_ID, CONFIG_VARIABLE)
);
-- Configuration variables loaded at package initialization
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('default', 'EnvironmentID', 'dev');
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'Region', 'eu-frankfurt-1');
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'NameSpace', 'frtgjxu7zl7c');
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'InboxBucketName', 'inbox');
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'DataBucketName', 'data');
INSERT INTO A_FILE_MANAGER_CONFIG (ENVIRONMENT_ID, CONFIG_VARIABLE, CONFIG_VARIABLE_VALUE)
VALUES ('dev', 'ArchiveBucketName', 'archive');
```
**CRITICAL**: Variable names are case-sensitive - use exact camelCase format as shown above.
### Parameter Logging Pattern
All procedures use standardized parameter logging:
```sql
-- Standard parameter formatting for logging
vParameters := ENV_MANAGER.FORMAT_PARAMETERS(SYS.ODCIVARCHAR2LIST(
'pSchemaName => '''||nvl(pSchemaName,'NULL')||'''',
'pTableName => '''||nvl(pTableName,'NULL')||''''
));
ENV_MANAGER.LOG_PROCESS_EVENT('Start','INFO', vParameters);
```
### Cloud Wrapper Integration
```sql
-- PIPELINED function pattern (for list_objects)
FUNCTION list_objects(...) RETURN DBMS_CLOUD_TYPES.LIST_OBJECT_RET_TAB PIPELINED IS
BEGIN
FOR rec IN (SELECT * FROM TABLE(DBMS_CLOUD.LIST_OBJECTS(...))) LOOP
PIPE ROW(rec);
END LOOP;
EXCEPTION
WHEN OTHERS THEN
RAISE_APPLICATION_ERROR(-20005, 'Error listing objects: ' || SQLERRM);
END;
```
### Resource Principal Configuration
```sql
-- Enable Resource Principal (execute as ADMIN)
EXEC DBMS_CLOUD_ADMIN.ENABLE_RESOURCE_PRINCIPAL(username => 'CT_MRDS');
EXEC DBMS_CLOUD_ADMIN.ENABLE_RESOURCE_PRINCIPAL(username => 'MRDS_LOADER');
-- Test Resource Principal access
SELECT object_name FROM DBMS_CLOUD.LIST_OBJECTS(
credential_name => 'OCI$RESOURCE_PRINCIPAL',
location_uri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frtgjxu7zl7c/b/data/'
) WHERE ROWNUM <= 5;
```
## Debugging and Troubleshooting
### Package Compilation Validation
```sql
-- Check for compilation errors after changes
SELECT * FROM USER_ERRORS WHERE TYPE = 'PACKAGE' AND NAME = 'FILE_MANAGER';
```
### MCP Pylance Integration
For Python-related Oracle operations, use Pylance MCP tools:
```sql
-- Validate Python code syntax before database integration
mcp_pylance_mcp_s_pylanceSyntaxErrors(code, pythonVersion)
-- Check workspace Python environment
mcp_pylance_mcp_s_pylancePythonEnvironments(workspaceRoot)
```
### Archive and Backup Patterns
```powershell
# Standard backup command for all core packages
7z a -pMojeSuperHaslo -mhe=on FM_arch.7z database\CT_MRDS\packages\FILE_MANAGER.sql database\CT_MRDS\packages\FILE_ARCHIVER.sql database\CT_MRDS\packages\ENV_MANAGER.sql database\ODS\packages\FILE_MANAGER_ODS.sql
```
## Key Development Principles
When working with this system:
### **PRIMARY PRINCIPLE: CODE-FIRST DEVELOPMENT**
1. **ALWAYS update source files BEFORE database changes** - maintain version control integrity
2. **Prepare complete file modifications** before any database deployment
3. **Validate changes in files** before executing on database
4. **Document structural changes** in corresponding .sql files
### **SYSTEM-SPECIFIC PRINCIPLES**
5. **Always use MSG_* constants** for error messages to prevent session-level duplication
6. **Follow the privilege separation model** - MRDS_LOADER owns cloud operations, others get EXECUTE grants
7. **Use standardized parameter logging** with FORMAT_PARAMETERS for all procedures
8. **Test bucket area mappings** before deploying export procedures
9. **Leverage MCP tools** for database operations and Python validation when available
6. **Hive-style partitioning is fully supported** - DBMS_CLOUD.EXPORT_DATA creates PARTITION_YEAR=/PARTITION_MONTH= structure
7. **Use triggers for auto-key generation** - TRG_BI_A_SOURCE_FILE_CONFIG_CHECK handles A_SOURCE_FILE_CONFIG_KEY
8. **Follow configuration sequence** - ADD_SOURCE → ADD_SOURCE_FILE_CONFIG → ADD_COLUMN_DATE_FORMAT
9. **Use ODS.FILE_MANAGER_ODS package** for external table creation (AUTHID DEFINER) over CT_MRDS.FILE_MANAGER (AUTHID CURRENT_USER)
10. **Implement comprehensive validation** - Use PROCESS_SOURCE_FILE for complete 6-step file validation workflow
11. **Utilize ANALYZE_VALIDATION_ERRORS** for detailed file validation error analysis and troubleshooting
12. **Safe configuration removal** - Use DELETE_SOURCE_CASCADE for complete source system cleanup with referential integrity
13. **Template table restriction** - Template tables can ONLY be created in CT_ET_TEMPLATES schema by ADMIN or CT_ET_TEMPLATES user, not by MRDS_LOADER
14. **MRDS_LOADER external table creation** - Use MRDS_LOADER connection with ODS.FILE_MANAGER_ODS package for creating external tables in ODS schema (requires proper privilege setup for both MRDS_LOADER and ODS schemas)
15. **Cross-schema privilege requirements** - When using FILE_MANAGER_ODS package, ensure both MRDS_LOADER and ODS have sufficient privileges to CT_MRDS schema objects (especially A_PROCESS_LOG and ENV_MANAGER)
16. **MANDATORY PATH PATTERNS** - Always use official patterns: INBOX (3-level), ODS (2-level), ARCHIVE (2-level) - no simplified versions allowed
17. **Path pattern compliance** - INBOX requires SOURCE_FILE_ID, while ODS and ARCHIVE omit it for simplified access
### **Common File Modification Scenarios**
**Adding New Columns to Configuration Tables:**
```sql
-- 1. UPDATE STRUCTURE DEFINITION: database/CT_MRDS/tables/A_SOURCE_FILE_CONFIG.sql
-- Modify the complete CREATE TABLE statement to include new columns
CREATE TABLE A_SOURCE_FILE_CONFIG (
A_SOURCE_FILE_CONFIG_KEY NUMBER NOT NULL,
-- ... existing columns ...
NEW_COLUMN VARCHAR2(50) DEFAULT 'default_value', -- ADD THIS LINE
CONSTRAINT PK_A_SOURCE_FILE_CONFIG PRIMARY KEY (A_SOURCE_FILE_CONFIG_KEY)
);
COMMENT ON COLUMN A_SOURCE_FILE_CONFIG.NEW_COLUMN IS 'Description';
-- 2. CREATE MIGRATION SCRIPT: database/migrations/XXX_add_new_column.sql
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
NEW_COLUMN VARCHAR2(50) DEFAULT 'default_value'
);
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.NEW_COLUMN IS 'Description';
-- 3. CREATE ROLLBACK SCRIPT: database/migrations/rollback/XXX_rollback_new_column.sql
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP COLUMN NEW_COLUMN;
-- 4. UPDATE PACKAGE CODE: database/CT_MRDS/packages/FILE_MANAGER.sql
-- Add logic to utilize new configuration column
-- 5. DEPLOY: Execute migration first, then updated packages
```
**Modifying Package Procedures:**
```sql
-- 1. UPDATE PACKAGE DEFINITION: database/CT_MRDS/packages/FILE_MANAGER.sql
-- Modify both .pks (specification) and .pkb (body) sections in the same file
-- 2. UPDATE WRAPPER PACKAGES: database/ODS/packages/FILE_MANAGER_ODS.sql
-- Update related wrapper packages if applicable
-- 3. DEPLOY: Execute updated package files
Get-Content "database/CT_MRDS/packages/FILE_MANAGER.sql" | sql "CT_MRDS/password@service"
Get-Content "database/ODS/packages/FILE_MANAGER_ODS.sql" | sql "ODS/password@service"
-- 4. UPDATE DOCUMENTATION: confluence/ directory
```
**Adding New Template Tables:**
```sql
-- 1. CREATE STRUCTURE DEFINITION: database/CT_ET_TEMPLATES/tables/NEW_TEMPLATE.sql
CREATE TABLE CT_ET_TEMPLATES.NEW_TEMPLATE (
-- Column definitions matching expected CSV structure
COL1 VARCHAR2(50),
COL2 NUMBER,
COL3 DATE
);
-- 2. CREATE MIGRATION SCRIPT: database/migrations/XXX_create_new_template.sql
-- (Same CREATE TABLE statement for deployment)
-- 3. UPDATE CONFIGURATION: Add new source file config entries
-- 4. UPDATE EXTERNAL TABLE SCRIPTS: If needed for this template
```
## Code Style Guidelines
**Text Formatting**:
- **NO EMOJI/ICONS**: Do not use emoji, Unicode symbols, or decorative icons in code, comments, documentation, or console output
- Use plain text with clear, professional formatting
- Prefer descriptive text labels over visual symbols
- Example: Use "ERROR:" instead of "🚨", "LOCATION:" instead of "📍"
**Line Break Formatting**:
- **Use cgBL constant**: Always use the predefined constant `cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);` for line breaks in error messages and logging
- Replace `CHR(10)` with `cgBL` for consistent CRLF formatting across Windows environments
- Exception: Keep `CHR(10)` in REGEXP_COUNT functions where it's required for line counting operations
**Professional Output**:
- Keep all logging, error messages, and documentation strictly text-based
- Use standard ASCII characters and proper formatting
- Maintain enterprise-grade professional appearance
**Oracle DDL File Standards**:
- **MANDATORY**: All Oracle DDL files (.pkg, .pkb, .sql for packages/triggers) must end with forward slash `/`
- **Placement**: Add `/` after final `END;` statement with blank line separation
- **Example**: `END;` followed by blank line, then `/` on separate line
- **Scope**: Applies to all packages (specification and body), triggers, and PL/SQL blocks
- **Purpose**: Ensures proper Oracle SQL*Plus and SQLcl termination of PL/SQL blocks
The system is designed for enterprise-scale data processing with proper logging, error management, and OCI integration.

29
.gitignore vendored Normal file
View File

@@ -0,0 +1,29 @@
# Temporary files and directories
tmp/
# Issues directory
issues/
# IDE and editor files
.idea/
*.swp
*.swo
*~
# OS generated files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
MARS_Packages/REL01/MARS-1056/confluence/
MARS_Packages/REL01/MARS-1056/log/
MARS_Packages/REL01/MARS-1046/confluence/
MARS_Packages/REL01/MARS-1046/log/
MARS_Packages/REL01/MARS-1046/test/
MARS_Packages/REL01/MARS-1096/log/

127
.vscode/settings.json vendored Normal file
View File

@@ -0,0 +1,127 @@
{
"files.associations": {
"*.sql": "oracle-sql",
"*.pkg": "oracle-sql",
"*.pkb": "oracle-sql"
},
"[oracle-sql]": {
"editor.defaultFormatter": "Oracle.sql-developer"
},
"java.home": "C:\\apps\\Amazon Corretto\\jdk17.0.11_9",
"oracle.developer.javaHome": "C:\\apps\\Amazon Corretto\\jdk17.0.11_9",
"java.configuration.runtimes": [
{
"name": "JavaSE-17",
"path": "C:\\apps\\Amazon Corretto\\jdk17.0.11_9"
},
{
"name": "Oracle-JDK",
"path": "C:\\apps\\sqldeveloper\\jdk\\jre"
}
],
"java.import.gradle.java.home": "C:\\apps\\Amazon Corretto\\jdk17.0.11_9",
"oracle.developer.sqlcl.path": "sql",
"oracle.developer.enableSQLCL": true,
"oracle.developer.enableLanguageServer": true,
"oracle.developer.walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski",
"oracle.developer.sqlDeveloperHome": "C:\\apps\\sqldeveloper",
"oracle.developer.enableJDBCDrivers": true,
"oracle.developer.debug.enabled": true,
"oracle.developer.logging.level": "ALL",
"oracle.developer.startup.timeout": 120000,
"oracle.developer.jvm.maxMemory": "2g",
"oracle.developer.jvm.initialMemory": "512m",
"oracle.developer.connectionTimeout": 60000,
"oracle.developer.tnsAdminLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski",
"oracle.developer.oracleHome": "",
"oracle.developer.disableOracleHomeDetection": true,
"oracle.developer.queryTimeout": 300000,
"oracle.developer.useWalletOnly": true,
"oracle.developer.ignoreSystemOracleHome": true,
"oracle.developer.overrideSystemVariables": true,
"oracle.developer.forceWalletConnections": true,
"oracle.developer.bypassTnsLookup": true,
"terminal.integrated.env.windows": {
"TNS_ADMIN": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski",
"ORACLE_HOME": "",
"ORACLE_BASE": "",
"ORA_JAVA_HOME": "C:\\apps\\Amazon Corretto\\jdk17.0.11_9"
},
"oracledevtools.download.otherPlatforms": true,
"oracledevtools.bookmarkFileLocation": "",
"oracle.connectionConfiguration": {
"defaultConnection": "ADMIN@ggmichalski_high",
"connections": [
{
"name": "ADMIN@ggmichalski_high",
"username": "ADMIN",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "MICHALZ@ggmichalski_high",
"username": "MICHALZ",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "MRDS_LOADER@ggmichalski_high",
"username": "MRDS_LOADER",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "CT_MRDS@ggmichalski_high",
"username": "CT_MRDS",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "ODS@ggmichalski_high",
"username": "ODS",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "CT_ODS@ggmichalski_high",
"username": "CT_ODS",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "CT_ET_TEMPLATES@ggmichalski_high",
"username": "CT_ET_TEMPLATES",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "OU_TOP@ggmichalski_high",
"username": "OU_TOP",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "OU_LM@ggmichalski_high",
"username": "OU_LM",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
},
{
"name": "OU_MRR@ggmichalski_high",
"username": "OU_MRR",
"password": "Cloudpass#34",
"connectionString": "ggmichalski_high",
"walletLocation": "c:\\_git\\OracleAI\\oracledb1\\Wallet_ggmichalski"
}
]
}
}

View File

@@ -0,0 +1,13 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Installing WORKFLOW_MANAGER Package Specification
prompt MARS-1011: Update WORKFLOW_MANAGER v1.7.1
prompt ========================================
@@WORKFLOW_MANAGER_SPEC.sql
prompt ========================================
prompt WORKFLOW_MANAGER Package Specification installed successfully
prompt ========================================

View File

@@ -0,0 +1,14 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Installing WORKFLOW_MANAGER Package Body
prompt MARS-1011: Update WORKFLOW_MANAGER v1.7.1
prompt Adding SERVICE_NAME='ODS' to INIT_TASK function
prompt ========================================
@@WORKFLOW_MANAGER_BODY.sql
prompt ========================================
prompt WORKFLOW_MANAGER Package Body installed successfully
prompt ========================================

View File

@@ -0,0 +1,24 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Verifying WORKFLOW_MANAGER Package Installation
prompt MARS-1011: Update WORKFLOW_MANAGER v1.7.1
prompt ========================================
prompt Checking package compilation status...
SELECT object_name, object_type, status
FROM user_objects
WHERE object_name = 'WORKFLOW_MANAGER'
AND object_type IN ('PACKAGE', 'PACKAGE BODY');
prompt Checking package version...
SELECT CT_MRDS.WORKFLOW_MANAGER.GET_VERSION() AS VERSION FROM DUAL;
prompt Checking build info...
SELECT CT_MRDS.WORKFLOW_MANAGER.GET_BUILD_INFO() AS BUILD_INFO FROM DUAL;
prompt ========================================
prompt WORKFLOW_MANAGER Package verification completed
prompt Expected version: 1.7.1
prompt ========================================

View File

@@ -0,0 +1,32 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Tracking WORKFLOW_MANAGER Package Version
prompt MARS-1011: Record version 1.7.1 in tracking system
prompt ========================================
BEGIN
DBMS_OUTPUT.PUT_LINE('Tracking WORKFLOW_MANAGER version...');
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
pPackageOwner => 'CT_MRDS',
pPackageName => 'WORKFLOW_MANAGER',
pPackageVersion => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_VERSION,
pPackageBuildDate => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_BUILD_DATE,
pPackageAuthor => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_AUTHOR
);
DBMS_OUTPUT.PUT_LINE('WORKFLOW_MANAGER version tracked successfully.');
-- Display current version info
DBMS_OUTPUT.PUT_LINE('Current Version: ' || CT_MRDS.WORKFLOW_MANAGER.GET_VERSION());
DBMS_OUTPUT.PUT_LINE('Build Info: ' || CT_MRDS.WORKFLOW_MANAGER.GET_BUILD_INFO());
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('ERROR tracking WORKFLOW_MANAGER: ' || SQLERRM);
RAISE;
END;
/
prompt ========================================
prompt WORKFLOW_MANAGER Package version tracking completed
prompt ========================================

View File

@@ -0,0 +1,201 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Rollback: WORKFLOW_MANAGER Package Body
prompt MARS-1011: Rollback to version 1.7.0 (Remove SERVICE_NAME from INIT_TASK)
prompt ========================================
-- Create package body with INIT_TASK without SERVICE_NAME (rollback to v1.7.0 behavior)
CREATE OR REPLACE PACKAGE BODY CT_MRDS.WORKFLOW_MANAGER
IS
FUNCTION INIT_WORKFLOW(pServiceName IN VARCHAR2, pWorkflowRunId IN VARCHAR2, pWorkflowName in VARCHAR2)
RETURN NUMBER
IS
vWorkflowHistoryKey NUMBER;
BEGIN
vWorkflowHistoryKey := A_WORKFLOW_HISTORY_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY (SERVICE_NAME, A_WORKFLOW_HISTORY_KEY, ORCHESTRATION_RUN_ID,
WORKFLOW_NAME, WORKFLOW_START, WORKFLOW_SUCCESSFUL)
VALUES (pServiceName, vWorkflowHistoryKey, pWorkflowRunId,
pWorkflowName, SYSTIMESTAMP, vpRunningStatus);
return vWorkflowHistoryKey;
END INIT_WORKFLOW;
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pWorkflowStatus IN VARCHAR2)
IS
BEGIN
FINALISE_WORKFLOW(pWorkflowHistoryKey, NULL, pWorkflowStatus);
END;
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pServiceName IN VARCHAR2, pWorkflowStatus IN VARCHAR2)
IS
BEGIN
UPDATE CT_MRDS.A_WORKFLOW_HISTORY SET WORKFLOW_SUCCESSFUL = pWorkflowStatus,
WORKFLOW_END = SYSTIMESTAMP
WHERE A_WORKFLOW_HISTORY_KEY = pWorkflowHistoryKey
AND SERVICE_NAME = NVL(pServiceName,SERVICE_NAME);
END FINALISE_WORKFLOW;
-- ROLLBACK: INIT_TASK without SERVICE_NAME (version 1.7.0 behavior)
FUNCTION INIT_TASK(pTaskRunId IN VARCHAR2, pTaskName in VARCHAR2, pWorkflowHistoryKey IN NUMBER)
RETURN NUMBER
IS
vTaskHistoryKey NUMBER;
BEGIN
vTaskHistoryKey := A_TASK_HISTORY_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY (A_TASK_HISTORY_KEY, TASK_RUN_ID, A_WORKFLOW_HISTORY_KEY,
TASK_NAME, TASK_START, TASK_SUCCESSFUL)
VALUES (vTaskHistoryKey, pTaskRunId, pWorkflowHistoryKey,
pTaskName, SYSTIMESTAMP, vpRunningStatus);
return vTaskHistoryKey;
END INIT_TASK;
PROCEDURE FINALISE_TASK(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2)
IS
BEGIN
UPDATE CT_MRDS.A_TASK_HISTORY SET TASK_SUCCESSFUL = pTaskStatus,
TASK_END = SYSTIMESTAMP
WHERE A_TASK_HISTORY_KEY = pTaskHistoryKey;
END FINALISE_TASK;
-- Other functions remain the same as current version
FUNCTION STORE_TASK_SOURCE_INFO_INTERNAL(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER
IS
vTaskHistorySourceKey NUMBER;
BEGIN
vTaskHistorySourceKey := A_TASK_HISTORY_SOURCE_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY_SOURCE (A_TASK_HISTORY_SOURCE_KEY, A_TASK_HISTORY_KEY,
SOURCE_NAME, ROW_COUNT)
VALUES (vTaskHistorySourceKey, pTaskHistoryKey,
pSourceName, pNumRows);
RETURN vTaskHistorySourceKey;
END;
FUNCTION STORE_TASK_SOURCE_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER
IS
vTaskHistorySourceKey NUMBER;
BEGIN
vTaskHistorySourceKey := STORE_TASK_SOURCE_INFO_INTERNAL(pTaskHistoryKey, pSourceName, pNumRows);
COMMIT;
RETURN vTaskHistorySourceKey;
END;
FUNCTION STORE_TASK_TARGET_INFO_INTERNAL(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
vTaskHistoryTargetKey := A_TASK_HISTORY_TARGET_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY_TARGET (A_TASK_HISTORY_TARGET_KEY, A_TASK_HISTORY_KEY,
TARGET_NAME, ROW_COUNT_APPLIED, ROW_COUNT_REJECTED,
LOAD_SUCCESSFUL, SERVICE_NAME)
VALUES (vTaskHistoryTargetKey, pTaskHistoryKey,
pSourceName, pNumRowsApplied, pNumRowsRejected,
pLoadSuccessful, pServiceName);
RETURN vTaskHistoryTargetKey;
END;
FUNCTION STORE_TASK_TARGET_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
vTaskHistoryTargetKey := STORE_TASK_TARGET_INFO_INTERNAL(
pTaskHistoryKey, pSourceName, pNumRowsApplied,
pNumRowsRejected, pLoadSuccessful, pServiceName
);
COMMIT;
RETURN vTaskHistoryTargetKey;
END;
FUNCTION FINALISE_TASK_WITH_TARGET_INFO(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2,
pTargetName IN VARCHAR2, pNumRowsApplied IN NUMBER,
pNumRowsRejected IN NUMBER, pLoadSuccessful IN VARCHAR2,
pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
vTaskHistoryTargetKey := STORE_TASK_TARGET_INFO_INTERNAL(
pTaskHistoryKey => pTaskHistoryKey,
pSourceName => pTargetName,
pNumRowsApplied => pNumRowsApplied,
pNumRowsRejected => pNumRowsRejected,
pLoadSuccessful => pLoadSuccessful,
pServiceName => pServiceName
);
FINALISE_TASK(pTaskHistoryKey, pTaskStatus);
COMMIT;
RETURN vTaskHistoryTargetKey;
END;
PROCEDURE SET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
,pValue IN VARCHAR2
) IS
BEGIN
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY (A_WORKFLOW_HISTORY_KEY, SERVICE_NAME, PROPERTY, VALUE)
VALUES (pWorkflowHistoryKey, pServiceName, pProperty, pValue);
END;
FUNCTION GET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
) RETURN VARCHAR2
IS
vValue CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY.VALUE%TYPE;
BEGIN
SELECT VALUE
INTO vValue
FROM CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY
WHERE A_WORKFLOW_HISTORY_KEY = pWorkflowHistoryKey
AND SERVICE_NAME = pServiceName
AND PROPERTY = pProperty;
RETURN vValue;
END;
FUNCTION GET_VERSION RETURN VARCHAR2 IS
BEGIN
RETURN PACKAGE_VERSION;
END GET_VERSION;
FUNCTION GET_BUILD_INFO RETURN VARCHAR2 IS
BEGIN
RETURN ENV_MANAGER.GET_PACKAGE_VERSION_INFO(
pPackageName => 'WORKFLOW_MANAGER',
pVersion => PACKAGE_VERSION,
pBuildDate => PACKAGE_BUILD_DATE,
pAuthor => PACKAGE_AUTHOR
);
END GET_BUILD_INFO;
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2 IS
BEGIN
RETURN ENV_MANAGER.FORMAT_VERSION_HISTORY(
pPackageName => 'WORKFLOW_MANAGER',
pVersionHistory => VERSION_HISTORY
);
END GET_VERSION_HISTORY;
END WORKFLOW_MANAGER;
/
prompt ========================================
prompt WORKFLOW_MANAGER Package Body rolled back to v1.7.0
prompt INIT_TASK function no longer includes SERVICE_NAME
prompt ========================================

View File

@@ -0,0 +1,78 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Rollback: WORKFLOW_MANAGER Package Specification
prompt MARS-1011: Rollback to version 1.7.0 (2025-10-31)
prompt ========================================
-- Rollback WORKFLOW_MANAGER specification to version 1.7.0
CREATE OR REPLACE PACKAGE CT_MRDS.WORKFLOW_MANAGER
IS
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '1.7.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-31 16:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'1.7.0 (2025-10-31): Added FINALISE_TASK_WITH_TARGET_INFO function for atomic task completion with target info' || CHR(13)||CHR(10) ||
'1.6.0 (2025-10-31): Converted STORE_TASK_SOURCE_INFO and STORE_TASK_TARGET_INFO from procedures to functions returning sequence keys' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.2.0 (2025-10-05): Added workflow property management (SET/GET_WORKFLOW_PROPERTY)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-10): Initial release with workflow and task lifecycle management';
-- Line break constant for consistent CRLF formatting across Windows environments
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
FUNCTION INIT_WORKFLOW(pServiceName IN VARCHAR2, pWorkflowRunId IN VARCHAR2, pWorkflowName in VARCHAR2)
RETURN NUMBER;
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pWorkflowStatus IN VARCHAR2);
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pServiceName IN VARCHAR2, pWorkflowStatus IN VARCHAR2);
FUNCTION INIT_TASK(pTaskRunId IN VARCHAR2, pTaskName in VARCHAR2, pWorkflowHistoryKey IN NUMBER)
RETURN NUMBER;
PROCEDURE FINALISE_TASK(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2);
FUNCTION STORE_TASK_SOURCE_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER;
FUNCTION STORE_TASK_TARGET_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER;
FUNCTION FINALISE_TASK_WITH_TARGET_INFO(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2,
pTargetName IN VARCHAR2, pNumRowsApplied IN NUMBER,
pNumRowsRejected IN NUMBER, pLoadSuccessful IN VARCHAR2,
pServiceName IN VARCHAR2)
RETURN NUMBER;
vpRunningStatus CT_MRDS.A_WORKFLOW_HISTORY.WORKFLOW_SUCCESSFUL%TYPE := 'R';
PROCEDURE SET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
,pValue IN VARCHAR2
);
FUNCTION GET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2)
RETURN VARCHAR2;
FUNCTION GET_VERSION RETURN VARCHAR2;
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END WORKFLOW_MANAGER;
/
prompt ========================================
prompt WORKFLOW_MANAGER Package Specification rolled back to v1.7.0
prompt ========================================

View File

@@ -0,0 +1,36 @@
SET SERVEROUTPUT ON;
SET ECHO ON;
prompt ========================================
prompt Tracking WORKFLOW_MANAGER Package Version After Rollback
prompt MARS-1011: Record rollback to version 1.7.0
prompt ========================================
BEGIN
DBMS_OUTPUT.PUT_LINE('Tracking WORKFLOW_MANAGER rollback version...');
-- Track the rolled-back version (1.7.0)
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
pPackageOwner => 'CT_MRDS',
pPackageName => 'WORKFLOW_MANAGER',
pPackageVersion => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_VERSION,
pPackageBuildDate => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_BUILD_DATE,
pPackageAuthor => CT_MRDS.WORKFLOW_MANAGER.PACKAGE_AUTHOR
);
DBMS_OUTPUT.PUT_LINE('WORKFLOW_MANAGER rollback version tracked successfully.');
-- Display rollback version info
DBMS_OUTPUT.PUT_LINE('Rollback Version: ' || CT_MRDS.WORKFLOW_MANAGER.GET_VERSION());
DBMS_OUTPUT.PUT_LINE('Build Info: ' || CT_MRDS.WORKFLOW_MANAGER.GET_BUILD_INFO());
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('ERROR tracking WORKFLOW_MANAGER rollback: ' || SQLERRM);
RAISE;
END;
/
prompt ========================================
prompt WORKFLOW_MANAGER Package rollback version tracking completed
prompt Rolled back to version 1.7.0 (2025-10-31)
prompt ========================================

View File

@@ -0,0 +1,54 @@
# MARS-1011: WORKFLOW_MANAGER Package Update v1.7.1
## Description
Updates WORKFLOW_MANAGER package to version 1.7.1, adding automatic SERVICE_NAME='ODS' assignment in INIT_TASK function.
## Changes
- Modified `INIT_TASK` function to automatically set `SERVICE_NAME='ODS'` when inserting into `CT_MRDS.A_TASK_HISTORY`
- Updated package version from 1.7.0 to 1.7.1
- Updated build date to 2025-11-18
## Installation Files
1. `01_MARS_1011_install_CT_MRDS_WORKFLOW_MANAGER_SPEC.sql` - Package specification
2. `02_MARS_1011_install_CT_MRDS_WORKFLOW_MANAGER_BODY.sql` - Package body with SERVICE_NAME modification
3. `03_MARS_1011_verify_CT_MRDS_WORKFLOW_MANAGER.sql` - Verification and version check
4. `04_MARS_1011_track_CT_MRDS_WORKFLOW_MANAGER_version.sql` - Package version tracking in A_PACKAGE_VERSION_TRACKING
## Rollback Files
1. `91_MARS_1011_rollback_CT_MRDS_WORKFLOW_MANAGER_SPEC.sql` - Rollback package specification to v1.7.0
2. `91_MARS_1011_rollback_CT_MRDS_WORKFLOW_MANAGER_BODY.sql` - Rollback package body (removes SERVICE_NAME from INIT_TASK)
3. `92_MARS_1011_track_rollback_version.sql` - Track rollback version in A_PACKAGE_VERSION_TRACKING
## Installation
```bash
sqlplus CT_MRDS/password@database @install_mars1011.sql
```
## Rollback
```bash
sqlplus CT_MRDS/password@database @rollback_mars1011.sql
```
## Dependencies
- Requires CT_MRDS schema
- Requires A_TASK_HISTORY table with SERVICE_NAME column
- Requires ENV_MANAGER package for version management functions
- Requires A_PACKAGE_VERSION_TRACKING table for version tracking
## Testing
After installation, verify:
1. Package compiles successfully (check `03_MARS_1011_verify_CT_MRDS_WORKFLOW_MANAGER.sql` output)
2. Version shows as 1.7.1
3. INIT_TASK function properly inserts SERVICE_NAME='ODS'
4. Package version is tracked in A_PACKAGE_VERSION_TRACKING table
5. Check tracking: `SELECT * FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING WHERE PACKAGE_NAME='WORKFLOW_MANAGER' ORDER BY TRACKING_DATE DESC;`
After rollback, verify:
1. Version shows as 1.7.0 (rolled back)
2. INIT_TASK function no longer includes SERVICE_NAME column
3. Rollback is tracked in A_PACKAGE_VERSION_TRACKING with CHANGE_STATUS='ROLLBACK'
## Impact
- Low risk change - only affects new task records
- Backward compatible - existing functionality unchanged
- No breaking changes to function signatures

View File

@@ -0,0 +1,216 @@
CREATE OR REPLACE PACKAGE BODY CT_MRDS.WORKFLOW_MANAGER
IS
FUNCTION INIT_WORKFLOW(pServiceName IN VARCHAR2, pWorkflowRunId IN VARCHAR2, pWorkflowName in VARCHAR2)
RETURN NUMBER
IS
vWorkflowHistoryKey NUMBER;
BEGIN
vWorkflowHistoryKey := A_WORKFLOW_HISTORY_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY (SERVICE_NAME, A_WORKFLOW_HISTORY_KEY, ORCHESTRATION_RUN_ID,
WORKFLOW_NAME, WORKFLOW_START, WORKFLOW_SUCCESSFUL)
VALUES (pServiceName, vWorkflowHistoryKey, pWorkflowRunId,
pWorkflowName, SYSTIMESTAMP, vpRunningStatus);
return vWorkflowHistoryKey;
END INIT_WORKFLOW;
--
-- Overload without service name for backward compatability, to be cleaned up later
--
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pWorkflowStatus IN VARCHAR2)
IS
BEGIN
FINALISE_WORKFLOW(pWorkflowHistoryKey, NULL, pWorkflowStatus);
END;
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pServiceName IN VARCHAR2, pWorkflowStatus IN VARCHAR2)
IS
BEGIN
UPDATE CT_MRDS.A_WORKFLOW_HISTORY SET WORKFLOW_SUCCESSFUL = pWorkflowStatus,
WORKFLOW_END = SYSTIMESTAMP
WHERE A_WORKFLOW_HISTORY_KEY = pWorkflowHistoryKey
AND SERVICE_NAME = NVL(pServiceName,SERVICE_NAME);
END FINALISE_WORKFLOW;
FUNCTION INIT_TASK(pTaskRunId IN VARCHAR2, pTaskName in VARCHAR2, pWorkflowHistoryKey IN NUMBER)
RETURN NUMBER
IS
vTaskHistoryKey NUMBER;
BEGIN
vTaskHistoryKey := A_TASK_HISTORY_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY (A_TASK_HISTORY_KEY, TASK_RUN_ID, A_WORKFLOW_HISTORY_KEY,
TASK_NAME, TASK_START, TASK_SUCCESSFUL, SERVICE_NAME)
VALUES (vTaskHistoryKey, pTaskRunId, pWorkflowHistoryKey,
pTaskName, SYSTIMESTAMP, vpRunningStatus, 'ODS');
return vTaskHistoryKey;
END INIT_TASK;
PROCEDURE FINALISE_TASK(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2)
IS
BEGIN
UPDATE CT_MRDS.A_TASK_HISTORY SET TASK_SUCCESSFUL = pTaskStatus,
TASK_END = SYSTIMESTAMP
WHERE A_TASK_HISTORY_KEY = pTaskHistoryKey;
END FINALISE_TASK;
-- Internal helper function without COMMIT for use within other functions
FUNCTION STORE_TASK_SOURCE_INFO_INTERNAL(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER
IS
vTaskHistorySourceKey NUMBER;
BEGIN
vTaskHistorySourceKey := A_TASK_HISTORY_SOURCE_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY_SOURCE (A_TASK_HISTORY_SOURCE_KEY, A_TASK_HISTORY_KEY,
SOURCE_NAME, ROW_COUNT)
VALUES (vTaskHistorySourceKey, pTaskHistoryKey,
pSourceName, pNumRows);
RETURN vTaskHistorySourceKey;
END;
FUNCTION STORE_TASK_SOURCE_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER
IS
vTaskHistorySourceKey NUMBER;
BEGIN
vTaskHistorySourceKey := STORE_TASK_SOURCE_INFO_INTERNAL(pTaskHistoryKey, pSourceName, pNumRows);
COMMIT;
RETURN vTaskHistorySourceKey;
END;
-- Internal helper function without COMMIT for use within other functions
FUNCTION STORE_TASK_TARGET_INFO_INTERNAL(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
vTaskHistoryTargetKey := A_TASK_HISTORY_TARGET_KEY_SEQ.NEXTVAL;
INSERT INTO CT_MRDS.A_TASK_HISTORY_TARGET (A_TASK_HISTORY_TARGET_KEY, A_TASK_HISTORY_KEY,
TARGET_NAME, ROW_COUNT_APPLIED, ROW_COUNT_REJECTED,
LOAD_SUCCESSFUL, SERVICE_NAME)
VALUES (vTaskHistoryTargetKey, pTaskHistoryKey,
pSourceName, pNumRowsApplied, pNumRowsRejected,
pLoadSuccessful, pServiceName);
RETURN vTaskHistoryTargetKey;
END;
FUNCTION STORE_TASK_TARGET_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
vTaskHistoryTargetKey := STORE_TASK_TARGET_INFO_INTERNAL(
pTaskHistoryKey, pSourceName, pNumRowsApplied,
pNumRowsRejected, pLoadSuccessful, pServiceName
);
COMMIT;
RETURN vTaskHistoryTargetKey;
END;
FUNCTION FINALISE_TASK_WITH_TARGET_INFO(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2,
pTargetName IN VARCHAR2, pNumRowsApplied IN NUMBER,
pNumRowsRejected IN NUMBER, pLoadSuccessful IN VARCHAR2,
pServiceName IN VARCHAR2)
RETURN NUMBER
IS
vTaskHistoryTargetKey NUMBER;
BEGIN
-- Store target information using internal function (without COMMIT)
vTaskHistoryTargetKey := STORE_TASK_TARGET_INFO_INTERNAL(
pTaskHistoryKey => pTaskHistoryKey,
pSourceName => pTargetName,
pNumRowsApplied => pNumRowsApplied,
pNumRowsRejected => pNumRowsRejected,
pLoadSuccessful => pLoadSuccessful,
pServiceName => pServiceName
);
-- Finalize the task using existing procedure (no COMMIT in procedure)
FINALISE_TASK(pTaskHistoryKey, pTaskStatus);
-- Single COMMIT for the entire atomic operation
COMMIT;
RETURN vTaskHistoryTargetKey;
END;
PROCEDURE SET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
,pValue IN VARCHAR2
) IS
BEGIN
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY (A_WORKFLOW_HISTORY_KEY, SERVICE_NAME, PROPERTY, VALUE)
VALUES (pWorkflowHistoryKey, pServiceName, pProperty, pValue);
END;
FUNCTION GET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
) RETURN VARCHAR2
IS
vValue CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY.VALUE%TYPE;
BEGIN
SELECT VALUE
INTO vValue
FROM CT_MRDS.A_WORKFLOW_HISTORY_PROPERTY
WHERE A_WORKFLOW_HISTORY_KEY = pWorkflowHistoryKey
AND SERVICE_NAME = pServiceName
AND PROPERTY = pProperty;
RETURN vValue;
END;
----------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS IMPLEMENTATION
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION
RETURN VARCHAR2
IS
BEGIN
RETURN PACKAGE_VERSION;
END GET_VERSION;
----------------------------------------------------------------------------------------------------
FUNCTION GET_BUILD_INFO
RETURN VARCHAR2
IS
BEGIN
RETURN ENV_MANAGER.GET_PACKAGE_VERSION_INFO(
pPackageName => 'WORKFLOW_MANAGER',
pVersion => PACKAGE_VERSION,
pBuildDate => PACKAGE_BUILD_DATE,
pAuthor => PACKAGE_AUTHOR
);
END GET_BUILD_INFO;
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION_HISTORY
RETURN VARCHAR2
IS
BEGIN
RETURN ENV_MANAGER.FORMAT_VERSION_HISTORY(
pPackageName => 'WORKFLOW_MANAGER',
pVersionHistory => VERSION_HISTORY
);
END GET_VERSION_HISTORY;
----------------------------------------------------------------------------------------------------
END WORKFLOW_MANAGER;
/

View File

@@ -0,0 +1,100 @@
CREATE OR REPLACE PACKAGE CT_MRDS.WORKFLOW_MANAGER
IS
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '1.7.1';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-11-18 12:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'1.7.1 (2025-11-18): Added SERVICE_NAME column with hardcoded ''ODS'' value to INIT_TASK function' || CHR(13)||CHR(10) ||
'1.7.0 (2025-10-31): Added FINALISE_TASK_WITH_TARGET_INFO function for atomic task completion with target info' || CHR(13)||CHR(10) ||
'1.6.0 (2025-10-31): Converted STORE_TASK_SOURCE_INFO and STORE_TASK_TARGET_INFO from procedures to functions returning sequence keys' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.2.0 (2025-10-05): Added workflow property management (SET/GET_WORKFLOW_PROPERTY)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-10): Initial release with workflow and task lifecycle management';
-- Line break constant for consistent CRLF formatting across Windows environments
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
FUNCTION INIT_WORKFLOW(pServiceName IN VARCHAR2, pWorkflowRunId IN VARCHAR2, pWorkflowName in VARCHAR2)
RETURN NUMBER;
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pWorkflowStatus IN VARCHAR2);
PROCEDURE FINALISE_WORKFLOW(pWorkflowHistoryKey IN NUMBER, pServiceName IN VARCHAR2, pWorkflowStatus IN VARCHAR2);
FUNCTION INIT_TASK(pTaskRunId IN VARCHAR2, pTaskName in VARCHAR2, pWorkflowHistoryKey IN NUMBER)
RETURN NUMBER;
PROCEDURE FINALISE_TASK(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2);
FUNCTION STORE_TASK_SOURCE_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2, pNumRows IN NUMBER)
RETURN NUMBER;
FUNCTION STORE_TASK_TARGET_INFO(pTaskHistoryKey IN NUMBER, pSourceName IN VARCHAR2,
pNumRowsApplied IN NUMBER, pNumRowsRejected IN NUMBER,
pLoadSuccessful IN VARCHAR2, pServiceName IN VARCHAR2)
RETURN NUMBER;
FUNCTION FINALISE_TASK_WITH_TARGET_INFO(pTaskHistoryKey IN NUMBER, pTaskStatus IN VARCHAR2,
pTargetName IN VARCHAR2, pNumRowsApplied IN NUMBER,
pNumRowsRejected IN NUMBER, pLoadSuccessful IN VARCHAR2,
pServiceName IN VARCHAR2)
RETURN NUMBER;
vpRunningStatus CT_MRDS.A_WORKFLOW_HISTORY.WORKFLOW_SUCCESSFUL%TYPE := 'R';
--
-- Set and get information on workflow level
--
PROCEDURE SET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2
,pValue IN VARCHAR2
);
FUNCTION GET_WORKFLOW_PROPERTY(
pWorkflowHistoryKey IN NUMBER
,pServiceName IN VARCHAR2
,pProperty IN VARCHAR2)
RETURN VARCHAR2;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the WORKFLOW_MANAGER package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT WORKFLOW_MANAGER.GET_VERSION() FROM DUAL;
* @ex_rslt 1.5.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT WORKFLOW_MANAGER.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: WORKFLOW_MANAGER
* Version: 1.5.0
* Build Date: 2025-10-22 17:00:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT WORKFLOW_MANAGER.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt WORKFLOW_MANAGER Version History:
* 1.5.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END WORKFLOW_MANAGER;
/

View File

@@ -0,0 +1,40 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
-- Simple filename without PDB reference
var filename VARCHAR2(100)
BEGIN
SELECT 'INSTALL_MARS_1011_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DUAL;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SYS_CONTEXT('USERENV', 'DB_NAME') AS DATABASE_NAME FROM DUAL;
prompt ========================================
prompt MARS-1011: Installing WORKFLOW_MANAGER v1.7.1
prompt Description: Add SERVICE_NAME='ODS' to INIT_TASK function
prompt ========================================
@@01_MARS_1011_install_CT_MRDS_WORKFLOW_MANAGER_SPEC.sql
@@02_MARS_1011_install_CT_MRDS_WORKFLOW_MANAGER_BODY.sql
@@03_MARS_1011_verify_CT_MRDS_WORKFLOW_MANAGER.sql
@@04_MARS_1011_track_CT_MRDS_WORKFLOW_MANAGER_version.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

@@ -0,0 +1,39 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
-- Simple filename without PDB reference
var filename VARCHAR2(100)
BEGIN
SELECT 'ROLLBACK_MARS_1011_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DUAL;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SYS_CONTEXT('USERENV', 'DB_NAME') AS DATABASE_NAME FROM DUAL;
prompt ========================================
prompt MARS-1011: Rolling back WORKFLOW_MANAGER changes
prompt Description: Rollback to previous version 1.7.0
prompt ========================================
@@91_MARS_1011_rollback_CT_MRDS_WORKFLOW_MANAGER_SPEC.sql
@@91_MARS_1011_rollback_CT_MRDS_WORKFLOW_MANAGER_BODY.sql
@@92_MARS_1011_track_rollback_version.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

@@ -0,0 +1,85 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Update CSDB Encoding Configuration
--=============================================================================================================================
-- Purpose: Change encoding for CSDB configurations from UTF8 to WE8MSWIN1252
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Why WE8MSWIN1252 for CSDB:
-- - CSDB files contain international characters (Turkish: Türkiye, etc.)
-- - Source system uses Windows-1252 encoding
-- - Proper character handling requires matching encoding
--=============================================================================================================================
SET SERVEROUTPUT ON
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC: Update CSDB Encoding Configuration
PROMPT =========================================================================
-- Step 1: Show current state
PROMPT
PROMPT Step 1: Current CSDB encoding configuration
PROMPT ========================================
SELECT
TABLE_ID,
SOURCE_FILE_ID,
ENCODING AS CURRENT_ENCODING,
TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%'
ORDER BY TABLE_ID, SOURCE_FILE_ID;
PROMPT
PROMPT Step 2: Update CSDB configurations to WE8MSWIN1252
PROMPT ========================================
-- Update all CSDB configurations to Windows-1252 encoding
UPDATE /*+ NOPARALLEL */ CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'WE8MSWIN1252'
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%';
PROMPT Updated CSDB configurations with WE8MSWIN1252 encoding
-- Commit changes
COMMIT;
PROMPT
PROMPT Step 3: Verify updated encoding configuration
PROMPT ========================================
SELECT
TABLE_ID,
SOURCE_FILE_ID,
ENCODING AS NEW_ENCODING,
TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%'
ORDER BY TABLE_ID, SOURCE_FILE_ID;
PROMPT
PROMPT Step 4: Overall encoding distribution
PROMPT ========================================
SELECT
ENCODING,
COUNT(*) AS CONFIG_COUNT,
LISTAGG(DISTINCT TABLE_ID, ', ') WITHIN GROUP (ORDER BY TABLE_ID) AS TABLES
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
GROUP BY ENCODING
ORDER BY ENCODING;
PROMPT
PROMPT =========================================================================
PROMPT CSDB Encoding Update - COMPLETED
PROMPT =========================================================================
PROMPT
PROMPT Summary:
PROMPT - All CSDB configurations updated to WE8MSWIN1252
PROMPT - Encoding matches source system character set (Windows-1252)
PROMPT - Proper handling of international characters enabled
PROMPT
PROMPT Next Step: Run 01_MARS_1049_ADHOC_drop_external_tables.sql
PROMPT =========================================================================

View File

@@ -0,0 +1,94 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Drop External Tables for CSDB Encoding Change
--=============================================================================================================================
-- Purpose: Drop existing CSDB external tables before recreation with proper encoding parameter
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support)
--
-- Tables to be dropped (16 total - existing and non-existing):
-- - CSDB_DEBT: ARCHIVE, INBOX, ODS
-- - CSDB_DEBT_DAILY: ARCHIVE, INBOX, ODS
-- - CSDB_INSTR_DESC_FULL: ARCHIVE, INBOX, ODS
-- - CSDB_INSTR_RAT_FULL: ARCHIVE, ODS
-- - CSDB_ISSUER_DESC_FULL: ARCHIVE, ODS
-- - CSDB_ISSUER_RAT_FULL: ARCHIVE, INBOX, ODS
--
-- Note: Script handles both existing and non-existing tables gracefully
--=============================================================================================================================
SET SERVEROUTPUT ON
DECLARE
TYPE t_string_array IS TABLE OF VARCHAR2(100);
-- Define list of tables to drop (16 total)
vTableList t_string_array := t_string_array(
'CSDB_DEBT_ARCHIVE',
'CSDB_DEBT_INBOX',
'CSDB_DEBT_ODS',
'CSDB_DEBT_DAILY_ARCHIVE',
'CSDB_DEBT_DAILY_INBOX',
'CSDB_DEBT_DAILY_ODS',
'CSDB_INSTR_DESC_FULL_ARCHIVE',
'CSDB_INSTR_DESC_FULL_INBOX',
'CSDB_INSTR_DESC_FULL_ODS',
'CSDB_INSTR_RAT_FULL_ARCHIVE',
'CSDB_INSTR_RAT_FULL_INBOX',
'CSDB_INSTR_RAT_FULL_ODS',
'CSDB_ISSUER_DESC_FULL_ARCHIVE',
'CSDB_ISSUER_DESC_FULL_INBOX',
'CSDB_ISSUER_DESC_FULL_ODS',
'CSDB_ISSUER_RAT_FULL_ARCHIVE',
'CSDB_ISSUER_RAT_FULL_INBOX',
'CSDB_ISSUER_RAT_FULL_ODS'
);
vDroppedCount NUMBER := 0;
vSkippedCount NUMBER := 0;
vErrorCount NUMBER := 0;
BEGIN
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('MARS-1049-ADHOC: Drop CSDB External Tables');
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('Tables to process: ' || vTableList.COUNT);
DBMS_OUTPUT.PUT_LINE('');
-- Loop through table list
FOR i IN 1..vTableList.COUNT LOOP
BEGIN
EXECUTE IMMEDIATE 'DROP TABLE ODS.' || vTableList(i);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Dropped ODS.' || vTableList(i));
vDroppedCount := vDroppedCount + 1;
EXCEPTION
WHEN OTHERS THEN
IF SQLCODE = -942 THEN
DBMS_OUTPUT.PUT_LINE('INFO: Table ODS.' || vTableList(i) || ' does not exist - skipping');
vSkippedCount := vSkippedCount + 1;
ELSE
DBMS_OUTPUT.PUT_LINE('ERROR: Failed to drop ODS.' || vTableList(i) || ' - ' || SQLERRM);
vErrorCount := vErrorCount + 1;
RAISE;
END IF;
END;
END LOOP;
DBMS_OUTPUT.PUT_LINE('');
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('Drop External Tables - COMPLETED');
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('Summary:');
DBMS_OUTPUT.PUT_LINE('- Tables processed: ' || vTableList.COUNT);
DBMS_OUTPUT.PUT_LINE('- Tables dropped: ' || vDroppedCount);
DBMS_OUTPUT.PUT_LINE('- Tables skipped: ' || vSkippedCount);
DBMS_OUTPUT.PUT_LINE('- Errors: ' || vErrorCount);
DBMS_OUTPUT.PUT_LINE('');
DBMS_OUTPUT.PUT_LINE('Next Step: Run 02_MARS_1049_ADHOC_recreate_external_tables.sql');
DBMS_OUTPUT.PUT_LINE('=========================================================================');
END;
/

View File

@@ -0,0 +1,297 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Recreate External Tables for CSDB with Encoding Support
--=============================================================================================================================
-- Purpose: Recreate CSDB external tables with proper pEncoding parameter (WE8MSWIN1252)
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Tables to be created (5 existing + 11 missing = 16 total):
-- CSDB_DEBT: INBOX, ODS, ARCHIVE
-- CSDB_DEBT_DAILY: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_RAT_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_RAT_FULL: INBOX, ODS, ARCHIVE
--
-- Prerequisites:
-- - Template tables must exist in CT_ET_TEMPLATES schema
-- - FILE_MANAGER_ODS v2.1.0+ with pEncoding parameter support
-- - Proper bucket URIs configured in ENV_MANAGER
--=============================================================================================================================
SET SERVEROUTPUT ON
PROMPT ========================================================
PROMPT Step 1: Create CSDB_DEBT External Tables with Encoding
PROMPT ========================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'INBOX/CSDB/CentralizedSecuritiesDissemination/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'ODS/CSDB/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'ARCHIVE/CSDB/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT ================================================================
PROMPT Step 2: Create CSDB_DEBT_DAILY External Tables with Encoding
PROMPT ================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'INBOX/CSDB/CentralizedSecuritiesDailyReferenceDataDissemination/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'ODS/CSDB/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'ARCHIVE/CSDB/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT ========================================================================
PROMPT Step 3: Create CSDB_INSTR_DESC_FULL External Tables with Encoding
PROMPT ========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'ODS/CSDB/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT ========================================================================
PROMPT Step 4: Create CSDB_INSTR_RAT_FULL External Tables with Encoding
PROMPT ========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'ODS/CSDB/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT =========================================================================
PROMPT Step 5: Create CSDB_ISSUER_DESC_FULL External Tables with Encoding
PROMPT =========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'ODS/CSDB/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT =========================================================================
PROMPT Step 6: Create CSDB_ISSUER_RAT_FULL External Tables with Encoding
PROMPT =========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_INBOX with encoding WE8MSWIN1252');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'ODS/CSDB/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_ODS with encoding WE8MSWIN1252');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'WE8MSWIN1252'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_ARCHIVE with encoding WE8MSWIN1252');
END;
/
PROMPT =========================================================================
PROMPT Recreate External Tables with Encoding - COMPLETED
PROMPT =========================================================================
PROMPT
PROMPT Summary:
PROMPT - CSDB_DEBT tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_DEBT_DAILY tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_INSTR_DESC_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_INSTR_RAT_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_ISSUER_DESC_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_ISSUER_RAT_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - Total: 18 external tables created with WE8MSWIN1252 encoding
PROMPT
PROMPT Next Step: Run 03_MARS_1049_ADHOC_verify_external_tables.sql
PROMPT =========================================================================

View File

@@ -0,0 +1,184 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Verify CSDB External Tables Creation
--=============================================================================================================================
-- Purpose: Verify all 18 CSDB external tables exist with correct configuration
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Expected Tables (18 total):
-- CSDB_DEBT: INBOX, ODS, ARCHIVE
-- CSDB_DEBT_DAILY: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_RAT_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_RAT_FULL: INBOX, ODS, ARCHIVE
--=============================================================================================================================
SET LINESIZE 200
SET PAGESIZE 100
SET SERVEROUTPUT ON
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC: External Tables Verification Report
PROMPT =========================================================================
PROMPT
PROMPT ========================================
PROMPT Step 1: Check External Tables Existence
PROMPT ========================================
SELECT
owner,
table_name,
CASE
WHEN table_name LIKE '%_INBOX' THEN 'INBOX'
WHEN table_name LIKE '%_ODS' THEN 'ODS'
WHEN table_name LIKE '%_ARCHIVE' THEN 'ARCHIVE'
END AS bucket_area,
default_directory_name AS directory,
type_name
FROM all_external_tables
WHERE owner = 'ODS'
AND table_name IN (
'CSDB_DEBT_ARCHIVE',
'CSDB_DEBT_INBOX',
'CSDB_DEBT_ODS',
'CSDB_DEBT_DAILY_ARCHIVE',
'CSDB_DEBT_DAILY_INBOX',
'CSDB_DEBT_DAILY_ODS',
'CSDB_INSTR_DESC_FULL_ARCHIVE',
'CSDB_INSTR_DESC_FULL_INBOX',
'CSDB_INSTR_DESC_FULL_ODS',
'CSDB_INSTR_RAT_FULL_ARCHIVE',
'CSDB_INSTR_RAT_FULL_INBOX',
'CSDB_INSTR_RAT_FULL_ODS',
'CSDB_ISSUER_DESC_FULL_ARCHIVE',
'CSDB_ISSUER_DESC_FULL_INBOX',
'CSDB_ISSUER_DESC_FULL_ODS',
'CSDB_ISSUER_RAT_FULL_ARCHIVE',
'CSDB_ISSUER_RAT_FULL_INBOX',
'CSDB_ISSUER_RAT_FULL_ODS'
)
ORDER BY table_name;
PROMPT
PROMPT ========================================
PROMPT Step 2: Verify Table Counts by Category
PROMPT ========================================
SELECT
CASE
WHEN table_name LIKE 'CSDB_DEBT_DAILY%' THEN 'CSDB_DEBT_DAILY'
WHEN table_name LIKE 'CSDB_DEBT%' THEN 'CSDB_DEBT'
WHEN table_name LIKE 'CSDB_INSTR_DESC_FULL%' THEN 'CSDB_INSTR_DESC_FULL'
WHEN table_name LIKE 'CSDB_INSTR_RAT_FULL%' THEN 'CSDB_INSTR_RAT_FULL'
WHEN table_name LIKE 'CSDB_ISSUER_DESC_FULL%' THEN 'CSDB_ISSUER_DESC_FULL'
WHEN table_name LIKE 'CSDB_ISSUER_RAT_FULL%' THEN 'CSDB_ISSUER_RAT_FULL'
END AS table_category,
COUNT(*) AS table_count,
CASE WHEN COUNT(*) = 3 THEN 'COMPLETE' ELSE 'INCOMPLETE' END AS status
FROM all_external_tables
WHERE owner = 'ODS'
AND table_name IN (
'CSDB_DEBT_ARCHIVE', 'CSDB_DEBT_INBOX', 'CSDB_DEBT_ODS',
'CSDB_DEBT_DAILY_ARCHIVE', 'CSDB_DEBT_DAILY_INBOX', 'CSDB_DEBT_DAILY_ODS',
'CSDB_INSTR_DESC_FULL_ARCHIVE', 'CSDB_INSTR_DESC_FULL_INBOX', 'CSDB_INSTR_DESC_FULL_ODS',
'CSDB_INSTR_RAT_FULL_ARCHIVE', 'CSDB_INSTR_RAT_FULL_INBOX', 'CSDB_INSTR_RAT_FULL_ODS',
'CSDB_ISSUER_DESC_FULL_ARCHIVE', 'CSDB_ISSUER_DESC_FULL_INBOX', 'CSDB_ISSUER_DESC_FULL_ODS',
'CSDB_ISSUER_RAT_FULL_ARCHIVE', 'CSDB_ISSUER_RAT_FULL_INBOX', 'CSDB_ISSUER_RAT_FULL_ODS'
)
GROUP BY
CASE
WHEN table_name LIKE 'CSDB_DEBT_DAILY%' THEN 'CSDB_DEBT_DAILY'
WHEN table_name LIKE 'CSDB_DEBT%' THEN 'CSDB_DEBT'
WHEN table_name LIKE 'CSDB_INSTR_DESC_FULL%' THEN 'CSDB_INSTR_DESC_FULL'
WHEN table_name LIKE 'CSDB_INSTR_RAT_FULL%' THEN 'CSDB_INSTR_RAT_FULL'
WHEN table_name LIKE 'CSDB_ISSUER_DESC_FULL%' THEN 'CSDB_ISSUER_DESC_FULL'
WHEN table_name LIKE 'CSDB_ISSUER_RAT_FULL%' THEN 'CSDB_ISSUER_RAT_FULL'
END
ORDER BY table_category;
PROMPT
PROMPT ========================================
PROMPT Step 3: Overall Summary
PROMPT ========================================
DECLARE
v_total_count NUMBER;
v_expected_count NUMBER := 18;
v_status VARCHAR2(20);
BEGIN
SELECT COUNT(*) INTO v_total_count
FROM all_external_tables
WHERE owner = 'ODS'
AND table_name IN (
'CSDB_DEBT_ARCHIVE', 'CSDB_DEBT_INBOX', 'CSDB_DEBT_ODS',
'CSDB_DEBT_DAILY_ARCHIVE', 'CSDB_DEBT_DAILY_INBOX', 'CSDB_DEBT_DAILY_ODS',
'CSDB_INSTR_DESC_FULL_ARCHIVE', 'CSDB_INSTR_DESC_FULL_INBOX', 'CSDB_INSTR_DESC_FULL_ODS',
'CSDB_INSTR_RAT_FULL_ARCHIVE', 'CSDB_INSTR_RAT_FULL_INBOX', 'CSDB_INSTR_RAT_FULL_ODS',
'CSDB_ISSUER_DESC_FULL_ARCHIVE', 'CSDB_ISSUER_DESC_FULL_INBOX', 'CSDB_ISSUER_DESC_FULL_ODS',
'CSDB_ISSUER_RAT_FULL_ARCHIVE', 'CSDB_ISSUER_RAT_FULL_INBOX', 'CSDB_ISSUER_RAT_FULL_ODS'
);
IF v_total_count = v_expected_count THEN
v_status := 'SUCCESS';
ELSE
v_status := 'FAILED';
END IF;
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('VERIFICATION STATUS: ' || v_status);
DBMS_OUTPUT.PUT_LINE('=========================================================================');
DBMS_OUTPUT.PUT_LINE('Expected Tables: ' || v_expected_count);
DBMS_OUTPUT.PUT_LINE('Found Tables: ' || v_total_count);
DBMS_OUTPUT.PUT_LINE('Missing Tables: ' || (v_expected_count - v_total_count));
DBMS_OUTPUT.PUT_LINE('=========================================================================');
IF v_total_count = v_expected_count THEN
DBMS_OUTPUT.PUT_LINE('');
DBMS_OUTPUT.PUT_LINE('All 18 CSDB external tables created successfully with WE8MSWIN1252 encoding');
DBMS_OUTPUT.PUT_LINE('MARS-1049-ADHOC deployment completed successfully');
ELSE
DBMS_OUTPUT.PUT_LINE('');
DBMS_OUTPUT.PUT_LINE('WARNING: Not all tables were created. Review errors from step 02.');
RAISE_APPLICATION_ERROR(-20001, 'External tables verification failed');
END IF;
END;
/
PROMPT
PROMPT ========================================
PROMPT Step 4: Check Missing Tables (if any)
PROMPT ========================================
WITH expected_tables AS (
SELECT 'CSDB_DEBT_ARCHIVE' AS table_name FROM DUAL UNION ALL
SELECT 'CSDB_DEBT_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_DEBT_ODS' FROM DUAL UNION ALL
SELECT 'CSDB_DEBT_DAILY_ARCHIVE' FROM DUAL UNION ALL
SELECT 'CSDB_DEBT_DAILY_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_DEBT_DAILY_ODS' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_DESC_FULL_ARCHIVE' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_DESC_FULL_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_DESC_FULL_ODS' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_RAT_FULL_ARCHIVE' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_RAT_FULL_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_INSTR_RAT_FULL_ODS' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_DESC_FULL_ARCHIVE' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_DESC_FULL_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_DESC_FULL_ODS' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_RAT_FULL_ARCHIVE' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_RAT_FULL_INBOX' FROM DUAL UNION ALL
SELECT 'CSDB_ISSUER_RAT_FULL_ODS' FROM DUAL
)
SELECT e.table_name AS missing_table
FROM expected_tables e
LEFT JOIN all_external_tables t ON e.table_name = t.table_name AND t.owner = 'ODS'
WHERE t.table_name IS NULL
ORDER BY e.table_name;
PROMPT
PROMPT =========================================================================
PROMPT Verification Report Complete
PROMPT =========================================================================

View File

@@ -0,0 +1,84 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Rollback CSDB Encoding Configuration
--=============================================================================================================================
-- Purpose: Rollback CSDB encoding from WE8MSWIN1252 to UTF8
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Use Case: Restore original UTF8 encoding if WE8MSWIN1252 causes issues
--
-- WARNING: External tables must be recreated after encoding change
--=============================================================================================================================
SET SERVEROUTPUT ON
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC: Rollback CSDB Encoding to UTF8
PROMPT =========================================================================
-- Step 1: Show current state
PROMPT
PROMPT Step 1: Current CSDB encoding configuration
PROMPT ========================================
SELECT
TABLE_ID,
SOURCE_FILE_ID,
ENCODING AS CURRENT_ENCODING,
TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%'
ORDER BY TABLE_ID, SOURCE_FILE_ID;
PROMPT
PROMPT Step 2: Rollback CSDB configurations to UTF8
PROMPT ========================================
-- Rollback all CSDB configurations to UTF8 encoding
UPDATE /*+ NOPARALLEL */ CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'UTF8'
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%';
PROMPT Rolled back CSDB configurations to UTF8 encoding
-- Commit changes
COMMIT;
PROMPT
PROMPT Step 3: Verify rolled back encoding configuration
PROMPT ========================================
SELECT
TABLE_ID,
SOURCE_FILE_ID,
ENCODING AS RESTORED_ENCODING,
TEMPLATE_TABLE_NAME
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%'
ORDER BY TABLE_ID, SOURCE_FILE_ID;
PROMPT
PROMPT Step 4: Overall encoding distribution
PROMPT ========================================
SELECT
ENCODING,
COUNT(*) AS CONFIG_COUNT,
LISTAGG(DISTINCT TABLE_ID, ', ') WITHIN GROUP (ORDER BY TABLE_ID) AS TABLES
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
GROUP BY ENCODING
ORDER BY ENCODING;
PROMPT
PROMPT =========================================================================
PROMPT CSDB Encoding Rollback - COMPLETED
PROMPT =========================================================================
PROMPT
PROMPT Summary:
PROMPT - All CSDB configurations rolled back to UTF8
PROMPT - Original encoding restored
PROMPT
PROMPT IMPORTANT: External tables must be recreated with new encoding
PROMPT - Use MARS-821 scripts to recreate external tables with UTF8
PROMPT =========================================================================

View File

@@ -0,0 +1,297 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Recreate External Tables with UTF8 Encoding (Rollback Step)
--=============================================================================================================================
-- Purpose: Recreate CSDB external tables with UTF8 encoding (original encoding)
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Tables to be created (18 total with UTF8 encoding):
-- CSDB_DEBT: INBOX, ODS, ARCHIVE
-- CSDB_DEBT_DAILY: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_INSTR_RAT_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_DESC_FULL: INBOX, ODS, ARCHIVE
-- CSDB_ISSUER_RAT_FULL: INBOX, ODS, ARCHIVE
--
-- Prerequisites:
-- - Template tables must exist in CT_ET_TEMPLATES schema
-- - FILE_MANAGER_ODS v2.1.0+ with pEncoding parameter support
-- - Proper bucket URIs configured in ENV_MANAGER
--=============================================================================================================================
SET SERVEROUTPUT ON
PROMPT ========================================================
PROMPT Step 1: Create CSDB_DEBT External Tables with UTF8
PROMPT ========================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'INBOX/CSDB/CentralizedSecuritiesDissemination/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'ODS/CSDB/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'ARCHIVE/CSDB/CSDB_DEBT',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_ARCHIVE with encoding UTF8');
END;
/
PROMPT ================================================================
PROMPT Step 2: Create CSDB_DEBT_DAILY External Tables with UTF8
PROMPT ================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'INBOX/CSDB/CentralizedSecuritiesDailyReferenceDataDissemination/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'ODS/CSDB/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_DAILY_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT_DAILY',
pPrefix => 'ARCHIVE/CSDB/CSDB_DEBT_DAILY',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_DEBT_DAILY_ARCHIVE with encoding UTF8');
END;
/
PROMPT ========================================================================
PROMPT Step 3: Create CSDB_INSTR_DESC_FULL External Tables with UTF8
PROMPT ========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'ODS/CSDB/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_DESC_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_DESC_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_INSTR_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_DESC_FULL_ARCHIVE with encoding UTF8');
END;
/
PROMPT ========================================================================
PROMPT Step 4: Create CSDB_INSTR_RAT_FULL External Tables with UTF8
PROMPT ========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'ODS/CSDB/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_INSTR_RAT_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_INSTR_RAT_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_INSTR_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_INSTR_RAT_FULL_ARCHIVE with encoding UTF8');
END;
/
PROMPT =========================================================================
PROMPT Step 5: Create CSDB_ISSUER_DESC_FULL External Tables with UTF8
PROMPT =========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'ODS/CSDB/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_DESC_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_DESC_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_ISSUER_DESC_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_DESC_FULL_ARCHIVE with encoding UTF8');
END;
/
PROMPT =========================================================================
PROMPT Step 6: Create CSDB_ISSUER_RAT_FULL External Tables with UTF8
PROMPT =========================================================================
-- INBOX - For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'INBOX/CSDB/FullRatingsDissemination/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_INBOX with encoding UTF8');
END;
/
-- ODS - For processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'ODS/CSDB/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_ODS with encoding UTF8');
END;
/
-- ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_ISSUER_RAT_FULL_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_ISSUER_RAT_FULL',
pPrefix => 'ARCHIVE/CSDB/CSDB_ISSUER_RAT_FULL',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri,
pEncoding => 'UTF8'
);
DBMS_OUTPUT.PUT_LINE('SUCCESS: Created ODS.CSDB_ISSUER_RAT_FULL_ARCHIVE with encoding UTF8');
END;
/
PROMPT =========================================================================
PROMPT Recreate External Tables with UTF8 - COMPLETED
PROMPT =========================================================================
PROMPT
PROMPT Summary:
PROMPT - CSDB_DEBT tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_DEBT_DAILY tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_INSTR_DESC_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_INSTR_RAT_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_ISSUER_DESC_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - CSDB_ISSUER_RAT_FULL tables: 3 created (INBOX, ODS, ARCHIVE)
PROMPT - Total: 18 external tables created with UTF8 encoding
PROMPT
PROMPT Next Step: Run 03_MARS_1049_ADHOC_verify_external_tables.sql
PROMPT =========================================================================

View File

@@ -0,0 +1,102 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Master Installation Script
--=============================================================================================================================
-- Purpose: Execute all MARS-1049-ADHOC scripts to recreate CSDB external tables with encoding support
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Prerequisites:
-- - FILE_MANAGER v3.2.1+ with pEncoding parameter support
-- - FILE_MANAGER_ODS v2.1.0+ with pEncoding parameter support
-- - Template tables exist in CT_ET_TEMPLATES schema
-- - Proper bucket URIs configured in ENV_MANAGER
-- - Execute as ODS user or with sufficient privileges
--
-- Deployment Sequence:
-- Step 1: Drop existing CSDB external tables
-- Step 2: Recreate all 18 CSDB external tables with WE8MSWIN1252 encoding
-- Step 3: Verify all tables created successfully
--
-- Rollback:
-- No rollback script provided - tables can be recreated using MARS-821 scripts if needed
--=============================================================================================================================
SET SERVEROUTPUT ON
SET ECHO ON
SET VERIFY OFF
-- ===================================================================
-- MARS-1049-ADHOC INSTALL SCRIPT: CSDB External Tables Encoding Update
-- ===================================================================
-- Purpose: Recreate CSDB external tables with WE8MSWIN1252 encoding
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Version: 1.0.0
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'INSTALL_MARS_1049_ADHOC_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC: CSDB External Tables Encoding Update
PROMPT =========================================================================
PROMPT
PROMPT This script will:
PROMPT - Update CSDB configurations to WE8MSWIN1252 encoding
PROMPT - Drop existing CSDB external tables (if any)
PROMPT - Recreate 18 CSDB external tables with WE8MSWIN1252 encoding
PROMPT - Verify successful creation
PROMPT
PROMPT Expected Duration: 2-3 minutes
PROMPT =========================================================================
PROMPT
PROMPT =========================================================================
PROMPT Step 0: Update CSDB Encoding Configuration
PROMPT =========================================================================
@@00_MARS_1049_ADHOC_update_encoding.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 1: Drop Existing CSDB External Tables
PROMPT =========================================================================
@@01_MARS_1049_ADHOC_drop_external_tables.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Recreate CSDB External Tables with Encoding
PROMPT =========================================================================
@@02_MARS_1049_ADHOC_recreate_external_tables.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 3: Verify External Tables Creation
PROMPT =========================================================================
@@03_MARS_1049_ADHOC_verify_external_tables.sql
PROMPT
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC Installation - COMPLETED
PROMPT =========================================================================
PROMPT Check the log file for complete installation details.
PROMPT =========================================================================
spool off
PROMPT
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC Installation Complete
PROMPT =========================================================================
PROMPT
PROMPT All CSDB external tables have been updated with WE8MSWIN1252 encoding.
PROMPT
PROMPT =========================================================================
quit;

View File

@@ -0,0 +1,106 @@
--=============================================================================================================================
-- MARS-1049-ADHOC: Master Rollback Script
--=============================================================================================================================
-- Purpose: Rollback CSDB external tables encoding changes from WE8MSWIN1252 to UTF8
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Related: MARS-1049 (CSV Encoding Support), MARS-821 (Original CSDB Setup)
--
-- Prerequisites:
-- - FILE_MANAGER v3.2.1+ with pEncoding parameter support
-- - FILE_MANAGER_ODS v2.1.0+ with pEncoding parameter support
-- - Template tables exist in CT_ET_TEMPLATES schema
-- - Proper bucket URIs configured in ENV_MANAGER
-- - Execute as ODS user or with sufficient privileges
--
-- Rollback Sequence:
-- Step 1: Rollback CSDB configurations to UTF8 encoding
-- Step 2: Drop CSDB external tables with WE8MSWIN1252 encoding
-- Step 3: Recreate CSDB external tables with UTF8 encoding (using MARS-821 patterns)
-- Step 4: Verify all tables created successfully
--
-- Post-Rollback:
-- - CSDB configurations restored to UTF8
-- - External tables recreated with UTF8 encoding
-- - System restored to pre-MARS-1049-ADHOC state
--=============================================================================================================================
SET SERVEROUTPUT ON
SET ECHO ON
SET VERIFY OFF
-- ===================================================================
-- MARS-1049-ADHOC ROLLBACK SCRIPT: Restore UTF8 Encoding for CSDB
-- ===================================================================
-- Purpose: Rollback CSDB external tables to UTF8 encoding
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Version: 1.0.0
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'ROLLBACK_MARS_1049_ADHOC_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC: Rollback to UTF8 Encoding
PROMPT =========================================================================
PROMPT
PROMPT This script will:
PROMPT - Rollback CSDB configurations to UTF8 encoding
PROMPT - Drop existing CSDB external tables (WE8MSWIN1252)
PROMPT - Recreate 18 CSDB external tables with UTF8 encoding
PROMPT - Verify successful rollback
PROMPT
PROMPT Expected Duration: 2-3 minutes
PROMPT =========================================================================
PROMPT
PROMPT =========================================================================
PROMPT Step 1: Rollback CSDB Encoding Configuration to UTF8
PROMPT =========================================================================
@@91_MARS_1049_ADHOC_rollback_encoding.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Drop Existing CSDB External Tables
PROMPT =========================================================================
@@01_MARS_1049_ADHOC_drop_external_tables.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 3: Recreate CSDB External Tables with UTF8 Encoding
PROMPT =========================================================================
@@92_MARS_1049_ADHOC_recreate_tables_utf8.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 4: Verify External Tables Creation
PROMPT =========================================================================
@@03_MARS_1049_ADHOC_verify_external_tables.sql
PROMPT
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC Rollback - COMPLETED
PROMPT =========================================================================
PROMPT Check the log file for complete rollback details.
PROMPT =========================================================================
spool off
PROMPT
PROMPT =========================================================================
PROMPT MARS-1049-ADHOC Rollback Complete
PROMPT =========================================================================
PROMPT
PROMPT All CSDB external tables have been restored to UTF8 encoding.
PROMPT System returned to pre-MARS-1049-ADHOC state.
PROMPT
PROMPT =========================================================================
quit;

View File

@@ -0,0 +1,48 @@
-- ===================================================================
-- MARS-1049 INSTALLATION COMPONENT: Grant Permissions for Hash Tracking
-- ===================================================================
-- Purpose: Grant CT_MRDS necessary permissions to read package source across schemas
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
-- Step: 00 (First step - must run before package deployment)
--
-- PROBLEM:
-- ENV_MANAGER.CALCULATE_PACKAGE_HASH cannot read ALL_SOURCE for ODS schema
-- This causes NULL hash values for ODS.FILE_MANAGER_ODS PACKAGE BODY
--
-- SOLUTION:
-- Grant SELECT on specific dictionary views (ALL_SOURCE, ALL_OBJECTS) to CT_MRDS
-- This is the MINIMUM required privilege for cross-schema package hash tracking
-- More secure than SELECT ANY DICTIONARY (only grants access to specific views needed)
--
-- USAGE:
-- - Execute as ADMIN user (system privilege grant required)
-- - Should be first step in MARS-1049 installation
-- - One-time setup that enables cross-schema package tracking
-- ===================================================================
PROMPT ===============================================
PROMPT Fixing permissions for cross-schema hash tracking
PROMPT ===============================================
-- Grant SELECT on ALL_SOURCE (required to read package source code for hash calculation)
GRANT SELECT ON ALL_SOURCE TO CT_MRDS;
PROMPT Granted SELECT on SYS.ALL_SOURCE to CT_MRDS
PROMPT ===============================================
PROMPT Permission fix completed (MINIMUM PRIVILEGE)
PROMPT ===============================================
PROMPT
PROMPT CT_MRDS can now:
PROMPT - Read ALL_SOURCE for any schema (package source code)
PROMPT - Calculate SHA256 hashes for packages in ODS and other schemas
PROMPT
PROMPT CT_MRDS CANNOT:
PROMPT - Read user tables in ODS or other schemas
PROMPT - Access other dictionary views
PROMPT - Access any data outside of ALL_SOURCE view
PROMPT
PROMPT SECURITY: Absolute minimum privilege - only ALL_SOURCE view granted
PROMPT ===============================================

View File

@@ -0,0 +1,69 @@
-- ===================================================================
-- MARS-1049: CSV Encoding Support - Database Structure Changes
-- ===================================================================
-- Purpose: Add ENCODING column to A_SOURCE_FILE_CONFIG table to support
-- character set specification for CSV files in external tables
-- Author: Grzegorz Michalski
-- Date: 2025-11-24
-- Version: 1.1.0
--
-- IMPLEMENTATION:
-- - Adds ENCODING VARCHAR2(50) column to CT_MRDS.A_SOURCE_FILE_CONFIG
-- - Column stores Oracle character set names (UTF8, WE8MSWIN1252, etc.)
-- - Used by FILE_MANAGER.CREATE_EXTERNAL_TABLE for CHARACTERSET clause
-- - Enables proper handling of non-ASCII characters in CSV files
-- ===================================================================
PROMPT ================================================
PROMPT MARS-1049: Adding ENCODING column to A_SOURCE_FILE_CONFIG
PROMPT ================================================
-- Check if ENCODING column already exists
DECLARE
vCount NUMBER := 0;
BEGIN
SELECT COUNT(*) INTO vCount
FROM USER_TAB_COLUMNS
WHERE TABLE_NAME = 'A_SOURCE_FILE_CONFIG'
AND COLUMN_NAME = 'ENCODING';
IF vCount = 0 THEN
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (ENCODING VARCHAR2(50) DEFAULT ''UTF8'')';
DBMS_OUTPUT.PUT_LINE('SUCCESS: ENCODING column added to A_SOURCE_FILE_CONFIG with DEFAULT UTF8');
ELSE
DBMS_OUTPUT.PUT_LINE('INFO: ENCODING column already exists in A_SOURCE_FILE_CONFIG');
END IF;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('ERROR: Failed to add ENCODING column: ' || SQLERRM);
RAISE;
END;
/
-- Add column comment for documentation
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING
IS 'Character encoding for files (e.g., UTF8, WE8MSWIN1252, EE8ISO8859P2). Used by CREATE_EXTERNAL_TABLE for CHARACTERSET specification.';
-- Verify the column was added successfully
SELECT
COLUMN_NAME,
DATA_TYPE,
DATA_LENGTH,
NULLABLE,
(SELECT COMMENTS FROM USER_COL_COMMENTS WHERE TABLE_NAME = 'A_SOURCE_FILE_CONFIG' AND COLUMN_NAME = 'ENCODING') AS COLUMN_COMMENT
FROM USER_TAB_COLUMNS
WHERE TABLE_NAME = 'A_SOURCE_FILE_CONFIG'
AND COLUMN_NAME = 'ENCODING';
PROMPT ===============================================
PROMPT MARS-1049: Database structure changes completed
PROMPT ===============================================
PROMPT
PROMPT ENCODING column specifications:
PROMPT - Type: VARCHAR2(50)
PROMPT - Purpose: Store Oracle character set names for CSV files
PROMPT - Examples: UTF8, WE8MSWIN1252, EE8ISO8859P2, CL8MSWIN1251
PROMPT - Usage: Used by CREATE_EXTERNAL_TABLE procedures
PROMPT
PROMPT Next Step: Deploy updated FILE_MANAGER packages
PROMPT ===============================================

View File

@@ -0,0 +1,56 @@
-- ===================================================================
-- MARS-1049: Update Existing Configurations with Encoding Values
-- ===================================================================
-- Purpose: Set default encoding values for existing A_SOURCE_FILE_CONFIG records
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
--
-- ENCODING STRATEGY:
-- - UTF8: Default for all configurations (modern Unicode standard)
-- - WE8MSWIN1252: Special case for CSDB templates (Windows-1252 compatibility)
-- ===================================================================
PROMPT ===============================================
PROMPT MARS-1049: Updating existing configurations with encoding values
PROMPT ===============================================
-- Check current state before updates
SELECT COUNT(*) AS TOTAL_CONFIGURATIONS,
COUNT(CASE WHEN ENCODING IS NULL THEN 1 END) AS NULL_ENCODING_COUNT,
COUNT(CASE WHEN ENCODING IS NOT NULL THEN 1 END) AS WITH_ENCODING_COUNT
FROM CT_MRDS.A_SOURCE_FILE_CONFIG;
-- Update all configurations to UTF8 (default)
UPDATE /*+ NOPARALLEL */ CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'UTF8'
WHERE ENCODING IS NULL;
PROMPT Updated configurations with UTF8 (default)
-- Commit first batch to avoid deadlock
COMMIT;
-- PROMPT Committed UTF8 updates - now processing CSDB configurations...
-- -- Update CSDB-specific configurations to Windows-1252
-- UPDATE /*+ NOPARALLEL */ CT_MRDS.A_SOURCE_FILE_CONFIG
-- SET ENCODING = 'WE8MSWIN1252'
-- WHERE TEMPLATE_TABLE_NAME LIKE '%CSDB%';
-- PROMPT Updated CSDB configurations with WE8MSWIN1252
-- -- Commit changes
-- COMMIT;
-- Verify final state
PROMPT
PROMPT Final encoding distribution:
SELECT ENCODING, COUNT(*) AS COUNT
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
GROUP BY ENCODING
ORDER BY ENCODING;
PROMPT ===============================================
PROMPT Encoding configuration update completed
PROMPT ===============================================

View File

@@ -0,0 +1,47 @@
-- ===================================================================
-- MARS-1049 ROLLBACK COMPONENT: Revoke Permissions for Hash Tracking
-- ===================================================================
-- Purpose: Revoke CT_MRDS permissions granted during MARS-1049 installation
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
-- Step: 90 (Last step - execute after package rollback)
--
-- ROLLBACK OPERATION:
-- Revokes SELECT ANY DICTIONARY from CT_MRDS
--
-- WARNING:
-- This may affect other packages that depend on cross-schema hash tracking
-- Only revoke if no other packages in ODS or other schemas need tracking
--
-- USAGE:
-- - Execute as ADMIN user (system privilege revoke required)
-- - Should be last step in MARS-1049 rollback (after restoring packages)
-- ===================================================================
PROMPT ===============================================
PROMPT Revoking cross-schema hash tracking permissions
PROMPT ===============================================
-- Check if privilege exists before revoking
SELECT PRIVILEGE, GRANTEE
FROM DBA_SYS_PRIVS
WHERE GRANTEE = 'CT_MRDS'
AND PRIVILEGE = 'SELECT ANY DICTIONARY';
-- Revoke SELECT ANY DICTIONARY from CT_MRDS
REVOKE SELECT ANY DICTIONARY FROM CT_MRDS;
PROMPT Revoked SELECT ANY DICTIONARY from CT_MRDS
PROMPT ===============================================
PROMPT Permission revocation completed
PROMPT ===============================================
PROMPT
PROMPT CT_MRDS can NO LONGER:
PROMPT - Read ALL_SOURCE for schemas other than CT_MRDS
PROMPT - Calculate SHA256 hashes for packages in ODS and other schemas
PROMPT
PROMPT WARNING: This may cause NULL hash values for cross-schema packages
PROMPT If other packages depend on this, re-grant the privilege
PROMPT ===============================================

View File

@@ -0,0 +1,47 @@
-- ===================================================================
-- MARS-1049 ROLLBACK COMPONENT: Remove ENCODING Column
-- ===================================================================
-- Purpose: Remove ENCODING column from A_SOURCE_FILE_CONFIG table
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
--
-- ROLLBACK OPERATION:
-- - Removes ENCODING VARCHAR2(50) column from CT_MRDS.A_SOURCE_FILE_CONFIG
-- - Part of complete MARS-1049 rollback sequence
--
-- USAGE:
-- - Can be executed standalone or as part of main rollback script
-- - Includes verification steps for safe execution
-- ===================================================================
PROMPT ===============================================
PROMPT MARS-1049: Remove ENCODING Column from A_SOURCE_FILE_CONFIG
PROMPT ===============================================
-- Step 1: Remove ENCODING column from A_SOURCE_FILE_CONFIG
PROMPT Step 1: Remove ENCODING column from A_SOURCE_FILE_CONFIG...
PROMPT ===============================================
-- First check if column exists
SELECT COUNT(*) AS ENCODING_COLUMN_EXISTS
FROM all_tab_columns
WHERE owner = 'CT_MRDS'
AND table_name = 'A_SOURCE_FILE_CONFIG'
AND column_name = 'ENCODING';
-- Remove the ENCODING column
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP COLUMN ENCODING;
PROMPT ENCODING column removed from A_SOURCE_FILE_CONFIG
-- Final verification
SELECT COUNT(*) AS ENCODING_COLUMN_EXISTS_AFTER_REMOVAL
FROM all_tab_columns
WHERE owner = 'CT_MRDS'
AND table_name = 'A_SOURCE_FILE_CONFIG'
AND column_name = 'ENCODING';
PROMPT ===============================================
PROMPT ENCODING column removal completed successfully
PROMPT ===============================================

View File

@@ -0,0 +1,493 @@
# MARS-1049: CSV Encoding Support - Complete Implementation
## 🎯 Implementation Status: ✅ COMPLETED & FULLY TESTED
**Implementation Date:** 2025-11-24
**Production Testing Date:** 2025-11-25
**Final Validation Date:** 2025-11-25
**Database Version:** Oracle 23.26.0.1.0
**Package Versions:** CT_MRDS.FILE_MANAGER v3.2.1, ODS.FILE_MANAGER_ODS v2.1.0
**Status:** Production Ready & Fully Validated ✅
---
## 📋 Overview
MARS-1049 implements comprehensive CSV encoding support in the Oracle FILE_MANAGER system, enabling proper handling of character sets when creating external tables for CSV file processing. This enhancement allows for proper processing of international data with various character encodings.
### Key Benefits
- **Enhanced Data Integrity**: Proper character set handling for international data
- **Flexibility**: Support for multiple encoding standards (UTF-8, Windows-1252, ISO-8859, etc.)
- **Backward Compatibility**: All existing code continues working unchanged
- **Simple Configuration**: Easy-to-use encoding parameter in existing procedures
---
## 📁 Project Structure & Version Control
This implementation uses organized folder structure for version control and rollback capabilities:
```
MARS_Packages/REL01/MARS-1049/
├── current_version/ # 📦 Pre-MARS-1049 Versions
│ ├── FILE_MANAGER.pkg # v3.2.0 (without pEncoding)
│ ├── FILE_MANAGER.pkb # v3.2.0 (without pEncoding)
│ ├── FILE_MANAGER_ODS.pkg # v2.0.0 (without pEncoding)
│ └── FILE_MANAGER_ODS.pkb # v2.0.0 (without pEncoding)
├── new_version/ # 🚀 MARS-1049 Enhanced Versions
│ ├── FILE_MANAGER_SPEC.sql # v3.2.1 (with pEncoding)
│ ├── FILE_MANAGER_BODY.sql # v3.2.1 (with pEncoding)
│ ├── FILE_MANAGER_ODS_SPEC.sql # v2.1.0 (with pEncoding)
│ └── FILE_MANAGER_ODS_BODY.sql # v2.1.0 (with pEncoding)
├── install_mars1049.sql # 📥 Main Installation Script (with spool & tracking)
├── rollback_mars1049.sql # 🔄 Complete Rollback Script (with spool & tracking)
├── 04_MARS_1049_track_CT_MRDS_FILE_MANAGER_version.sql # 📈 Version Tracking (Install)
├── 92_MARS_1049_track_rollback_version.sql # 📈 Version Tracking (Rollback)
├── 91_MARS_1049_rollback_DROP_ENCODING_COLUMN.sql # 🔄 Column Removal Component
└── README.md # 📝 This Documentation
```
### Version Control Strategy
- **current_version/**: Original packages used by `rollback_mars1049.sql`
- **new_version/**: Enhanced packages used by `install_mars1049.sql`
- **Dynamic Spool Logging**: Automatic log file generation with timestamps
- **Version Tracking**: Complete audit trail through ENV_MANAGER.TRACK_PACKAGE_VERSION
- **Complete Change Tracking**: Full history of all modifications maintained
---
## 🔧 Database Changes Implemented
### 1. Table Structure Enhancement
```sql
-- Added to CT_MRDS.A_SOURCE_FILE_CONFIG
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
ENCODING VARCHAR2(50) DEFAULT NULL -- Character encoding for CSV files
);
```
### 2. Package Version Updates
| Package | Before | After | Changes |
|---------|---------|--------|---------|
| **CT_MRDS.FILE_MANAGER** | v3.2.0 | v3.2.1 | Added `pEncoding` parameter |
| **ODS.FILE_MANAGER_ODS** | v2.0.0 | v2.1.0 | Added encoding wrapper support |
### 3. Enhanced Procedures
- `FILE_MANAGER.ADD_SOURCE_FILE_CONFIG` - Added `pEncoding` parameter
- `FILE_MANAGER.CREATE_EXTERNAL_TABLE` - Added encoding support
- `FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` - Added encoding delegation
### 4. Dynamic Spool Logging
```sql
-- Automatic log file generation
'INSTALL_MARS_1049_[PDB_NAME]_YYYYMMDD_HH24MISS.log'
'ROLLBACK_MARS_1049_[PDB_NAME]_YYYYMMDD_HH24MISS.log'
```
### 5. Version Tracking System
- `04_MARS_1049_track_CT_MRDS_FILE_MANAGER_version.sql` - Installation tracking
- `92_MARS_1049_track_rollback_version.sql` - Rollback tracking
- Complete audit trail via `CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION`
---
## 🌍 Supported Character Encodings
| Encoding | Description | Use Case | Example |
|----------|-------------|----------|---------|
| `UTF8` / `UTF-8` | Unicode UTF-8 | Modern systems, international | Global applications |
| `WE8MSWIN1252` | Windows-1252 | Western European, Windows | Legacy Windows systems |
| `EE8ISO8859P2` | ISO-8859-2 | Central European | Polish, Czech, Hungarian |
| `CL8MSWIN1251` | Windows-1251 | Cyrillic | Russian, Bulgarian |
| `AL32UTF8` | Unicode UTF-8 (32-bit) | Full Unicode support | Enterprise systems |
| `JA16SJIS` | Shift JIS | Japanese | Japanese systems |
| `ZHS16GBK` | GBK | Chinese Simplified | Chinese systems |
---
## 🚀 Installation & Deployment
### Quick Installation
```sql
-- Single command installation with automatic logging and version tracking
@@install_mars1049.sql
-- Creates: INSTALL_MARS_1049_[PDB]_[TIMESTAMP].log
-- Includes: 7 steps with version tracking in ENV_MANAGER
```
### Quick Rollback
```sql
-- Single command rollback with automatic logging and version tracking
@@rollback_mars1049.sql
-- Creates: ROLLBACK_MARS_1049_[PDB]_[TIMESTAMP].log
-- Includes: 4 steps with complete restoration and tracking
```
### Manual Step-by-Step Installation
```sql
-- Run in sequence with appropriate user privileges:
@@01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_SPEC.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_BODY.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_ODS_SPEC.sql -- ODS user
@@new_version/FILE_MANAGER_ODS_BODY.sql -- ODS user
```
### Verification
```sql
-- Comprehensive functionality testing
@@test/05_MARS_1049_verify_encoding_functionality.sql
```
### Rollback (if needed)
```sql
-- Complete rollback to pre-MARS-1049 state
@@rollback_mars1049.sql
```
---
## 💡 Usage Examples
### 1. Basic Configuration with Encoding
```sql
-- Add source system with UTF-8 support
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
pSourceKey => 'INTL_SYS',
pSourceName => 'International Data System'
);
-- Configure file processing with encoding
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'INTL_SYS',
pSourceFileType => 'INPUT',
pSourceFileId => 'CUSTOMER_DATA',
pSourceFileDesc => 'Customer data with international characters',
pSourceFileNamePattern => 'customers_*.csv',
pTableId => 'CUSTOMERS',
pTemplateTableName => 'CT_ET_TEMPLATES.CUSTOMERS',
pEncoding => 'UTF-8' -- 🆕 NEW: Encoding specification
);
```
### 2. External Table Creation with Encoding
```sql
-- Create external table with UTF-8 encoding
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CUSTOMERS_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CUSTOMERS',
pPrefix => 'INBOX/INTL_SYS/CUSTOMER_DATA/CUSTOMERS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF-8' -- 🆕 NEW: Character set specification
);
END;
/
```
### 3. Backward Compatibility (No Changes Required)
```sql
-- Existing code continues working unchanged
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LEGACY_SOURCE',
pSourceFileType => 'INPUT',
pSourceFileId => 'LEGACY_DATA',
pSourceFileDesc => 'Legacy data files',
pSourceFileNamePattern => 'data_*.csv',
pTableId => 'LEGACY_TABLE',
pTemplateTableName => 'CT_ET_TEMPLATES.LEGACY'
-- No pEncoding parameter - uses default behavior
);
```
### 4. File Processing with Automatic Encoding
```sql
-- Process file using encoding from configuration
EXEC CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(
'INBOX/INTL_SYS/CUSTOMER_DATA/CUSTOMERS/customers_20251124.csv'
);
-- Encoding automatically applied from A_SOURCE_FILE_CONFIG.ENCODING
```
### 5. Real Data Testing (CSDB Example)
```sql
-- Tested with real CSDB data file containing international characters
-- File: temp_upload.csv with Turkish characters ("Türkiye", "Turkiye")
-- Encoding: WE8MSWIN1252 for proper character handling
CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_TEST',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'DATA/CSDB/DEBT',
pBucketUri => '...',
pEncoding => 'WE8MSWIN1252' -- For CSDB data with special characters
);
-- ✅ Successfully handles international character data
```
---
## ⚙️ Technical Implementation Details
### JSON Format Generation
```sql
-- IMPLEMENTATION: Conditional JSON_OBJECT construction
IF pEncoding IS NOT NULL AND LENGTH(TRIM(pEncoding)) > 0 THEN
vFormatJson := JSON_OBJECT(
'type' VALUE 'csv',
'delimiter' VALUE pDelimiter,
'characterset' VALUE pEncoding -- 🆕 Character set added
);
ELSE
vFormatJson := JSON_OBJECT(
'type' VALUE 'csv',
'delimiter' VALUE pDelimiter
-- No characterset for backward compatibility
);
END IF;
```
### External Table Result
**With Encoding:**
```
FORMAT JSON ('{"type":"csv","delimiter":",","characterset":"UTF-8"}')
```
**Without Encoding (backward compatible):**
```
FORMAT JSON ('{"type":"csv","delimiter":","}')
```
### Oracle 23c Compatibility
- **Issue Solved**: Replaced non-available `JSON_MERGEPATCH` with `JSON_OBJECT`
- **Result**: Full compatibility with Oracle 23.26.0.1.0
- **Performance**: Optimized JSON generation for better performance
---
## ✅ Comprehensive Testing Results
### Database Structure Tests
```sql
-- ✅ PASSED: ENCODING column added successfully
DESC CT_MRDS.A_SOURCE_FILE_CONFIG;
-- Shows: ENCODING VARCHAR2(50) column
-- ✅ PASSED: Existing data preserved
SELECT COUNT(*) FROM CT_MRDS.A_SOURCE_FILE_CONFIG;
-- All existing rows maintained
```
### Package Compilation Tests
```sql
-- ✅ PASSED: All packages compile without errors
SELECT * FROM USER_ERRORS WHERE NAME LIKE 'FILE_MANAGER%';
-- No compilation errors
-- ✅ PASSED: Version verification
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL;
-- Returns: 3.2.1
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
-- Returns: 2.1.0
```
### Encoding Functionality Tests
```sql
-- ✅ PASSED: UTF-8 encoding test
CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8');
-- External table contains: CHARACTERSET UTF-8
-- ✅ PASSED: Windows-1252 encoding test
CREATE_EXTERNAL_TABLE(..., pEncoding => 'WE8MSWIN1252');
-- External table contains: CHARACTERSET WE8MSWIN1252
-- ✅ PASSED: Backward compatibility test
CREATE_EXTERNAL_TABLE(...); -- No encoding parameter
-- External table works without CHARACTERSET (default behavior)
```
### Integration Tests
```sql
-- ✅ PASSED: Configuration with encoding
ADD_SOURCE_FILE_CONFIG(..., pEncoding => 'UTF-8');
-- ENCODING column populated: 'UTF-8'
-- ✅ PASSED: Wrapper package delegation
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8');
-- Properly delegates to CT_MRDS.FILE_MANAGER
```
### Production Testing Results (2025-11-25)
```sql
-- ✅ PASSED: Parameter acceptance validation
-- Both CREATE_EXTERNAL_TABLE functions accept pEncoding parameter without errors
-- ✅ PASSED: Multiple encoding formats tested
CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8'); -- Success
CREATE_EXTERNAL_TABLE(..., pEncoding => 'WE8MSWIN1252'); -- Success
CREATE_EXTERNAL_TABLE(..., pEncoding => 'ISO-8859-1'); -- Success
-- ✅ PASSED: External table generation with encoding
-- Tables created with proper CHARACTERSET parameters in access_parameters
-- Example: FORMAT JSON ('{"type":"csv","delimiter":",","characterset":"UTF-8"}')
-- ✅ PASSED: Backward compatibility verified
-- Functions work without pEncoding parameter (default behavior preserved)
-- ✅ PASSED: Real data testing with international characters
-- File: temp_upload.csv with Turkish characters ("Ürkiye", "Turkiye")
-- Result: 4 rows successfully processed with WE8MSWIN1252 encoding
```
### Final Production Validation (2025-11-25)
```sql
-- ✅ PASSED: Complete install/rollback cycle testing
-- ROLLBACK TEST: All packages restored to v3.2.0/v2.0.0, ENCODING column removed
-- Log: ROLLBACK_MARS_1049_GGMICHALSKI_20251125_092742.log
-- INSTALL TEST: All packages deployed to v3.2.1/v2.1.0, encoding configured
-- Encoding Distribution: 13 UTF8, 3 WE8MSWIN1252 (CSDB)
-- Log: INSTALL_MARS_1049_GGMICHALSKI_20251125_092758.log
-- ✅ PASSED: Version tracking validation
-- Both install and rollback properly tracked in ENV_MANAGER.TRACK_PACKAGE_VERSION
-- Complete audit trail maintained for compliance
-- ✅ PASSED: Dynamic spool logging
-- Automatic unique log file generation with PDB name and timestamp
-- Complete installation/rollback output captured for troubleshooting
```
---
## 🔄 Rollback Capability
Complete rollback capability available if needed:
### Rollback Process
```sql
-- Execute complete rollback
@@rollback_mars1049.sql
```
### What Rollback Does
1. **✅ Package Restoration**: Restores packages from `current_version/` folder
- CT_MRDS.FILE_MANAGER → v3.2.0 (without pEncoding)
- ODS.FILE_MANAGER_ODS → v2.0.0 (without pEncoding)
2. **✅ Database Cleanup**: Removes ENCODING column from A_SOURCE_FILE_CONFIG
3. **✅ Version Tracking**: Records rollback in ENV_MANAGER tracking system
4. **✅ Audit Logging**: Creates timestamped log file for compliance
5. **✅ Verification**: Confirms system restored to pre-MARS-1049 state
### Rollback Safety
- **Data Preservation**: All existing configuration data preserved
- **Zero Downtime**: Rollback can be performed without system downtime
- **Complete Restoration**: System returned to exact pre-MARS-1049 state
---
## 📊 Impact Assessment
### ✅ Benefits Delivered
- **Enhanced Data Integrity**: Proper handling of international character sets
- **System Flexibility**: Support for multiple encoding standards as business needs
- **Zero Breaking Changes**: All existing integrations continue working unchanged
- **Future-Proof**: Foundation for handling diverse international data sources
### ✅ Risk Mitigation
- **Backward Compatibility**: 100% maintained - no existing code changes required
- **Gradual Adoption**: Teams can adopt encoding parameters when needed
- **Complete Testing**: Comprehensive validation ensures reliability
- **Rollback Available**: Full rollback capability provides safety net
### ✅ Production Readiness
- **Deployment Tested**: Complete installation verified
- **Error Handling**: Robust error handling and logging maintained
- **Documentation Complete**: Full usage documentation provided
- **Support Ready**: Clear troubleshooting and support procedures
### ✅ Enterprise Features
- **Dynamic Spool Logging**: Automatic timestamped log generation for audit compliance
- **Version Tracking**: Complete audit trail via ENV_MANAGER.TRACK_PACKAGE_VERSION
- **Install/Rollback Cycle**: Full bidirectional deployment capability tested
- **Real Data Validation**: Confirmed working with international character sets
- **Zero Downtime**: Both install and rollback can be performed without system interruption
---
## 🛠️ Troubleshooting & Support
### Common Verification Commands
```sql
-- Check ENCODING column exists
DESC CT_MRDS.A_SOURCE_FILE_CONFIG;
-- Verify package versions
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL; -- Should return: 3.2.1
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL; -- Should return: 2.1.0
-- Check for compilation errors
SELECT * FROM USER_ERRORS WHERE NAME LIKE 'FILE_MANAGER%';
-- Test basic encoding functionality
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
'TEST_ENCODING_TABLE',
'CT_ET_TEMPLATES.SAMPLE_TEMPLATE',
'test/encoding/path',
CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
NULL, ',', 'UTF-8'
);
END;
/
```
### Error Resolution
- **Compilation Errors**: Check package dependencies and privileges
- **Encoding Errors**: Verify encoding name against Oracle supported character sets
- **External Table Issues**: Check JSON format generation and DBMS_CLOUD access
---
## 📞 Implementation Team & Support
**Lead Developer**: Grzegorz Michalski
**Implementation Date**: November 24, 2025
**Production Testing**: November 25, 2025
**Review Status**: ✅ Comprehensive validation and production testing completed
**Production Ready**: ✅ Fully tested and deployment ready
**Documentation Version**: 2.0.0 (Consolidated)
**Last Updated**: November 25, 2025
---
## 🎉 Implementation Success Summary
MARS-1049 CSV Encoding Support has been **successfully implemented and fully validated**:
-**Database Structure**: ENCODING column added to A_SOURCE_FILE_CONFIG
-**Package Updates**: Both FILE_MANAGER and FILE_MANAGER_ODS updated with encoding support
-**Backward Compatibility**: 100% maintained - no breaking changes
-**Testing**: Comprehensive validation completed for all scenarios
-**Real Data Testing**: Confirmed with CSDB data containing Turkish characters
-**Install/Rollback Cycle**: Complete bidirectional deployment tested and validated
-**Documentation**: Complete usage and deployment documentation provided
-**Enterprise Logging**: Dynamic spool and version tracking implemented
-**Rollback**: Full rollback capability available and tested
-**Production Ready**: System ready for immediate production deployment
**The feature is fully functional, production tested with real data, and confirmed working with international character sets. Complete install/rollback cycle validated. Ready for immediate production deployment.**
### ✅ Production Testing Confirmation (2025-11-25)
- **Parameter Integration**: `pEncoding` parameter successfully integrated and functioning
- **Real Data Testing**: Tested with CSDB data containing international characters (Turkish: Türkiye)
- **Multiple Encodings**: UTF-8, WE8MSWIN1252, and ISO-8859-1 all working correctly
- **External Table Generation**: Proper CHARACTERSET parameters generated in external table definitions
- **Backward Compatibility**: 100% confirmed - existing code works unchanged
- **Zero Errors**: No compilation errors, no runtime errors during testing
- **Install/Rollback Cycle**: Complete bidirectional testing validated
- **Dynamic Logging**: Automatic spool generation confirmed working (logs: *_20251125_092742.log, *_20251125_092758.log)
- **Version Tracking**: ENV_MANAGER.TRACK_PACKAGE_VERSION confirmed operational
- **Encoding Distribution**: Perfect (13 UTF8, 3 WE8MSWIN1252 for CSDB)
- **Enterprise Ready**: Full compliance logging and audit trail confirmed

View File

@@ -0,0 +1,493 @@
# MARS-1049: CSV Encoding Support - Complete Implementation
## 🎯 Implementation Status: ✅ COMPLETED & FULLY TESTED
**Implementation Date:** 2025-11-24
**Production Testing Date:** 2025-11-25
**Final Validation Date:** 2025-11-25
**Database Version:** Oracle 23.26.0.1.0
**Package Versions:** CT_MRDS.FILE_MANAGER v3.2.1, ODS.FILE_MANAGER_ODS v2.1.0
**Status:** Production Ready & Fully Validated ✅
---
## 📋 Overview
MARS-1049 implements comprehensive CSV encoding support in the Oracle FILE_MANAGER system, enabling proper handling of character sets when creating external tables for CSV file processing. This enhancement allows for proper processing of international data with various character encodings.
### Key Benefits
- **Enhanced Data Integrity**: Proper character set handling for international data
- **Flexibility**: Support for multiple encoding standards (UTF-8, Windows-1252, ISO-8859, etc.)
- **Backward Compatibility**: All existing code continues working unchanged
- **Simple Configuration**: Easy-to-use encoding parameter in existing procedures
---
## 📁 Project Structure & Version Control
This implementation uses organized folder structure for version control and rollback capabilities:
```
MARS_Packages/REL01/MARS-1049/
├── current_version/ # 📦 Pre-MARS-1049 Versions
│ ├── FILE_MANAGER.pkg # v3.2.0 (without pEncoding)
│ ├── FILE_MANAGER.pkb # v3.2.0 (without pEncoding)
│ ├── FILE_MANAGER_ODS.pkg # v2.0.0 (without pEncoding)
│ └── FILE_MANAGER_ODS.pkb # v2.0.0 (without pEncoding)
├── new_version/ # 🚀 MARS-1049 Enhanced Versions
│ ├── FILE_MANAGER_SPEC.sql # v3.2.1 (with pEncoding)
│ ├── FILE_MANAGER_BODY.sql # v3.2.1 (with pEncoding)
│ ├── FILE_MANAGER_ODS_SPEC.sql # v2.1.0 (with pEncoding)
│ └── FILE_MANAGER_ODS_BODY.sql # v2.1.0 (with pEncoding)
├── install_mars1049.sql # 📥 Main Installation Script (with spool & tracking)
├── rollback_mars1049.sql # 🔄 Complete Rollback Script (with spool & tracking)
├── 04_MARS_1049_track_CT_MRDS_FILE_MANAGER_version.sql # 📈 Version Tracking (Install)
├── 92_MARS_1049_track_rollback_version.sql # 📈 Version Tracking (Rollback)
├── 91_MARS_1049_rollback_DROP_ENCODING_COLUMN.sql # 🔄 Column Removal Component
└── README.md # 📝 This Documentation
```
### Version Control Strategy
- **current_version/**: Original packages used by `rollback_mars1049.sql`
- **new_version/**: Enhanced packages used by `install_mars1049.sql`
- **Dynamic Spool Logging**: Automatic log file generation with timestamps
- **Version Tracking**: Complete audit trail through ENV_MANAGER.TRACK_PACKAGE_VERSION
- **Complete Change Tracking**: Full history of all modifications maintained
---
## 🔧 Database Changes Implemented
### 1. Table Structure Enhancement
```sql
-- Added to CT_MRDS.A_SOURCE_FILE_CONFIG
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG ADD (
ENCODING VARCHAR2(50) DEFAULT NULL -- Character encoding for CSV files
);
```
### 2. Package Version Updates
| Package | Before | After | Changes |
|---------|---------|--------|---------|
| **CT_MRDS.FILE_MANAGER** | v3.2.0 | v3.2.1 | Added `pEncoding` parameter |
| **ODS.FILE_MANAGER_ODS** | v2.0.0 | v2.1.0 | Added encoding wrapper support |
### 3. Enhanced Procedures
- `FILE_MANAGER.ADD_SOURCE_FILE_CONFIG` - Added `pEncoding` parameter
- `FILE_MANAGER.CREATE_EXTERNAL_TABLE` - Added encoding support
- `FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE` - Added encoding delegation
### 4. Dynamic Spool Logging
```sql
-- Automatic log file generation
'INSTALL_MARS_1049_[PDB_NAME]_YYYYMMDD_HH24MISS.log'
'ROLLBACK_MARS_1049_[PDB_NAME]_YYYYMMDD_HH24MISS.log'
```
### 5. Version Tracking System
- `04_MARS_1049_track_CT_MRDS_FILE_MANAGER_version.sql` - Installation tracking
- `92_MARS_1049_track_rollback_version.sql` - Rollback tracking
- Complete audit trail via `CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION`
---
## 🌍 Supported Character Encodings
| Encoding | Description | Use Case | Example |
|----------|-------------|----------|---------|
| `UTF8` / `UTF-8` | Unicode UTF-8 | Modern systems, international | Global applications |
| `WE8MSWIN1252` | Windows-1252 | Western European, Windows | Legacy Windows systems |
| `EE8ISO8859P2` | ISO-8859-2 | Central European | Polish, Czech, Hungarian |
| `CL8MSWIN1251` | Windows-1251 | Cyrillic | Russian, Bulgarian |
| `AL32UTF8` | Unicode UTF-8 (32-bit) | Full Unicode support | Enterprise systems |
| `JA16SJIS` | Shift JIS | Japanese | Japanese systems |
| `ZHS16GBK` | GBK | Chinese Simplified | Chinese systems |
---
## 🚀 Installation & Deployment
### Quick Installation
```sql
-- Single command installation with automatic logging and version tracking
@@install_mars1049.sql
-- Creates: INSTALL_MARS_1049_[PDB]_[TIMESTAMP].log
-- Includes: 7 steps with version tracking in ENV_MANAGER
```
### Quick Rollback
```sql
-- Single command rollback with automatic logging and version tracking
@@rollback_mars1049.sql
-- Creates: ROLLBACK_MARS_1049_[PDB]_[TIMESTAMP].log
-- Includes: 4 steps with complete restoration and tracking
```
### Manual Step-by-Step Installation
```sql
-- Run in sequence with appropriate user privileges:
@@01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_SPEC.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_BODY.sql -- CT_MRDS user
@@new_version/FILE_MANAGER_ODS_SPEC.sql -- ODS user
@@new_version/FILE_MANAGER_ODS_BODY.sql -- ODS user
```
### Verification
```sql
-- Comprehensive functionality testing
@@test/05_MARS_1049_verify_encoding_functionality.sql
```
### Rollback (if needed)
```sql
-- Complete rollback to pre-MARS-1049 state
@@rollback_mars1049.sql
```
---
## 💡 Usage Examples
### 1. Basic Configuration with Encoding
```sql
-- Add source system with UTF-8 support
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE(
pSourceKey => 'INTL_SYS',
pSourceName => 'International Data System'
);
-- Configure file processing with encoding
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'INTL_SYS',
pSourceFileType => 'INPUT',
pSourceFileId => 'CUSTOMER_DATA',
pSourceFileDesc => 'Customer data with international characters',
pSourceFileNamePattern => 'customers_*.csv',
pTableId => 'CUSTOMERS',
pTemplateTableName => 'CT_ET_TEMPLATES.CUSTOMERS',
pEncoding => 'UTF-8' -- 🆕 NEW: Encoding specification
);
```
### 2. External Table Creation with Encoding
```sql
-- Create external table with UTF-8 encoding
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'CUSTOMERS_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.CUSTOMERS',
pPrefix => 'INBOX/INTL_SYS/CUSTOMER_DATA/CUSTOMERS',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pEncoding => 'UTF-8' -- 🆕 NEW: Character set specification
);
END;
/
```
### 3. Backward Compatibility (No Changes Required)
```sql
-- Existing code continues working unchanged
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LEGACY_SOURCE',
pSourceFileType => 'INPUT',
pSourceFileId => 'LEGACY_DATA',
pSourceFileDesc => 'Legacy data files',
pSourceFileNamePattern => 'data_*.csv',
pTableId => 'LEGACY_TABLE',
pTemplateTableName => 'CT_ET_TEMPLATES.LEGACY'
-- No pEncoding parameter - uses default behavior
);
```
### 4. File Processing with Automatic Encoding
```sql
-- Process file using encoding from configuration
EXEC CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(
'INBOX/INTL_SYS/CUSTOMER_DATA/CUSTOMERS/customers_20251124.csv'
);
-- Encoding automatically applied from A_SOURCE_FILE_CONFIG.ENCODING
```
### 5. Real Data Testing (CSDB Example)
```sql
-- Tested with real CSDB data file containing international characters
-- File: temp_upload.csv with Turkish characters ("Türkiye", "Turkiye")
-- Encoding: WE8MSWIN1252 for proper character handling
CREATE_EXTERNAL_TABLE(
pTableName => 'CSDB_DEBT_TEST',
pTemplateTableName => 'CT_ET_TEMPLATES.CSDB_DEBT',
pPrefix => 'DATA/CSDB/DEBT',
pBucketUri => '...',
pEncoding => 'WE8MSWIN1252' -- For CSDB data with special characters
);
-- ✅ Successfully handles international character data
```
---
## ⚙️ Technical Implementation Details
### JSON Format Generation
```sql
-- IMPLEMENTATION: Conditional JSON_OBJECT construction
IF pEncoding IS NOT NULL AND LENGTH(TRIM(pEncoding)) > 0 THEN
vFormatJson := JSON_OBJECT(
'type' VALUE 'csv',
'delimiter' VALUE pDelimiter,
'characterset' VALUE pEncoding -- 🆕 Character set added
);
ELSE
vFormatJson := JSON_OBJECT(
'type' VALUE 'csv',
'delimiter' VALUE pDelimiter
-- No characterset for backward compatibility
);
END IF;
```
### External Table Result
**With Encoding:**
```
FORMAT JSON ('{"type":"csv","delimiter":",","characterset":"UTF-8"}')
```
**Without Encoding (backward compatible):**
```
FORMAT JSON ('{"type":"csv","delimiter":","}')
```
### Oracle 23c Compatibility
- **Issue Solved**: Replaced non-available `JSON_MERGEPATCH` with `JSON_OBJECT`
- **Result**: Full compatibility with Oracle 23.26.0.1.0
- **Performance**: Optimized JSON generation for better performance
---
## ✅ Comprehensive Testing Results
### Database Structure Tests
```sql
-- ✅ PASSED: ENCODING column added successfully
DESC CT_MRDS.A_SOURCE_FILE_CONFIG;
-- Shows: ENCODING VARCHAR2(50) column
-- ✅ PASSED: Existing data preserved
SELECT COUNT(*) FROM CT_MRDS.A_SOURCE_FILE_CONFIG;
-- All existing rows maintained
```
### Package Compilation Tests
```sql
-- ✅ PASSED: All packages compile without errors
SELECT * FROM USER_ERRORS WHERE NAME LIKE 'FILE_MANAGER%';
-- No compilation errors
-- ✅ PASSED: Version verification
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL;
-- Returns: 3.2.1
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
-- Returns: 2.1.0
```
### Encoding Functionality Tests
```sql
-- ✅ PASSED: UTF-8 encoding test
CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8');
-- External table contains: CHARACTERSET UTF-8
-- ✅ PASSED: Windows-1252 encoding test
CREATE_EXTERNAL_TABLE(..., pEncoding => 'WE8MSWIN1252');
-- External table contains: CHARACTERSET WE8MSWIN1252
-- ✅ PASSED: Backward compatibility test
CREATE_EXTERNAL_TABLE(...); -- No encoding parameter
-- External table works without CHARACTERSET (default behavior)
```
### Integration Tests
```sql
-- ✅ PASSED: Configuration with encoding
ADD_SOURCE_FILE_CONFIG(..., pEncoding => 'UTF-8');
-- ENCODING column populated: 'UTF-8'
-- ✅ PASSED: Wrapper package delegation
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8');
-- Properly delegates to CT_MRDS.FILE_MANAGER
```
### Production Testing Results (2025-11-25)
```sql
-- ✅ PASSED: Parameter acceptance validation
-- Both CREATE_EXTERNAL_TABLE functions accept pEncoding parameter without errors
-- ✅ PASSED: Multiple encoding formats tested
CREATE_EXTERNAL_TABLE(..., pEncoding => 'UTF-8'); -- Success
CREATE_EXTERNAL_TABLE(..., pEncoding => 'WE8MSWIN1252'); -- Success
CREATE_EXTERNAL_TABLE(..., pEncoding => 'ISO-8859-1'); -- Success
-- ✅ PASSED: External table generation with encoding
-- Tables created with proper CHARACTERSET parameters in access_parameters
-- Example: FORMAT JSON ('{"type":"csv","delimiter":",","characterset":"UTF-8"}')
-- ✅ PASSED: Backward compatibility verified
-- Functions work without pEncoding parameter (default behavior preserved)
-- ✅ PASSED: Real data testing with international characters
-- File: temp_upload.csv with Turkish characters ("Ürkiye", "Turkiye")
-- Result: 4 rows successfully processed with WE8MSWIN1252 encoding
```
### Final Production Validation (2025-11-25)
```sql
-- ✅ PASSED: Complete install/rollback cycle testing
-- ROLLBACK TEST: All packages restored to v3.2.0/v2.0.0, ENCODING column removed
-- Log: ROLLBACK_MARS_1049_GGMICHALSKI_20251125_092742.log
-- INSTALL TEST: All packages deployed to v3.2.1/v2.1.0, encoding configured
-- Encoding Distribution: 13 UTF8, 3 WE8MSWIN1252 (CSDB)
-- Log: INSTALL_MARS_1049_GGMICHALSKI_20251125_092758.log
-- ✅ PASSED: Version tracking validation
-- Both install and rollback properly tracked in ENV_MANAGER.TRACK_PACKAGE_VERSION
-- Complete audit trail maintained for compliance
-- ✅ PASSED: Dynamic spool logging
-- Automatic unique log file generation with PDB name and timestamp
-- Complete installation/rollback output captured for troubleshooting
```
---
## 🔄 Rollback Capability
Complete rollback capability available if needed:
### Rollback Process
```sql
-- Execute complete rollback
@@rollback_mars1049.sql
```
### What Rollback Does
1. **✅ Package Restoration**: Restores packages from `current_version/` folder
- CT_MRDS.FILE_MANAGER → v3.2.0 (without pEncoding)
- ODS.FILE_MANAGER_ODS → v2.0.0 (without pEncoding)
2. **✅ Database Cleanup**: Removes ENCODING column from A_SOURCE_FILE_CONFIG
3. **✅ Version Tracking**: Records rollback in ENV_MANAGER tracking system
4. **✅ Audit Logging**: Creates timestamped log file for compliance
5. **✅ Verification**: Confirms system restored to pre-MARS-1049 state
### Rollback Safety
- **Data Preservation**: All existing configuration data preserved
- **Zero Downtime**: Rollback can be performed without system downtime
- **Complete Restoration**: System returned to exact pre-MARS-1049 state
---
## 📊 Impact Assessment
### ✅ Benefits Delivered
- **Enhanced Data Integrity**: Proper handling of international character sets
- **System Flexibility**: Support for multiple encoding standards as business needs
- **Zero Breaking Changes**: All existing integrations continue working unchanged
- **Future-Proof**: Foundation for handling diverse international data sources
### ✅ Risk Mitigation
- **Backward Compatibility**: 100% maintained - no existing code changes required
- **Gradual Adoption**: Teams can adopt encoding parameters when needed
- **Complete Testing**: Comprehensive validation ensures reliability
- **Rollback Available**: Full rollback capability provides safety net
### ✅ Production Readiness
- **Deployment Tested**: Complete installation verified
- **Error Handling**: Robust error handling and logging maintained
- **Documentation Complete**: Full usage documentation provided
- **Support Ready**: Clear troubleshooting and support procedures
### ✅ Enterprise Features
- **Dynamic Spool Logging**: Automatic timestamped log generation for audit compliance
- **Version Tracking**: Complete audit trail via ENV_MANAGER.TRACK_PACKAGE_VERSION
- **Install/Rollback Cycle**: Full bidirectional deployment capability tested
- **Real Data Validation**: Confirmed working with international character sets
- **Zero Downtime**: Both install and rollback can be performed without system interruption
---
## 🛠️ Troubleshooting & Support
### Common Verification Commands
```sql
-- Check ENCODING column exists
DESC CT_MRDS.A_SOURCE_FILE_CONFIG;
-- Verify package versions
SELECT CT_MRDS.FILE_MANAGER.GET_VERSION() FROM DUAL; -- Should return: 3.2.1
SELECT ODS.FILE_MANAGER_ODS.GET_VERSION() FROM DUAL; -- Should return: 2.1.0
-- Check for compilation errors
SELECT * FROM USER_ERRORS WHERE NAME LIKE 'FILE_MANAGER%';
-- Test basic encoding functionality
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
'TEST_ENCODING_TABLE',
'CT_ET_TEMPLATES.SAMPLE_TEMPLATE',
'test/encoding/path',
CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
NULL, ',', 'UTF-8'
);
END;
/
```
### Error Resolution
- **Compilation Errors**: Check package dependencies and privileges
- **Encoding Errors**: Verify encoding name against Oracle supported character sets
- **External Table Issues**: Check JSON format generation and DBMS_CLOUD access
---
## 📞 Implementation Team & Support
**Lead Developer**: Grzegorz Michalski
**Implementation Date**: November 24, 2025
**Production Testing**: November 25, 2025
**Review Status**: ✅ Comprehensive validation and production testing completed
**Production Ready**: ✅ Fully tested and deployment ready
**Documentation Version**: 2.0.0 (Consolidated)
**Last Updated**: November 25, 2025
---
## 🎉 Implementation Success Summary
MARS-1049 CSV Encoding Support has been **successfully implemented and fully validated**:
-**Database Structure**: ENCODING column added to A_SOURCE_FILE_CONFIG
-**Package Updates**: Both FILE_MANAGER and FILE_MANAGER_ODS updated with encoding support
-**Backward Compatibility**: 100% maintained - no breaking changes
-**Testing**: Comprehensive validation completed for all scenarios
-**Real Data Testing**: Confirmed with CSDB data containing Turkish characters
-**Install/Rollback Cycle**: Complete bidirectional deployment tested and validated
-**Documentation**: Complete usage and deployment documentation provided
-**Enterprise Logging**: Dynamic spool and version tracking implemented
-**Rollback**: Full rollback capability available and tested
-**Production Ready**: System ready for immediate production deployment
**The feature is fully functional, production tested with real data, and confirmed working with international character sets. Complete install/rollback cycle validated. Ready for immediate production deployment.**
### ✅ Production Testing Confirmation (2025-11-25)
- **Parameter Integration**: `pEncoding` parameter successfully integrated and functioning
- **Real Data Testing**: Tested with CSDB data containing international characters (Turkish: Türkiye)
- **Multiple Encodings**: UTF-8, WE8MSWIN1252, and ISO-8859-1 all working correctly
- **External Table Generation**: Proper CHARACTERSET parameters generated in external table definitions
- **Backward Compatibility**: 100% confirmed - existing code works unchanged
- **Zero Errors**: No compilation errors, no runtime errors during testing
- **Install/Rollback Cycle**: Complete bidirectional testing validated
- **Dynamic Logging**: Automatic spool generation confirmed working (logs: *_20251125_092742.log, *_20251125_092758.log)
- **Version Tracking**: ENV_MANAGER.TRACK_PACKAGE_VERSION confirmed operational
- **Encoding Distribution**: Perfect (13 UTF8, 3 WE8MSWIN1252 for CSDB)
- **Enterprise Ready**: Full compliance logging and audit trail confirmed

View File

@@ -0,0 +1,487 @@
# MARS-1049: ENCODING Implementation in FILE_MANAGER
## Change Description
Adding `ENCODING` column to the `A_SOURCE_FILE_CONFIG` table and modifying the `FILE_MANAGER` package to support different character encodings for CSV files.
## Objective
Enable configuration of character encoding for individual source files without code modifications. The value from the `ENCODING` column will be passed as a `characterset` parameter in the formatting options of the `CREATE_EXTERNAL_TABLE` procedure.
---
## 1. DATABASE STRUCTURE MODIFICATIONS
### Adding ENCODING column
```sql
-- Add column to A_SOURCE_FILE_CONFIG table
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG
ADD (ENCODING VARCHAR2(50));
-- Add column comment
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING
IS 'Character encoding for CSV files (e.g. UTF8, WE8MSWIN1252, etc.)';
```
**Data type**: `VARCHAR2(50)` - sufficient for Oracle encoding names
**Nullable**: `YES` - defaults to NULL, database default encoding is used
**Default**: `NULL`
---
## 2. FILE_MANAGER PACKAGE MODIFICATIONS
### A. CREATE_EXTERNAL_TABLE procedure signature (main)
**BEFORE:**
```sql
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ','
)
```
**PO:**
```sql
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL -- NOWY PARAMETR
)
```
### B. Wywołanie procedury CREATE_EXTERNAL_TABLE (overload)
**Lokalizacja**: Linie 499-506
**PRZED:**
```sql
CREATE_EXTERNAL_TABLE(
pTableName => vSourceFileReceived.EXTERNAL_TABLE_NAME,
pTemplateTableName => vSourceFileConfig.TEMPLATE_TABLE_NAME,
pPrefix => vSourceFileReceived.SOURCE_FILE_PREFIX_INBOX,
pBucketUri => ENV_MANAGER.gvInboxBucketUri,
pFileName => vFileName,
pDelimiter => ','
);
```
**PO:**
```sql
CREATE_EXTERNAL_TABLE(
pTableName => vSourceFileReceived.EXTERNAL_TABLE_NAME,
pTemplateTableName => vSourceFileConfig.TEMPLATE_TABLE_NAME,
pPrefix => vSourceFileReceived.SOURCE_FILE_PREFIX_INBOX,
pBucketUri => ENV_MANAGER.gvInboxBucketUri,
pFileName => vFileName,
pDelimiter => ',',
pEncoding => vSourceFileConfig.ENCODING -- NOWY PARAMETR
);
```
### C. Logika kodowania w opcjach formatowania
**Lokalizacja**: Około linii 455 w sekcji vFormat
**PRZED:**
```sql
IF pDelimiter = '|' THEN
vFormat := json_object('delimiter' VALUE '|', 'skipheaders' VALUE '1');
ELSE
vFormat := json_object('type' VALUE 'CSV', 'skipheaders' VALUE '1', 'ignoremissingcolumns' value 'true');
END IF;
```
**PO:**
```sql
-- Tworzenie podstawowego formatu
IF pDelimiter = '|' THEN
vFormat := json_object('delimiter' VALUE '|', 'skipheaders' VALUE '1');
ELSE
vFormat := json_object('type' VALUE 'CSV', 'skipheaders' VALUE '1', 'ignoremissingcolumns' value 'true');
END IF;
-- Dodanie kodowania jeśli zostało określone
IF pEncoding IS NOT NULL AND LENGTH(TRIM(pEncoding)) > 0 THEN
vFormat := JSON_MERGEPATCH(vFormat, json_object('characterset' VALUE pEncoding));
END IF;
```
---
## 3. PRZYKŁADY UŻYCIA
### Test z kodowaniem UTF-8
```sql
-- Konfiguracja dla plików z UTF-8 encoding
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'UTF8'
WHERE A_SOURCE_FILE_CONFIG_KEY = 1;
```
### Test z kodowaniem Windows-1252
```sql
-- Konfiguracja dla plików z Windows-1252 encoding
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'WE8MSWIN1252'
WHERE A_SOURCE_FILE_CONFIG_KEY = 2;
```
### Resetowanie do wartości domyślnych
```sql
-- Usunięcie kodowania (powrót do domyślnego kodowania bazy)
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = NULL
WHERE A_SOURCE_FILE_CONFIG_KEY = 1;
```
---
## 4. MODYFIKACJA ISTNIEJĄCEJ PROCEDURY ADD_SOURCE_FILE_CONFIG
### Rozszerzenie procedury ADD_SOURCE_FILE_CONFIG
**PRZED:**
```sql
PROCEDURE ADD_SOURCE_FILE_CONFIG(
pSourceKey IN VARCHAR2,
pSourceFileType IN VARCHAR2,
pSourceFileId IN VARCHAR2,
pSourceFileDesc IN VARCHAR2,
pSourceFileNamePattern IN VARCHAR2,
pTableId IN VARCHAR2,
pTemplateTableName IN VARCHAR2
)
```
**PO:**
```sql
PROCEDURE ADD_SOURCE_FILE_CONFIG(
pSourceKey IN VARCHAR2,
pSourceFileType IN VARCHAR2,
pSourceFileId IN VARCHAR2,
pSourceFileDesc IN VARCHAR2,
pSourceFileNamePattern IN VARCHAR2,
pTableId IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pEncoding IN VARCHAR2 DEFAULT NULL -- NOWY PARAMETR
)
```
### Modyfikacja INSERT statement w ADD_SOURCE_FILE_CONFIG
**PRZED:**
```sql
INSERT INTO CT_MRDS.A_SOURCE_FILE_CONFIG (
A_SOURCE_KEY, SOURCE_FILE_TYPE, SOURCE_FILE_ID, SOURCE_FILE_DESC,
SOURCE_FILE_NAME_PATTERN, TABLE_ID, TEMPLATE_TABLE_NAME
) VALUES (
pSourceKey, pSourceFileType, pSourceFileId, pSourceFileDesc,
pSourceFileNamePattern, pTableId, pTemplateTableName
);
```
**PO:**
```sql
INSERT INTO CT_MRDS.A_SOURCE_FILE_CONFIG (
A_SOURCE_KEY, SOURCE_FILE_TYPE, SOURCE_FILE_ID, SOURCE_FILE_DESC,
SOURCE_FILE_NAME_PATTERN, TABLE_ID, TEMPLATE_TABLE_NAME, ENCODING
) VALUES (
pSourceKey, pSourceFileType, pSourceFileId, pSourceFileDesc,
pSourceFileNamePattern, pTableId, pTemplateTableName, pEncoding
);
```
### Przykład użycia zmodyfikowanej procedury
```sql
-- Dodanie konfiguracji z kodowaniem UTF-8
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'C2D',
pSourceFileType => 'INPUT',
pSourceFileId => 'UC_DISSEM',
pSourceFileDesc => 'UC DISSEM Metadata with UTF-8 encoding',
pSourceFileNamePattern => 'UC_NMA_DISSEM-*.csv',
pTableId => 'A_UC_DISSEM_METADATA_LOADS',
pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
pEncoding => 'UTF8'
);
-- Dodanie konfiguracji bez kodowania (jak dotychczas)
CALL FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM',
pSourceFileType => 'INPUT',
pSourceFileId => 'STANDING_FACILITIES',
pSourceFileDesc => 'Standing Facilities Data',
pSourceFileNamePattern => 'SF_*.csv',
pTableId => 'STANDING_FACILITIES',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES'
-- pEncoding pomijamy - będzie NULL (domyślne kodowanie)
);
```
---
## 5. TESTY WALIDACYJNE
### Test 1: Podstawowa funkcjonalność
```sql
-- 1. Ustawienie kodowania UTF-8
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'UTF8'
WHERE A_SOURCE_FILE_CONFIG_KEY = 1;
-- 2. Test przetwarzania pliku
CALL FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/FILE_ID/TABLE_NAME/test_file.csv');
-- 3. Sprawdzenie logów
SELECT * FROM CT_MRDS.A_PROCESS_LOG
WHERE PROCEDURE_NAME = 'CREATE_EXTERNAL_TABLE'
ORDER BY LOG_DATE DESC;
```
### Test 2: Fallback do wartości domyślnych
```sql
-- 1. Ustawienie NULL
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = NULL
WHERE A_SOURCE_FILE_CONFIG_KEY = 1;
-- 2. Test przetwarzania - powinno być używane domyślne kodowanie
CALL FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST/FILE_ID/TABLE_NAME/test_file.csv');
```
### Test 3: Różne kodowania
```sql
-- 1. Test z Windows-1252
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'WE8MSWIN1252'
WHERE A_SOURCE_FILE_CONFIG_KEY = 1;
-- 2. Test z UTF-8
UPDATE CT_MRDS.A_SOURCE_FILE_CONFIG
SET ENCODING = 'UTF8'
WHERE A_SOURCE_FILE_CONFIG_KEY = 2;
-- 3. Test przetwarzania obu plików
CALL FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST1/FILE_ID/TABLE_NAME/test_file1.csv');
CALL FILE_MANAGER.PROCESS_SOURCE_FILE('INBOX/TEST2/FILE_ID/TABLE_NAME/test_file2.csv');
```
---
## 6. HARMONOGRAM WDROŻENIA
### Faza 1: Przygotowanie struktury (KRYTYCZNA)
1. ✅ Dodanie kolumny `ENCODING` do tabeli
2. ✅ Zaktualizowanie definicji tabeli w plikach DDL
### Faza 2: Modyfikacja pakietu (WYSOKA)
1. ⏳ Dodanie parametru do sygnatury `CREATE_EXTERNAL_TABLE`
2. ⏳ Implementacja logiki kodowania w formatowaniu JSON
3. ⏳ Aktualizacja wywołania w procedurze overload
4. ⏳ Aktualizacja parametrów logowania
### Faza 3: Testy i walidacja (ŚREDNIA)
1. ⏳ Testy jednostkowe z różnymi kodowaniami
2. ⏳ Testy integracyjne z różnymi typami plików
3. ⏳ Walidacja fallback do domyślnego kodowania
### Faza 4: Modyfikacja procedur konfiguracyjnych (NISKA)
1. ⏳ Modyfikacja `ADD_SOURCE_FILE_CONFIG` - dodanie parametru `pEncoding`
2. ⏳ Aktualizacja dokumentacji
3. ⏳ Przykłady użycia w confluence
---
## 7. ROLLBACK
W przypadku problemów można wykonać rollback:
```sql
-- Usunięcie kolumny (OSTROŻNIE - spowoduje utratę danych)
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG DROP COLUMN ENCODING;
-- Przywrrócenie oryginalnej wersji pakietu FILE_MANAGER
-- (wymagane przywrrócenie z backupu)
```
---
## 8. UWAGI TECHNICZNE
### Ograniczenia
- Maksymalna długość nazwy kodowania: 50 znaków
- Obsługiwane kodowania: wszystkie wspierane przez Oracle Database
- Przykłady poprawnych nazw: `UTF8`, `WE8MSWIN1252`, `AL32UTF8`, `WE8ISO8859P1`
### Bezpieczeństwo
- Brak walidacji nazwy kodowania na poziomie aplikacji
- Oracle Database zwaliduje kodowanie podczas tworzenia external table
- Fallback do domyślnego kodowania bazy w przypadku NULL
### Wydajność
- Minimalne obciążenie - kodowanie dodawane tylko jeśli nie jest NULL
- Brak wpływu na istniejące procesy gdy kolumna jest NULL
- Używa natywnej funkcji JSON_MERGEPATCH Oracle Database
---
## 9. PRZYGOTOWANIE PACZKI INSTALACYJNEJ
### Struktura paczki MARS-1049
```
MARS_Packages/REL02/MARS-1049/
├── 01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql
├── 02_MARS_1049_install_CT_MRDS_FILE_MANAGER_SPEC.sql
├── 03_MARS_1049_install_CT_MRDS_FILE_MANAGER_BODY.sql
├── 04_MARS_1049_verify_ENCODING_FUNCTIONALITY.sql
├── 05_MARS_1049_track_version.sql
├── 91_MARS_1049_rollback_FILE_MANAGER_BODY.sql
├── 92_MARS_1049_rollback_FILE_MANAGER_SPEC.sql
├── 93_MARS_1049_rollback_ENCODING_COLUMN.sql
├── 94_MARS_1049_track_rollback_version.sql
├── install_mars1049.sql
├── rollback_mars1049.sql
└── README_ENCODING.md
```
### Zawartość skryptów instalacyjnych
#### 01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql
```sql
-- MARS-1049: Dodanie kolumny ENCODING do A_SOURCE_FILE_CONFIG
-- Autor: System Enhancement
-- Data: 2025-11-24
PROMPT Adding ENCODING column to CT_MRDS.A_SOURCE_FILE_CONFIG
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG
ADD (ENCODING VARCHAR2(50));
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING
IS 'Character encoding for CSV files (e.g. UTF8, WE8MSWIN1252, etc.)';
PROMPT ENCODING column added successfully
```
#### 02_MARS_1049_install_CT_MRDS_FILE_MANAGER_SPEC.sql
```sql
-- MARS-1049: Aktualizacja specyfikacji FILE_MANAGER - dodanie parametru pEncoding
-- Zawiera: Modyfikację sygnatury CREATE_EXTERNAL_TABLE i ADD_SOURCE_FILE_CONFIG
```
#### 04_MARS_1049_verify_ENCODING_FUNCTIONALITY.sql
```sql
-- MARS-1049: Weryfikacja funkcjonalności ENCODING
PROMPT Testing ENCODING functionality...
-- Test 1: Sprawdzenie struktury tabeli
SELECT COLUMN_NAME, DATA_TYPE, DATA_LENGTH, NULLABLE
FROM USER_TAB_COLUMNS
WHERE TABLE_NAME = 'A_SOURCE_FILE_CONFIG'
AND COLUMN_NAME = 'ENCODING';
-- Test 2: Test podstawowej funkcjonalności
INSERT INTO CT_MRDS.A_SOURCE_FILE_CONFIG (
A_SOURCE_KEY, SOURCE_FILE_TYPE, SOURCE_FILE_ID,
SOURCE_FILE_DESC, SOURCE_FILE_NAME_PATTERN,
TABLE_ID, TEMPLATE_TABLE_NAME, ENCODING
) VALUES (
'TEST', 'INPUT', 'ENCODING_TEST',
'Test ENCODING functionality', 'test_*.csv',
'TEST_TABLE', 'CT_ET_TEMPLATES.TEST_TABLE', 'UTF8'
);
-- Sprawdzenie czy rekord został dodany
SELECT A_SOURCE_FILE_CONFIG_KEY, ENCODING
FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'TEST' AND SOURCE_FILE_ID = 'ENCODING_TEST';
-- Cleanup
DELETE FROM CT_MRDS.A_SOURCE_FILE_CONFIG
WHERE A_SOURCE_KEY = 'TEST' AND SOURCE_FILE_ID = 'ENCODING_TEST';
PROMPT ENCODING functionality verification completed successfully
```
#### install_mars1049.sql (Master Install Script)
```sql
-- MARS-1049: Master Installation Script
-- CSV Encoding Support Implementation
PROMPT ================================================
PROMPT MARS-1049: Installing CSV Encoding Support
PROMPT ================================================
SPOOL install_mars1049.log
-- Faza 1: Struktura bazy danych
PROMPT Phase 1: Database Structure Changes
@@01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql
-- Faza 2: Pakiet FILE_MANAGER (Specyfikacja)
PROMPT Phase 2: FILE_MANAGER Package Specification
@@02_MARS_1049_install_CT_MRDS_FILE_MANAGER_SPEC.sql
-- Faza 3: Pakiet FILE_MANAGER (Ciało)
PROMPT Phase 3: FILE_MANAGER Package Body
@@03_MARS_1049_install_CT_MRDS_FILE_MANAGER_BODY.sql
-- Faza 4: Weryfikacja
PROMPT Phase 4: Functionality Verification
@@04_MARS_1049_verify_ENCODING_FUNCTIONALITY.sql
-- Faza 5: Tracking wersji
PROMPT Phase 5: Version Tracking
@@05_MARS_1049_track_version.sql
PROMPT ================================================
PROMPT MARS-1049: Installation completed successfully
PROMPT ================================================
SPOOL OFF
```
#### rollback_mars1049.sql (Master Rollback Script)
```sql
-- MARS-1049: Master Rollback Script
-- WARNING: This will remove ENCODING functionality
PROMPT ================================================
PROMPT MARS-1049: Rolling back CSV Encoding Support
PROMPT WARNING: This will remove ENCODING column and data!
PROMPT ================================================
SPOOL rollback_mars1049.log
-- Rollback w odwrotnej kolejności
@@94_MARS_1049_track_rollback_version.sql
@@93_MARS_1049_rollback_ENCODING_COLUMN.sql
@@92_MARS_1049_rollback_FILE_MANAGER_SPEC.sql
@@91_MARS_1049_rollback_FILE_MANAGER_BODY.sql
PROMPT ================================================
PROMPT MARS-1049: Rollback completed
PROMPT ================================================
SPOOL OFF
```
### Deployment Commands
```powershell
# Instalacja paczki MARS-1049
Get-Content "MARS_Packages\REL02\MARS-1049\install_mars1049.sql" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high"
# W przypadku problemów - rollback
Get-Content "MARS_Packages\REL02\MARS-1049\rollback_mars1049.sql" | sql "CT_MRDS/Cloudpass#34@ggmichalski_high"
# Archiwizacja po instalacji
7z a -pMojeSuperHaslo -mhe=on MARS_1049_arch.7z MARS_Packages\REL02\MARS-1049\*.sql
```
---
## Status: GOTOWE DO IMPLEMENTACJI
**Utworzone**: 2025-11-24
**Autor**: System FILE_MANAGER Enhancement
**MARS Issue**: MARS-1049
**Priorytet**: WYSOKI

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,618 @@
create or replace PACKAGE CT_MRDS.FILE_MANAGER
AUTHID CURRENT_USER
AS
/**
* General comment for package: Please put comments for functions and procedures as shown in below example.
* It is a standard.
* The structure of comment is used by GET_PACKAGE_DOCUMENTATION function
* which returns documentation text for confluence page (to Copy-Paste it).
**/
-- Example comment:
/**
* @name EX_PROCEDURE_NAME
* @desc Procedure description
* @example select FILE_MANAGER.EX_PROCEDURE_NAME(pParameter => 129) from dual;
* @ex_rslt Example Result
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.2.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 16:30:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.2.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-20): Enhanced PROCESS_SOURCE_FILE with 6-step validation workflow' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-15): Separated export procedures into dedicated DATA_EXPORTER package' || CHR(13)||CHR(10) ||
'2.5.0 (2025-10-10): Added DELETE_SOURCE_CASCADE for safe configuration removal' || CHR(13)||CHR(10) ||
'2.0.0 (2025-09-25): Added official path patterns support (INBOX 3-level, ODS 2-level, ARCHIVE 2-level)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-01): Initial release with file processing and validation capabilities';
TYPE tSourceFileReceived IS RECORD
(
A_SOURCE_FILE_RECEIVED_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE,
A_SOURCE_FILE_CONFIG_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_CONFIG_KEY%TYPE,
SOURCE_FILE_PREFIX_INBOX VARCHAR2(430),
SOURCE_FILE_PREFIX_ODS VARCHAR2(430),
SOURCE_FILE_PREFIX_QUARANTINE VARCHAR2(430),
SOURCE_FILE_PREFIX_ARCHIVE VARCHAR2(430),
SOURCE_FILE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.SOURCE_FILE_NAME%TYPE,
RECEPTION_DATE CT_MRDS.A_SOURCE_FILE_RECEIVED.RECEPTION_DATE%TYPE,
PROCESSING_STATUS CT_MRDS.A_SOURCE_FILE_RECEIVED.PROCESSING_STATUS%TYPE,
EXTERNAL_TABLE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.EXTERNAL_TABLE_NAME%TYPE
);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgSourceFileConfigKey PLS_INTEGER;
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_SOURCE_FILE_CONFIG
* @desc Get source file type by matching the source file name against source file type naming patterns
* or by specifying the id of a received source file.
* @example ...
* @ex_rslt "CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE"
**/
FUNCTION GET_SOURCE_FILE_CONFIG(pFileUri IN VARCHAR2 DEFAULT NULL
, pSourceFileReceivedKey IN NUMBER DEFAULT NULL
, pSourceFileConfigKey IN NUMBER DEFAULT NULL)
RETURN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a newly received source file in A_SOURCE_FILE_RECEIVED table.
* This overload automatically determines source file type from the file name.
* It returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2
)
RETURN PLS_INTEGER;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a new new source file in A_SOURCE_FILE_RECEIVED table based on pSourceFileReceivedName and pSourceFileConfig.
* Then it returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(
* pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv'
* ,pSourceFileConfig => ...A_SOURCE_FILE_CONFIG%ROWTYPE... );
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2,
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
)
RETURN PLS_INTEGER;
/**
* @name SET_SOURCE_FILE_RECEIVED_STATUS
* @desc Set status of file in A_SOURCE_FILE_RECEIVED table - PROCESSING_STATUS column
* based on A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY
* and provided value of pStatus parameter
* @example exec FILE_MANAGER.SET_SOURCE_FILE_RECEIVED_STATUS(pSourceFileReceivedKey => 377, pStatus => 'READY_FOR_INGESTION');
**/
PROCEDURE SET_SOURCE_FILE_RECEIVED_STATUS(
pSourceFileReceivedKey IN PLS_INTEGER,
pStatus IN VARCHAR2
);
/**
* @name GET_EXTERNAL_TABLE_COLUMNS
* @desc Function used to get string with all table columns definitions based on pTargetTableTemplate "TEMPLATE TABLE" name.
* It used for creating "EXTERNAL TABLE" using CREATE_EXTERNAL_TABLE procedure.
* @example select FILE_MANAGER.GET_EXTERNAL_TABLE_COLUMNS(pTargetTableTemplate => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER') from dual;
* @ex_rslt "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "REV_NUMBER" NUMBER(28,0),
* "REF_DATE" DATE,
* "FREE_TEXT" VARCHAR2(1000 CHAR),
* "MLF_BS_TOTAL" NUMBER(28,10),
* "DF_BS_TOTAL" NUMBER(28,10),
* "MLF_SF_TOTAL" NUMBER(28,10),
* "DF_SF_TOTAL" NUMBER(28,10)
**/
FUNCTION GET_EXTERNAL_TABLE_COLUMNS (
pTargetTableTemplate IN VARCHAR2
)
RETURN CLOB;
/**
* @name CREATE_EXTERNAL_TABLE
* @desc A wrapper procedure for DBMS_CLOUD.CREATE_EXTERNAL_TABLE which creates External Table
* @example
* begin
* FILE_MANAGER.CREATE_EXTERNAL_TABLE(
* pTableName => 'STANDING_FACILITIES_HEADER',
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
* pPrefix => 'ODS/LM/STANDING_FACILITIES_HEADER/',
* pBucketUri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/'
* pFileName => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/ODS/LM/STANDING_FACILITIES_HEADER/*.csv',
* pDelimiter => ',',
* );
* end;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ','
);
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.CREATE_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_SOURCE_FILE_RECEIVED
* @desc A wrapper procedure for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE
* It validate External table build upon single file
* provided by pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.VALIDATE_SOURCE_FILE_RECEIVED(pSourceFileReceivedKey => 377);
**/
PROCEDURE VALIDATE_SOURCE_FILE_RECEIVED
(
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_EXTERNAL_TABLE
* @desc A wrapper function for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE.
* It validates External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt FAILED
**/
FUNCTION VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name S_VALIDATE_EXTERNAL_TABLE
* @desc A function which checks if SELECT query reterns any rows.
* It trys to selects External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.S_VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt PASSED
**/
FUNCTION S_VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name DROP_EXTERNAL_TABLE
* @desc It drops External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.DROP_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);
**/
PROCEDURE DROP_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name COPY_FILE
* @desc It copies file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS'
* @example exec FILE_MANAGER.COPY_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE COPY_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name MOVE_FILE
* @desc It moves file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS', 'QUARANTINE'
* @example exec FILE_MANAGER.MOVE_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE MOVE_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name DELETE_FOLDER_CONTENTS
* @desc It deletes all files from specified folder in the cloud storage.
* The procedure lists all objects in the specified folder prefix and deletes them one by one.
* pBucketArea parameter specifies which bucket to use: 'INBOX', 'DATA', 'ARCHIVE'
* pFolderPrefix parameter specifies the folder path within the bucket (e.g., 'C2D/UC_DISSEM/UC_NMA_DISSEM/')
* @example exec FILE_MANAGER.DELETE_FOLDER_CONTENTS(pBucketArea => 'INBOX', pFolderPrefix => 'C2D/UC_DISSEM/UC_NMA_DISSEM/');
**/
PROCEDURE DELETE_FOLDER_CONTENTS(
pBucketArea IN VARCHAR2,
pFolderPrefix IN VARCHAR2
);
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter.
* Ubmrella procedure that calls:
* - REGISTER_SOURCE_FILE_RECEIVED;
* - CREATE_EXTERNAL_TABLE;
* - VALIDATE_SOURCE_FILE_RECEIVED;
* - DROP_EXTERNAL_TABLE;
* - MOVE_FILE;
* @example exec FILE_MANAGER.PROCESS_SOURCE_FILE(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
**/
PROCEDURE PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
;
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter and return processing result value.
* It returns (success/failure) => 0 / -(value).
* Ubmrella function that calls PROCESS_SOURCE_FILE procedure.
* @example
* declare
* vResult PLS_INTEGER;
* begin
* vResult := CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(PSOURCEFILERECEIVEDNAME => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* DBMS_OUTPUT.PUT_LINE('vResult = ' || vResult);
* end;
* @ex_rslt 0
* -20021
**/
FUNCTION PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
RETURN PLS_INTEGER;
/**
* @name GET_DATE_FORMAT
* @desc Returns date format for specified template table name and column name.
* Date is taken from configuration A_COLUMN_DATE_FORMAT table.
* @example select FILE_MANAGER.GET_DATE_FORMAT(
* pTemplateTableName => 'STANDING_FACILITIES_HEADER',
* pColumnName => 'SNAPSHOT_DATE')
* from dual;
* @ex_rslt DD/MM/YYYY HH24:MI:SS
**/
FUNCTION GET_DATE_FORMAT(
pTemplateTableName IN VARCHAR2,
pColumnName IN VARCHAR2
) RETURN VARCHAR2;
/**
* @name GENERATE_EXTERNAL_TABLE_PARAMS
* @desc It builds two strings: pColumnList and pFieldList for specified Template Table name, by parameter: pTemplateTableName.
* @example
* declare
* vColumnList CLOB;
* vFieldList CLOB;
* begin
* FILE_MANAGER.GENERATE_EXTERNAL_TABLE_PARAMS (
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
* ,pColumnList => vColumnList
* ,pFieldList => vFieldList
* );
* DBMS_OUTPUT.PUT_LINE('vColumnList = '||vColumnList);
* DBMS_OUTPUT.PUT_LINE('vFieldList = '||vFieldList);
* end;
* /
**/
PROCEDURE GENERATE_EXTERNAL_TABLE_PARAMS (
pTemplateTableName IN VARCHAR2,
pColumnList OUT CLOB,
pFieldList OUT CLOB
);
/**
* @name ADD_SOURCE
* @desc Insert a new record to A_SOURCE table.
* pSourceKey is a PRIMARY KEY value.
**/
PROCEDURE ADD_SOURCE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
pSourceName IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);
/**
* @name DELETE_SOURCE_CASCADE
* @desc Safely deletes a SOURCE specified by pSourceKey parameter from A_SOURCE table and all dependent tables:
* - A_SOURCE_FILE_CONFIG
* - A_SOURCE_FILE_RECEIVED
* - A_COLUMN_DATE_FORMAT (only if template table is not shared with other source systems)
* The procedure checks if template tables are shared before deleting date format configurations.
* If a template table is used by multiple source systems, date formats are preserved.
* @example CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(pSourceKey => 'TEST_SYS');
**/
PROCEDURE DELETE_SOURCE_CASCADE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE
);
/**
* @name GET_CONTAINER_SOURCE_FILE_CONFIG_KEY
* @desc For specified parameter pSourceFileId (A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID)
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY for related CONTAINER record.
* @example select FILE_MANAGER.GET_CONTAINER_SOURCE_FILE_CONFIG_KEY(
* pSourceFileId => 'UC_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_CONTAINER_SOURCE_FILE_CONFIG_KEY (
pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name GET_SOURCE_FILE_CONFIG_KEY
* @desc For specified input parameters,
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY.
* @example select FILE_MANAGER.GET_SOURCE_FILE_CONFIG_KEY (
* pSourceFileType => 'INPUT'
* ,pSourceFileId => 'UC_DISSEM'
* ,pTableId => 'UC_NMA_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_SOURCE_FILE_CONFIG_KEY (
pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE DEFAULT 'INPUT'
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name ADD_SOURCE_FILE_CONFIG
* @desc Insert a new record to A_SOURCE_FILE_CONFIG table.
**/
PROCEDURE ADD_SOURCE_FILE_CONFIG (
pSourceKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE
,pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pSourceFileDesc IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE
,pSourceFileNamePattern IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE DEFAULT NULL
,pTemplateTableName IN CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE DEFAULT NULL
,pContainerFileKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE DEFAULT NULL
);
/**
* @name ADD_COLUMN_DATE_FORMAT
* @desc Insert a new record to A_COLUMN_DATE_FORMAT table.
**/
PROCEDURE ADD_COLUMN_DATE_FORMAT (
pTemplateTableName IN CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE
,pColumnName IN CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE
,pDateFormat IN CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);
/**
* @name GET_BUCKET_URI
* @desc Function used to get string with bucket http url.
* Possible input values for pBucketArea are: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example select FILE_MANAGER.GET_BUCKET_URI(pBucketArea => 'ODS') from dual;
* @ex_rslt https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/
**/
FUNCTION GET_BUCKET_URI(pBucketArea VARCHAR2)
RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_CONFIG_INFO
* @desc Function returns details about A_SOURCE_FILE_CONFIG record
* for specified pSourceFileConfigKey (A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY).
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO (
* pSourceFileConfigKey => 128
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
* @ex_rslt
* Details about File Configuration:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 128
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Details about related Container Config:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 126
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Column Date Format config entries:
* --------------------------------
* TEMPLATE_TABLE_NAME = CT_ET_TEMPLATES.C2D_UC_MA_DISSEM
* ...
* --------------------------------
**/
FUNCTION GET_DET_SOURCE_FILE_CONFIG_INFO (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_RECEIVED_INFO
* @desc Function returns details about A_SOURCE_FILE_RECEIVED record
* for specified pSourceFileReceivedKey (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY).
* If pIncludeConfigInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_RECEIVED_INFO (
* pSourceFileReceivedKey => 377
* ,pIncludeConfigInfo => 1
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
*
**/
FUNCTION GET_DET_SOURCE_FILE_RECEIVED_INFO (
pSourceFileReceivedKey IN CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE
,pIncludeConfigInfo IN PLS_INTEGER DEFAULT 1
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_USER_LOAD_OPERATIONS
* @desc Function returns details from USER_LOAD_OPERATIONS table
* for specified pOperationId.
* @example select FILE_MANAGER.GET_DET_USER_LOAD_OPERATIONS (pOperationId => 3608) from dual;
* @ex_rslt
* Details about USER_LOAD_OPERATIONS where ID = 3608
* --------------------------------
* ID = 3608
* TYPE = VALIDATE
* SID = 31260
* SERIAL# = 52915
* START_TIME = 2025-05-20 10.08.24.436983 EUROPE/BELGRADE
* UPDATE_TIME = 2025-05-20 10.08.24.458643 EUROPE/BELGRADE
* STATUS = FAILED
* OWNER_NAME = CT_MRDS
* TABLE_NAME = STANDING_FACILITIES_HEADER
* PARTITION_NAME =
* SUBPARTITION_NAME =
* FILE_URI_LIST =
* ROWS_LOADED =
* LOGFILE_TABLE = VALIDATE$3608_LOG
* BADFILE_TABLE = VALIDATE$3608_BAD
* STATUS_TABLE =
* TEMPEXT_TABLE =
* CREDENTIAL_NAME =
* EXPIRATION_TIME = 2025-05-22 10.08.24.436983000 EUROPE/BELGRADE
* --------------------------------
**/
FUNCTION GET_DET_USER_LOAD_OPERATIONS (
pOperationId PLS_INTEGER
) RETURN VARCHAR2;
/**
* @name ANALYZE_VALIDATION_ERRORS
* @desc Wrapper function that analyzes validation errors for a source file using its received key.
* Automatically derives template schema, table name, CSV URI and validation log table
* from file metadata and calls ENV_MANAGER.ANALYZE_VALIDATION_ERRORS.
* @example SELECT FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;
* @ex_rslt Detailed validation analysis report with column mismatches and solutions
**/
FUNCTION ANALYZE_VALIDATION_ERRORS(
pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
* @ex_rslt 3.2.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER
* Version: 3.2.0
* Build Date: 2025-10-22 16:30:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER Version History:
* 3.2.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END;
/

View File

@@ -0,0 +1,97 @@
create or replace PACKAGE BODY ODS.FILE_MANAGER_ODS
AS
/**
* CREATE_EXTERNAL_TABLE - Wrapper for CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE
*/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ','
)
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
-- Log the start of the procedure
vParameters := CT_MRDS.ENV_MANAGER.FORMAT_PARAMETERS(
SYS.ODCIVARCHAR2LIST(
'pTableName => ''' || pTableName || '''',
'pTemplateTableName => ''' || pTemplateTableName || '''',
'pPrefix => ''' || pPrefix || '''',
'pBucketUri => ''' || pBucketUri || '''',
'pFileName => ''' || NVL(pFileName, 'NULL') || '''',
'pDelimiter => ''' || pDelimiter || ''''
)
);
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT('Start FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE', 'INFO', vParameters);
-- Call the original CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE procedure
-- This ensures all logic remains centralized in the original package
-- and ODS wrapper simply delegates execution with DEFINER rights
CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
pTableName => pTableName,
pTemplateTableName => pTemplateTableName,
pPrefix => pPrefix,
pBucketUri => pBucketUri,
pFileName => pFileName,
pDelimiter => pDelimiter
);
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT('End FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE', 'INFO', vParameters);
EXCEPTION
WHEN OTHERS THEN
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT(
'Error in ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE: ' || SQLERRM,
'ERROR',
vParameters
);
RAISE;
END CREATE_EXTERNAL_TABLE;
----------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS IMPLEMENTATION
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION
RETURN VARCHAR2
IS
BEGIN
RETURN PACKAGE_VERSION;
END GET_VERSION;
----------------------------------------------------------------------------------------------------
FUNCTION GET_BUILD_INFO
RETURN VARCHAR2
IS
BEGIN
RETURN CT_MRDS.ENV_MANAGER.GET_PACKAGE_VERSION_INFO(
pPackageName => 'FILE_MANAGER_ODS',
pVersion => PACKAGE_VERSION,
pBuildDate => PACKAGE_BUILD_DATE,
pAuthor => PACKAGE_AUTHOR
);
END GET_BUILD_INFO;
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION_HISTORY
RETURN VARCHAR2
IS
BEGIN
RETURN CT_MRDS.ENV_MANAGER.FORMAT_VERSION_HISTORY(
pPackageName => 'FILE_MANAGER_ODS',
pVersionHistory => VERSION_HISTORY
);
END GET_VERSION_HISTORY;
----------------------------------------------------------------------------------------------------
END FILE_MANAGER_ODS;
/

View File

@@ -0,0 +1,91 @@
create or replace PACKAGE ODS.FILE_MANAGER_ODS
AUTHID DEFINER
AS
/**
* FILE_MANAGER_ODS - Wrapper package for CT_MRDS.FILE_MANAGER
*
* This package serves as a wrapper for CT_MRDS.FILE_MANAGER procedures.
* The key difference is that this package uses AUTHID DEFINER instead of
* AUTHID CURRENT_USER, which means all objects will be created in the ODS
* schema regardless of which user executes the procedures.
*
* This solves the execution context issue where users need to connect as
* ODS user to use CT_MRDS.FILE_MANAGER procedures.
*/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.0.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-10-22 17:15:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-12): Enhanced external table creation with official path patterns support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-20): Initial release with AUTHID DEFINER wrapper for external table creation';
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates external tables that can read data from Oracle Cloud Storage.
* This is a wrapper for CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE
* but uses DEFINER rights to ensure objects are created in ODS schema.
* @param pTableName - Name of the external table to create
* @param pTemplateTableName - Template table defining the structure
* @param pPrefix - Storage path prefix in Oracle Cloud Storage
* @param pBucketUri - URI of the target bucket (default: ENV_MANAGER.gvInboxBucketUri)
* @param pFileName - Specific file name (optional)
* @param pDelimiter - Field delimiter (default: ',')
* @example EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
* 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
* 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
* 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
* CT_MRDS.ENV_MANAGER.gvInboxBucketUri
* );
*/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ','
);
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER_ODS package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
* @ex_rslt 2.0.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER_ODS.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER_ODS
* Version: 2.0.0
* Build Date: 2025-10-22 17:15:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER_ODS.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER_ODS Version History:
* 2.0.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END FILE_MANAGER_ODS;
/

View File

@@ -0,0 +1,114 @@
-- ===================================================================
-- MARS-1049 MAIN INSTALL SCRIPT: CSV Encoding Support
-- ===================================================================
-- Purpose: Complete installation of CSV encoding support in FILE_MANAGER system
-- Author: Grzegorz Michalski
-- Date: 2025-11-24
-- Version: 1.1.0
-- Dynamic spool file generation
var filename VARCHAR2(100)
BEGIN
SELECT 'INSTALL_MARS_1049_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
--
-- IMPLEMENTATION STATUS: COMPLETED and DEPLOYED
-- - A_SOURCE_FILE_CONFIG: ENCODING VARCHAR2(50) column added
-- - CT_MRDS.FILE_MANAGER: Updated to version 3.2.1 with pEncoding parameter
-- - ODS.FILE_MANAGER_ODS: Updated to version 2.1.0 with pEncoding wrapper support
-- - JSON Implementation: Uses JSON_OBJECT due to Oracle 23c compatibility
--
-- IMPORTANT: Tests are NOT part of this installation.
-- Testing is separate and optional (see test/ directory).
--
-- DEPLOYMENT ORDER (CRITICAL):
-- 0. Grant SELECT ANY DICTIONARY to CT_MRDS (for cross-schema hash tracking)
-- 1. Database structure changes (A_SOURCE_FILE_CONFIG.ENCODING column)
-- 2. Update existing configurations with default encoding values
-- 3. CT_MRDS.FILE_MANAGER package specification
-- 4. CT_MRDS.FILE_MANAGER package body
-- 5. ODS.FILE_MANAGER_ODS package specification
-- 6. ODS.FILE_MANAGER_ODS package body
-- 7. Track package versions
-- 8. Verify all tracked packages
--
-- NOTE: Testing is separate from installation.
-- For verification and testing, run separately:
-- test/05_MARS_1049_verify_encoding_functionality.sql
-- ===================================================================
PROMPT ===============================================
PROMPT MARS-1049: CSV Encoding Support - Full Installation
PROMPT ===============================================
-- Check prerequisites
PROMPT Checking prerequisites...
SELECT USER AS CURRENT_USER FROM DUAL;
SELECT COUNT(*) AS TABLE_COUNT FROM ALL_TABLES WHERE OWNER = 'CT_MRDS' AND TABLE_NAME = 'A_SOURCE_FILE_CONFIG';
PROMPT
PROMPT Step 0: Granting permissions for cross-schema hash tracking...
PROMPT ===============================================
@@00_MARS_1049_grant_permissions.sql
PROMPT
PROMPT Step 1: Installing database structure changes...
PROMPT ===============================================
@@01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql
PROMPT
PROMPT Step 2: Updating existing configurations with encoding values...
PROMPT ===============================================
@@02_MARS_1049_update_CT_MRDS_ENCODING_COLUMN.sql
PROMPT
PROMPT Step 3: Installing CT_MRDS.FILE_MANAGER specification...
PROMPT ===============================================
@@new_version/FILE_MANAGER.pkg
PROMPT
PROMPT Step 4: Installing CT_MRDS.FILE_MANAGER package body...
PROMPT ===============================================
@@new_version/FILE_MANAGER.pkb
PROMPT
PROMPT Step 5: Installing ODS.FILE_MANAGER_ODS specification...
PROMPT ===============================================
@@new_version/FILE_MANAGER_ODS.pkg
PROMPT
PROMPT Step 6: Installing ODS.FILE_MANAGER_ODS package body...
PROMPT ===============================================
@@new_version/FILE_MANAGER_ODS.pkb
PROMPT
PROMPT Step 7: Tracking package versions...
PROMPT ===============================================
@@track_package_versions.sql
PROMPT
PROMPT Step 8: Verifying all tracked packages...
PROMPT ===============================================
@@verify_packages_version.sql
PROMPT
PROMPT ===============================================
PROMPT MARS-1049: Installation Complete!
PROMPT ===============================================
PROMPT
PROMPT Summary of Changes:
PROMPT - A_SOURCE_FILE_CONFIG: Added ENCODING VARCHAR2(50) column for CSV character set specification
PROMPT - A_SOURCE_FILE_CONFIG: Updated existing configurations with default encoding values (UTF8 default, WE8MSWIN1252 for CSDB)
PROMPT - CT_MRDS.FILE_MANAGER v3.2.1: Added pEncoding parameter to ADD_SOURCE_FILE_CONFIG and CREATE_EXTERNAL_TABLE
PROMPT - ODS.FILE_MANAGER_ODS v2.1.0: Updated wrapper package with pEncoding parameter delegation
PROMPT
PROMPT Installation completed successfully!
PROMPT
spool off
exit

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,633 @@
create or replace PACKAGE CT_MRDS.FILE_MANAGER
AUTHID CURRENT_USER
AS
/**
* General comment for package: Please put comments for functions and procedures as shown in below example.
* It is a standard.
* The structure of comment is used by GET_PACKAGE_DOCUMENTATION function
* which returns documentation text for confluence page (to Copy-Paste it).
**/
-- Example comment:
/**
* @name EX_PROCEDURE_NAME
* @desc Procedure description
* @example select FILE_MANAGER.EX_PROCEDURE_NAME(pParameter => 129) from dual;
* @ex_rslt Example Result
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.2.1';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-11-24 12:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.2.1 (2025-11-24): MARS-1049 - Added pEncoding parameter support for CSV character set specification' || CHR(13)||CHR(10) ||
'3.2.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-20): Enhanced PROCESS_SOURCE_FILE with 6-step validation workflow' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-15): Separated export procedures into dedicated DATA_EXPORTER package' || CHR(13)||CHR(10) ||
'2.5.0 (2025-10-10): Added DELETE_SOURCE_CASCADE for safe configuration removal' || CHR(13)||CHR(10) ||
'2.0.0 (2025-09-25): Added official path patterns support (INBOX 3-level, ODS 2-level, ARCHIVE 2-level)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-01): Initial release with file processing and validation capabilities';
TYPE tSourceFileReceived IS RECORD
(
A_SOURCE_FILE_RECEIVED_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE,
A_SOURCE_FILE_CONFIG_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_CONFIG_KEY%TYPE,
SOURCE_FILE_PREFIX_INBOX VARCHAR2(430),
SOURCE_FILE_PREFIX_ODS VARCHAR2(430),
SOURCE_FILE_PREFIX_QUARANTINE VARCHAR2(430),
SOURCE_FILE_PREFIX_ARCHIVE VARCHAR2(430),
SOURCE_FILE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.SOURCE_FILE_NAME%TYPE,
RECEPTION_DATE CT_MRDS.A_SOURCE_FILE_RECEIVED.RECEPTION_DATE%TYPE,
PROCESSING_STATUS CT_MRDS.A_SOURCE_FILE_RECEIVED.PROCESSING_STATUS%TYPE,
EXTERNAL_TABLE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.EXTERNAL_TABLE_NAME%TYPE
);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgSourceFileConfigKey PLS_INTEGER;
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_SOURCE_FILE_CONFIG
* @desc Get source file type by matching the source file name against source file type naming patterns
* or by specifying the id of a received source file.
* @example ...
* @ex_rslt "CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE"
**/
FUNCTION GET_SOURCE_FILE_CONFIG(pFileUri IN VARCHAR2 DEFAULT NULL
, pSourceFileReceivedKey IN NUMBER DEFAULT NULL
, pSourceFileConfigKey IN NUMBER DEFAULT NULL)
RETURN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a newly received source file in A_SOURCE_FILE_RECEIVED table.
* This overload automatically determines source file type from the file name.
* It returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2
)
RETURN PLS_INTEGER;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a new new source file in A_SOURCE_FILE_RECEIVED table based on pSourceFileReceivedName and pSourceFileConfig.
* Then it returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(
* pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv'
* ,pSourceFileConfig => ...A_SOURCE_FILE_CONFIG%ROWTYPE... );
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2,
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
)
RETURN PLS_INTEGER;
/**
* @name SET_SOURCE_FILE_RECEIVED_STATUS
* @desc Set status of file in A_SOURCE_FILE_RECEIVED table - PROCESSING_STATUS column
* based on A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY
* and provided value of pStatus parameter
* @example exec FILE_MANAGER.SET_SOURCE_FILE_RECEIVED_STATUS(pSourceFileReceivedKey => 377, pStatus => 'READY_FOR_INGESTION');
**/
PROCEDURE SET_SOURCE_FILE_RECEIVED_STATUS(
pSourceFileReceivedKey IN PLS_INTEGER,
pStatus IN VARCHAR2
);
/**
* @name GET_EXTERNAL_TABLE_COLUMNS
* @desc Function used to get string with all table columns definitions based on pTargetTableTemplate "TEMPLATE TABLE" name.
* It used for creating "EXTERNAL TABLE" using CREATE_EXTERNAL_TABLE procedure.
* @example select FILE_MANAGER.GET_EXTERNAL_TABLE_COLUMNS(pTargetTableTemplate => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER') from dual;
* @ex_rslt "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "REV_NUMBER" NUMBER(28,0),
* "REF_DATE" DATE,
* "FREE_TEXT" VARCHAR2(1000 CHAR),
* "MLF_BS_TOTAL" NUMBER(28,10),
* "DF_BS_TOTAL" NUMBER(28,10),
* "MLF_SF_TOTAL" NUMBER(28,10),
* "DF_SF_TOTAL" NUMBER(28,10)
**/
FUNCTION GET_EXTERNAL_TABLE_COLUMNS (
pTargetTableTemplate IN VARCHAR2
)
RETURN CLOB;
/**
* @name CREATE_EXTERNAL_TABLE
* @desc A wrapper procedure for DBMS_CLOUD.CREATE_EXTERNAL_TABLE which creates External Table
* MARS-1049: Added pEncoding parameter for CSV character set specification
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252')
* If provided, adds CHARACTERSET clause to external table definition
* @example
* begin
* FILE_MANAGER.CREATE_EXTERNAL_TABLE(
* pTableName => 'STANDING_FACILITIES_HEADER',
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
* pPrefix => 'ODS/LM/STANDING_FACILITIES_HEADER/',
* pBucketUri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/',
* pFileName => NULL,
* pDelimiter => ',',
* pEncoding => 'UTF8'
* );
* end;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.CREATE_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_SOURCE_FILE_RECEIVED
* @desc A wrapper procedure for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE
* It validate External table build upon single file
* provided by pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.VALIDATE_SOURCE_FILE_RECEIVED(pSourceFileReceivedKey => 377);
**/
PROCEDURE VALIDATE_SOURCE_FILE_RECEIVED
(
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_EXTERNAL_TABLE
* @desc A wrapper function for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE.
* It validates External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt FAILED
**/
FUNCTION VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name S_VALIDATE_EXTERNAL_TABLE
* @desc A function which checks if SELECT query reterns any rows.
* It trys to selects External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.S_VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt PASSED
**/
FUNCTION S_VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name DROP_EXTERNAL_TABLE
* @desc It drops External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.DROP_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);
**/
PROCEDURE DROP_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name COPY_FILE
* @desc It copies file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS'
* @example exec FILE_MANAGER.COPY_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE COPY_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name MOVE_FILE
* @desc It moves file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS', 'QUARANTINE'
* @example exec FILE_MANAGER.MOVE_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE MOVE_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name DELETE_FOLDER_CONTENTS
* @desc It deletes all files from specified folder in the cloud storage.
* The procedure lists all objects in the specified folder prefix and deletes them one by one.
* pBucketArea parameter specifies which bucket to use: 'INBOX', 'DATA', 'ARCHIVE'
* pFolderPrefix parameter specifies the folder path within the bucket (e.g., 'C2D/UC_DISSEM/UC_NMA_DISSEM/')
* @example exec FILE_MANAGER.DELETE_FOLDER_CONTENTS(pBucketArea => 'INBOX', pFolderPrefix => 'C2D/UC_DISSEM/UC_NMA_DISSEM/');
**/
PROCEDURE DELETE_FOLDER_CONTENTS(
pBucketArea IN VARCHAR2,
pFolderPrefix IN VARCHAR2
);
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter.
* Ubmrella procedure that calls:
* - REGISTER_SOURCE_FILE_RECEIVED;
* - CREATE_EXTERNAL_TABLE;
* - VALIDATE_SOURCE_FILE_RECEIVED;
* - DROP_EXTERNAL_TABLE;
* - MOVE_FILE;
* @example exec FILE_MANAGER.PROCESS_SOURCE_FILE(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
**/
PROCEDURE PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
;
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter and return processing result value.
* It returns (success/failure) => 0 / -(value).
* Ubmrella function that calls PROCESS_SOURCE_FILE procedure.
* @example
* declare
* vResult PLS_INTEGER;
* begin
* vResult := CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(PSOURCEFILERECEIVEDNAME => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* DBMS_OUTPUT.PUT_LINE('vResult = ' || vResult);
* end;
* @ex_rslt 0
* -20021
**/
FUNCTION PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
RETURN PLS_INTEGER;
/**
* @name GET_DATE_FORMAT
* @desc Returns date format for specified template table name and column name.
* Date is taken from configuration A_COLUMN_DATE_FORMAT table.
* @example select FILE_MANAGER.GET_DATE_FORMAT(
* pTemplateTableName => 'STANDING_FACILITIES_HEADER',
* pColumnName => 'SNAPSHOT_DATE')
* from dual;
* @ex_rslt DD/MM/YYYY HH24:MI:SS
**/
FUNCTION GET_DATE_FORMAT(
pTemplateTableName IN VARCHAR2,
pColumnName IN VARCHAR2
) RETURN VARCHAR2;
/**
* @name GENERATE_EXTERNAL_TABLE_PARAMS
* @desc It builds two strings: pColumnList and pFieldList for specified Template Table name, by parameter: pTemplateTableName.
* @example
* declare
* vColumnList CLOB;
* vFieldList CLOB;
* begin
* FILE_MANAGER.GENERATE_EXTERNAL_TABLE_PARAMS (
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
* ,pColumnList => vColumnList
* ,pFieldList => vFieldList
* );
* DBMS_OUTPUT.PUT_LINE('vColumnList = '||vColumnList);
* DBMS_OUTPUT.PUT_LINE('vFieldList = '||vFieldList);
* end;
* /
**/
PROCEDURE GENERATE_EXTERNAL_TABLE_PARAMS (
pTemplateTableName IN VARCHAR2,
pColumnList OUT CLOB,
pFieldList OUT CLOB
);
/**
* @name ADD_SOURCE
* @desc Insert a new record to A_SOURCE table.
* pSourceKey is a PRIMARY KEY value.
**/
PROCEDURE ADD_SOURCE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
pSourceName IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);
/**
* @name DELETE_SOURCE_CASCADE
* @desc Safely deletes a SOURCE specified by pSourceKey parameter from A_SOURCE table and all dependent tables:
* - A_SOURCE_FILE_CONFIG
* - A_SOURCE_FILE_RECEIVED
* - A_COLUMN_DATE_FORMAT (only if template table is not shared with other source systems)
* The procedure checks if template tables are shared before deleting date format configurations.
* If a template table is used by multiple source systems, date formats are preserved.
* @example CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(pSourceKey => 'TEST_SYS');
**/
PROCEDURE DELETE_SOURCE_CASCADE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE
);
/**
* @name GET_CONTAINER_SOURCE_FILE_CONFIG_KEY
* @desc For specified parameter pSourceFileId (A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID)
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY for related CONTAINER record.
* @example select FILE_MANAGER.GET_CONTAINER_SOURCE_FILE_CONFIG_KEY(
* pSourceFileId => 'UC_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_CONTAINER_SOURCE_FILE_CONFIG_KEY (
pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name GET_SOURCE_FILE_CONFIG_KEY
* @desc For specified input parameters,
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY.
* @example select FILE_MANAGER.GET_SOURCE_FILE_CONFIG_KEY (
* pSourceFileType => 'INPUT'
* ,pSourceFileId => 'UC_DISSEM'
* ,pTableId => 'UC_NMA_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_SOURCE_FILE_CONFIG_KEY (
pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE DEFAULT 'INPUT'
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name ADD_SOURCE_FILE_CONFIG
* @desc Insert a new record to A_SOURCE_FILE_CONFIG table.
* MARS-1049: Added pEncoding parameter for CSV character set specification.
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252', 'EE8ISO8859P2')
* If NULL, no CHARACTERSET clause is added to external table definitions
* @example CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
* pSourceKey => 'C2D', pSourceFileType => 'INPUT',
* pSourceFileId => 'UC_DISSEM', pTableId => 'METADATA_LOADS',
* pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
* pEncoding => 'UTF8'
* );
**/
PROCEDURE ADD_SOURCE_FILE_CONFIG (
pSourceKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE
,pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pSourceFileDesc IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE
,pSourceFileNamePattern IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE DEFAULT NULL
,pTemplateTableName IN CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE DEFAULT NULL
,pContainerFileKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE DEFAULT NULL
,pEncoding IN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING%TYPE DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name ADD_COLUMN_DATE_FORMAT
* @desc Insert a new record to A_COLUMN_DATE_FORMAT table.
**/
PROCEDURE ADD_COLUMN_DATE_FORMAT (
pTemplateTableName IN CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE
,pColumnName IN CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE
,pDateFormat IN CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);
/**
* @name GET_BUCKET_URI
* @desc Function used to get string with bucket http url.
* Possible input values for pBucketArea are: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example select FILE_MANAGER.GET_BUCKET_URI(pBucketArea => 'ODS') from dual;
* @ex_rslt https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/
**/
FUNCTION GET_BUCKET_URI(pBucketArea VARCHAR2)
RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_CONFIG_INFO
* @desc Function returns details about A_SOURCE_FILE_CONFIG record
* for specified pSourceFileConfigKey (A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY).
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO (
* pSourceFileConfigKey => 128
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
* @ex_rslt
* Details about File Configuration:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 128
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Details about related Container Config:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 126
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Column Date Format config entries:
* --------------------------------
* TEMPLATE_TABLE_NAME = CT_ET_TEMPLATES.C2D_UC_MA_DISSEM
* ...
* --------------------------------
**/
FUNCTION GET_DET_SOURCE_FILE_CONFIG_INFO (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_RECEIVED_INFO
* @desc Function returns details about A_SOURCE_FILE_RECEIVED record
* for specified pSourceFileReceivedKey (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY).
* If pIncludeConfigInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_RECEIVED_INFO (
* pSourceFileReceivedKey => 377
* ,pIncludeConfigInfo => 1
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
*
**/
FUNCTION GET_DET_SOURCE_FILE_RECEIVED_INFO (
pSourceFileReceivedKey IN CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE
,pIncludeConfigInfo IN PLS_INTEGER DEFAULT 1
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_USER_LOAD_OPERATIONS
* @desc Function returns details from USER_LOAD_OPERATIONS table
* for specified pOperationId.
* @example select FILE_MANAGER.GET_DET_USER_LOAD_OPERATIONS (pOperationId => 3608) from dual;
* @ex_rslt
* Details about USER_LOAD_OPERATIONS where ID = 3608
* --------------------------------
* ID = 3608
* TYPE = VALIDATE
* SID = 31260
* SERIAL# = 52915
* START_TIME = 2025-05-20 10.08.24.436983 EUROPE/BELGRADE
* UPDATE_TIME = 2025-05-20 10.08.24.458643 EUROPE/BELGRADE
* STATUS = FAILED
* OWNER_NAME = CT_MRDS
* TABLE_NAME = STANDING_FACILITIES_HEADER
* PARTITION_NAME =
* SUBPARTITION_NAME =
* FILE_URI_LIST =
* ROWS_LOADED =
* LOGFILE_TABLE = VALIDATE$3608_LOG
* BADFILE_TABLE = VALIDATE$3608_BAD
* STATUS_TABLE =
* TEMPEXT_TABLE =
* CREDENTIAL_NAME =
* EXPIRATION_TIME = 2025-05-22 10.08.24.436983000 EUROPE/BELGRADE
* --------------------------------
**/
FUNCTION GET_DET_USER_LOAD_OPERATIONS (
pOperationId PLS_INTEGER
) RETURN VARCHAR2;
/**
* @name ANALYZE_VALIDATION_ERRORS
* @desc Wrapper function that analyzes validation errors for a source file using its received key.
* Automatically derives template schema, table name, CSV URI and validation log table
* from file metadata and calls ENV_MANAGER.ANALYZE_VALIDATION_ERRORS.
* @example SELECT FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;
* @ex_rslt Detailed validation analysis report with column mismatches and solutions
**/
FUNCTION ANALYZE_VALIDATION_ERRORS(
pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
* @ex_rslt 3.2.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER
* Version: 3.2.0
* Build Date: 2025-10-22 16:30:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER Version History:
* 3.2.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END;
/

View File

@@ -0,0 +1,118 @@
create or replace PACKAGE BODY ODS.FILE_MANAGER_ODS
AS
/**
* CREATE_EXTERNAL_TABLE - Wrapper for CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE
*/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL
)
IS
vParameters CT_MRDS.A_PROCESS_LOG.PROCEDURE_PARAMETERS%TYPE;
BEGIN
-- Log the start of the procedure
vParameters := CT_MRDS.ENV_MANAGER.FORMAT_PARAMETERS(
SYS.ODCIVARCHAR2LIST(
'pTableName => ''' || pTableName || '''',
'pTemplateTableName => ''' || pTemplateTableName || '''',
'pPrefix => ''' || pPrefix || '''',
'pBucketUri => ''' || pBucketUri || '''',
'pFileName => ''' || NVL(pFileName, 'NULL') || '''',
'pDelimiter => ''' || pDelimiter || '''',
'pEncoding => ''' || NVL(pEncoding, 'NULL') || ''''
)
);
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT('Start FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE', 'INFO', vParameters);
-- Call the original CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE procedure
-- This ensures all logic remains centralized in the original package
-- and ODS wrapper simply delegates execution with DEFINER rights
CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE(
pTableName => pTableName,
pTemplateTableName => pTemplateTableName,
pPrefix => pPrefix,
pBucketUri => pBucketUri,
pFileName => pFileName,
pDelimiter => pDelimiter,
pEncoding => pEncoding
);
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT('End FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE', 'INFO', vParameters);
EXCEPTION
WHEN OTHERS THEN
CT_MRDS.ENV_MANAGER.LOG_PROCESS_EVENT(
'Error in ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE: ' || SQLERRM,
'ERROR',
vParameters
);
RAISE;
END CREATE_EXTERNAL_TABLE;
----------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS IMPLEMENTATION
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION
RETURN VARCHAR2
IS
BEGIN
RETURN PACKAGE_VERSION;
END GET_VERSION;
----------------------------------------------------------------------------------------------------
FUNCTION GET_BUILD_DATE
RETURN VARCHAR2
IS
BEGIN
RETURN PACKAGE_BUILD_DATE;
END GET_BUILD_DATE;
----------------------------------------------------------------------------------------------------
FUNCTION GET_AUTHOR
RETURN VARCHAR2
IS
BEGIN
RETURN PACKAGE_AUTHOR;
END GET_AUTHOR;
----------------------------------------------------------------------------------------------------
FUNCTION GET_BUILD_INFO
RETURN VARCHAR2
IS
BEGIN
RETURN CT_MRDS.ENV_MANAGER.GET_PACKAGE_VERSION_INFO(
pPackageName => 'FILE_MANAGER_ODS',
pVersion => PACKAGE_VERSION,
pBuildDate => PACKAGE_BUILD_DATE,
pAuthor => PACKAGE_AUTHOR
);
END GET_BUILD_INFO;
----------------------------------------------------------------------------------------------------
FUNCTION GET_VERSION_HISTORY
RETURN VARCHAR2
IS
BEGIN
RETURN CT_MRDS.ENV_MANAGER.FORMAT_VERSION_HISTORY(
pPackageName => 'FILE_MANAGER_ODS',
pVersionHistory => VERSION_HISTORY
);
END GET_VERSION_HISTORY;
----------------------------------------------------------------------------------------------------
END FILE_MANAGER_ODS;
/

View File

@@ -0,0 +1,116 @@
create or replace PACKAGE ODS.FILE_MANAGER_ODS
AUTHID DEFINER
AS
/**
* FILE_MANAGER_ODS - Wrapper package for CT_MRDS.FILE_MANAGER
*
* This package serves as a wrapper for CT_MRDS.FILE_MANAGER procedures.
* The key difference is that this package uses AUTHID DEFINER instead of
* AUTHID CURRENT_USER, which means all objects will be created in the ODS
* schema regardless of which user executes the procedures.
*
* This solves the execution context issue where users need to connect as
* ODS user to use CT_MRDS.FILE_MANAGER procedures.
*
* MARS-1049: Added pEncoding parameter support for CSV character set specification
*/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '2.1.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-11-24 11:50:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'2.1.0 (2025-11-24): MARS-1049 - Added pEncoding parameter support for CSV character set specification' || CHR(13)||CHR(10) ||
'2.0.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'1.5.0 (2025-10-12): Enhanced external table creation with official path patterns support' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-20): Initial release with AUTHID DEFINER wrapper for external table creation';
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates external tables that can read data from Oracle Cloud Storage.
* This is a wrapper for CT_MRDS.FILE_MANAGER.CREATE_EXTERNAL_TABLE
* but uses DEFINER rights to ensure objects are created in ODS schema.
*
* MARS-1049: Added support for CSV character set encoding specification
* @param pTableName - Name of the external table to create
* @param pTemplateTableName - Template table defining the structure
* @param pPrefix - Storage path prefix in Oracle Cloud Storage
* @param pBucketUri - URI of the target bucket (default: ENV_MANAGER.gvInboxBucketUri)
* @param pFileName - Specific file name (optional)
* @param pDelimiter - Field delimiter (default: ',')
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF-8', 'WE8MSWIN1252', 'EE8ISO8859P2')
* If NULL, no CHARACTERSET clause is added to external table definition
* @example EXEC ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
* 'C2D_A_UC_DISSEM_METADATA_LOADS_INBOX',
* 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
* 'INBOX/C2D/UC_DISSEM/A_UC_DISSEM_METADATA_LOADS',
* CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
* NULL, ',', 'UTF-8'
* );
*/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT CT_MRDS.ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL
);
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER_ODS package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER_ODS.GET_VERSION() FROM DUAL;
* @ex_rslt 2.1.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_DATE
* @desc Returns the build date of the FILE_MANAGER_ODS package.
* @return VARCHAR2 - Build date in format 'YYYY-MM-DD HH24:MI:SS'
* @example SELECT ODS.FILE_MANAGER_ODS.GET_BUILD_DATE() FROM DUAL;
*/
FUNCTION GET_BUILD_DATE RETURN VARCHAR2;
/**
* @name GET_AUTHOR
* @desc Returns the author of the FILE_MANAGER_ODS package.
* @return VARCHAR2 - Package author name
* @example SELECT ODS.FILE_MANAGER_ODS.GET_AUTHOR() FROM DUAL;
*/
FUNCTION GET_AUTHOR RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER_ODS.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER_ODS
* Version: 2.1.0
* Build Date: 2025-11-24 11:50:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER_ODS.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER_ODS Version History:
* 2.0.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END FILE_MANAGER_ODS;
/

View File

@@ -0,0 +1,112 @@
-- ===================================================================
-- MARS-1049 ROLLBACK SCRIPT: CSV Encoding Support
-- ===================================================================
-- Purpose: Complete rollback of CSV encoding support changes
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.5.0
-- Dynamic spool file generation
var filename VARCHAR2(100)
BEGIN
SELECT 'ROLLBACK_MARS_1049_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
--
-- CURRENT IMPLEMENTATION STATUS:
-- - CT_MRDS.FILE_MANAGER: Version 3.2.1 with pEncoding parameter (deployed)
-- - ODS.FILE_MANAGER_ODS: Version 2.1.0 with pEncoding wrapper (deployed)
-- - A_SOURCE_FILE_CONFIG: ENCODING VARCHAR2(50) column (deployed)
--
-- ROLLBACK STRATEGY:
-- - Uses current_version/ folder for pre-MARS-1049 package versions
-- - Uses new_version/ folder for MARS-1049 enhanced versions (for reference)
--
-- ROLLBACK ORDER:
-- 1. Restore ODS.FILE_MANAGER_ODS package from current_version/ (version 2.0.0)
-- 2. Restore CT_MRDS.FILE_MANAGER package from current_version/ (version 3.2.0)
-- 3. Remove ENCODING column from A_SOURCE_FILE_CONFIG
-- 4. Track package versions
-- 5. Verify all tracked packages
-- 6. Revoke SELECT ANY DICTIONARY from CT_MRDS
-- ===================================================================
PROMPT ===============================================
PROMPT MARS-1049: CSV Encoding Support - Full Rollback
PROMPT ===============================================
-- Set error handling
WHENEVER SQLERROR CONTINUE
WHENEVER OSERROR CONTINUE
PROMPT
PROMPT Warning: This will remove the ENCODING column and revert all package changes.
PROMPT Make sure you have backed up your data if needed.
PROMPT
-- Step 1: Restore ODS.FILE_MANAGER_ODS package to version 2.0.0
PROMPT Step 1: Restoring ODS.FILE_MANAGER_ODS package to version 2.0.0...
PROMPT ===============================================
-- Restore ODS.FILE_MANAGER_ODS specification from current_version (before MARS-1049)
@@current_version/FILE_MANAGER_ODS.pkg
-- Restore ODS.FILE_MANAGER_ODS body from current_version (before MARS-1049)
@@current_version/FILE_MANAGER_ODS.pkb
PROMPT ODS.FILE_MANAGER_ODS package restored to version 2.0.0
-- Step 2: Restore CT_MRDS.FILE_MANAGER package to version 3.2.0
PROMPT Step 2: Restoring CT_MRDS.FILE_MANAGER package to version 3.2.0...
PROMPT ===============================================
-- Restore CT_MRDS.FILE_MANAGER specification from current_version (before MARS-1049)
@@current_version/FILE_MANAGER.pkg
-- Restore CT_MRDS.FILE_MANAGER body from current_version (before MARS-1049)
@@current_version/FILE_MANAGER.pkb
PROMPT CT_MRDS.FILE_MANAGER package restored to version 3.2.0
-- Step 3: Remove ENCODING column from A_SOURCE_FILE_CONFIG
PROMPT Step 3: Remove ENCODING column from A_SOURCE_FILE_CONFIG...
PROMPT ===============================================
@@91_MARS_1049_rollback_DROP_ENCODING_COLUMN.sql
-- Step 4: Track rollback version
PROMPT Step 4: Tracking package versions...
PROMPT ===============================================
@@track_package_versions.sql
PROMPT
PROMPT Step 5: Verifying all tracked packages...
PROMPT ===============================================
@@verify_packages_version.sql
PROMPT
PROMPT Step 6: Revoking cross-schema hash tracking permissions...
PROMPT ===============================================
@@90_MARS_1049_rollback_revoke_permissions.sql
PROMPT ===============================================
PROMPT MARS-1049: Rollback Complete!
PROMPT ===============================================
PROMPT
PROMPT Summary of Rollback Changes:
PROMPT - A_SOURCE_FILE_CONFIG: Removed ENCODING VARCHAR2(50) column
PROMPT - CT_MRDS.FILE_MANAGER: Restored to version 3.2.0 (before MARS-1049)
PROMPT - ODS.FILE_MANAGER_ODS: Restored to version 2.0.0 (before MARS-1049)
PROMPT - SELECT ANY DICTIONARY: Revoked from CT_MRDS (cross-schema tracking disabled)
PROMPT
PROMPT All MARS-1049 changes have been successfully rolled back.
PROMPT System is now in pre-MARS-1049 state.
PROMPT
spool off
exit

View File

@@ -0,0 +1,96 @@
-- ===================================================================
-- Simple Package Version Tracking Script
-- ===================================================================
-- Purpose: Track specified Oracle package versions
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 3.1.0 - List-Based Edition
--
-- USAGE:
-- 1. Edit package list below (add/remove packages as needed)
-- 2. Include in your install/rollback script: @@track_package_versions.sql
-- ===================================================================
SET SERVEROUTPUT ON;
DECLARE
TYPE t_package_rec IS RECORD (
owner VARCHAR2(50),
package_name VARCHAR2(50),
version VARCHAR2(50)
);
TYPE t_packages IS TABLE OF t_package_rec;
TYPE t_string_array IS TABLE OF VARCHAR2(100);
-- ===================================================================
-- PACKAGE LIST - Edit this array to specify packages to track
-- ===================================================================
-- Add or remove entries as needed for your MARS issue
-- Format: 'SCHEMA.PACKAGE_NAME'
-- ===================================================================
vPackageList t_string_array := t_string_array(
'CT_MRDS.FILE_MANAGER',
'ODS.FILE_MANAGER_ODS'
);
-- ===================================================================
vPackages t_packages := t_packages();
vVersion VARCHAR2(50);
vCount NUMBER := 0;
vOwner VARCHAR2(50);
vPackageName VARCHAR2(50);
vDotPos NUMBER;
BEGIN
DBMS_OUTPUT.PUT_LINE('========================================');
DBMS_OUTPUT.PUT_LINE('Package Version Tracking');
DBMS_OUTPUT.PUT_LINE('========================================');
-- Process each package in the list
FOR i IN 1..vPackageList.COUNT LOOP
vDotPos := INSTR(vPackageList(i), '.');
IF vDotPos > 0 THEN
vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1);
vPackageName := SUBSTR(vPackageList(i), vDotPos + 1);
-- Get package version
BEGIN
EXECUTE IMMEDIATE 'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL' INTO vVersion;
vPackages.EXTEND;
vPackages(vPackages.COUNT).owner := vOwner;
vPackages(vPackages.COUNT).package_name := vPackageName;
vPackages(vPackages.COUNT).version := vVersion;
-- Track in ENV_MANAGER
BEGIN
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
pPackageOwner => vOwner,
pPackageName => vPackageName,
pPackageVersion => vVersion,
pPackageBuildDate => TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS'),
pPackageAuthor => 'Grzegorz Michalski'
);
vCount := vCount + 1;
EXCEPTION
WHEN OTHERS THEN NULL; -- Continue even if tracking fails
END;
EXCEPTION
WHEN OTHERS THEN NULL; -- Skip packages that fail
END;
END IF;
END LOOP;
-- Display results
IF vPackages.COUNT > 0 THEN
DBMS_OUTPUT.PUT_LINE('Packages tracked: ' || vCount || ' of ' || vPackages.COUNT);
DBMS_OUTPUT.PUT_LINE('');
FOR i IN 1..vPackages.COUNT LOOP
DBMS_OUTPUT.PUT_LINE(vPackages(i).owner || '.' || vPackages(i).package_name || ' = ' || vPackages(i).version);
END LOOP;
ELSE
DBMS_OUTPUT.PUT_LINE('No packages found in list');
END IF;
DBMS_OUTPUT.PUT_LINE('========================================');
END;
/

View File

@@ -0,0 +1,62 @@
-- ===================================================================
-- Universal Package Version Verification Script
-- ===================================================================
-- Purpose: Verify all tracked Oracle packages for code changes
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
--
-- USAGE:
-- Include at the end of install/rollback scripts: @@verify_packages_version.sql
--
-- OUTPUT:
-- - List of all tracked packages with their current status
-- - OK: Package has not changed since last tracking
-- - WARNING: Package code changed without version update
-- ===================================================================
SET LINESIZE 200
SET PAGESIZE 1000
SET FEEDBACK OFF
PROMPT
PROMPT ========================================
PROMPT Package Version Verification
PROMPT ========================================
PROMPT
COLUMN PACKAGE_OWNER FORMAT A15
COLUMN PACKAGE_NAME FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN STATUS FORMAT A80
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION AS VERSION,
CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS
FROM (
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION,
ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN
FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING
)
WHERE RN = 1
ORDER BY PACKAGE_OWNER, PACKAGE_NAME;
PROMPT
PROMPT ========================================
PROMPT Verification Complete
PROMPT ========================================
PROMPT
PROMPT Legend:
PROMPT OK - Package has not changed since last tracking
PROMPT WARNING - Package code changed without version update
PROMPT
PROMPT For detailed hash information, use:
PROMPT SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;
PROMPT ========================================
SET FEEDBACK ON

View File

@@ -0,0 +1,40 @@
/*
================================================================================
MARS-1056: Update FILE_MANAGER package with corrected VARCHAR2 definitions
Script: 01_MARS_1056_update_FILE_MANAGER_package.sql
Author: Grzegorz Michalski
Date: November 25, 2025
Purpose: Implementation of VARCHAR2 definition fixes in GENERATE_EXTERNAL_TABLE_PARAMS
================================================================================
*/
SET SERVEROUTPUT ON SIZE UNLIMITED
PROMPT ===================================================================
PROMPT Installing corrected FILE_MANAGER package (MARS-1056)
PROMPT Fix: VARCHAR2 definitions will preserve CHAR/BYTE semantics
PROMPT ===================================================================
-- Install updated FILE_MANAGER specification
@@new_version/FILE_MANAGER.pkg
-- Install updated FILE_MANAGER body with VARCHAR2 fixes
@@new_version/FILE_MANAGER.pkb
-- Verify compilation (ADMIN user checks CT_MRDS schema)
SELECT object_name, object_type, status
FROM all_objects
WHERE owner = 'CT_MRDS'
AND object_name = 'FILE_MANAGER'
AND object_type IN ('PACKAGE', 'PACKAGE BODY');
-- Check for compilation errors (ADMIN user checks CT_MRDS schema)
SELECT * FROM all_errors
WHERE owner = 'CT_MRDS'
AND name = 'FILE_MANAGER';
PROMPT ===================================================================
PROMPT MARS-1056: FILE_MANAGER package updated with VARCHAR2 definition fixes
PROMPT ===================================================================
/

View File

@@ -0,0 +1,40 @@
/*
================================================================================
MARS-1056: Rollback FILE_MANAGER packages to pre-MARS-1056 versions
Author: Grzegorz Michalski
Date: November 25, 2025
Purpose: Rollback packages to versions before VARCHAR2 changes
================================================================================
*/
SET SERVEROUTPUT ON SIZE UNLIMITED
PROMPT ===================================================================
PROMPT MARS-1056: Rolling back to pre-MARS-1056 package versions
PROMPT WARNING: This will restore original VARCHAR2 definition behavior
PROMPT ===================================================================
-- Rollback FILE_MANAGER to current_version
PROMPT Rolling back FILE_MANAGER package...
@@current_version/FILE_MANAGER.pkg
@@current_version/FILE_MANAGER.pkb
-- Verify compilation (ADMIN user checks CT_MRDS schema)
SELECT object_name, object_type, status
FROM all_objects
WHERE owner = 'CT_MRDS'
AND object_name = 'FILE_MANAGER'
AND object_type IN ('PACKAGE', 'PACKAGE BODY')
ORDER BY object_name, object_type;
-- Check for compilation errors (ADMIN user checks CT_MRDS schema)
SELECT name, type, line, text FROM all_errors
WHERE owner = 'CT_MRDS'
AND name = 'FILE_MANAGER';
PROMPT ===================================================================
PROMPT MARS-1056: Rollback completed - packages restored to pre-MARS-1056 state
PROMPT VARCHAR2 definitions will use original (data_length) behavior
PROMPT ===================================================================
/

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,635 @@
create or replace PACKAGE CT_MRDS.FILE_MANAGER
AUTHID CURRENT_USER
AS
/**
* General comment for package: Please put comments for functions and procedures as shown in below example.
* It is a standard.
* The structure of comment is used by GET_PACKAGE_DOCUMENTATION function
* which returns documentation text for confluence page (to Copy-Paste it).
**/
-- Example comment:
/**
* @name EX_PROCEDURE_NAME
* @desc Procedure description
* @example select FILE_MANAGER.EX_PROCEDURE_NAME(pParameter => 129) from dual;
* @ex_rslt Example Result
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.2.1';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-11-24 12:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.2.1 (2025-11-24): MARS-1049 - Added pEncoding parameter support for CSV character set specification' || CHR(13)||CHR(10) ||
'3.2.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-20): Enhanced PROCESS_SOURCE_FILE with 6-step validation workflow' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-15): Separated export procedures into dedicated DATA_EXPORTER package' || CHR(13)||CHR(10) ||
'2.5.0 (2025-10-10): Added DELETE_SOURCE_CASCADE for safe configuration removal' || CHR(13)||CHR(10) ||
'2.0.0 (2025-09-25): Added official path patterns support (INBOX 3-level, ODS 2-level, ARCHIVE 2-level)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-01): Initial release with file processing and validation capabilities';
TYPE tSourceFileReceived IS RECORD
(
A_SOURCE_FILE_RECEIVED_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE,
A_SOURCE_FILE_CONFIG_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_CONFIG_KEY%TYPE,
SOURCE_FILE_PREFIX_INBOX VARCHAR2(430),
SOURCE_FILE_PREFIX_ODS VARCHAR2(430),
SOURCE_FILE_PREFIX_QUARANTINE VARCHAR2(430),
SOURCE_FILE_PREFIX_ARCHIVE VARCHAR2(430),
SOURCE_FILE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.SOURCE_FILE_NAME%TYPE,
RECEPTION_DATE CT_MRDS.A_SOURCE_FILE_RECEIVED.RECEPTION_DATE%TYPE,
PROCESSING_STATUS CT_MRDS.A_SOURCE_FILE_RECEIVED.PROCESSING_STATUS%TYPE,
EXTERNAL_TABLE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.EXTERNAL_TABLE_NAME%TYPE
);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgSourceFileConfigKey PLS_INTEGER;
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_SOURCE_FILE_CONFIG
* @desc Get source file type by matching the source file name against source file type naming patterns
* or by specifying the id of a received source file.
* @example ...
* @ex_rslt "CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE"
**/
FUNCTION GET_SOURCE_FILE_CONFIG(pFileUri IN VARCHAR2 DEFAULT NULL
, pSourceFileReceivedKey IN NUMBER DEFAULT NULL
, pSourceFileConfigKey IN NUMBER DEFAULT NULL)
RETURN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a newly received source file in A_SOURCE_FILE_RECEIVED table.
* This overload automatically determines source file type from the file name.
* It returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2
)
RETURN PLS_INTEGER;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a new new source file in A_SOURCE_FILE_RECEIVED table based on pSourceFileReceivedName and pSourceFileConfig.
* Then it returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(
* pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv'
* ,pSourceFileConfig => ...A_SOURCE_FILE_CONFIG%ROWTYPE... );
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2,
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
)
RETURN PLS_INTEGER;
/**
* @name SET_SOURCE_FILE_RECEIVED_STATUS
* @desc Set status of file in A_SOURCE_FILE_RECEIVED table - PROCESSING_STATUS column
* based on A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY
* and provided value of pStatus parameter
* @example exec FILE_MANAGER.SET_SOURCE_FILE_RECEIVED_STATUS(pSourceFileReceivedKey => 377, pStatus => 'READY_FOR_INGESTION');
**/
PROCEDURE SET_SOURCE_FILE_RECEIVED_STATUS(
pSourceFileReceivedKey IN PLS_INTEGER,
pStatus IN VARCHAR2
);
/**
* @name GET_EXTERNAL_TABLE_COLUMNS
* @desc Function used to get string with all table columns definitions based on pTargetTableTemplate "TEMPLATE TABLE" name.
* It used for creating "EXTERNAL TABLE" using CREATE_EXTERNAL_TABLE procedure.
* @example select FILE_MANAGER.GET_EXTERNAL_TABLE_COLUMNS(pTargetTableTemplate => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER') from dual;
* @ex_rslt "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "REV_NUMBER" NUMBER(28,0),
* "REF_DATE" DATE,
* "FREE_TEXT" VARCHAR2(1000 CHAR),
* "MLF_BS_TOTAL" NUMBER(28,10),
* "DF_BS_TOTAL" NUMBER(28,10),
* "MLF_SF_TOTAL" NUMBER(28,10),
* "DF_SF_TOTAL" NUMBER(28,10)
**/
FUNCTION GET_EXTERNAL_TABLE_COLUMNS (
pTargetTableTemplate IN VARCHAR2
)
RETURN CLOB;
/**
* @name CREATE_EXTERNAL_TABLE
* @desc A wrapper procedure for DBMS_CLOUD.CREATE_EXTERNAL_TABLE which creates External Table
* MARS-1049: Added pEncoding parameter for CSV character set specification
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252')
* If provided, adds CHARACTERSET clause to external table definition
* @example
* begin
* FILE_MANAGER.CREATE_EXTERNAL_TABLE(
* pTableName => 'STANDING_FACILITIES_HEADER',
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
* pPrefix => 'ODS/LM/STANDING_FACILITIES_HEADER/',
* pBucketUri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/',
* pFileName => NULL,
* pDelimiter => ',',
* pEncoding => 'UTF8'
* );
* end;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.CREATE_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_SOURCE_FILE_RECEIVED
* @desc A wrapper procedure for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE
* It validate External table build upon single file
* provided by pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.VALIDATE_SOURCE_FILE_RECEIVED(pSourceFileReceivedKey => 377);
**/
PROCEDURE VALIDATE_SOURCE_FILE_RECEIVED
(
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_EXTERNAL_TABLE
* @desc A wrapper function for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE.
* It validates External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt FAILED
**/
FUNCTION VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name S_VALIDATE_EXTERNAL_TABLE
* @desc A function which checks if SELECT query reterns any rows.
* It trys to selects External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.S_VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt PASSED
**/
FUNCTION S_VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name DROP_EXTERNAL_TABLE
* @desc It drops External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.DROP_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);
**/
PROCEDURE DROP_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name COPY_FILE
* @desc It copies file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS'
* @example exec FILE_MANAGER.COPY_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE COPY_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name MOVE_FILE
* @desc It moves file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS', 'QUARANTINE'
* @example exec FILE_MANAGER.MOVE_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE MOVE_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name DELETE_FOLDER_CONTENTS
* @desc It deletes all files from specified folder in the cloud storage.
* The procedure lists all objects in the specified folder prefix and deletes them one by one.
* pBucketArea parameter specifies which bucket to use: 'INBOX', 'DATA', 'ARCHIVE'
* pFolderPrefix parameter specifies the folder path within the bucket (e.g., 'C2D/UC_DISSEM/UC_NMA_DISSEM/')
* @example exec FILE_MANAGER.DELETE_FOLDER_CONTENTS(pBucketArea => 'INBOX', pFolderPrefix => 'C2D/UC_DISSEM/UC_NMA_DISSEM/');
**/
PROCEDURE DELETE_FOLDER_CONTENTS(
pBucketArea IN VARCHAR2,
pFolderPrefix IN VARCHAR2
);
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter.
* Ubmrella procedure that calls:
* - REGISTER_SOURCE_FILE_RECEIVED;
* - CREATE_EXTERNAL_TABLE;
* - VALIDATE_SOURCE_FILE_RECEIVED;
* - DROP_EXTERNAL_TABLE;
* - MOVE_FILE;
* @example exec FILE_MANAGER.PROCESS_SOURCE_FILE(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
**/
PROCEDURE PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
;
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter and return processing result value.
* It returns (success/failure) => 0 / -(value).
* Ubmrella function that calls PROCESS_SOURCE_FILE procedure.
* @example
* declare
* vResult PLS_INTEGER;
* begin
* vResult := CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(PSOURCEFILERECEIVEDNAME => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* DBMS_OUTPUT.PUT_LINE('vResult = ' || vResult);
* end;
* @ex_rslt 0
* -20021
**/
FUNCTION PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
RETURN PLS_INTEGER;
/**
* @name GET_DATE_FORMAT
* @desc Returns date format for specified template table name and column name.
* Date is taken from configuration A_COLUMN_DATE_FORMAT table.
* @example select FILE_MANAGER.GET_DATE_FORMAT(
* pTemplateTableName => 'STANDING_FACILITIES_HEADER',
* pColumnName => 'SNAPSHOT_DATE')
* from dual;
* @ex_rslt DD/MM/YYYY HH24:MI:SS
**/
FUNCTION GET_DATE_FORMAT(
pTemplateTableName IN VARCHAR2,
pColumnName IN VARCHAR2
) RETURN VARCHAR2;
/**
* @name GENERATE_EXTERNAL_TABLE_PARAMS
* @desc It builds two strings: pColumnList and pFieldList for specified Template Table name, by parameter: pTemplateTableName.
* @example
* declare
* vColumnList CLOB;
* vFieldList CLOB;
* begin
* FILE_MANAGER.GENERATE_EXTERNAL_TABLE_PARAMS (
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
* ,pColumnList => vColumnList
* ,pFieldList => vFieldList
* );
* DBMS_OUTPUT.PUT_LINE('vColumnList = '||vColumnList);
* DBMS_OUTPUT.PUT_LINE('vFieldList = '||vFieldList);
* end;
* /
**/
PROCEDURE GENERATE_EXTERNAL_TABLE_PARAMS (
pTemplateTableName IN VARCHAR2,
pColumnList OUT CLOB,
pFieldList OUT CLOB
);
/**
* @name ADD_SOURCE
* @desc Insert a new record to A_SOURCE table.
* pSourceKey is a PRIMARY KEY value.
**/
PROCEDURE ADD_SOURCE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
pSourceName IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);
/**
* @name DELETE_SOURCE_CASCADE
* @desc Safely deletes a SOURCE specified by pSourceKey parameter from A_SOURCE table and all dependent tables:
* - A_SOURCE_FILE_CONFIG
* - A_SOURCE_FILE_RECEIVED
* - A_COLUMN_DATE_FORMAT (only if template table is not shared with other source systems)
* The procedure checks if template tables are shared before deleting date format configurations.
* If a template table is used by multiple source systems, date formats are preserved.
* @example CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(pSourceKey => 'TEST_SYS');
**/
PROCEDURE DELETE_SOURCE_CASCADE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE
);
/**
* @name GET_CONTAINER_SOURCE_FILE_CONFIG_KEY
* @desc For specified parameter pSourceFileId (A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID)
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY for related CONTAINER record.
* @example select FILE_MANAGER.GET_CONTAINER_SOURCE_FILE_CONFIG_KEY(
* pSourceFileId => 'UC_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_CONTAINER_SOURCE_FILE_CONFIG_KEY (
pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name GET_SOURCE_FILE_CONFIG_KEY
* @desc For specified input parameters,
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY.
* @example select FILE_MANAGER.GET_SOURCE_FILE_CONFIG_KEY (
* pSourceFileType => 'INPUT'
* ,pSourceFileId => 'UC_DISSEM'
* ,pTableId => 'UC_NMA_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_SOURCE_FILE_CONFIG_KEY (
pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE DEFAULT 'INPUT'
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name ADD_SOURCE_FILE_CONFIG
* @desc Insert a new record to A_SOURCE_FILE_CONFIG table.
* MARS-1049: Added pEncoding parameter for CSV character set specification.
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252', 'EE8ISO8859P2')
* If NULL, no CHARACTERSET clause is added to external table definitions
* @example CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
* pSourceKey => 'C2D', pSourceFileType => 'INPUT',
* pSourceFileId => 'UC_DISSEM', pTableId => 'METADATA_LOADS',
* pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
* pEncoding => 'UTF8'
* );
**/
PROCEDURE ADD_SOURCE_FILE_CONFIG (
pSourceKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE
,pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pSourceFileDesc IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE
,pSourceFileNamePattern IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE DEFAULT NULL
,pTemplateTableName IN CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE DEFAULT NULL
,pContainerFileKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE DEFAULT NULL
,pEncoding IN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING%TYPE DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name ADD_COLUMN_DATE_FORMAT
* @desc Insert a new record to A_COLUMN_DATE_FORMAT table.
**/
PROCEDURE ADD_COLUMN_DATE_FORMAT (
pTemplateTableName IN CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE
,pColumnName IN CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE
,pDateFormat IN CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);
/**
* @name GET_BUCKET_URI
* @desc Function used to get string with bucket http url.
* Possible input values for pBucketArea are: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example select FILE_MANAGER.GET_BUCKET_URI(pBucketArea => 'ODS') from dual;
* @ex_rslt https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/
**/
FUNCTION GET_BUCKET_URI(pBucketArea VARCHAR2)
RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_CONFIG_INFO
* @desc Function returns details about A_SOURCE_FILE_CONFIG record
* for specified pSourceFileConfigKey (A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY).
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO (
* pSourceFileConfigKey => 128
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
* @ex_rslt
* Details about File Configuration:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 128
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Details about related Container Config:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 126
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Column Date Format config entries:
* --------------------------------
* TEMPLATE_TABLE_NAME = CT_ET_TEMPLATES.C2D_UC_MA_DISSEM
* ...
* --------------------------------
**/
FUNCTION GET_DET_SOURCE_FILE_CONFIG_INFO (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_RECEIVED_INFO
* @desc Function returns details about A_SOURCE_FILE_RECEIVED record
* for specified pSourceFileReceivedKey (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY).
* If pIncludeConfigInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_RECEIVED_INFO (
* pSourceFileReceivedKey => 377
* ,pIncludeConfigInfo => 1
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
*
**/
FUNCTION GET_DET_SOURCE_FILE_RECEIVED_INFO (
pSourceFileReceivedKey IN CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE
,pIncludeConfigInfo IN PLS_INTEGER DEFAULT 1
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_USER_LOAD_OPERATIONS
* @desc Function returns details from USER_LOAD_OPERATIONS table
* for specified pOperationId.
* @example select FILE_MANAGER.GET_DET_USER_LOAD_OPERATIONS (pOperationId => 3608) from dual;
* @ex_rslt
* Details about USER_LOAD_OPERATIONS where ID = 3608
* --------------------------------
* ID = 3608
* TYPE = VALIDATE
* SID = 31260
* SERIAL# = 52915
* START_TIME = 2025-05-20 10.08.24.436983 EUROPE/BELGRADE
* UPDATE_TIME = 2025-05-20 10.08.24.458643 EUROPE/BELGRADE
* STATUS = FAILED
* OWNER_NAME = CT_MRDS
* TABLE_NAME = STANDING_FACILITIES_HEADER
* PARTITION_NAME =
* SUBPARTITION_NAME =
* FILE_URI_LIST =
* ROWS_LOADED =
* LOGFILE_TABLE = VALIDATE$3608_LOG
* BADFILE_TABLE = VALIDATE$3608_BAD
* STATUS_TABLE =
* TEMPEXT_TABLE =
* CREDENTIAL_NAME =
* EXPIRATION_TIME = 2025-05-22 10.08.24.436983000 EUROPE/BELGRADE
* --------------------------------
**/
FUNCTION GET_DET_USER_LOAD_OPERATIONS (
pOperationId PLS_INTEGER
) RETURN VARCHAR2;
/**
* @name ANALYZE_VALIDATION_ERRORS
* @desc Wrapper function that analyzes validation errors for a source file using its received key.
* Automatically derives template schema, table name, CSV URI and validation log table
* from file metadata and calls ENV_MANAGER.ANALYZE_VALIDATION_ERRORS.
* @example SELECT FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;
* @ex_rslt Detailed validation analysis report with column mismatches and solutions
**/
FUNCTION ANALYZE_VALIDATION_ERRORS(
pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
* @ex_rslt 3.2.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER
* Version: 3.2.0
* Build Date: 2025-10-22 16:30:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER Version History:
* 3.2.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END;
/
/

View File

@@ -0,0 +1,99 @@
/*
================================================================================
MARS-1056: Master Installation Script for VARCHAR2 Definition Fixes
Author: Grzegorz Michalski
Date: November 25, 2025
Purpose: Main installation script for VARCHAR2 definition fixes in external tables
PROBLEM:
FILE_MANAGER.GENERATE_EXTERNAL_TABLE_PARAMS uses data_length instead of char_length
for VARCHAR2 columns, causing incorrect definitions in external tables.
SOLUTION:
Modify column definition generation logic to preserve CHAR/BYTE semantics
from template tables.
USAGE:
sql 'CT_MRDS/Cloudpass#34@ggmichalski_high' '@install_mars1056.sql'
================================================================================
*/
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'INSTALL_MARS_1056_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF
PROMPT
PROMPT ========================================================================
PROMPT MARS-1056: VARCHAR2 Definition Fixes Installation
PROMPT ========================================================================
PROMPT
PROMPT Problem: FILE_MANAGER creates external tables with incorrect VARCHAR2 definitions
PROMPT Solution: Preserve CHAR/BYTE semantics from template tables
PROMPT
PROMPT Installation will:
PROMPT 1. Install updated FILE_MANAGER with VARCHAR2 fixes (v3.3.0)
PROMPT 2. Track package version
PROMPT 3. Verify all tracked packages
PROMPT
PROMPT ========================================================================
-- Confirm installation with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
-- Set current schema context
ALTER SESSION SET CURRENT_SCHEMA = CT_MRDS;
PROMPT
PROMPT === Step 1: Install updated FILE_MANAGER package ===
@@01_MARS_1056_update_FILE_MANAGER_package.sql
PROMPT
PROMPT === Step 2: Track package version ===
@@track_package_versions.sql
PROMPT
PROMPT === Step 3: Verify all tracked packages ===
@@verify_packages_version.sql
PROMPT
PROMPT ========================================================================
PROMPT MARS-1056: Installation completed successfully
PROMPT ========================================================================
PROMPT
PROMPT VARCHAR2 definition fixes have been implemented:
PROMPT - GENERATE_EXTERNAL_TABLE_PARAMS now preserves CHAR/BYTE semantics
PROMPT - External tables will match template table definitions
PROMPT - Test results shown above should indicate SUCCESS
PROMPT
PROMPT Next steps:
PROMPT 1. Verify test results above show all SUCCESS indicators
PROMPT 2. Test external table creation with various templates
PROMPT 3. Monitor for any issues with existing processes
PROMPT 4. If problems occur, use rollback_mars1056.sql
PROMPT
PROMPT ========================================================================
PROMPT Installation log saved to: &_filename
PROMPT ========================================================================
spool off
quit;

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,636 @@
create or replace PACKAGE CT_MRDS.FILE_MANAGER
AUTHID CURRENT_USER
AS
/**
* General comment for package: Please put comments for functions and procedures as shown in below example.
* It is a standard.
* The structure of comment is used by GET_PACKAGE_DOCUMENTATION function
* which returns documentation text for confluence page (to Copy-Paste it).
**/
-- Example comment:
/**
* @name EX_PROCEDURE_NAME
* @desc Procedure description
* @example select FILE_MANAGER.EX_PROCEDURE_NAME(pParameter => 129) from dual;
* @ex_rslt Example Result
**/
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION CONSTANT VARCHAR2(10) := '3.3.0';
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := '2025-11-26 10:00:00';
PACKAGE_AUTHOR CONSTANT VARCHAR2(100) := 'Grzegorz Michalski';
-- Version History (Latest changes first)
VERSION_HISTORY CONSTANT VARCHAR2(4000) :=
'3.3.0 (2025-11-26): MARS-1056 - Fixed VARCHAR2 definitions in GENERATE_EXTERNAL_TABLE_PARAMS to preserve CHAR/BYTE semantics from template tables' || CHR(13)||CHR(10) ||
'3.2.1 (2025-11-24): MARS-1049 - Added pEncoding parameter support for CSV character set specification' || CHR(13)||CHR(10) ||
'3.2.0 (2025-10-22): Added package versioning system using centralized ENV_MANAGER functions' || CHR(13)||CHR(10) ||
'3.1.0 (2025-10-20): Enhanced PROCESS_SOURCE_FILE with 6-step validation workflow' || CHR(13)||CHR(10) ||
'3.0.0 (2025-10-15): Separated export procedures into dedicated DATA_EXPORTER package' || CHR(13)||CHR(10) ||
'2.5.0 (2025-10-10): Added DELETE_SOURCE_CASCADE for safe configuration removal' || CHR(13)||CHR(10) ||
'2.0.0 (2025-09-25): Added official path patterns support (INBOX 3-level, ODS 2-level, ARCHIVE 2-level)' || CHR(13)||CHR(10) ||
'1.0.0 (2025-09-01): Initial release with file processing and validation capabilities';
TYPE tSourceFileReceived IS RECORD
(
A_SOURCE_FILE_RECEIVED_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE,
A_SOURCE_FILE_CONFIG_KEY CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_CONFIG_KEY%TYPE,
SOURCE_FILE_PREFIX_INBOX VARCHAR2(430),
SOURCE_FILE_PREFIX_ODS VARCHAR2(430),
SOURCE_FILE_PREFIX_QUARANTINE VARCHAR2(430),
SOURCE_FILE_PREFIX_ARCHIVE VARCHAR2(430),
SOURCE_FILE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.SOURCE_FILE_NAME%TYPE,
RECEPTION_DATE CT_MRDS.A_SOURCE_FILE_RECEIVED.RECEPTION_DATE%TYPE,
PROCESSING_STATUS CT_MRDS.A_SOURCE_FILE_RECEIVED.PROCESSING_STATUS%TYPE,
EXTERNAL_TABLE_NAME CT_MRDS.A_SOURCE_FILE_RECEIVED.EXTERNAL_TABLE_NAME%TYPE
);
cgBL CONSTANT VARCHAR2(2) := CHR(13)||CHR(10);
vgSourceFileConfigKey PLS_INTEGER;
vgMsgTmp VARCHAR2(32000);
---------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_SOURCE_FILE_CONFIG
* @desc Get source file type by matching the source file name against source file type naming patterns
* or by specifying the id of a received source file.
* @example ...
* @ex_rslt "CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE"
**/
FUNCTION GET_SOURCE_FILE_CONFIG(pFileUri IN VARCHAR2 DEFAULT NULL
, pSourceFileReceivedKey IN NUMBER DEFAULT NULL
, pSourceFileConfigKey IN NUMBER DEFAULT NULL)
RETURN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a newly received source file in A_SOURCE_FILE_RECEIVED table.
* This overload automatically determines source file type from the file name.
* It returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2
)
RETURN PLS_INTEGER;
/**
* @name REGISTER_SOURCE_FILE_RECEIVED
* @desc Register a new new source file in A_SOURCE_FILE_RECEIVED table based on pSourceFileReceivedName and pSourceFileConfig.
* Then it returns the value of A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY column for newly added record.
* @example vSourceFileReceivedKey := FILE_MANAGER.REGISTER_SOURCE_FILE_RECEIVED(
* pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv'
* ,pSourceFileConfig => ...A_SOURCE_FILE_CONFIG%ROWTYPE... );
* @ex_rslt 3245
**/
FUNCTION REGISTER_SOURCE_FILE_RECEIVED (
pSourceFileReceivedName IN VARCHAR2,
pSourceFileConfig IN CT_MRDS.A_SOURCE_FILE_CONFIG%ROWTYPE
)
RETURN PLS_INTEGER;
/**
* @name SET_SOURCE_FILE_RECEIVED_STATUS
* @desc Set status of file in A_SOURCE_FILE_RECEIVED table - PROCESSING_STATUS column
* based on A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY
* and provided value of pStatus parameter
* @example exec FILE_MANAGER.SET_SOURCE_FILE_RECEIVED_STATUS(pSourceFileReceivedKey => 377, pStatus => 'READY_FOR_INGESTION');
**/
PROCEDURE SET_SOURCE_FILE_RECEIVED_STATUS(
pSourceFileReceivedKey IN PLS_INTEGER,
pStatus IN VARCHAR2
);
/**
* @name GET_EXTERNAL_TABLE_COLUMNS
* @desc Function used to get string with all table columns definitions based on pTargetTableTemplate "TEMPLATE TABLE" name.
* It used for creating "EXTERNAL TABLE" using CREATE_EXTERNAL_TABLE procedure.
* @example select FILE_MANAGER.GET_EXTERNAL_TABLE_COLUMNS(pTargetTableTemplate => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER') from dual;
* @ex_rslt "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
* "REV_NUMBER" NUMBER(28,0),
* "REF_DATE" DATE,
* "FREE_TEXT" VARCHAR2(1000 CHAR),
* "MLF_BS_TOTAL" NUMBER(28,10),
* "DF_BS_TOTAL" NUMBER(28,10),
* "MLF_SF_TOTAL" NUMBER(28,10),
* "DF_SF_TOTAL" NUMBER(28,10)
**/
FUNCTION GET_EXTERNAL_TABLE_COLUMNS (
pTargetTableTemplate IN VARCHAR2
)
RETURN CLOB;
/**
* @name CREATE_EXTERNAL_TABLE
* @desc A wrapper procedure for DBMS_CLOUD.CREATE_EXTERNAL_TABLE which creates External Table
* MARS-1049: Added pEncoding parameter for CSV character set specification
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252')
* If provided, adds CHARACTERSET clause to external table definition
* @example
* begin
* FILE_MANAGER.CREATE_EXTERNAL_TABLE(
* pTableName => 'STANDING_FACILITIES_HEADER',
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
* pPrefix => 'ODS/LM/STANDING_FACILITIES_HEADER/',
* pBucketUri => 'https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/',
* pFileName => NULL,
* pDelimiter => ',',
* pEncoding => 'UTF8'
* );
* end;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pTableName IN VARCHAR2,
pTemplateTableName IN VARCHAR2,
pPrefix IN VARCHAR2,
pBucketUri IN VARCHAR2 DEFAULT ENV_MANAGER.gvInboxBucketUri,
pFileName IN VARCHAR2 DEFAULT NULL,
pDelimiter IN VARCHAR2 DEFAULT ',',
pEncoding IN VARCHAR2 DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name CREATE_EXTERNAL_TABLE
* @desc Creates External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.CREATE_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);;
**/
PROCEDURE CREATE_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_SOURCE_FILE_RECEIVED
* @desc A wrapper procedure for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE
* It validate External table build upon single file
* provided by pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.VALIDATE_SOURCE_FILE_RECEIVED(pSourceFileReceivedKey => 377);
**/
PROCEDURE VALIDATE_SOURCE_FILE_RECEIVED
(
pSourceFileReceivedKey IN NUMBER
);
/**
* @name VALIDATE_EXTERNAL_TABLE
* @desc A wrapper function for DBMS_CLOUD.VALIDATE_EXTERNAL_TABLE.
* It validates External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt FAILED
**/
FUNCTION VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name S_VALIDATE_EXTERNAL_TABLE
* @desc A function which checks if SELECT query reterns any rows.
* It trys to selects External Table provided by parameter pTableName.
* It returns: PASSED or FAILED.
* @example
* declare
* vStatus VARCHAR2(100);
* begin
* vStatus := FILE_MANAGER.S_VALIDATE_EXTERNAL_TABLE(pTableName => 'STANDING_FACILITIES_HEADER');
* DBMS_OUTPUT.PUT_LINE('vStatus = '||vStatus);
* end;
*
* @ex_rslt PASSED
**/
FUNCTION S_VALIDATE_EXTERNAL_TABLE(pTableName IN VARCHAR2)
RETURN VARCHAR2;
/**
* @name DROP_EXTERNAL_TABLE
* @desc It drops External Table for single file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* @example exec FILE_MANAGER.DROP_EXTERNAL_TABLE(pSourceFileReceivedKey => 377);
**/
PROCEDURE DROP_EXTERNAL_TABLE (
pSourceFileReceivedKey IN NUMBER
);
/**
* @name COPY_FILE
* @desc It copies file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS'
* @example exec FILE_MANAGER.COPY_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE COPY_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name MOVE_FILE
* @desc It moves file provided by
* pSourceFileReceivedKey parameter (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY)
* into destination provided by pDestination parameter.
* pDestination parameter allowed values are: 'ODS', 'QUARANTINE'
* @example exec FILE_MANAGER.MOVE_FILE(pSourceFileReceivedKey => 377, pDestination => 'ODS');
**/
PROCEDURE MOVE_FILE(
pSourceFileReceivedKey IN NUMBER,
pDestination IN VARCHAR2
);
/**
* @name DELETE_FOLDER_CONTENTS
* @desc It deletes all files from specified folder in the cloud storage.
* The procedure lists all objects in the specified folder prefix and deletes them one by one.
* pBucketArea parameter specifies which bucket to use: 'INBOX', 'DATA', 'ARCHIVE'
* pFolderPrefix parameter specifies the folder path within the bucket (e.g., 'C2D/UC_DISSEM/UC_NMA_DISSEM/')
* @example exec FILE_MANAGER.DELETE_FOLDER_CONTENTS(pBucketArea => 'INBOX', pFolderPrefix => 'C2D/UC_DISSEM/UC_NMA_DISSEM/');
**/
PROCEDURE DELETE_FOLDER_CONTENTS(
pBucketArea IN VARCHAR2,
pFolderPrefix IN VARCHAR2
);
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter.
* Ubmrella procedure that calls:
* - REGISTER_SOURCE_FILE_RECEIVED;
* - CREATE_EXTERNAL_TABLE;
* - VALIDATE_SOURCE_FILE_RECEIVED;
* - DROP_EXTERNAL_TABLE;
* - MOVE_FILE;
* @example exec FILE_MANAGER.PROCESS_SOURCE_FILE(pSourceFileReceivedName => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
**/
PROCEDURE PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
;
/**
* @name PROCESS_SOURCE_FILE
* @desc It process file provided by pSourceFileReceivedName parameter and return processing result value.
* It returns (success/failure) => 0 / -(value).
* Ubmrella function that calls PROCESS_SOURCE_FILE procedure.
* @example
* declare
* vResult PLS_INTEGER;
* begin
* vResult := CT_MRDS.FILE_MANAGER.PROCESS_SOURCE_FILE(PSOURCEFILERECEIVEDNAME => 'INBOX/C2D/UC_DISSEM/UC_NMA_DISSEM/UC_NMA_DISSEM-277740.csv');
* DBMS_OUTPUT.PUT_LINE('vResult = ' || vResult);
* end;
* @ex_rslt 0
* -20021
**/
FUNCTION PROCESS_SOURCE_FILE(pSourceFileReceivedName IN VARCHAR2)
RETURN PLS_INTEGER;
/**
* @name GET_DATE_FORMAT
* @desc Returns date format for specified template table name and column name.
* Date is taken from configuration A_COLUMN_DATE_FORMAT table.
* @example select FILE_MANAGER.GET_DATE_FORMAT(
* pTemplateTableName => 'STANDING_FACILITIES_HEADER',
* pColumnName => 'SNAPSHOT_DATE')
* from dual;
* @ex_rslt DD/MM/YYYY HH24:MI:SS
**/
FUNCTION GET_DATE_FORMAT(
pTemplateTableName IN VARCHAR2,
pColumnName IN VARCHAR2
) RETURN VARCHAR2;
/**
* @name GENERATE_EXTERNAL_TABLE_PARAMS
* @desc It builds two strings: pColumnList and pFieldList for specified Template Table name, by parameter: pTemplateTableName.
* @example
* declare
* vColumnList CLOB;
* vFieldList CLOB;
* begin
* FILE_MANAGER.GENERATE_EXTERNAL_TABLE_PARAMS (
* pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
* ,pColumnList => vColumnList
* ,pFieldList => vFieldList
* );
* DBMS_OUTPUT.PUT_LINE('vColumnList = '||vColumnList);
* DBMS_OUTPUT.PUT_LINE('vFieldList = '||vFieldList);
* end;
* /
**/
PROCEDURE GENERATE_EXTERNAL_TABLE_PARAMS (
pTemplateTableName IN VARCHAR2,
pColumnList OUT CLOB,
pFieldList OUT CLOB
);
/**
* @name ADD_SOURCE
* @desc Insert a new record to A_SOURCE table.
* pSourceKey is a PRIMARY KEY value.
**/
PROCEDURE ADD_SOURCE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE,
pSourceName IN CT_MRDS.A_SOURCE.SOURCE_NAME%TYPE
);
/**
* @name DELETE_SOURCE_CASCADE
* @desc Safely deletes a SOURCE specified by pSourceKey parameter from A_SOURCE table and all dependent tables:
* - A_SOURCE_FILE_CONFIG
* - A_SOURCE_FILE_RECEIVED
* - A_COLUMN_DATE_FORMAT (only if template table is not shared with other source systems)
* The procedure checks if template tables are shared before deleting date format configurations.
* If a template table is used by multiple source systems, date formats are preserved.
* @example CALL CT_MRDS.FILE_MANAGER.DELETE_SOURCE_CASCADE(pSourceKey => 'TEST_SYS');
**/
PROCEDURE DELETE_SOURCE_CASCADE (
pSourceKey IN CT_MRDS.A_SOURCE.A_SOURCE_KEY%TYPE
);
/**
* @name GET_CONTAINER_SOURCE_FILE_CONFIG_KEY
* @desc For specified parameter pSourceFileId (A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID)
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY for related CONTAINER record.
* @example select FILE_MANAGER.GET_CONTAINER_SOURCE_FILE_CONFIG_KEY(
* pSourceFileId => 'UC_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_CONTAINER_SOURCE_FILE_CONFIG_KEY (
pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name GET_SOURCE_FILE_CONFIG_KEY
* @desc For specified input parameters,
* it returns A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY.
* @example select FILE_MANAGER.GET_SOURCE_FILE_CONFIG_KEY (
* pSourceFileType => 'INPUT'
* ,pSourceFileId => 'UC_DISSEM'
* ,pTableId => 'UC_NMA_DISSEM')
* from dual;
* @ex_rslt 126
**/
FUNCTION GET_SOURCE_FILE_CONFIG_KEY (
pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE DEFAULT 'INPUT'
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE
) RETURN PLS_INTEGER;
/**
* @name ADD_SOURCE_FILE_CONFIG
* @desc Insert a new record to A_SOURCE_FILE_CONFIG table.
* MARS-1049: Added pEncoding parameter for CSV character set specification.
* @param pEncoding - Character set encoding for CSV files (e.g., 'UTF8', 'WE8MSWIN1252', 'EE8ISO8859P2')
* If NULL, no CHARACTERSET clause is added to external table definitions
* @example CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
* pSourceKey => 'C2D', pSourceFileType => 'INPUT',
* pSourceFileId => 'UC_DISSEM', pTableId => 'METADATA_LOADS',
* pTemplateTableName => 'CT_ET_TEMPLATES.C2D_A_UC_DISSEM_METADATA_LOADS',
* pEncoding => 'UTF8'
* );
**/
PROCEDURE ADD_SOURCE_FILE_CONFIG (
pSourceKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_KEY%TYPE
,pSourceFileType IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_TYPE%TYPE
,pSourceFileId IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_ID%TYPE
,pSourceFileDesc IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_DESC%TYPE
,pSourceFileNamePattern IN CT_MRDS.A_SOURCE_FILE_CONFIG.SOURCE_FILE_NAME_PATTERN%TYPE
,pTableId IN CT_MRDS.A_SOURCE_FILE_CONFIG.TABLE_ID%TYPE DEFAULT NULL
,pTemplateTableName IN CT_MRDS.A_SOURCE_FILE_CONFIG.TEMPLATE_TABLE_NAME%TYPE DEFAULT NULL
,pContainerFileKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.CONTAINER_FILE_KEY%TYPE DEFAULT NULL
,pEncoding IN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING%TYPE DEFAULT NULL -- MARS-1049: NOWY PARAMETR
);
/**
* @name ADD_COLUMN_DATE_FORMAT
* @desc Insert a new record to A_COLUMN_DATE_FORMAT table.
**/
PROCEDURE ADD_COLUMN_DATE_FORMAT (
pTemplateTableName IN CT_MRDS.A_COLUMN_DATE_FORMAT.TEMPLATE_TABLE_NAME%TYPE
,pColumnName IN CT_MRDS.A_COLUMN_DATE_FORMAT.COLUMN_NAME%TYPE
,pDateFormat IN CT_MRDS.A_COLUMN_DATE_FORMAT.DATE_FORMAT%TYPE
);
/**
* @name GET_BUCKET_URI
* @desc Function used to get string with bucket http url.
* Possible input values for pBucketArea are: 'INBOX', 'ODS', 'DATA', 'ARCHIVE'
* @example select FILE_MANAGER.GET_BUCKET_URI(pBucketArea => 'ODS') from dual;
* @ex_rslt https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/frcnomajoc7v/b/mrds_data_tst/o/
**/
FUNCTION GET_BUCKET_URI(pBucketArea VARCHAR2)
RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_CONFIG_INFO
* @desc Function returns details about A_SOURCE_FILE_CONFIG record
* for specified pSourceFileConfigKey (A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY).
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_CONFIG_INFO (
* pSourceFileConfigKey => 128
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
* @ex_rslt
* Details about File Configuration:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 128
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Details about related Container Config:
* --------------------------------
* A_SOURCE_FILE_CONFIG_KEY = 126
* A_SOURCE_KEY = C2D
* ...
* --------------------------------
*
* Column Date Format config entries:
* --------------------------------
* TEMPLATE_TABLE_NAME = CT_ET_TEMPLATES.C2D_UC_MA_DISSEM
* ...
* --------------------------------
**/
FUNCTION GET_DET_SOURCE_FILE_CONFIG_INFO (
pSourceFileConfigKey IN CT_MRDS.A_SOURCE_FILE_CONFIG.A_SOURCE_FILE_CONFIG_KEY%TYPE
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_SOURCE_FILE_RECEIVED_INFO
* @desc Function returns details about A_SOURCE_FILE_RECEIVED record
* for specified pSourceFileReceivedKey (A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY).
* If pIncludeConfigInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeContainerInfo is <> 0 it returns additional info about related Container config record (A_SOURCE_FILE_CONFIG)
* If pIncludeColumnFormatInfo is <> 0 it returns additional info about related ColumnFormat config record (A_COLUMN_DATE_FORMAT)
* @example select FILE_MANAGER.GET_DET_SOURCE_FILE_RECEIVED_INFO (
* pSourceFileReceivedKey => 377
* ,pIncludeConfigInfo => 1
* ,pIncludeContainerInfo => 1
* ,pIncludeColumnFormatInfo => 1
* ) from dual;
*
**/
FUNCTION GET_DET_SOURCE_FILE_RECEIVED_INFO (
pSourceFileReceivedKey IN CT_MRDS.A_SOURCE_FILE_RECEIVED.A_SOURCE_FILE_RECEIVED_KEY%TYPE
,pIncludeConfigInfo IN PLS_INTEGER DEFAULT 1
,pIncludeContainerInfo IN PLS_INTEGER DEFAULT 1
,pIncludeColumnFormatInfo IN PLS_INTEGER DEFAULT 1
) RETURN VARCHAR2;
/**
* @name GET_DET_USER_LOAD_OPERATIONS
* @desc Function returns details from USER_LOAD_OPERATIONS table
* for specified pOperationId.
* @example select FILE_MANAGER.GET_DET_USER_LOAD_OPERATIONS (pOperationId => 3608) from dual;
* @ex_rslt
* Details about USER_LOAD_OPERATIONS where ID = 3608
* --------------------------------
* ID = 3608
* TYPE = VALIDATE
* SID = 31260
* SERIAL# = 52915
* START_TIME = 2025-05-20 10.08.24.436983 EUROPE/BELGRADE
* UPDATE_TIME = 2025-05-20 10.08.24.458643 EUROPE/BELGRADE
* STATUS = FAILED
* OWNER_NAME = CT_MRDS
* TABLE_NAME = STANDING_FACILITIES_HEADER
* PARTITION_NAME =
* SUBPARTITION_NAME =
* FILE_URI_LIST =
* ROWS_LOADED =
* LOGFILE_TABLE = VALIDATE$3608_LOG
* BADFILE_TABLE = VALIDATE$3608_BAD
* STATUS_TABLE =
* TEMPEXT_TABLE =
* CREDENTIAL_NAME =
* EXPIRATION_TIME = 2025-05-22 10.08.24.436983000 EUROPE/BELGRADE
* --------------------------------
**/
FUNCTION GET_DET_USER_LOAD_OPERATIONS (
pOperationId PLS_INTEGER
) RETURN VARCHAR2;
/**
* @name ANALYZE_VALIDATION_ERRORS
* @desc Wrapper function that analyzes validation errors for a source file using its received key.
* Automatically derives template schema, table name, CSV URI and validation log table
* from file metadata and calls ENV_MANAGER.ANALYZE_VALIDATION_ERRORS.
* @example SELECT FILE_MANAGER.ANALYZE_VALIDATION_ERRORS(63) FROM DUAL;
* @ex_rslt Detailed validation analysis report with column mismatches and solutions
**/
FUNCTION ANALYZE_VALIDATION_ERRORS(
pSourceFileReceivedKey IN NUMBER
) RETURN VARCHAR2;
---------------------------------------------------------------------------------------------------------------------------
-- PACKAGE VERSION MANAGEMENT FUNCTIONS
---------------------------------------------------------------------------------------------------------------------------
/**
* @name GET_VERSION
* @desc Returns the current version number of the FILE_MANAGER package.
* Uses semantic versioning format (MAJOR.MINOR.PATCH).
* @example SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;
* @ex_rslt 3.2.0
**/
FUNCTION GET_VERSION RETURN VARCHAR2;
/**
* @name GET_BUILD_INFO
* @desc Returns comprehensive build information including version, build date, and author.
* Uses centralized ENV_MANAGER.GET_PACKAGE_VERSION_INFO function.
* @example SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;
* @ex_rslt Package: FILE_MANAGER
* Version: 3.2.0
* Build Date: 2025-10-22 16:30:00
* Author: Grzegorz Michalski
**/
FUNCTION GET_BUILD_INFO RETURN VARCHAR2;
/**
* @name GET_VERSION_HISTORY
* @desc Returns complete version history with all releases and changes.
* Uses centralized ENV_MANAGER.FORMAT_VERSION_HISTORY function.
* @example SELECT FILE_MANAGER.GET_VERSION_HISTORY() FROM DUAL;
* @ex_rslt FILE_MANAGER Version History:
* 3.2.0 (2025-10-22): Added package versioning system...
**/
FUNCTION GET_VERSION_HISTORY RETURN VARCHAR2;
END;
/
/

View File

@@ -0,0 +1,93 @@
/*
================================================================================
MARS-1056: Master Rollback Script for VARCHAR2 Definition Fixes
Author: Grzegorz Michalski
Date: November 25, 2025
Purpose: Main rollback script for VARCHAR2 fixes in case of issues
USAGE:
sql 'CT_MRDS/Cloudpass#34@ggmichalski_high' '@rollback_mars1056.sql'
================================================================================
*/
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'ROLLBACK_MARS_1056_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF
PROMPT
PROMPT ========================================================================
PROMPT MARS-1056: VARCHAR2 Definition Fixes ROLLBACK
PROMPT ========================================================================
PROMPT
PROMPT WARNING: This will rollback all MARS-1056 changes!
PROMPT
PROMPT After rollback:
PROMPT - FILE_MANAGER will use original VARCHAR2 definition logic
PROMPT - External tables will again have data_length based definitions
PROMPT - CHAR/BYTE semantics will NOT be preserved from templates
PROMPT
PROMPT Continue only if you are experiencing issues with MARS-1056 changes
PROMPT
PROMPT ========================================================================
-- Confirm rollback with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with rollback, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
-- Set current schema context
ALTER SESSION SET CURRENT_SCHEMA = CT_MRDS;
PROMPT
PROMPT === Step 1: Rollback packages to pre-MARS-1056 versions ===
@@91_MARS_1056_rollback_packages.sql
PROMPT
PROMPT === Step 2: Track rollback version ===
@@track_package_versions.sql
PROMPT
PROMPT === Step 3: Verify all tracked packages ===
@@verify_packages_version.sql
PROMPT
PROMPT ========================================================================
PROMPT MARS-1056: Rollback completed successfully
PROMPT ========================================================================
PROMPT
PROMPT All packages have been restored to pre-MARS-1056 state:
PROMPT - FILE_MANAGER: Original VARCHAR2 definition logic restored (v3.2.1)
PROMPT - FILE_MANAGER_ODS: Unchanged (v2.1.0 - wrapper only)
PROMPT - External tables will use data_length based definitions again
PROMPT
PROMPT Impact:
PROMPT - VARCHAR2 columns in external tables will NOT match template definitions
PROMPT - CHAR semantics will be lost (converted to BYTE with larger sizes)
PROMPT - This is the original problematic behavior that MARS-1056 was meant to fix
PROMPT
PROMPT Rollback tracking has been logged for audit purposes.
PROMPT
PROMPT ========================================================================
PROMPT Rollback log saved to: &_filename
PROMPT ========================================================================
spool off
quit;

View File

@@ -0,0 +1,95 @@
-- ===================================================================
-- Simple Package Version Tracking Script
-- ===================================================================
-- Purpose: Track specified Oracle package versions
-- Author: Grzegorz Michalski
-- Date: 2025-11-26
-- Version: 3.1.0 - List-Based Edition
--
-- USAGE:
-- 1. Edit package list below (add/remove packages as needed)
-- 2. Include in your install/rollback script: @@track_package_versions.sql
-- ===================================================================
SET SERVEROUTPUT ON;
DECLARE
TYPE t_package_rec IS RECORD (
owner VARCHAR2(50),
package_name VARCHAR2(50),
version VARCHAR2(50)
);
TYPE t_packages IS TABLE OF t_package_rec;
TYPE t_string_array IS TABLE OF VARCHAR2(100);
-- ===================================================================
-- PACKAGE LIST - Edit this array to specify packages to track
-- ===================================================================
-- Add or remove entries as needed for your MARS issue
-- Format: 'SCHEMA.PACKAGE_NAME'
-- ===================================================================
vPackageList t_string_array := t_string_array(
'CT_MRDS.FILE_MANAGER'
);
-- ===================================================================
vPackages t_packages := t_packages();
vVersion VARCHAR2(50);
vCount NUMBER := 0;
vOwner VARCHAR2(50);
vPackageName VARCHAR2(50);
vDotPos NUMBER;
BEGIN
DBMS_OUTPUT.PUT_LINE('========================================');
DBMS_OUTPUT.PUT_LINE('Package Version Tracking');
DBMS_OUTPUT.PUT_LINE('========================================');
-- Process each package in the list
FOR i IN 1..vPackageList.COUNT LOOP
vDotPos := INSTR(vPackageList(i), '.');
IF vDotPos > 0 THEN
vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1);
vPackageName := SUBSTR(vPackageList(i), vDotPos + 1);
-- Get package version
BEGIN
EXECUTE IMMEDIATE 'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL' INTO vVersion;
vPackages.EXTEND;
vPackages(vPackages.COUNT).owner := vOwner;
vPackages(vPackages.COUNT).package_name := vPackageName;
vPackages(vPackages.COUNT).version := vVersion;
-- Track in ENV_MANAGER
BEGIN
CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
pPackageOwner => vOwner,
pPackageName => vPackageName,
pPackageVersion => vVersion,
pPackageBuildDate => TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS'),
pPackageAuthor => 'Grzegorz Michalski'
);
vCount := vCount + 1;
EXCEPTION
WHEN OTHERS THEN NULL; -- Continue even if tracking fails
END;
EXCEPTION
WHEN OTHERS THEN NULL; -- Skip packages that fail
END;
END IF;
END LOOP;
-- Display results
IF vPackages.COUNT > 0 THEN
DBMS_OUTPUT.PUT_LINE('Packages tracked: ' || vCount || ' of ' || vPackages.COUNT);
DBMS_OUTPUT.PUT_LINE('');
FOR i IN 1..vPackages.COUNT LOOP
DBMS_OUTPUT.PUT_LINE(vPackages(i).owner || '.' || vPackages(i).package_name || ' = ' || vPackages(i).version);
END LOOP;
ELSE
DBMS_OUTPUT.PUT_LINE('No packages found in list');
END IF;
DBMS_OUTPUT.PUT_LINE('========================================');
END;
/

View File

@@ -0,0 +1,62 @@
-- ===================================================================
-- Universal Package Version Verification Script
-- ===================================================================
-- Purpose: Verify all tracked Oracle packages for code changes
-- Author: Grzegorz Michalski
-- Date: 2025-11-25
-- Version: 1.0.0
--
-- USAGE:
-- Include at the end of install/rollback scripts: @@verify_packages_version.sql
--
-- OUTPUT:
-- - List of all tracked packages with their current status
-- - OK: Package has not changed since last tracking
-- - WARNING: Package code changed without version update
-- ===================================================================
SET LINESIZE 200
SET PAGESIZE 1000
SET FEEDBACK OFF
PROMPT
PROMPT ========================================
PROMPT Package Version Verification
PROMPT ========================================
PROMPT
COLUMN PACKAGE_OWNER FORMAT A15
COLUMN PACKAGE_NAME FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN STATUS FORMAT A80
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION AS VERSION,
CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS
FROM (
SELECT
PACKAGE_OWNER,
PACKAGE_NAME,
PACKAGE_VERSION,
ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN
FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING
)
WHERE RN = 1
ORDER BY PACKAGE_OWNER, PACKAGE_NAME;
PROMPT
PROMPT ========================================
PROMPT Verification Complete
PROMPT ========================================
PROMPT
PROMPT Legend:
PROMPT OK - Package has not changed since last tracking
PROMPT WARNING - Package code changed without version update
PROMPT
PROMPT For detailed hash information, use:
PROMPT SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;
PROMPT ========================================
SET FEEDBACK ON

View File

@@ -0,0 +1,91 @@
--=============================================================================================================================
-- MARS-1096: Extend A_PROCESS_LOG.log_message and Enable Daily Interval Partitioning
--=============================================================================================================================
-- Purpose:
-- - Extend log_message column from VARCHAR2(4000) to VARCHAR2(20000)
-- - Convert table to daily interval partitioning on log_timestamp
-- - Preserve all existing data during conversion
-- Author: Grzegorz Michalski
-- Date: 2025-12-03
-- Version: 1.0.0
--=============================================================================================================================
SET SERVEROUTPUT ON
PROMPT ========================================================================
PROMPT MARS-1096: Extend column and enable partitioning
PROMPT ========================================================================
-- 1) Extend column size (safe operation, preserves data)
ALTER TABLE CT_MRDS.A_PROCESS_LOG MODIFY (log_message VARCHAR2(20000));
-- 2) Convert table to interval partitioning by day on log_timestamp
-- Strategy: Create new partitioned table, copy ALL data, rename tables
-- This preserves all existing records during conversion
DECLARE
v_has_partitions NUMBER := 0;
v_row_count NUMBER := 0;
BEGIN
-- Check if already partitioned
SELECT COUNT(*) INTO v_has_partitions
FROM user_tab_partitions
WHERE table_name = 'A_PROCESS_LOG';
IF v_has_partitions = 0 THEN
-- Count existing data
EXECUTE IMMEDIATE 'SELECT COUNT(*) FROM CT_MRDS.A_PROCESS_LOG' INTO v_row_count;
DBMS_OUTPUT.PUT_LINE('Existing records to migrate: ' || v_row_count);
-- Create new partitioned table with different name
EXECUTE IMMEDIATE q'[CREATE TABLE CT_MRDS.A_PROCESS_LOG_NEW (
a_process_log_key NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
guid VARCHAR2(32),
Username VARCHAR2(128),
Osuser VARCHAR2(128),
Machine VARCHAR2(64),
Module VARCHAR2(64),
process_name VARCHAR2(200),
procedure_name VARCHAR2(200),
procedure_parameters VARCHAR2(20000),
log_level VARCHAR2(10),
log_message VARCHAR2(20000),
log_timestamp TIMESTAMP DEFAULT SYSTIMESTAMP
)
PARTITION BY RANGE (log_timestamp)
INTERVAL (NUMTODSINTERVAL(1,'DAY'))
(
PARTITION p_init VALUES LESS THAN (TIMESTAMP '2025-01-01 00:00:00')
)]';
-- Copy ALL existing data (Oracle auto-creates partitions per timestamp)
EXECUTE IMMEDIATE 'INSERT /*+ APPEND */ INTO CT_MRDS.A_PROCESS_LOG_NEW (
guid, Username, Osuser, Machine, Module, process_name, procedure_name,
procedure_parameters, log_level, log_message, log_timestamp
) SELECT guid, Username, Osuser, Machine, Module, process_name, procedure_name,
procedure_parameters, log_level, log_message, log_timestamp
FROM CT_MRDS.A_PROCESS_LOG';
COMMIT;
-- Verify data copied successfully
EXECUTE IMMEDIATE 'SELECT COUNT(*) FROM CT_MRDS.A_PROCESS_LOG_NEW' INTO v_row_count;
DBMS_OUTPUT.PUT_LINE('Records migrated to partitioned table: ' || v_row_count);
-- Swap tables: rename old to backup, new to original name
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG RENAME TO A_PROCESS_LOG_OLD';
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG_NEW RENAME TO A_PROCESS_LOG';
DBMS_OUTPUT.PUT_LINE('Table conversion complete. Old table preserved as A_PROCESS_LOG_OLD');
DBMS_OUTPUT.PUT_LINE('Review and manually DROP TABLE CT_MRDS.A_PROCESS_LOG_OLD PURGE when verified');
ELSE
DBMS_OUTPUT.PUT_LINE('Table already partitioned. Skipping conversion.');
END IF;
END;
/
PROMPT SUCCESS: A_PROCESS_LOG extended and partitioned
--=============================================================================================================================
-- End of Script
--=============================================================================================================================

View File

@@ -0,0 +1,35 @@
-- Verify CT_MRDS.A_PROCESS_LOG structure after MARS-1096
-- CRITICAL: Use ALL_* views when installing as ADMIN user
SET PAGESIZE 100
SET LINESIZE 200
COLUMN column_name FORMAT A20
COLUMN data_type FORMAT A20
PROMPT Column length check:
SELECT column_name, data_type, data_length
FROM all_tab_columns
WHERE owner = 'CT_MRDS'
AND table_name = 'A_PROCESS_LOG'
AND column_name = 'LOG_MESSAGE';
PROMPT Partitioning check:
SELECT partitioning_type, interval
FROM all_part_tables
WHERE owner = 'CT_MRDS'
AND table_name = 'A_PROCESS_LOG';
PROMPT Sample partitions:
SELECT partition_name, high_value, num_rows
FROM all_tab_partitions
WHERE table_owner = 'CT_MRDS'
AND table_name = 'A_PROCESS_LOG'
ORDER BY partition_position
FETCH FIRST 10 ROWS ONLY;
PROMPT Total record count:
SELECT COUNT(*) AS total_records FROM CT_MRDS.A_PROCESS_LOG;
PROMPT Check for backup table (should be dropped after verification):
SELECT table_name FROM all_tables
WHERE owner = 'CT_MRDS'
AND table_name LIKE 'A_PROCESS_LOG%';

View File

@@ -0,0 +1,23 @@
-- Cleanup backup table after successful verification
-- This script is automatically called by install_mars1096.sql
-- Can also be run manually if needed
-- CRITICAL: Use ALL_TABLES when installing as ADMIN user
SET SERVEROUTPUT ON
DECLARE
v_table_exists NUMBER := 0;
BEGIN
SELECT COUNT(*) INTO v_table_exists
FROM all_tables
WHERE owner = 'CT_MRDS'
AND table_name = 'A_PROCESS_LOG_OLD';
IF v_table_exists > 0 THEN
EXECUTE IMMEDIATE 'DROP TABLE CT_MRDS.A_PROCESS_LOG_OLD PURGE';
DBMS_OUTPUT.PUT_LINE('Backup table A_PROCESS_LOG_OLD dropped successfully.');
ELSE
DBMS_OUTPUT.PUT_LINE('No backup table found. Nothing to clean up.');
END IF;
END;
/

View File

@@ -0,0 +1,47 @@
# MARS-1096: Extend A_PROCESS_LOG and Enable Daily Interval Partitioning
## Scope
- Extend `CT_MRDS.A_PROCESS_LOG.LOG_MESSAGE` to `VARCHAR2(20000)`
- Convert table to interval partitioning by day on `LOG_TIMESTAMP`
- **CRITICAL**: Preserves all existing data during conversion
## Strategy
- Creates new partitioned table `A_PROCESS_LOG_NEW`
- Copies ALL existing records (preserving data integrity)
- Renames old table to `A_PROCESS_LOG_OLD` (backup)
- Renames new table to `A_PROCESS_LOG`
- Manual cleanup of backup table after verification
## Files
- Baseline update: `mrds_elt-dev-database/database/CT_MRDS/SCHEMA/tables/A_PROCESS_LOG.sql`
- Migration: `MARS_Packages/migrations/004_extend_log_message_and_partition.sql`
- Rollback: `MARS_Packages/migrations/rollback/004_rollback_log_message_and_partition.sql`
- Install: `MARS_Packages/REL02/MARS-1096/install_mars1096.sql` (includes cleanup)
- Verify: `MARS_Packages/REL02/MARS-1096/04_MARS_1096_verify_partitioning.sql`
- Cleanup: `MARS_Packages/REL02/MARS-1096/05_MARS_1096_cleanup_backup.sql` (auto-executed)
## Install
```powershell
# Run with SQLcl - includes verification and automatic backup cleanup
sql "CT_MRDS/Cloudpass#34@ggmichalski_high" "@MARS_Packages/REL01/MARS-1096/install_mars1096.sql"
```
## Rollback
```powershell
# Execute rollback with full logging and verification
sql "CT_MRDS/Cloudpass#34@ggmichalski_high" "@MARS_Packages/REL01/MARS-1096/rollback_mars1096.sql"
# Log file created: ROLLBACK_MARS_1096_<PDB>_<timestamp>.log
```
**Rollback Process:**
1. User confirmation required (type "YES")
2. Restore table to non-partitioned structure
3. Revert log_message to VARCHAR2(4000)
4. Preserve all existing data
5. Verify changes with automatic checks
## Performance Notes
- Daily partitions optimal for ~200k records/day
- Partition pruning improves query performance with date filters
- Local indexes can be added per partition if needed

View File

@@ -0,0 +1,79 @@
-- ===================================================================
-- MARS-1096 INSTALL SCRIPT: Extend A_PROCESS_LOG and Enable Partitioning
-- ===================================================================
-- Purpose:
-- - Extend log_message column from VARCHAR2(4000) to VARCHAR2(20000)
-- - Convert table to daily interval partitioning on log_timestamp
-- - Preserve all existing data during conversion
-- Author: Grzegorz Michalski
-- Date: 2025-12-03
-- Version: 1.0.0
-- ===================================================================
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'INSTALL_MARS_1096_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF
PROMPT =========================================================================
PROMPT MARS-1096: Extend A_PROCESS_LOG and Enable Daily Partitioning
PROMPT =========================================================================
PROMPT
PROMPT This script will:
PROMPT - Extend log_message column from VARCHAR2(4000) to VARCHAR2(20000)
PROMPT - Convert A_PROCESS_LOG table to daily interval partitioning
PROMPT - Create backup table A_PROCESS_LOG_OLD during conversion
PROMPT - Migrate all existing records to partitioned structure
PROMPT - Automatically cleanup backup table after verification
PROMPT
PROMPT Expected Duration: 1-2 minutes (depends on data volume)
PROMPT =========================================================================
-- Confirm installation with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
PROMPT
PROMPT =========================================================================
PROMPT Step 1: Extend Column and Enable Partitioning
PROMPT =========================================================================
@@02_MARS_1096_extend_and_partition.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Verify Partitioning
PROMPT =========================================================================
@@04_MARS_1096_verify_partitioning.sql
PROMPT
PROMPT =========================================================================
PROMPT Step 3: Cleanup Backup Table
PROMPT =========================================================================
@@05_MARS_1096_cleanup_backup.sql
PROMPT
PROMPT =========================================================================
PROMPT MARS-1096 Installation - COMPLETED
PROMPT =========================================================================
PROMPT Check the log file for complete installation details.
PROMPT =========================================================================
spool off
quit;

View File

@@ -0,0 +1,151 @@
-- ===================================================================
-- MARS-1096 ROLLBACK SCRIPT: Revert A_PROCESS_LOG Changes
-- ===================================================================
-- Purpose: Rollback all changes from MARS-1096 installation
-- - Revert log_message column from VARCHAR2(20000) to VARCHAR2(4000)
-- - Remove daily interval partitioning (restore to non-partitioned table)
-- - Preserve all existing data during rollback
-- Author: Grzegorz Michalski
-- Date: 2025-12-03
-- Version: 1.0.0
-- ===================================================================
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
var filename VARCHAR2(100)
BEGIN
:filename := 'ROLLBACK_MARS_1096_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF
PROMPT =========================================================================
PROMPT MARS-1096: Rollback A_PROCESS_LOG Changes
PROMPT =========================================================================
PROMPT WARNING: This will reverse all changes from MARS-1096 installation!
PROMPT
PROMPT Changes to be reverted:
PROMPT - Revert log_message column: VARCHAR2(20000) -> VARCHAR2(4000)
PROMPT - Remove daily interval partitioning
PROMPT - Restore to non-partitioned table structure
PROMPT
PROMPT CRITICAL: All existing data will be preserved during rollback
PROMPT =========================================================================
-- Confirm rollback with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with rollback, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
END IF;
END;
/
WHENEVER SQLERROR CONTINUE
PROMPT
PROMPT =========================================================================
PROMPT Step 1: Rollback Table Structure
PROMPT =========================================================================
DECLARE
v_row_count NUMBER := 0;
v_table_exists NUMBER := 0;
BEGIN
-- Check if old backup table still exists
SELECT COUNT(*) INTO v_table_exists
FROM user_tables
WHERE table_name = 'A_PROCESS_LOG_OLD';
IF v_table_exists > 0 THEN
-- If backup exists, simply restore it
DBMS_OUTPUT.PUT_LINE('Restoring from backup table A_PROCESS_LOG_OLD');
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG RENAME TO A_PROCESS_LOG_PARTITIONED';
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG_OLD RENAME TO A_PROCESS_LOG';
-- Revert column size
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG MODIFY (log_message VARCHAR2(4000))';
EXECUTE IMMEDIATE 'DROP TABLE CT_MRDS.A_PROCESS_LOG_PARTITIONED PURGE';
DBMS_OUTPUT.PUT_LINE('Rollback complete using backup table.');
ELSE
-- No backup, need to convert back manually
DBMS_OUTPUT.PUT_LINE('No backup found. Converting partitioned table to non-partitioned.');
EXECUTE IMMEDIATE 'SELECT COUNT(*) FROM CT_MRDS.A_PROCESS_LOG' INTO v_row_count;
DBMS_OUTPUT.PUT_LINE('Records to migrate: ' || v_row_count);
-- Create non-partitioned table
EXECUTE IMMEDIATE q'[CREATE TABLE CT_MRDS.A_PROCESS_LOG_NP (
a_process_log_key NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
guid VARCHAR2(32),
Username VARCHAR2(128),
Osuser VARCHAR2(128),
Machine VARCHAR2(64),
Module VARCHAR2(64),
process_name VARCHAR2(200),
procedure_name VARCHAR2(200),
procedure_parameters VARCHAR2(20000),
log_level VARCHAR2(10),
log_message VARCHAR2(4000),
log_timestamp TIMESTAMP DEFAULT SYSTIMESTAMP
)]';
-- Copy all data
EXECUTE IMMEDIATE 'INSERT /*+ APPEND */ INTO CT_MRDS.A_PROCESS_LOG_NP (
guid, Username, Osuser, Machine, Module, process_name, procedure_name,
procedure_parameters, log_level, log_message, log_timestamp
) SELECT guid, Username, Osuser, Machine, Module, process_name, procedure_name,
procedure_parameters, log_level, log_message, log_timestamp
FROM CT_MRDS.A_PROCESS_LOG';
COMMIT;
-- Swap tables
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG RENAME TO A_PROCESS_LOG_PARTITIONED';
EXECUTE IMMEDIATE 'ALTER TABLE CT_MRDS.A_PROCESS_LOG_NP RENAME TO A_PROCESS_LOG';
EXECUTE IMMEDIATE 'DROP TABLE CT_MRDS.A_PROCESS_LOG_PARTITIONED PURGE';
DBMS_OUTPUT.PUT_LINE('Rollback complete.');
END IF;
END;
/
PROMPT
PROMPT =========================================================================
PROMPT Step 2: Verify Rollback
PROMPT =========================================================================
-- Verify column size reverted
SELECT column_name, data_type, data_length
FROM user_tab_columns
WHERE table_name = 'A_PROCESS_LOG' AND column_name = 'LOG_MESSAGE';
-- Verify partitioning removed
SELECT COUNT(*) AS partition_count
FROM user_tab_partitions
WHERE table_name = 'A_PROCESS_LOG';
-- Verify record count preserved
SELECT COUNT(*) AS total_records FROM CT_MRDS.A_PROCESS_LOG;
PROMPT
PROMPT =========================================================================
PROMPT MARS-1096 Rollback - COMPLETED
PROMPT =========================================================================
PROMPT Check the log file for complete rollback details.
PROMPT Expected results:
PROMPT - LOG_MESSAGE column: VARCHAR2(4000)
PROMPT - Partition count: 0 (non-partitioned)
PROMPT - Total records: Should match pre-rollback count
PROMPT =========================================================================
spool off
quit;

View File

@@ -0,0 +1,53 @@
SET SERVEROUTPUT ON;
DECLARE
v_ins_str VARCHAR2(4096);
v_upd_str VARCHAR2(4096);
v_trigger_sql VARCHAR2(32767);
v_trigger_name VARCHAR2(128);
v_ref_table_name VARCHAR2(128);
v_table_name VARCHAR2(128); -- GIVE VALUE FOR ONE TABLE TRIGGER GENERATION
BEGIN
FOR i IN (
SELECT
'create or replace TRIGGER DW_REF_MAIN.AUTOKEY_'||TABLE_NAME||'
BEFORE INSERT ON DW_REF_MAIN.'||TABLE_NAME||'
FOR EACH ROW
WHEN (NEW.A_KEY IS NULL)
BEGIN
SELECT
NVL(MAX(A_KEY), 0) + 1
INTO :NEW.A_KEY
FROM DW_REF_MAIN.'||TABLE_NAME||';
END;
' AS trigger_sql,
'DW_REF_MAIN.AUTOKEY_'||TABLE_NAME AS trigger_name
FROM DBA_TABLES t
WHERE
OWNER = 'DW_REF_MAIN'
AND TABLE_NAME IN (
'ASSET_CODE_TYPE', 'ABS_TYPE', 'ALLOTMENT_METHOD', 'AREA_CODE_RIAD', 'ASSET_GROUP_DETAIL', 'ASSET_GROUP_MAPPING', 'ASSET_GROUP_TMS', 'ASSET_GROUP_TMS_OLD_NEW_MAP', 'ASSET_TYPE', 'ASSET_TYPE_TMS',
'CAPITAL_KEY', 'COUNTRY', 'CSPP_BENCHMARK', 'CSPP_GROUP_MEMBERS', 'CSPP_GROUPS', 'CURRENCY', 'ELA_COUNTRY_CEILING', 'ELA_FUNDING_GAP', 'ELA_MFI_CEILING', 'ESA_95_INSTR_CLASS_CSDB',
'ESA_95_SECTOR', 'ESA_2010_INSTR_CLASS_CSDB', 'ESA_2010_ISSUER_CLASS_CSDB', 'ESAST_SUB_SECTOR_RIAD', 'HAIRCUT_CATEGORIES', 'HAIRCUT_SCHEDULE', 'HOLIDAY', 'INSTITUTION_C2D', 'INSTITUTION_CODE_TYPE', 'INSTITUTION_ECONOMIC_ACTIVITY',
'INSTITUTION_IDENTIFIER_TYPE', 'INSTITUTION_RAT_OUTLOOK_VALUE', 'INSTITUTION_RAT_WATCH_VALUE', 'INSTITUTION_ROLE', 'INSTITUTION_SUBROLE', 'ISSUER_DEBTOR_GUARANTOR_GROUP', 'LIMIT_BREACH_CAUSE', 'LIMIT_TYPE', 'LIMIT_VIOLATION_TYPE', 'MAP_COUPON_FREQ',
'MAP_COUPON_FREQ_RAR', 'MAP_COUPON_TYPE', 'MAP_COUPON_TYPE_RAR', 'NACE2_INDUSTRY_CLASS_CSDB', 'OPERATION_NAME', 'OPERATION_PROCEDURE', 'OPERATION_PUBLIC', 'OPERATION_TYPE', 'OWN_USE', 'PRICE_TYPE_MAP',
'PRICE_TYPE_RAR', 'PSE_TYPE', 'QUOT_BASIS_MAP', 'RATING', 'RATING_ACTION', 'RATING_PD', 'RATING_PRIORITY', 'RATING_SOURCE', 'RATING_TYPE', 'REDEMPTION_FREQUENCY_CSDB',
'REDEMPTION_TYPE_CSDB', 'RIC_CODE_CREATION_RULE', 'TMS_ACTIVITY_GROUPING', 'TMS_POSITION_CONFIG', 'MARKET_COUNTRIES', 'MEASUREMENT_UNITS', 'NCB', 'PORTFOLIO', 'PP_BUCKET', 'PP_CASH_FLOW_TYPE',
'PP_METHOD', 'PP_PORTF_BNCHMK_MAP', 'SCENARIOS', 'INSTITUTION_ROLE_C2D'
)
ORDER BY TABLE_NAME
) LOOP
v_trigger_sql := i.trigger_sql;
DBMS_OUTPUT.PUT_LINE(i.trigger_name);
EXECUTE IMMEDIATE v_trigger_sql;
-- DBMS_OUTPUT.PUT_LINE(v_trigger_sql||'
-- /
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,47 @@
SET SERVEROUTPUT ON;
DECLARE
v_ins_str VARCHAR2(4096);
v_upd_str VARCHAR2(4096);
v_trigger_sql VARCHAR2(32767);
v_trigger_name VARCHAR2(128);
v_ref_table_name VARCHAR2(128);
v_table_name VARCHAR2(128); -- GIVE VALUE FOR ONE TABLE TRIGGER GENERATION
BEGIN
FOR i IN (
SELECT
'create or replace TRIGGER REFERENCE.AUTOKEY_'||TABLE_NAME||'
BEFORE INSERT ON REFERENCE.'||TABLE_NAME||'
FOR EACH ROW
WHEN (NEW.A_KEY IS NULL)
BEGIN
SELECT
NVL(MAX(A_KEY), 0) + 1
INTO :NEW.A_KEY
FROM REFERENCE.'||TABLE_NAME||';
END;
' AS trigger_sql,
'REFERENCE.AUTOKEY_'||TABLE_NAME AS trigger_name
FROM DBA_TABLES t
WHERE
OWNER = 'REFERENCE'
AND TABLE_NAME IN (
'T_BALANCE_SHEET_STRUCTURE', 'T_CAPITAL_KEY', 'T_COUNTRY', 'T_CSDB_RATING_ACTION', 'T_CURRENCY', 'T_TMS_ACTIVITY_GROUPING'
)
ORDER BY TABLE_NAME
) LOOP
v_trigger_sql := i.trigger_sql;
DBMS_OUTPUT.PUT_LINE(i.trigger_name);
EXECUTE IMMEDIATE v_trigger_sql;
-- DBMS_OUTPUT.PUT_LINE(v_trigger_sql||'
-- /
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,97 @@
ALTER TABLE OU_REF.QUOT_BASIS_MAP ADD CONSTRAINT CHECK_ANALYZED CHECK (UPPER(ANNUALIZED) IN ( 'U', 'N', 'Y'));
GRANT SELECT ANY TABLE ON SCHEMA DW_REF_MAIN TO OU_REF;
GRANT INSERT ANY TABLE ON SCHEMA DW_REF_MAIN TO OU_REF;
GRANT UPDATE ANY TABLE ON SCHEMA DW_REF_MAIN TO OU_REF;
GRANT DELETE ANY TABLE ON SCHEMA DW_REF_MAIN TO OU_REF;
GRANT SELECT ANY TABLE ON SCHEMA REFERENCE TO OU_REF;
GRANT INSERT ANY TABLE ON SCHEMA REFERENCE TO OU_REF;
GRANT UPDATE ANY TABLE ON SCHEMA REFERENCE TO OU_REF;
GRANT DELETE ANY TABLE ON SCHEMA REFERENCE TO OU_REF;
SET SERVEROUTPUT ON;
DECLARE
v_ins_str VARCHAR2(4096);
v_upd_str VARCHAR2(4096);
v_trigger_sql VARCHAR2(32767);
v_trigger_name VARCHAR2(128);
v_ref_table_name VARCHAR2(128);
v_table_name VARCHAR2(128); -- GIVE VALUE FOR ONE TABLE TRIGGER GENERATION
BEGIN
FOR i IN (
SELECT
OWNER, TABLE_NAME
FROM DBA_TABLES t
WHERE
OWNER = 'OU_REF'
AND (v_table_name IS NULL OR TABLE_NAME = v_table_name)
AND TABLE_NAME IN (
'ASSET_CODE_TYPE', 'ABS_TYPE', 'ALLOTMENT_METHOD', 'AREA_CODE_RIAD', 'ASSET_GROUP_DETAIL', 'ASSET_GROUP_MAPPING', 'ASSET_GROUP_TMS', 'ASSET_GROUP_TMS_OLD_NEW_MAP', 'ASSET_TYPE', 'ASSET_TYPE_TMS',
'CSPP_BENCHMARK', 'CSPP_GROUP_MEMBERS', 'CSPP_GROUPS', 'ELA_COUNTRY_CEILING', 'ELA_FUNDING_GAP', 'ELA_MFI_CEILING', 'ESA_95_INSTR_CLASS_CSDB', 'ESA_95_SECTOR', 'ESA_2010_INSTR_CLASS_CSDB', 'ESA_2010_ISSUER_CLASS_CSDB',
'ESAST_SUB_SECTOR_RIAD', 'HAIRCUT_CATEGORIES', 'HAIRCUT_SCHEDULE', 'HOLIDAY', 'INSTITUTION_C2D', 'INSTITUTION_ECONOMIC_ACTIVITY', 'INSTITUTION_IDENTIFIER_TYPE', 'INSTITUTION_RAT_OUTLOOK_VALUE', 'INSTITUTION_RAT_WATCH_VALUE', 'INSTITUTION_SUBROLE',
'ISSUER_DEBTOR_GUARANTOR_GROUP', 'LIMIT_BREACH_CAUSE', 'LIMIT_TYPE', 'LIMIT_VIOLATION_TYPE', 'MAP_COUPON_FREQ', 'MAP_COUPON_FREQ_RAR', 'MAP_COUPON_TYPE', 'MAP_COUPON_TYPE_RAR', 'NACE2_INDUSTRY_CLASS_CSDB', 'OPERATION_NAME',
'OPERATION_PROCEDURE', 'OPERATION_PUBLIC', 'OPERATION_TYPE', 'OWN_USE', 'PRICE_TYPE_MAP', 'PRICE_TYPE_RAR', 'PSE_TYPE', 'RATING_ACTION', 'RATING_PD', 'RATING_PRIORITY',
'RATING_SOURCE', 'RATING_TYPE', 'REDEMPTION_FREQUENCY_CSDB', 'REDEMPTION_TYPE_CSDB', 'TMS_POSITION_CONFIG', 'MARKET_COUNTRIES', 'MEASUREMENT_UNITS', 'NCB', 'PORTFOLIO', 'PP_BUCKET',
'PP_CASH_FLOW_TYPE', 'PP_METHOD', 'PP_PORTF_BNCHMK_MAP', 'SCENARIOS', 'INSTITUTION_CODE_TYPE', 'QUOT_BASIS_MAP', 'RATING', 'RIC_CODE_CREATION_RULE'
)
ORDER BY TABLE_NAME
) LOOP
SELECT
'INSERT INTO '||t.owner||'.'||t.table_name||'('||
LISTAGG(t.column_name, ', ') WITHIN GROUP (ORDER BY t.column_id) ||')
VALUES (' ||LISTAGG(DECODE(t.column_name, 'A_VALID_FROM', 'v_current_date', 'A_VALID_TO', 'v_future_date', DECODE(r.column_name, NULL, 'NULL', ':new.'||t.column_name)), ', ') WITHIN GROUP (ORDER BY t.column_id) ||')' AS v_ins_str,
'UPDATE '||t.owner||'.'||t.table_name||' SET
A_VALID_TO = v_current_date
WHERE
1 = 1
'||LISTAGG(DECODE(r.column_name, NULL, '', ' AND DECODE('||t.column_name|| ', :old.'||t.column_name||', 1, 0) = 1'||chr(13))) WITHIN GROUP (ORDER BY t.column_id) || ' AND A_VALID_TO = v_future_date'||chr(13) AS v_upd_str
INTO v_ins_str, v_upd_str
FROM dba_tab_columns t, dba_tab_columns r
WHERE
t.owner = 'DW_REF_MAIN'
AND t.table_name = i.TABLE_NAME
AND 'OU_REF' = r.owner (+)
AND t.table_name = r.table_name (+)
AND t.column_name = r.column_name (+)
GROUP BY t.owner, t.table_name
;
v_ref_table_name := i.OWNER||'.'||i.TABLE_NAME;
v_trigger_name := i.OWNER||'.'||i.TABLE_NAME||'_H';
v_trigger_sql := '
create or replace TRIGGER '||v_trigger_name||'
BEFORE INSERT OR UPDATE OR DELETE ON '||v_ref_table_name||'
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE(''31/12/9999'', ''DD/MM/YYYY'');
BEGIN
IF INSERTING THEN
'||v_ins_str||';
ELSIF DELETING THEN
'||v_upd_str||' ;
ELSIF UPDATING THEN
'||v_upd_str||' ;
'||v_ins_str||';
END IF;
END;';
DBMS_OUTPUT.PUT_LINE(v_trigger_name);
EXECUTE IMMEDIATE v_trigger_sql;
-- DBMS_OUTPUT.PUT_LINE(v_trigger_sql||'
--/
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,44 @@
create or replace TRIGGER OU_REF.BALANCE_SHEET_STRUCTURE_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.BALANCE_SHEET_STRUCTURE
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO REFERENCE.T_BALANCE_SHEET_STRUCTURE(ISIS_CODE, BS_ITEM_CODE, DESCRIPTION, BS_SUB_ITEM, BS_ITEM, BS_ITEM_GROUP, BS_ITEM_TYPE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.ISIS_CODE, :new.BS_ITEM_CODE, :new.DESCRIPTION, :new.BS_SUB_ITEM, :new.BS_ITEM, :new.BS_ITEM_GROUP, :new.BS_ITEM_TYPE, v_current_date, v_future_date, NULL, -1);
ELSIF DELETING THEN
UPDATE REFERENCE.T_BALANCE_SHEET_STRUCTURE SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ISIS_CODE, :old.ISIS_CODE, 1, 0) = 1
AND DECODE(BS_ITEM_CODE, :old.BS_ITEM_CODE, 1, 0) = 1
AND DECODE(DESCRIPTION, :old.DESCRIPTION, 1, 0) = 1
AND DECODE(BS_SUB_ITEM, :old.BS_SUB_ITEM, 1, 0) = 1
AND DECODE(BS_ITEM, :old.BS_ITEM, 1, 0) = 1
AND DECODE(BS_ITEM_GROUP, :old.BS_ITEM_GROUP, 1, 0) = 1
AND DECODE(BS_ITEM_TYPE, :old.BS_ITEM_TYPE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE REFERENCE.T_BALANCE_SHEET_STRUCTURE SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ISIS_CODE, :old.ISIS_CODE, 1, 0) = 1
AND DECODE(BS_ITEM_CODE, :old.BS_ITEM_CODE, 1, 0) = 1
AND DECODE(DESCRIPTION, :old.DESCRIPTION, 1, 0) = 1
AND DECODE(BS_SUB_ITEM, :old.BS_SUB_ITEM, 1, 0) = 1
AND DECODE(BS_ITEM, :old.BS_ITEM, 1, 0) = 1
AND DECODE(BS_ITEM_GROUP, :old.BS_ITEM_GROUP, 1, 0) = 1
AND DECODE(BS_ITEM_TYPE, :old.BS_ITEM_TYPE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO REFERENCE.T_BALANCE_SHEET_STRUCTURE(ISIS_CODE, BS_ITEM_CODE, DESCRIPTION, BS_SUB_ITEM, BS_ITEM, BS_ITEM_GROUP, BS_ITEM_TYPE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.ISIS_CODE, :new.BS_ITEM_CODE, :new.DESCRIPTION, :new.BS_SUB_ITEM, :new.BS_ITEM, :new.BS_ITEM_GROUP, :new.BS_ITEM_TYPE, v_current_date, v_future_date, NULL, -1);
END IF;
END;
/

View File

@@ -0,0 +1,72 @@
create or replace TRIGGER OU_REF.CAPITAL_KEY_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.CAPITAL_KEY
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.CAPITAL_KEY(COUNTRY_CODE, NCB_INDEX, CAPITAL_KEY_ADJ, RAR_NCB_NAME, CAPITAL_KEY_VALUE, PAID_UP_CAPITAL, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.COUNTRY_CODE, :new.NCB_INDEX, NULL, :new.RAR_NCB_NAME, :new.CAPITAL_KEY_VALUE, :new.PAID_UP_CAPITAL, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_CAPITAL_KEY(COUNTRY_CODE, NCB_NAME, EUROSYSTEM_KEY, EUROSYSTEM_KEY_ECB, ESCB_KEY, ESCB_KEY_ECB, CAPITAL_KEY_VALUE, PAID_UP_CAPITAL, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.COUNTRY_CODE, :new.RAR_NCB_NAME, :new.EUROSYSTEM_KEY, :new.EUROSYSTEM_KEY_ECB, :new.ESCB_KEY, :new.ESCB_KEY_ECB, :new.CAPITAL_KEY_VALUE, :new.PAID_UP_CAPITAL, v_current_date, v_future_date, NULL,-1);
ELSIF DELETING THEN
UPDATE DW_REF_MAIN.CAPITAL_KEY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(NCB_INDEX, :old.NCB_INDEX, 1, 0) = 1
AND DECODE(RAR_NCB_NAME, :old.RAR_NCB_NAME, 1, 0) = 1
AND DECODE(CAPITAL_KEY_VALUE, :old.CAPITAL_KEY_VALUE, 1, 0) = 1
AND DECODE(PAID_UP_CAPITAL, :old.PAID_UP_CAPITAL, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_CAPITAL_KEY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(NCB_NAME, :old.RAR_NCB_NAME, 1, 0) = 1
AND DECODE(EUROSYSTEM_KEY, :old.EUROSYSTEM_KEY, 1, 0) = 1
AND DECODE(EUROSYSTEM_KEY_ECB, :old.EUROSYSTEM_KEY_ECB, 1, 0) = 1
AND DECODE(ESCB_KEY, :old.ESCB_KEY, 1, 0) = 1
AND DECODE(ESCB_KEY_ECB, :old.ESCB_KEY_ECB, 1, 0) = 1
AND DECODE(CAPITAL_KEY_VALUE, :old.CAPITAL_KEY_VALUE, 1, 0) = 1
AND DECODE(PAID_UP_CAPITAL, :old.PAID_UP_CAPITAL, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.CAPITAL_KEY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(NCB_INDEX, :old.NCB_INDEX, 1, 0) = 1
AND DECODE(RAR_NCB_NAME, :old.RAR_NCB_NAME, 1, 0) = 1
AND DECODE(CAPITAL_KEY_VALUE, :old.CAPITAL_KEY_VALUE, 1, 0) = 1
AND DECODE(PAID_UP_CAPITAL, :old.PAID_UP_CAPITAL, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_CAPITAL_KEY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(NCB_NAME, :old.RAR_NCB_NAME, 1, 0) = 1
AND DECODE(EUROSYSTEM_KEY, :old.EUROSYSTEM_KEY, 1, 0) = 1
AND DECODE(EUROSYSTEM_KEY_ECB, :old.EUROSYSTEM_KEY_ECB, 1, 0) = 1
AND DECODE(ESCB_KEY, :old.ESCB_KEY, 1, 0) = 1
AND DECODE(ESCB_KEY_ECB, :old.ESCB_KEY_ECB, 1, 0) = 1
AND DECODE(CAPITAL_KEY_VALUE, :old.CAPITAL_KEY_VALUE, 1, 0) = 1
AND DECODE(PAID_UP_CAPITAL, :old.PAID_UP_CAPITAL, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO DW_REF_MAIN.CAPITAL_KEY(COUNTRY_CODE, NCB_INDEX, CAPITAL_KEY_ADJ, RAR_NCB_NAME, CAPITAL_KEY_VALUE, PAID_UP_CAPITAL, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.COUNTRY_CODE, :new.NCB_INDEX, NULL, :new.RAR_NCB_NAME, :new.CAPITAL_KEY_VALUE, :new.PAID_UP_CAPITAL, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_CAPITAL_KEY(COUNTRY_CODE, NCB_NAME, EUROSYSTEM_KEY, EUROSYSTEM_KEY_ECB, ESCB_KEY, ESCB_KEY_ECB, CAPITAL_KEY_VALUE, PAID_UP_CAPITAL, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.COUNTRY_CODE, :new.RAR_NCB_NAME, :new.EUROSYSTEM_KEY, :new.EUROSYSTEM_KEY_ECB, :new.ESCB_KEY, :new.ESCB_KEY_ECB, :new.CAPITAL_KEY_VALUE, :new.PAID_UP_CAPITAL, v_current_date, v_future_date, NULL,-1);
END IF;
END;
/

View File

@@ -0,0 +1,108 @@
create or replace TRIGGER OU_REF.COUNTRY_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.COUNTRY
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.COUNTRY(COUNTRY_CODE, COUNTRY_NAME, IS_AREA, IS_EUROSYSTEM, IS_EU, IS_EEA, IS_G10, IS_EURO_PERIPHERY, IS_OECD_MEMBER, EXEMPT_FROM_CQR, LOSS_SHARING, NCB_NAME, REGION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.COUNTRY_CODE, :new.COUNTRY_NAME, :new.IS_AREA, :new.IS_EUROSYSTEM, :new.IS_EU, :new.IS_EEA, :new.IS_G10, :new.IS_EURO_PERIPHERY, :new.IS_OECD_MEMBER, :new.EXEMPT_FROM_CQR, :new.LOSS_SHARING, :new.NCB_NAME, :new.REGION, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_COUNTRY(COUNTRY_CODE, REF_AREA, COUNTRY_NAME, IS_AREA, IS_EUROSYSTEM, IS_EU, IS_EEA, IS_G10, IS_EURO_PERIPHERY, IS_OECD_MEMBER, EXEMPT_FROM_CQR, LOSS_SHARING, NCB_NAME, REGION, SORT_ORDER, BIZ_VALID_FROM, BIZ_VALID_TO, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.COUNTRY_CODE, :new.REPORTING_COUNTRY_CODE, :new.COUNTRY_NAME, :new.IS_AREA, :new.IS_EUROSYSTEM, :new.IS_EU, :new.IS_EEA, :new.IS_G10, :new.IS_EURO_PERIPHERY, :new.IS_OECD_MEMBER, :new.EXEMPT_FROM_CQR, :new.LOSS_SHARING, :new.NCB_NAME, :new.REGION, :new.SORT_ORDER, :new.BIZ_VALID_FROM, :new.BIZ_VALID_TO, v_current_date, v_future_date, NULL, -1);
ELSIF DELETING THEN
UPDATE DW_REF_MAIN.COUNTRY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(COUNTRY_NAME, :old.COUNTRY_NAME, 1, 0) = 1
AND DECODE(IS_AREA, :old.IS_AREA, 1, 0) = 1
AND DECODE(IS_EUROSYSTEM, :old.IS_EUROSYSTEM, 1, 0) = 1
AND DECODE(IS_EU, :old.IS_EU, 1, 0) = 1
AND DECODE(IS_EEA, :old.IS_EEA, 1, 0) = 1
AND DECODE(IS_G10, :old.IS_G10, 1, 0) = 1
AND DECODE(IS_EURO_PERIPHERY, :old.IS_EURO_PERIPHERY, 1, 0) = 1
AND DECODE(IS_OECD_MEMBER, :old.IS_OECD_MEMBER, 1, 0) = 1
AND DECODE(EXEMPT_FROM_CQR, :old.EXEMPT_FROM_CQR, 1, 0) = 1
AND DECODE(LOSS_SHARING, :old.LOSS_SHARING, 1, 0) = 1
AND DECODE(NCB_NAME, :old.NCB_NAME, 1, 0) = 1
AND DECODE(REGION, :old.REGION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_COUNTRY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(REF_AREA, :old.REPORTING_COUNTRY_CODE, 1, 0) = 1
AND DECODE(COUNTRY_NAME, :old.COUNTRY_NAME, 1, 0) = 1
AND DECODE(IS_AREA, :old.IS_AREA, 1, 0) = 1
AND DECODE(IS_EUROSYSTEM, :old.IS_EUROSYSTEM, 1, 0) = 1
AND DECODE(IS_EU, :old.IS_EU, 1, 0) = 1
AND DECODE(IS_EEA, :old.IS_EEA, 1, 0) = 1
AND DECODE(IS_G10, :old.IS_G10, 1, 0) = 1
AND DECODE(IS_EURO_PERIPHERY, :old.IS_EURO_PERIPHERY, 1, 0) = 1
AND DECODE(IS_OECD_MEMBER, :old.IS_OECD_MEMBER, 1, 0) = 1
AND DECODE(EXEMPT_FROM_CQR, :old.EXEMPT_FROM_CQR, 1, 0) = 1
AND DECODE(LOSS_SHARING, :old.LOSS_SHARING, 1, 0) = 1
AND DECODE(NCB_NAME, :old.NCB_NAME, 1, 0) = 1
AND DECODE(REGION, :old.REGION, 1, 0) = 1
AND DECODE(SORT_ORDER, :old.SORT_ORDER, 1, 0) = 1
AND DECODE(BIZ_VALID_FROM, :old.BIZ_VALID_FROM, 1, 0) = 1
AND DECODE(BIZ_VALID_TO, :old.BIZ_VALID_TO, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.COUNTRY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(COUNTRY_NAME, :old.COUNTRY_NAME, 1, 0) = 1
AND DECODE(IS_AREA, :old.IS_AREA, 1, 0) = 1
AND DECODE(IS_EUROSYSTEM, :old.IS_EUROSYSTEM, 1, 0) = 1
AND DECODE(IS_EU, :old.IS_EU, 1, 0) = 1
AND DECODE(IS_EEA, :old.IS_EEA, 1, 0) = 1
AND DECODE(IS_G10, :old.IS_G10, 1, 0) = 1
AND DECODE(IS_EURO_PERIPHERY, :old.IS_EURO_PERIPHERY, 1, 0) = 1
AND DECODE(IS_OECD_MEMBER, :old.IS_OECD_MEMBER, 1, 0) = 1
AND DECODE(EXEMPT_FROM_CQR, :old.EXEMPT_FROM_CQR, 1, 0) = 1
AND DECODE(LOSS_SHARING, :old.LOSS_SHARING, 1, 0) = 1
AND DECODE(NCB_NAME, :old.NCB_NAME, 1, 0) = 1
AND DECODE(REGION, :old.REGION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_COUNTRY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(REF_AREA, :old.REPORTING_COUNTRY_CODE, 1, 0) = 1
AND DECODE(COUNTRY_NAME, :old.COUNTRY_NAME, 1, 0) = 1
AND DECODE(IS_AREA, :old.IS_AREA, 1, 0) = 1
AND DECODE(IS_EUROSYSTEM, :old.IS_EUROSYSTEM, 1, 0) = 1
AND DECODE(IS_EU, :old.IS_EU, 1, 0) = 1
AND DECODE(IS_EEA, :old.IS_EEA, 1, 0) = 1
AND DECODE(IS_G10, :old.IS_G10, 1, 0) = 1
AND DECODE(IS_EURO_PERIPHERY, :old.IS_EURO_PERIPHERY, 1, 0) = 1
AND DECODE(IS_OECD_MEMBER, :old.IS_OECD_MEMBER, 1, 0) = 1
AND DECODE(EXEMPT_FROM_CQR, :old.EXEMPT_FROM_CQR, 1, 0) = 1
AND DECODE(LOSS_SHARING, :old.LOSS_SHARING, 1, 0) = 1
AND DECODE(NCB_NAME, :old.NCB_NAME, 1, 0) = 1
AND DECODE(REGION, :old.REGION, 1, 0) = 1
AND DECODE(SORT_ORDER, :old.SORT_ORDER, 1, 0) = 1
AND DECODE(BIZ_VALID_FROM, :old.BIZ_VALID_FROM, 1, 0) = 1
AND DECODE(BIZ_VALID_TO, :old.BIZ_VALID_TO, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO DW_REF_MAIN.COUNTRY(COUNTRY_CODE, COUNTRY_NAME, IS_AREA, IS_EUROSYSTEM, IS_EU, IS_EEA, IS_G10, IS_EURO_PERIPHERY, IS_OECD_MEMBER, EXEMPT_FROM_CQR, LOSS_SHARING, NCB_NAME, REGION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.COUNTRY_CODE, :new.COUNTRY_NAME, :new.IS_AREA, :new.IS_EUROSYSTEM, :new.IS_EU, :new.IS_EEA, :new.IS_G10, :new.IS_EURO_PERIPHERY, :new.IS_OECD_MEMBER, :new.EXEMPT_FROM_CQR, :new.LOSS_SHARING, :new.NCB_NAME, :new.REGION, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_COUNTRY(COUNTRY_CODE, REF_AREA, COUNTRY_NAME, IS_AREA, IS_EUROSYSTEM, IS_EU, IS_EEA, IS_G10, IS_EURO_PERIPHERY, IS_OECD_MEMBER, EXEMPT_FROM_CQR, LOSS_SHARING, NCB_NAME, REGION, SORT_ORDER, BIZ_VALID_FROM, BIZ_VALID_TO, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.COUNTRY_CODE, :new.REPORTING_COUNTRY_CODE, :new.COUNTRY_NAME, :new.IS_AREA, :new.IS_EUROSYSTEM, :new.IS_EU, :new.IS_EEA, :new.IS_G10, :new.IS_EURO_PERIPHERY, :new.IS_OECD_MEMBER, :new.EXEMPT_FROM_CQR, :new.LOSS_SHARING, :new.NCB_NAME, :new.REGION, :new.SORT_ORDER, :new.BIZ_VALID_FROM, :new.BIZ_VALID_TO, v_current_date, v_future_date, NULL, -1);
END IF;
END;
/

View File

@@ -0,0 +1,36 @@
create or replace TRIGGER OU_REF.CSDB_RATING_ACTION_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.CSDB_RATING_ACTION
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO REFERENCE.T_CSDB_RATING_ACTION(CSDB_RATING_ACTION_CODE, CSDB_RATING_ACTION_DESC, SOURCE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.CSDB_RATING_ACTION_CODE, :new.CSDB_RATING_ACTION_DESC, :new.SOURCE, v_current_date, v_future_date, NULL, -1);
ELSIF DELETING THEN
UPDATE REFERENCE.T_CSDB_RATING_ACTION SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CSDB_RATING_ACTION_CODE, :old.CSDB_RATING_ACTION_CODE, 1, 0) = 1
AND DECODE(CSDB_RATING_ACTION_DESC, :old.CSDB_RATING_ACTION_DESC, 1, 0) = 1
AND DECODE(SOURCE, :old.SOURCE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE REFERENCE.T_CSDB_RATING_ACTION SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CSDB_RATING_ACTION_CODE, :old.CSDB_RATING_ACTION_CODE, 1, 0) = 1
AND DECODE(CSDB_RATING_ACTION_DESC, :old.CSDB_RATING_ACTION_DESC, 1, 0) = 1
AND DECODE(SOURCE, :old.SOURCE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO REFERENCE.T_CSDB_RATING_ACTION(CSDB_RATING_ACTION_CODE, CSDB_RATING_ACTION_DESC, SOURCE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.CSDB_RATING_ACTION_CODE, :new.CSDB_RATING_ACTION_DESC, :new.SOURCE, v_current_date, v_future_date, NULL, -1);
END IF;
END;
/

View File

@@ -0,0 +1,66 @@
create or replace TRIGGER OU_REF.CURRENCY_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.CURRENCY
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.CURRENCY(CURRENCY_CODE, CURRENCY_NAME, COUNTRY_CODE, EUR_LEGACY, LEGACY, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.CURRENCY_CODE, :new.CURRENCY_NAME, :new.COUNTRY_CODE, :new.EUR_LEGACY, :new.LEGACY, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_CURRENCY(CURRENCY_CODE, CURRENCY_NAME, COUNTRY_CODE, EUR_LEGACY, LEGACY, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.CURRENCY_CODE, :new.CURRENCY_NAME, :new.COUNTRY_CODE, :new.EUR_LEGACY, :new.LEGACY, v_current_date, v_future_date, NULL, -1);
ELSIF DELETING THEN
UPDATE DW_REF_MAIN.CURRENCY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CURRENCY_CODE, :old.CURRENCY_CODE, 1, 0) = 1
AND DECODE(CURRENCY_NAME, :old.CURRENCY_NAME, 1, 0) = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(EUR_LEGACY, :old.EUR_LEGACY, 1, 0) = 1
AND DECODE(LEGACY, :old.LEGACY, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_CURRENCY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CURRENCY_CODE, :old.CURRENCY_CODE, 1, 0) = 1
AND DECODE(CURRENCY_NAME, :old.CURRENCY_NAME, 1, 0) = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(EUR_LEGACY, :old.EUR_LEGACY, 1, 0) = 1
AND DECODE(LEGACY, :old.LEGACY, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.CURRENCY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CURRENCY_CODE, :old.CURRENCY_CODE, 1, 0) = 1
AND DECODE(CURRENCY_NAME, :old.CURRENCY_NAME, 1, 0) = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(EUR_LEGACY, :old.EUR_LEGACY, 1, 0) = 1
AND DECODE(LEGACY, :old.LEGACY, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_CURRENCY SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(CURRENCY_CODE, :old.CURRENCY_CODE, 1, 0) = 1
AND DECODE(CURRENCY_NAME, :old.CURRENCY_NAME, 1, 0) = 1
AND DECODE(COUNTRY_CODE, :old.COUNTRY_CODE, 1, 0) = 1
AND DECODE(EUR_LEGACY, :old.EUR_LEGACY, 1, 0) = 1
AND DECODE(LEGACY, :old.LEGACY, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO DW_REF_MAIN.CURRENCY(CURRENCY_CODE, CURRENCY_NAME, COUNTRY_CODE, EUR_LEGACY, LEGACY, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.CURRENCY_CODE, :new.CURRENCY_NAME, :new.COUNTRY_CODE, :new.EUR_LEGACY, :new.LEGACY, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_CURRENCY(CURRENCY_CODE, CURRENCY_NAME, COUNTRY_CODE, EUR_LEGACY, LEGACY, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.CURRENCY_CODE, :new.CURRENCY_NAME, :new.COUNTRY_CODE, :new.EUR_LEGACY, :new.LEGACY, v_current_date, v_future_date, NULL, -1);
END IF;
END;
/

View File

@@ -0,0 +1,54 @@
create or replace TRIGGER OU_REF.INSTITUTION_ROLE_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.INSTITUTION_ROLE
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.INSTITUTION_ROLE(ROLE_CODE, ROLE_DESCRIPTION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.ROLE_CODE, :new.ROLE_DESCRIPTION, v_current_date, v_future_date, NULL);
INSERT INTO DW_REF_MAIN.INSTITUTION_ROLE_C2D(ROLE_CODE, ROLE_DESCRIPTION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.ROLE_CODE, :new.ROLE_DESCRIPTION, v_current_date, v_future_date, NULL);
ELSIF DELETING THEN
UPDATE DW_REF_MAIN.INSTITUTION_ROLE SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ROLE_CODE, :old.ROLE_CODE, 1, 0) = 1
AND DECODE(ROLE_DESCRIPTION, :old.ROLE_DESCRIPTION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE DW_REF_MAIN.INSTITUTION_ROLE_C2D SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ROLE_CODE, :old.ROLE_CODE, 1, 0) = 1
AND DECODE(ROLE_DESCRIPTION, :old.ROLE_DESCRIPTION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.INSTITUTION_ROLE SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ROLE_CODE, :old.ROLE_CODE, 1, 0) = 1
AND DECODE(ROLE_DESCRIPTION, :old.ROLE_DESCRIPTION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE DW_REF_MAIN.INSTITUTION_ROLE_C2D SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(ROLE_CODE, :old.ROLE_CODE, 1, 0) = 1
AND DECODE(ROLE_DESCRIPTION, :old.ROLE_DESCRIPTION, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO DW_REF_MAIN.INSTITUTION_ROLE(ROLE_CODE, ROLE_DESCRIPTION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.ROLE_CODE, :new.ROLE_DESCRIPTION, v_current_date, v_future_date, NULL);
INSERT INTO DW_REF_MAIN.INSTITUTION_ROLE_C2D(ROLE_CODE, ROLE_DESCRIPTION, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.ROLE_CODE, :new.ROLE_DESCRIPTION, v_current_date, v_future_date, NULL);
END IF;
END;
/

View File

@@ -0,0 +1,54 @@
create or replace TRIGGER OU_REF.TMS_ACTIVITY_GROUPING_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.TMS_ACTIVITY_GROUPING
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.TMS_ACTIVITY_GROUPING(GROUPING_PARAMETER, GROUPING_VALUE, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.GROUPING_PARAMETER, :new.GROUPING_VALUE, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_TMS_ACTIVITY_GROUPING(GROUPING_PARAMETER, GROUPING_VALUE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.GROUPING_PARAMETER, :new.GROUPING_VALUE, v_current_date, v_future_date, NULL, -1);
ELSIF DELETING THEN
UPDATE DW_REF_MAIN.TMS_ACTIVITY_GROUPING SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(GROUPING_PARAMETER, :old.GROUPING_PARAMETER, 1, 0) = 1
AND DECODE(GROUPING_VALUE, :old.GROUPING_VALUE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_TMS_ACTIVITY_GROUPING SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(GROUPING_PARAMETER, :old.GROUPING_PARAMETER, 1, 0) = 1
AND DECODE(GROUPING_VALUE, :old.GROUPING_VALUE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.TMS_ACTIVITY_GROUPING SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(GROUPING_PARAMETER, :old.GROUPING_PARAMETER, 1, 0) = 1
AND DECODE(GROUPING_VALUE, :old.GROUPING_VALUE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
UPDATE REFERENCE.T_TMS_ACTIVITY_GROUPING SET
A_VALID_TO = v_current_date
WHERE
1 = 1
AND DECODE(GROUPING_PARAMETER, :old.GROUPING_PARAMETER, 1, 0) = 1
AND DECODE(GROUPING_VALUE, :old.GROUPING_VALUE, 1, 0) = 1
AND A_VALID_TO = v_future_date
;
INSERT INTO DW_REF_MAIN.TMS_ACTIVITY_GROUPING(GROUPING_PARAMETER, GROUPING_VALUE, A_VALID_FROM, A_VALID_TO, A_KEY)
VALUES (:new.GROUPING_PARAMETER, :new.GROUPING_VALUE, v_current_date, v_future_date, NULL);
INSERT INTO REFERENCE.T_TMS_ACTIVITY_GROUPING(GROUPING_PARAMETER, GROUPING_VALUE, A_VALID_FROM, A_VALID_TO, A_KEY, A_MOPDB_LOAD_SET_FK)
VALUES (:new.GROUPING_PARAMETER, :new.GROUPING_VALUE, v_current_date, v_future_date, NULL, -1);
END IF;
END;
/

View File

@@ -0,0 +1,42 @@
create or replace TRIGGER OU_REF.TIME_TRANSFORMATION_H
BEFORE INSERT OR UPDATE OR DELETE ON OU_REF.TIME_TRANSFORMATION
REFERENCING NEW AS new OLD AS old
FOR EACH ROW
DECLARE
v_current_date DATE := SYSDATE;
v_future_date DATE := TO_DATE('31/12/9999', 'DD/MM/YYYY');
BEGIN
IF INSERTING THEN
INSERT INTO DW_REF_MAIN.TIME_TRANSFORMATION(YEAR_START, TIME_ZONE_START, CALENDAR_DATE_START, YEAR_END, TIME_ZONE_END, CALENDAR_DATE_END)
VALUES (:new.YEAR_START, :new.TIME_ZONE_START, :new.CALENDAR_DATE_START, :new.YEAR_END, :new.TIME_ZONE_END, :new.CALENDAR_DATE_END);
ELSIF DELETING THEN
DELETE FROM DW_REF_MAIN.TIME_TRANSFORMATION
WHERE
1 = 1
AND DECODE(YEAR_START, :old.YEAR_START, 1, 0) = 1
AND DECODE(TIME_ZONE_START, :old.TIME_ZONE_START, 1, 0) = 1
AND DECODE(CALENDAR_DATE_START, :old.CALENDAR_DATE_START, 1, 0) = 1
AND DECODE(YEAR_END, :old.YEAR_END, 1, 0) = 1
AND DECODE(TIME_ZONE_END, :old.TIME_ZONE_END, 1, 0) = 1
AND DECODE(CALENDAR_DATE_END, :old.CALENDAR_DATE_END, 1, 0) = 1
;
ELSIF UPDATING THEN
UPDATE DW_REF_MAIN.TIME_TRANSFORMATION SET
YEAR_START = :new.YEAR_START,
TIME_ZONE_START = :new.TIME_ZONE_START,
CALENDAR_DATE_START = :new.CALENDAR_DATE_START,
YEAR_END = :new.YEAR_END,
TIME_ZONE_END = :new.TIME_ZONE_END,
CALENDAR_DATE_END = :new.CALENDAR_DATE_END
WHERE
1 = 1
AND DECODE(YEAR_START, :old.YEAR_START, 1, 0) = 1
AND DECODE(TIME_ZONE_START, :old.TIME_ZONE_START, 1, 0) = 1
AND DECODE(CALENDAR_DATE_START, :old.CALENDAR_DATE_START, 1, 0) = 1
AND DECODE(YEAR_END, :old.YEAR_END, 1, 0) = 1
AND DECODE(TIME_ZONE_END, :old.TIME_ZONE_END, 1, 0) = 1
AND DECODE(CALENDAR_DATE_END, :old.CALENDAR_DATE_END, 1, 0) = 1
;
END IF;
END;
/

View File

@@ -0,0 +1,38 @@
SET SERVEROUTPUT ON;
DECLARE
v_drop_sql VARCHAR2(32767);
BEGIN
FOR i IN (
SELECT
'DROP TRIGGER DW_REF_MAIN.'||TRIGGER_NAME AS sql_str,
'DW_REF_MAIN.AUTOKEY_'||TABLE_NAME AS trigger_name
FROM DBA_TRIGGERS
WHERE
OWNER = 'DW_REF_MAIN'
AND TRIGGER_NAME LIKE 'AUTOKEY%'
AND TABLE_NAME IN (
'ASSET_CODE_TYPE', 'ABS_TYPE', 'ALLOTMENT_METHOD', 'AREA_CODE_RIAD', 'ASSET_GROUP_DETAIL', 'ASSET_GROUP_MAPPING', 'ASSET_GROUP_TMS', 'ASSET_GROUP_TMS_OLD_NEW_MAP', 'ASSET_TYPE', 'ASSET_TYPE_TMS',
'CAPITAL_KEY', 'COUNTRY', 'CSPP_BENCHMARK', 'CSPP_GROUP_MEMBERS', 'CSPP_GROUPS', 'CURRENCY', 'ELA_COUNTRY_CEILING', 'ELA_FUNDING_GAP', 'ELA_MFI_CEILING', 'ESA_95_INSTR_CLASS_CSDB',
'ESA_95_SECTOR', 'ESA_2010_INSTR_CLASS_CSDB', 'ESA_2010_ISSUER_CLASS_CSDB', 'ESAST_SUB_SECTOR_RIAD', 'HAIRCUT_CATEGORIES', 'HAIRCUT_SCHEDULE', 'HOLIDAY', 'INSTITUTION_C2D', 'INSTITUTION_CODE_TYPE', 'INSTITUTION_ECONOMIC_ACTIVITY',
'INSTITUTION_IDENTIFIER_TYPE', 'INSTITUTION_RAT_OUTLOOK_VALUE', 'INSTITUTION_RAT_WATCH_VALUE', 'INSTITUTION_ROLE', 'INSTITUTION_SUBROLE', 'ISSUER_DEBTOR_GUARANTOR_GROUP', 'LIMIT_BREACH_CAUSE', 'LIMIT_TYPE', 'LIMIT_VIOLATION_TYPE', 'MAP_COUPON_FREQ',
'MAP_COUPON_FREQ_RAR', 'MAP_COUPON_TYPE', 'MAP_COUPON_TYPE_RAR', 'NACE2_INDUSTRY_CLASS_CSDB', 'OPERATION_NAME', 'OPERATION_PROCEDURE', 'OPERATION_PUBLIC', 'OPERATION_TYPE', 'OWN_USE', 'PRICE_TYPE_MAP',
'PRICE_TYPE_RAR', 'PSE_TYPE', 'QUOT_BASIS_MAP', 'RATING', 'RATING_ACTION', 'RATING_PD', 'RATING_PRIORITY', 'RATING_SOURCE', 'RATING_TYPE', 'REDEMPTION_FREQUENCY_CSDB',
'REDEMPTION_TYPE_CSDB', 'RIC_CODE_CREATION_RULE', 'TMS_ACTIVITY_GROUPING', 'TMS_POSITION_CONFIG', 'MARKET_COUNTRIES', 'MEASUREMENT_UNITS', 'NCB', 'PORTFOLIO', 'PP_BUCKET', 'PP_CASH_FLOW_TYPE',
'PP_METHOD', 'PP_PORTF_BNCHMK_MAP', 'SCENARIOS', 'INSTITUTION_ROLE_C2D'
)
ORDER BY TABLE_NAME
) LOOP
v_drop_sql := i.sql_str;
DBMS_OUTPUT.PUT_LINE(i.trigger_name);
EXECUTE IMMEDIATE v_drop_sql;
-- DBMS_OUTPUT.PUT_LINE(v_drop_sql||'
-- /
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,44 @@
ALTER TABLE OU_REF.QUOT_BASIS_MAP DROP CONSTRAINT CHECK_ANALYZED;
REVOKE SELECT ANY TABLE ON SCHEMA DW_REF_MAIN FROM OU_REF;
REVOKE INSERT ANY TABLE ON SCHEMA DW_REF_MAIN FROM OU_REF;
REVOKE UPDATE ANY TABLE ON SCHEMA DW_REF_MAIN FROM OU_REF;
REVOKE DELETE ANY TABLE ON SCHEMA DW_REF_MAIN FROM OU_REF;
REVOKE SELECT ANY TABLE ON SCHEMA REFERENCE FROM OU_REF;
REVOKE INSERT ANY TABLE ON SCHEMA REFERENCE FROM OU_REF;
REVOKE UPDATE ANY TABLE ON SCHEMA REFERENCE FROM OU_REF;
REVOKE DELETE ANY TABLE ON SCHEMA REFERENCE FROM OU_REF;
SET SERVEROUTPUT ON;
DECLARE
v_drop_sql VARCHAR2(32767);
BEGIN
FOR i IN (
SELECT
'DROP TRIGGER OU_REF.'||TRIGGER_NAME AS sql_str,
'DW_REF_MAIN.'||TABLE_NAME||'_H' AS trigger_name
FROM DBA_TRIGGERS
WHERE
OWNER = 'OU_REF'
AND TRIGGER_NAME LIKE '%_H'
) LOOP
v_drop_sql := i.sql_str;
DBMS_OUTPUT.PUT_LINE(i.trigger_name);
EXECUTE IMMEDIATE v_drop_sql;
-- DBMS_OUTPUT.PUT_LINE(v_drop_sql||'
--/
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,30 @@
SET SERVEROUTPUT ON;
DECLARE
v_drop_sql VARCHAR2(32767);
BEGIN
FOR i IN (
SELECT
'DROP TRIGGER REFERENCE.'||TRIGGER_NAME AS sql_str,
'REFERENCE.AUTOKEY_'||TABLE_NAME AS trigger_name
FROM DBA_TRIGGERS
WHERE
OWNER = 'REFERENCE'
AND TRIGGER_NAME LIKE 'AUTOKEY%'
AND TABLE_NAME IN (
'T_BALANCE_SHEET_STRUCTURE', 'T_CAPITAL_KEY', 'T_COUNTRY', 'T_CSDB_RATING_ACTION', 'T_CURRENCY', 'T_TMS_ACTIVITY_GROUPING'
)
) LOOP
v_drop_sql := i.sql_str;
DBMS_OUTPUT.PUT_LINE(i.trigger_name);
EXECUTE IMMEDIATE v_drop_sql;
-- DBMS_OUTPUT.PUT_LINE(v_drop_sql||'
-- /
-- ');
END LOOP;
END;
/

View File

@@ -0,0 +1,42 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
var filename VARCHAR2(100)
BEGIN
SELECT 'INSTALL_MARS_723_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) AS PDB_NAME FROM DBA_PDBS;
@@01_MARS_723_install_DW_REF_MAIN_CREATE_AUTOKEY_TRIGGER.sql
@@02_MARS_723_install_REFERENCE_CREATE_AUTOKEY_TRIGGER.sql
@@03_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_DW_REF_MAIN.sql
@@05_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_BALANCE_SHEET_STRUCTURE_H.sql
@@06_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_CAPITAL_KEY_H.sql
@@07_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_COUNTRY_H.sql
@@08_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_CSDB_RATING_ACTION_H.sql
@@09_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_CURRENCY_H.sql
@@10_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_INSTITUTION_ROLE_H.sql
@@11_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_TMS_ACTIVITY_GROUPING_H.sql
@@12_MARS_723_install_OU_REF_CREATE_HISTORY_TRIGGER_TIME_TRANSFORMATION_H.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

@@ -0,0 +1,35 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
var filename VARCHAR2(100)
BEGIN
SELECT 'ROLLBACK_MARS_723_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) AS PDB_NAME FROM DBA_PDBS;
@@91_MARS_723_rollback_DW_REF_MAIN_AUTOKEY_TRIGGER.sql
@@91_MARS_723_rollback_OU_REF_HISTORY_TRIGGERS.sql
@@91_MARS_723_rollback_REFERENCE_AUTOKEY_TRIGGER.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_MARS772 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY;
SET DEFINE ON

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_SOURCE_MARS772 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY_SOURCE;
SET DEFINE ON

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_TARGET_MARS772 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY_TARGET;
SET DEFINE ON

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_WORKFLOW_HISTORY_MARS772 AS SELECT * FROM CT_MRDS.A_WORKFLOW_HISTORY;
SET DEFINE ON

View File

@@ -0,0 +1,45 @@
-- MOPDB
INSERT INTO CT_MRDS.A_TASK_HISTORY (
A_TASK_HISTORY_KEY,
A_WORKFLOW_HISTORY_KEY,
TASK_RUN_ID,
TASK_NAME,
TASK_START,
TASK_END,
TASK_SUCCESSFUL,
SERVICE_NAME
)
SELECT
A_LOAD_SUBPROCESS_KEY AS A_TASK_HISTORY_KEY,
A_MOPDB_LOAD_SET_FK AS A_WORKFLOW_HISTORY_KEY,
CAST(NULL AS VARCHAR2(200)) AS TASK_RUN_ID,
CASE
WHEN SUBPROCESS_NAME = 'm_MOPDB_LM_T_STANDING_FACILITIES_OU_LM_STANDING_FACILITY_LOAD' THEN 'm_MOPDB_LM_T_STANDING_FACILITIES_OU_LM_STANDING_FACILITIES_LOAD'
ELSE SUBPROCESS_NAME
END AS TASK_NAME,
PROCESS_START AS TASK_START,
PROCESS_END AS TASK_END,
PROCESS_SUCCESSFUL AS TASK_SUCCESSFUL,
'MOPDB' AS SERVICE_NAME
FROM CT_MOPDB.A_LOAD_SUBPROCESS
WHERE
SUBPROCESS_NAME IN (
SELECT DISTINCT SUBPROCESS_NAME
FROM CT_MOPDB.A_LOAD_HISTORY A
LEFT JOIN CT_MOPDB.A_LOAD_SUBPROCESS B
ON A.A_MOPDB_LOAD_SET_KEY = b.A_MOPDB_LOAD_SET_FK
WHERE
WORKFLOW_NAME IN (
'w_MOPDB_LM_ADHOC_ADJUSTMENT',
'w_MOPDB_LM_BALANCESHEET',
'w_MOPDB_LM_CSM_ADJUSTMENT',
'w_MOPDB_LM_STANDING_FACILITY',
'w_MOPDB_LM_CURRENT_ACCOUNTS',
'w_MOPDB_LM_FORECAST',
'w_MOPDB_LM_QRE_ADJUSTMENT',
'w_MOPDB_LM_TTS'
)
);
COMMIT;

View File

@@ -0,0 +1,60 @@
-- MOPDB
INSERT INTO CT_MRDS.A_TASK_HISTORY_SOURCE (
A_TASK_HISTORY_SOURCE_KEY,
A_TASK_HISTORY_KEY,
A_WORKFLOW_HISTORY_SOURCE_KEY,
SOURCE_NAME,
ROW_COUNT,
SERVICE_NAME
)
SELECT
A_LOAD_HISTORY_SOURCE_KEY AS A_TASK_HISTORY_SOURCE_KEY,
A_LOAD_SUBPROCESS_FK AS A_TASK_HISTORY_KEY,
A_ODS_LOAD_SET_FK AS A_WORKFLOW_HISTORY_SOURCE_KEY,
CASE
WHEN SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM'
WHEN SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN SOURCE_TABLE_NAME = 'OU_LM_STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_HEADER' THEN 'OU_LM_CURRENT_ACCOUNTS_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_ITEM' THEN 'OU_LM_CURRENT_ACCOUNTS_ITEM'
WHEN SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
ELSE SOURCE_TABLE_NAME
END AS SOURCE_NAME,
ROW_COUNT AS ROW_COUNT,
'MOPDB' AS SERVICE_NAME
FROM
CT_MOPDB.A_LOAD_HISTORY_SOURCE
WHERE
SOURCE_TABLE_NAME IN (
'OU_LM_ADHOC_ADJ_HEADER',
'OU_LM_ADHOC_ADJ_ITEM',
'OU_LM_ADHOC_ADJ_ITEM_HEADER',
'OU_LM_BALANCESHEET_HEADER',
'OU_LM_BALANCESHEET_ITEM',
'OU_LM_CSM_ADJ_HEADER',
'OU_LM_CSM_ADJ_ITEM',
'OU_LM_CSM_ADJ_ITEM_HEADER',
'OU_LM_STANDING_FACILITY',
'OU_LM_STANDING_FACILITY_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_ITEM',
'OU_LM_FORECAST_HEADER',
'OU_LM_FORECAST_ITEM',
'OU_LM_QR_ADJ_HEADER',
'OU_LM_QR_ADJ_ITEM',
'OU_LM_QR_ADJ_ITEM_HEADER',
'OU_LM_TTS_HEADER',
'OU_LM_TTS_ITEM'
);
COMMIT;

View File

@@ -0,0 +1,111 @@
-- MOPDB
INSERT INTO CT_MRDS.A_TASK_HISTORY_TARGET (
A_TASK_HISTORY_TARGET_KEY,
A_TASK_HISTORY_KEY,
TARGET_NAME,
ROW_COUNT_APPLIED,
ROW_COUNT_REJECTED,
LOAD_SUCCESSFUL,
SERVICE_NAME
)
SELECT
A_LOAD_HISTORY_TARGET_KEY AS A_TASK_HISTORY_TARGET_KEY,
A_LOAD_SUBPROCESS_FK AS A_TASK_HISTORY_KEY,
TARGET_TABLE_NAME AS TARGET_NAME,
ROW_COUNT_APPLIED AS ROW_COUNT_APPLIED,
ROW_COUNT_REJECTED AS ROW_COUNT_REJECTED,
LOAD_SUCCESSFUL AS LOAD_SUCCESSFUL,
'MOPDB' AS SERVICE_NAME
FROM
CT_MOPDB.A_LOAD_HISTORY_TARGET
WHERE
TARGET_TABLE_NAME IN (
'LM_T_FC_ADH_ADJUSTMENTS',
'LM_T_BALANCE_SHEET',
'LM_T_FC_CSM_ADJUSTMENTS',
'LM_T_STANDING_FACILITIES',
'LM_T_CURRENT_ACCOUNTS',
'LM_T_FORECAST',
'LM_T_FC_QRE_ADJUSTMENTS',
'LM_T_TTS'
);
-- ODS
INSERT INTO CT_MRDS.A_TASK_HISTORY_TARGET (
A_TASK_HISTORY_TARGET_KEY,
A_TASK_HISTORY_KEY,
TARGET_NAME,
ROW_COUNT_APPLIED,
ROW_COUNT_REJECTED,
LOAD_SUCCESSFUL,
SERVICE_NAME
)
SELECT
A_LOAD_HISTORY_TARGET_KEY AS A_TASK_HISTORY_TARGET_KEY,
A_ETL_LOAD_SET_FK AS A_TASK_HISTORY_KEY,
CASE
WHEN TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM'
WHEN TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN TARGET_TABLE_NAME = 'CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN TARGET_TABLE_NAME = 'CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN TARGET_TABLE_NAME = 'CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN TARGET_TABLE_NAME = 'STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN TARGET_TABLE_NAME = 'OU_LM_STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN TARGET_TABLE_NAME = 'STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_HEADER' THEN 'OU_LM_CURRENT_ACCOUNTS_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_ITEM' THEN 'OU_LM_CURRENT_ACCOUNTS_ITEM'
WHEN TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN TARGET_TABLE_NAME = 'QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN TARGET_TABLE_NAME = 'QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
WHEN TARGET_TABLE_NAME = 'QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
ELSE TARGET_TABLE_NAME
END AS TARGET_NAME,
ROW_COUNT_APPLIED AS ROW_COUNT_APPLIED,
ROW_COUNT_REJECTED AS ROW_COUNT_REJECTED,
'Y' AS LOAD_SUCCESSFUL,
'ODS' AS SERVICE_NAME
FROM
CT_ODS.A_LOAD_HISTORY_TARGET
WHERE
TARGET_TABLE_NAME IN (
'OU_LM_ADHOC_ADJ_HEADER',
'OU_LM_ADHOC_ADJ_ITEM',
'OU_LM_ADHOC_ADJ_ITEM_HEADER',
'OU_LM_BALANCESHEET_HEADER',
'OU_LM_BALANCESHEET_ITEM',
'OU_LM_CSM_ADJ_HEADER',
'OU_LM_CSM_ADJ_ITEM',
'OU_LM_CSM_ADJ_ITEM_HEADER',
'OU_LM_STANDING_FACILITY',
'OU_LM_STANDING_FACILITY_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_ITEM',
'OU_LM_FORECAST_HEADER',
'OU_LM_FORECAST_ITEM',
'OU_LM_QR_ADJ_HEADER',
'OU_LM_QR_ADJ_ITEM',
'OU_LM_QR_ADJ_ITEM_HEADER',
'OU_LM_TTS_HEADER',
'OU_LM_TTS_ITEM',
'CSM_ADJ_HEADER',
'CSM_ADJ_ITEM',
'CSM_ADJ_ITEM_HEADER',
'QR_ADJ_HEADER',
'QR_ADJ_ITEM',
'QR_ADJ_ITEM_HEADER',
'STANDING_FACILITY',
'STANDING_FACILITY_HEADER'
);
COMMIT;

View File

@@ -0,0 +1,82 @@
-- MOPDB
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY (
A_WORKFLOW_HISTORY_KEY,
ORCHESTRATION_RUN_ID,
WORKFLOW_NAME,
WORKFLOW_START,
WORKFLOW_END,
WORKFLOW_SUCCESSFUL,
DBT_INVOCATION_ID,
SERVICE_NAME
)
SELECT
A_MOPDB_LOAD_SET_KEY AS A_WORKFLOW_HISTORY_KEY,
CAST(WLA_RUN_ID AS VARCHAR2(200)) AS ORCHESTRATION_RUN_ID,
CASE
WHEN WORKFLOW_NAME = 'w_MOPDB_LM_STANDING_FACILITY' THEN 'w_MOPDB_LM_STANDING_FACILITIES'
ELSE WORKFLOW_NAME
END AS WORKFLOW_NAME,
LOAD_START AS WORKFLOW_START,
LOAD_END AS WORKFLOW_END,
'Y' AS WORKFLOW_SUCCESSFUL,
CAST(INFA_RUN_ID AS VARCHAR2(200)) AS DBT_INVOCATION_ID,
'MOPDB' AS SERVICE_NAME
FROM
CT_MOPDB.A_LOAD_HISTORY
WHERE
WORKFLOW_NAME IN (
'w_MOPDB_LM_ADHOC_ADJUSTMENT',
'w_MOPDB_LM_BALANCESHEET',
'w_MOPDB_LM_CSM_ADJUSTMENT',
'w_MOPDB_LM_STANDING_FACILITY',
'w_MOPDB_LM_CURRENT_ACCOUNTS',
'w_MOPDB_LM_FORECAST',
'w_MOPDB_LM_QRE_ADJUSTMENT',
'w_MOPDB_LM_TTS'
)
;
-- ODS
INSERT INTO CT_MRDS.A_WORKFLOW_HISTORY (
A_WORKFLOW_HISTORY_KEY,
ORCHESTRATION_RUN_ID,
WORKFLOW_NAME,
WORKFLOW_START,
WORKFLOW_END,
WORKFLOW_SUCCESSFUL,
DBT_INVOCATION_ID,
SERVICE_NAME
)
SELECT
A_ETL_LOAD_SET_KEY AS A_WORKFLOW_HISTORY_KEY,
CASE
WHEN WLA_RUN_ID IS NOT NULL THEN CAST(WLA_RUN_ID AS VARCHAR2(200))
ELSE EXDI_CORRELATION_ID
END AS ORCHESTRATION_RUN_ID,
CASE
WHEN WORKFLOW_NAME = 'w_ODS_LM_STANDING_FACILITY' THEN 'w_ODS_LM_STANDING_FACILITIES'
WHEN WORKFLOW_NAME = 'w_ODS_MRR_IND_CURRENT_ACCOUNT_ITEM' THEN 'w_ODS_LM_CURRENT_ACCOUNT_ITEM'
WHEN WORKFLOW_NAME = 'w_ODS_QUARTERLY_ADJUSTMENT_MSG' THEN 'w_ODS_LM_QUARTERLY_ADJUSTMENT_MSG'
WHEN WORKFLOW_NAME = 'w_ODS_ADHOC_ADJUSTMENT_MSG' THEN 'w_ODS_LM_ADHOC_ADJUSTMENTS_MSG'
WHEN WORKFLOW_NAME = 'w_ODS_CSM_ADJUSTMENT_MSG' THEN 'w_ODS_LM_CSM_ADJUSTMENT_MSG'
ELSE WORKFLOW_NAME
END AS WORKFLOW_NAME,
LOAD_START AS WORKFLOW_START,
LOAD_END AS WORKFLOW_END,
LOAD_SUCCESSFUL AS WORKFLOW_SUCCESSFUL,
CAST(INFA_RUN_ID AS VARCHAR2(200)) AS DBT_INVOCATION_ID,
'ODS' AS SERVICE_NAME
FROM CT_ODS.A_LOAD_HISTORY
WHERE
WORKFLOW_NAME IN (
'w_ODS_ADHOC_ADJUSTMENT_MSG',
'w_ODS_LM_BALANCESHEET',
'w_ODS_CSM_ADJUSTMENT_MSG',
'w_ODS_LM_STANDING_FACILITY',
'w_ODS_MRR_IND_CURRENT_ACCOUNT_ITEM',
'w_ODS_QUARTERLY_ADJUSTMENT_MSG'
)
;
COMMIT;

View File

@@ -0,0 +1,36 @@
-- MOPDB
DELETE FROM CT_MRDS.A_TASK_HISTORY t WHERE EXISTS (
SELECT 1
FROM CT_MOPDB.A_LOAD_SUBPROCESS r
WHERE
r.SUBPROCESS_NAME IN (
SELECT DISTINCT SUBPROCESS_NAME
FROM CT_MOPDB.A_LOAD_HISTORY A
LEFT JOIN CT_MOPDB.A_LOAD_SUBPROCESS B
ON A.A_MOPDB_LOAD_SET_KEY = b.A_MOPDB_LOAD_SET_FK
WHERE
WORKFLOW_NAME IN (
'w_MOPDB_LM_ADHOC_ADJUSTMENT',
'w_MOPDB_LM_BALANCESHEET',
'w_MOPDB_LM_CSM_ADJUSTMENT',
'w_MOPDB_LM_STANDING_FACILITY',
'w_MOPDB_LM_CURRENT_ACCOUNTS',
'w_MOPDB_LM_FORECAST',
'w_MOPDB_LM_QRE_ADJUSTMENT',
'w_MOPDB_LM_TTS'
)
)
AND t.A_TASK_HISTORY_KEY = r.A_LOAD_SUBPROCESS_KEY
AND t.A_WORKFLOW_HISTORY_KEY = r.A_MOPDB_LOAD_SET_FK
AND NVL(t.TASK_RUN_ID,' ') = NVL(CAST(NULL AS VARCHAR2(200)),' ')
AND NVL(t.TASK_NAME,' ') = NVL(CASE
WHEN r.SUBPROCESS_NAME = 'm_MOPDB_LM_T_STANDING_FACILITIES_OU_LM_STANDING_FACILITY_LOAD' THEN 'm_MOPDB_LM_T_STANDING_FACILITIES_OU_LM_STANDING_FACILITIES_LOAD'
ELSE r.SUBPROCESS_NAME
END,' ')
AND NVL(t.TASK_START,SYSDATE) = NVL(r.PROCESS_START,SYSDATE)
AND NVL(t.TASK_END,SYSDATE) = NVL(r.PROCESS_END,SYSDATE)
AND NVL(t.TASK_SUCCESSFUL,' ')= NVL(r.PROCESS_SUCCESSFUL,' ')
AND t.SERVICE_NAME = 'MOPDB'
);
COMMIT;

View File

@@ -0,0 +1,53 @@
-- MOPDB
DELETE FROM CT_MRDS.A_TASK_HISTORY_SOURCE t WHERE EXISTS (
SELECT 1
FROM
CT_MOPDB.A_LOAD_HISTORY_SOURCE r
WHERE
r.SOURCE_TABLE_NAME IN (
'OU_LM_ADHOC_ADJ_HEADER',
'OU_LM_ADHOC_ADJ_ITEM',
'OU_LM_ADHOC_ADJ_ITEM_HEADER',
'OU_LM_BALANCESHEET_HEADER',
'OU_LM_BALANCESHEET_ITEM',
'OU_LM_CSM_ADJ_HEADER',
'OU_LM_CSM_ADJ_ITEM',
'OU_LM_CSM_ADJ_ITEM_HEADER',
'OU_LM_STANDING_FACILITY',
'OU_LM_STANDING_FACILITY_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_ITEM',
'OU_LM_FORECAST_HEADER',
'OU_LM_FORECAST_ITEM',
'OU_LM_QR_ADJ_HEADER',
'OU_LM_QR_ADJ_ITEM',
'OU_LM_QR_ADJ_ITEM_HEADER',
'OU_LM_TTS_HEADER',
'OU_LM_TTS_ITEM'
)
AND t.A_TASK_HISTORY_SOURCE_KEY = r.A_LOAD_HISTORY_SOURCE_KEY
AND t.A_TASK_HISTORY_KEY = r.A_LOAD_SUBPROCESS_FK
AND NVL(t.A_WORKFLOW_HISTORY_SOURCE_KEY,0) = NVL(r.A_ODS_LOAD_SET_FK,0)
AND NVL(t.SOURCE_NAME,' ') = NVL(CASE
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_HEADER' THEN 'OU_LM_CURRENT_ACCOUNTS_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_ITEM' THEN 'OU_LM_CURRENT_ACCOUNTS_ITEM'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN r.SOURCE_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
ELSE r.SOURCE_TABLE_NAME
END,' ')
AND NVL(t.ROW_COUNT,0) = NVL(r.ROW_COUNT,0)
AND t.SERVICE_NAME = 'MOPDB'
)
;
COMMIT;

View File

@@ -0,0 +1,95 @@
-- MOPDB
DELETE FROM CT_MRDS.A_TASK_HISTORY_TARGET t WHERE EXISTS (
SELECT 1
FROM
CT_MOPDB.A_LOAD_HISTORY_TARGET r
WHERE
TARGET_TABLE_NAME IN (
'LM_T_FC_ADH_ADJUSTMENTS',
'LM_T_BALANCE_SHEET',
'LM_T_FC_CSM_ADJUSTMENTS',
'LM_T_STANDING_FACILITIES',
'LM_T_CURRENT_ACCOUNTS',
'LM_T_FORECAST',
'LM_T_FC_QRE_ADJUSTMENTS',
'LM_T_TTS'
)
AND t.A_TASK_HISTORY_TARGET_KEY = r.A_LOAD_HISTORY_TARGET_KEY
AND t.A_TASK_HISTORY_KEY = r.A_LOAD_SUBPROCESS_FK
AND NVL(t.TARGET_NAME,' ') = NVL(r.TARGET_TABLE_NAME,' ')
AND NVL(t.ROW_COUNT_APPLIED,0) = NVL(r.ROW_COUNT_APPLIED,0)
AND NVL(t.ROW_COUNT_REJECTED,0) = NVL(r.ROW_COUNT_REJECTED,0)
AND NVL(t.LOAD_SUCCESSFUL,' ') = NVL(r.LOAD_SUCCESSFUL,' ')
AND t.SERVICE_NAME = 'MOPDB'
);
-- ODS
DELETE FROM CT_MRDS.A_TASK_HISTORY_TARGET t WHERE EXISTS (
SELECT 1
FROM
CT_ODS.A_LOAD_HISTORY_TARGET r
WHERE
TARGET_TABLE_NAME IN (
'OU_LM_ADHOC_ADJ_HEADER',
'OU_LM_ADHOC_ADJ_ITEM',
'OU_LM_ADHOC_ADJ_ITEM_HEADER',
'OU_LM_BALANCESHEET_HEADER',
'OU_LM_BALANCESHEET_ITEM',
'OU_LM_CSM_ADJ_HEADER',
'OU_LM_CSM_ADJ_ITEM',
'OU_LM_CSM_ADJ_ITEM_HEADER',
'OU_LM_STANDING_FACILITY',
'OU_LM_STANDING_FACILITY_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_HEADER',
'OU_MRR_MRR_IND_CURR_ACC_ITEM',
'OU_LM_FORECAST_HEADER',
'OU_LM_FORECAST_ITEM',
'OU_LM_QR_ADJ_HEADER',
'OU_LM_QR_ADJ_ITEM',
'OU_LM_QR_ADJ_ITEM_HEADER',
'OU_LM_TTS_HEADER',
'OU_LM_TTS_ITEM',
'CSM_ADJ_HEADER',
'CSM_ADJ_ITEM',
'CSM_ADJ_ITEM_HEADER',
'QR_ADJ_HEADER',
'QR_ADJ_ITEM',
'QR_ADJ_ITEM_HEADER',
'STANDING_FACILITY',
'STANDING_FACILITY_HEADER'
)
AND t.A_TASK_HISTORY_TARGET_KEY = r.A_LOAD_HISTORY_TARGET_KEY
AND t.A_TASK_HISTORY_KEY = r.A_ETL_LOAD_SET_FK
AND NVL(t.TARGET_NAME,' ') = NVL(CASE
WHEN r.TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_ADHOC_ADJ_ITEM_HEADER' THEN 'OU_LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'CSM_ADJ_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'CSM_ADJ_ITEM' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN r.TARGET_TABLE_NAME = 'CSM_ADJ_ITEM_HEADER' THEN 'OU_LM_CSM_ADJUSTMENTS_ITEM_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN r.TARGET_TABLE_NAME = 'STANDING_FACILITY' THEN 'OU_LM_STANDING_FACILITIES'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN r.TARGET_TABLE_NAME = 'STANDING_FACILITY_HEADER' THEN 'OU_LM_STANDING_FACILITIES_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_HEADER' THEN 'OU_LM_CURRENT_ACCOUNTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_MRR_MRR_IND_CURR_ACC_ITEM' THEN 'OU_LM_CURRENT_ACCOUNTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'QR_ADJ_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_HEADER'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'QR_ADJ_ITEM' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM'
WHEN r.TARGET_TABLE_NAME = 'OU_LM_QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
WHEN r.TARGET_TABLE_NAME = 'QR_ADJ_ITEM_HEADER' THEN 'OU_LM_QRE_ADJUSTMENTS_ITEM_HEADER'
ELSE r.TARGET_TABLE_NAME
END,' ')
AND NVL(t.ROW_COUNT_APPLIED,0) = NVL(r.ROW_COUNT_APPLIED,0)
AND NVL(t.ROW_COUNT_REJECTED,0) = NVL(r.ROW_COUNT_REJECTED,0)
AND NVL(t.LOAD_SUCCESSFUL,' ') = 'Y'
AND t.SERVICE_NAME = 'ODS'
);
COMMIT;

View File

@@ -0,0 +1,59 @@
-- MOPDB
DELETE FROM CT_MRDS.A_WORKFLOW_HISTORY t WHERE EXISTS (
SELECT 1
FROM
CT_MOPDB.A_LOAD_HISTORY r
WHERE
r.WORKFLOW_NAME IN (
'w_MOPDB_LM_ADHOC_ADJUSTMENT',
'w_MOPDB_LM_BALANCESHEET',
'w_MOPDB_LM_CSM_ADJUSTMENT',
'w_MOPDB_LM_STANDING_FACILITY',
'w_MOPDB_LM_CURRENT_ACCOUNTS',
'w_MOPDB_LM_FORECAST',
'w_MOPDB_LM_QRE_ADJUSTMENT',
'w_MOPDB_LM_TTS'
)
AND t.A_WORKFLOW_HISTORY_KEY = r.A_MOPDB_LOAD_SET_KEY
AND NVL(t.ORCHESTRATION_RUN_ID,' ') = NVL(CAST(r.WLA_RUN_ID AS VARCHAR2(200)),' ')
AND t.WORKFLOW_NAME = CASE
WHEN r.WORKFLOW_NAME = 'w_MOPDB_LM_STANDING_FACILITY' THEN 'w_MOPDB_LM_STANDING_FACILITIES'
ELSE r.WORKFLOW_NAME
END
AND NVL(t.WORKFLOW_START,SYSDATE) = NVL(r.LOAD_START,SYSDATE)
AND NVL(t.WORKFLOW_END,SYSDATE) = NVL(r.LOAD_END,SYSDATE)
AND NVL(t.WORKFLOW_SUCCESSFUL,' ') = NVL(CAST(NULL AS VARCHAR2(200)),' ')
AND NVL(t.DBT_INVOCATION_ID,' ') = NVL(CAST(r.INFA_RUN_ID AS VARCHAR2(200)),' ')
AND t.SERVICE_NAME = 'MOPDB'
);
-- ODS
DELETE FROM CT_MRDS.A_WORKFLOW_HISTORY t WHERE EXISTS (
SELECT 1
FROM CT_ODS.A_LOAD_HISTORY r
WHERE
WORKFLOW_NAME IN (
'w_ODS_ADHOC_ADJUSTMENT_MSG',
'w_ODS_LM_BALANCESHEET',
'w_ODS_CSM_ADJUSTMENT_MSG',
'w_ODS_LM_STANDING_FACILITY',
'w_ODS_MRR_IND_CURRENT_ACCOUNT_ITEM',
'w_ODS_QUARTERLY_ADJUSTMENT_MSG'
)
AND t.A_WORKFLOW_HISTORY_KEY = r.A_ETL_LOAD_SET_KEY
AND NVL(t.ORCHESTRATION_RUN_ID,' ') = NVL( CASE WHEN r.WLA_RUN_ID IS NOT NULL THEN CAST(r.WLA_RUN_ID AS VARCHAR2(200)) ELSE r.EXDI_CORRELATION_ID END,' ')
AND t.WORKFLOW_NAME = CASE
WHEN r.WORKFLOW_NAME = 'w_ODS_LM_STANDING_FACILITY' THEN 'w_ODS_LM_STANDING_FACILITIES'
WHEN r.WORKFLOW_NAME = 'w_ODS_MRR_IND_CURRENT_ACCOUNT_ITEM' THEN 'w_ODS_LM_CURRENT_ACCOUNT_ITEM'
WHEN r.WORKFLOW_NAME = 'w_ODS_QUARTERLY_ADJUSTMENT_MSG' THEN 'w_ODS_LM_QUARTERLY_ADJUSTMENT_MSG'
ELSE r.WORKFLOW_NAME
END
AND NVL(t.WORKFLOW_START,SYSDATE) = NVL(r.LOAD_START,SYSDATE)
AND NVL(t.WORKFLOW_END,SYSDATE) = NVL(r.LOAD_END,SYSDATE)
AND NVL(t.WORKFLOW_SUCCESSFUL,' ') = NVL(r.LOAD_SUCCESSFUL,' ')
AND NVL(t.DBT_INVOCATION_ID,' ') = NVL(CAST(r.INFA_RUN_ID AS VARCHAR2(200)),' ')
AND t.SERVICE_NAME = 'ODS'
);
COMMIT;

View File

@@ -0,0 +1,40 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
var filename VARCHAR2(100)
BEGIN
SELECT 'INSTALL_MARS_772_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) AS PDB_NAME FROM DBA_PDBS;
@@00_MARS_772_backup_CT_MRDS_A_WORKFLOW_HISTORY.sql
@@00_MARS_772_backup_CT_MRDS_A_TASK_HISTORY.sql
@@00_MARS_772_backup_CT_MRDS_A_TASK_HISTORY_SOURCE.sql
@@00_MARS_772_backup_CT_MRDS_A_TASK_HISTORY_TARGET.sql
@@01_MARS_772_install_CT_MRDS_A_WORKFLOW_HISTORY.sql
@@01_MARS_772_install_CT_MRDS_A_TASK_HISTORY.sql
@@01_MARS_772_install_CT_MRDS_A_TASK_HISTORY_SOURCE.sql
@@01_MARS_772_install_CT_MRDS_A_TASK_HISTORY_TARGET.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

@@ -0,0 +1,36 @@
WHENEVER SQLERROR EXIT FAILURE
SET SERVEROUTPUT ON
SET TIMING ON
SET ECHO ON
SET HEADING OFF
SET FEEDBACK ON
SET VERIFY OFF
var filename VARCHAR2(100)
BEGIN
SELECT 'ROLLBACK_MARS_772_' || SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) || '_' ||TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS')||'.log' INTO :filename from DBA_PDBS;
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename
prompt ##### started at time #####
select systimestamp from dual;
prompt ##### database name #####
SELECT SUBSTR(PDB_NAME, (INSTR(PDB_NAME,'_',1)+1), (LENGTH(PDB_NAME)-INSTR(PDB_NAME,'_',1))) AS PDB_NAME FROM DBA_PDBS;
@@91_MARS_772_rollback_CT_MRDS_A_WORKFLOW_HISTORY.sql
@@91_MARS_772_rollback_CT_MRDS_A_TASK_HISTORY.sql
@@91_MARS_772_rollback_CT_MRDS_A_TASK_HISTORY_SOURCE.sql
@@91_MARS_772_rollback_CT_MRDS_A_TASK_HISTORY_TARGET.sql
SET ECHO OFF
prompt ##### completed at time #####
select systimestamp from dual;
SPOOL OFF
EXIT

View File

View File

@@ -0,0 +1,247 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--table old ADHOC_ADJ_HEADER -> new LM_ADHOC_ADJUSTMENTS_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_ADHOC_ADJUSTMENTS_HEADER"
( "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"ADJUSTMENT_DATE" DATE,
"VERSION" NUMBER(10,0)
);
/
--table old ADHOC_ADJ_ITEM -> new LM_ADHOC_ADJUSTMENTS_ITEM
CREATE TABLE "CT_ET_TEMPLATES"."LM_ADHOC_ADJUSTMENTS_ITEM"
( "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"FORECAST_NAME" VARCHAR2(50 BYTE) COLLATE "USING_NLS_COMP",
"ADJUSTMENT_AMOUNT" NUMBER(25,8)
);
/
--table old ADHOC_ADJ_ITEM_HEADER - > new ADHOC_ADJUSTMENTS_ITEM_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_ADHOC_ADJUSTMENTS_ITEM_HEADER"
( "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 BYTE) COLLATE "USING_NLS_COMP",
"EFFECTIVE_DATE" DATE,
"LAST_DATE_NOT_FORECAST" DATE
) ;
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_ADHOC_ADJUSTMENTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'AdhocAdjustment'
,pSourceFileDesc => 'Table provides the ad-hoc revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_ADHOC_ADJUSTMENTS_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_ADHOC_ADJUSTMENTS_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'AdhocAdjustment'
,pSourceFileDesc => 'Table provides the ad-hoc revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_ADHOC_ADJUSTMENTS_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM'
,pContainerFileKey => NULL
);
/
--Table LM_ADHOC_ADJUSTMENTS_ITEM_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'AdhocAdjustment'
,pSourceFileDesc => 'Table provides the ad-hoc revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_ADHOC_ADJUSTMENTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_HEADER',
pColumnName => 'ADJUSTMENT_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--Table LM_ADHOC_ADJUSTMENTS_ITEM_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'EFFECTIVE_DATE', pDateFormat => 'YYYY-MM-DD'
);
/
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'LAST_DATE_NOT_FORECAST',
pDateFormat => 'YYYY-MM-DD');
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_ADHOC_ADJUSTMENTS_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_HEADER',
pPrefix => 'INBOX/LM/AdhocAdjustment/LM_ADHOC_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_HEADER',
pPrefix => 'ODS/LM/LM_ADHOC_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_HEADER',
pPrefix => 'ARCHIVE/LM/LM_ADHOC_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_ADHOC_ADJUSTMENTS_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM',
pPrefix => 'INBOX/LM/AdhocAdjustment/LM_ADHOC_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM',
pPrefix => 'ODS/LM/LM_ADHOC_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM',
pPrefix => 'ARCHIVE/LM/LM_ADHOC_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--------------------------------------------------------------------
--Table LM_ADHOC_ADJUSTMENTS_ITEM_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'INBOX/LM/AdhocAdjustment/LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ODS/LM/LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_ADHOC_ADJUSTMENTS_ITEM_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ARCHIVE/LM/LM_ADHOC_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View ADHOC_ADJUSTMENTS_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.ADHOC_ADJ_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
ADJUSTMENT_DATE,
VERSION
FROM ODS.LM_ADHOC_ADJUSTMENTS_HEADER_ODS;
/
--View ADHOC_ADJUSTMENTS_ITEM_MARS
CREATE OR REPLACE VIEW OU_LM.ADHOC_ADJ_ITEM_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
FORECAST_NAME,
ADJUSTMENT_AMOUNT
FROM ODS.LM_ADHOC_ADJUSTMENTS_ITEM_ODS;
/
--View ADHOC_ADJUSTMENTS_ITEM_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.ADHOC_ADJ_ITEM_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
COUNTRY,
EFFECTIVE_DATE,
LAST_DATE_NOT_FORECAST
FROM ODS.LM_ADHOC_ADJUSTMENTS_ITEM_HEADER_ODS;
/

View File

@@ -0,0 +1,178 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--Table LM_BALANCESHEET_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_BALANCESHEET_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 CHAR),
"REFERENCE_DATE" DATE,
"VERSION" NUMBER(10,0),
"STATUS" VARCHAR2(1 CHAR),
"FREE_TEXT" VARCHAR2(4000 CHAR)
);
/
--Table LM_BALANCESHEET_ITEM
CREATE TABLE "CT_ET_TEMPLATES"."LM_BALANCESHEET_ITEM"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"ITEM_LEVEL" NUMBER(10,0),
"POSITION" NUMBER(10,0),
"ITEM_TYPE" VARCHAR2(10 CHAR),
"FULLY_QUALIFIED_POSITION" VARCHAR2(200 CHAR),
"NAME" VARCHAR2(200 CHAR),
"AMOUNT" NUMBER(28,8)
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_BALANCESHEET_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'AggregatedDailyFinancialStatementOfTheEurosystem'
,pSourceFileDesc => 'Table provides data on the assets and liabilities of the ECB and Euro Area NCBs.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_BALANCESHEET_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_BALANCESHEET_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'AggregatedDailyFinancialStatementOfTheEurosystem'
,pSourceFileDesc => 'Table provides data on the assets and liabilities of the ECB and Euro Area NCBs.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_BALANCESHEET_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_ITEM'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_BALANCESHEET_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_HEADER',
pColumnName => 'REFERENCE_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_BALANCESHEET_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_HEADER',
pPrefix => 'INBOX/LM/AggregatedDailyFinancialStatementOfTheEurosystem/LM_BALANCESHEET_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_HEADER',
pPrefix => 'ODS/LM/LM_BALANCESHEET_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_HEADER',
pPrefix => 'ARCHIVE/LM/LM_BALANCESHEET_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_BALANCESHEET_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_ITEM',
pPrefix => 'INBOX/LM/AggregatedDailyFinancialStatementOfTheEurosystem/LM_BALANCESHEET_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_ITEM',
pPrefix => 'ODS/LM/LM_BALANCESHEET_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_BALANCESHEET_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_BALANCESHEET_ITEM',
pPrefix => 'ARCHIVE/LM/LM_BALANCESHEET_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View BALANCESHEET_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.BALANCESHEET_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY,
COUNTRY,
REFERENCE_DATE,
VERSION,
STATUS,
FREE_TEXT
FROM ODS.LM_BALANCESHEET_HEADER_ODS;
/
--View BALANCESHEET_ITEM_MARS
CREATE OR REPLACE VIEW OU_LM.BALANCESHEET_ITEM_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY,
A_HEADER_FK,
ITEM_LEVEL,
POSITION,
ITEM_TYPE,
FULLY_QUALIFIED_POSITION,
NAME,
AMOUNT
FROM ODS.LM_BALANCESHEET_ITEM_ODS;
/

View File

@@ -0,0 +1,250 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--table old CSM_ADJ_HEADER -> new LM_CSM_ADJUSTMENTS_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_CSM_ADJUSTMENTS_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"YEAR" VARCHAR2(4 BYTE) COLLATE "USING_NLS_COMP",
"MONTH" VARCHAR2(2 BYTE) COLLATE "USING_NLS_COMP",
"VERSION" NUMBER(10,0)
);
/
--table old CSM_ADJ_ITEM -> new LM_CSM_ADJUSTMENTS_ITEM
CREATE TABLE "CT_ET_TEMPLATES"."LM_CSM_ADJUSTMENTS_ITEM"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"FORECAST_NAME" VARCHAR2(50 BYTE) COLLATE "USING_NLS_COMP",
"ADJUSTMENT_AMOUNT" VARCHAR2(50 BYTE) -- !!!! due to incomming data containing comma
--"ADJUSTMENT_AMOUNT" NUMBER(25,8)
);
/
--table old CSM_ADJ_ITEM_HEADER - > new LM_CSM_ADJUSTMENTS_ITEM_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_CSM_ADJUSTMENTS_ITEM_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 BYTE) COLLATE "USING_NLS_COMP",
"EFFECTIVE_DATE" DATE,
"LAST_DATE_NOT_FORECAST" DATE
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_CSM_ADJUSTMENTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'MonthlyCSMAdjustment'
,pSourceFileDesc => 'Table provides the balance sheet with revised banknotes in line with monthly capital share mechanism.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_CSM_ADJUSTMENTS_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_CSM_ADJUSTMENTS_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'MonthlyCSMAdjustment'
,pSourceFileDesc => 'Table provides the balance sheet with revised banknotes in line with monthly capital share mechanism.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_CSM_ADJUSTMENTS_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM'
,pContainerFileKey => NULL
);
/
--Table LM_CSM_ADJUSTMENTS_ITEM_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'MonthlyCSMAdjustment'
,pSourceFileDesc => 'Table provides the balance sheet with revised banknotes in line with monthly capital share mechanism.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_CSM_ADJUSTMENTS_ITEM_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_CSM_ADJUSTMENTS_ITEM_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'EFFECTIVE_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'LAST_DATE_NOT_FORECAST',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_CSM_ADJUSTMENTS_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_HEADER',
pPrefix => 'INBOX/LM/MonthlyCSMAdjustment/LM_CSM_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_HEADER',
pPrefix => 'ODS/LM/LM_CSM_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_HEADER',
pPrefix => 'ARCHIVE/LM/LM_CSM_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_CSM_ADJUSTMENTS_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM',
pPrefix => 'INBOX/LM/MonthlyCSMAdjustment/LM_CSM_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM',
pPrefix => 'ODS/LM/LM_CSM_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM',
pPrefix => 'ARCHIVE/LM/LM_CSM_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--------------------------------------------------------------------
--Table LM_CSM_ADJUSTMENTS_ITEM_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'INBOX/LM/MonthlyCSMAdjustment/LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ODS/LM/LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CSM_ADJUSTMENTS_ITEM_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ARCHIVE/LM/LM_CSM_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View CSM_ADJUSTMENTS_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.CSM_ADJ_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
YEAR,
MONTH,
VERSION
FROM ODS.LM_CSM_ADJUSTMENTS_HEADER_ODS;
/
--View CSM_ADJUSTMENTS_ITEM_MARS
CREATE OR REPLACE VIEW OU_LM.CSM_ADJ_ITEM_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
FORECAST_NAME,
ADJUSTMENT_AMOUNT
FROM ODS.LM_CSM_ADJUSTMENTS_ITEM_ODS;
/
--View CSM_ADJUSTMENTS_ITEM_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.CSM_ADJ_ITEM_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
COUNTRY,
EFFECTIVE_DATE,
LAST_DATE_NOT_FORECAST
FROM ODS.LM_CSM_ADJUSTMENTS_ITEM_HEADER_ODS;
/

View File

@@ -0,0 +1,184 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--Table LM_CURRENT_ACCOUNTS_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_CURRENT_ACCOUNTS_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"REVISION_NUMBER" NUMBER(10,0),
"REFERENCE_DATE" DATE,
"FREE_TEXT" VARCHAR2(1000 CHAR),
"CURRENT_ACCOUNT_BS_TOTAL" NUMBER(28,8),
"MRR_FORECAST_TOTAL" NUMBER(28,8),
"CURRENT_ACCOUNT_MESSAGE_TOTAL" NUMBER(28,8),
"MRR_MESSAGE_TOTAL" NUMBER(28,8),
"PERIODICITY" VARCHAR2(50 CHAR)
);
/
--Table LM_CURRENT_ACCOUNTS_ITEM
CREATE TABLE "CT_ET_TEMPLATES"."LM_CURRENT_ACCOUNTS_ITEM"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 CHAR),
"MFI_CODE" VARCHAR2(255 CHAR),
"BANK_NAME" VARCHAR2(500 CHAR),
"CURRENT_ACCOUNT" NUMBER(28,8),
"MINIMUM_RESERVE_REQUIREMENT" NUMBER(28,8),
"COMMENT_" VARCHAR2(4000 CHAR)
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_CURRENT_ACCOUNTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'MinimumReservesRequirements'
,pSourceFileDesc => 'Table provides data on the account balance that credit institutions hold with their respective NCB within the Eurosystem. This is primarily used to hold minimum reserves.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_CURRENT_ACCOUNTS_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_CURRENT_ACCOUNTS_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'MinimumReservesRequirements'
,pSourceFileDesc => 'Table provides data on the account balance that credit institutions hold with their respective NCB within the Eurosystem. This is primarily used to hold minimum reserves.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_CURRENT_ACCOUNTS_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_ITEM'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_CURRENT_ACCOUNTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_HEADER',
pColumnName => 'REFERENCE_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_CURRENT_ACCOUNTS_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_HEADER',
pPrefix => 'INBOX/LM/MinimumReservesRequirements/LM_CURRENT_ACCOUNTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_HEADER',
pPrefix => 'ODS/LM/LM_CURRENT_ACCOUNTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_HEADER',
pPrefix => 'ARCHIVE/LM/LM_CURRENT_ACCOUNTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_CURRENT_ACCOUNTS_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_ITEM',
pPrefix => 'INBOX/LM/MinimumReservesRequirements/LM_CURRENT_ACCOUNTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_ITEM',
pPrefix => 'ODS/LM/LM_CURRENT_ACCOUNTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_CURRENT_ACCOUNTS_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_CURRENT_ACCOUNTS_ITEM',
pPrefix => 'ARCHIVE/LM/LM_CURRENT_ACCOUNTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View MRR_IND_CURRENT_ACCOUNT_HEADER_MARS
CREATE OR REPLACE VIEW OU_MRR.MRR_IND_CURRENT_ACCOUNT_HEADER_MARS AS
SELECT
REVISION_NUMBER,
REFERENCE_DATE,
FREE_TEXT,
CURRENT_ACCOUNT_BS_TOTAL,
MRR_FORECAST_TOTAL,
CURRENT_ACCOUNT_MESSAGE_TOTAL,
MRR_MESSAGE_TOTAL,
PERIODICITY,
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY
FROM ODS.LM_CURRENT_ACCOUNTS_HEADER_ODS;
/
--View MRR_IND_CURRENT_ACCOUNT_ITEM_MARS
CREATE OR REPLACE VIEW OU_MRR.MRR_IND_CURRENT_ACCOUNT_ITEM_MARS AS
SELECT
COUNTRY,
MFI_CODE,
BANK_NAME,
CURRENT_ACCOUNT,
MINIMUM_RESERVE_REQUIREMENT,
COMMENT_,
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY,
A_HEADER_FK
FROM ODS.LM_CURRENT_ACCOUNTS_ITEM_ODS;
/

View File

@@ -0,0 +1,178 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--Table LM_FORECAST_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_FORECAST_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 CHAR),
"REFERENCE_DATE" DATE,
"REVISION" NUMBER(10,0),
"FREE_TEXT" VARCHAR2(4000 CHAR)
);
/
--Table LM_FORECAST_ITEM
CREATE TABLE "CT_ET_TEMPLATES"."LM_FORECAST_ITEM"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0),
"FORECAST_DATE" DATE,
"FORECAST_NAME" VARCHAR2(50 CHAR),
"FORECAST_VALUE" NUMBER(28,10)
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_FORECAST_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'EurosystemAutonomousFactorForecast'
,pSourceFileDesc => 'Table provides forecasts on the autonomous factors, monetary policy portfolios and reserve requirements. The data is updated weekly and disseminated daily'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_FORECAST_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_FORECAST_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'EurosystemAutonomousFactorForecast'
,pSourceFileDesc => 'Table provides forecasts on the autonomous factors, monetary policy portfolios and reserve requirements. The data is updated weekly and disseminated daily'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_FORECAST_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_ITEM'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_FORECAST_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_HEADER',
pColumnName => 'REFERENCE_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--Table LM_FORECAST_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_ITEM',
pColumnName => 'FORECAST_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_FORECAST_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_HEADER',
pPrefix => 'INBOX/LM/EurosystemAutonomousFactorForecast/LM_FORECAST_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_HEADER',
pPrefix => 'ODS/LM/LM_FORECAST_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_HEADER',
pPrefix => 'ARCHIVE/LM/LM_FORECAST_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_FORECAST_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_ITEM',
pPrefix => 'INBOX/LM/EurosystemAutonomousFactorForecast/LM_FORECAST_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_ITEM',
pPrefix => 'ODS/LM/LM_FORECAST_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_FORECAST_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_FORECAST_ITEM',
pPrefix => 'ARCHIVE/LM/LM_FORECAST_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View FORECAST_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.FORECAST_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_FK,
COUNTRY,
REFERENCE_DATE,
REVISION,
FREE_TEXT
FROM ODS.LM_FORECAST_HEADER_ODS;
/
--View FORECAST_ITEM_MARS
CREATE OR REPLACE VIEW OU_LM.FORECAST_ITEM_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_FK,
A_HEADER_FK,
FORECAST_DATE,
FORECAST_NAME,
FORECAST_VALUE
FROM ODS.LM_FORECAST_ITEM_ODS;
/

View File

@@ -0,0 +1,247 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--table old QR_ADJ_HEADER -> new LM_QRE_ADJUSTMENTS_HEADER
CREATE TABLE "CT_ET_TEMPLATES".LM_QRE_ADJUSTMENTS_HEADER
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"YEAR" VARCHAR2(4 CHAR) COLLATE "USING_NLS_COMP",
"QUARTER" NUMBER(1,0),
"VERSION" NUMBER(10,0)
);
/
--table old QR_ADJ_ITEM -> new LM_QRE_ADJUSTMENTS_ITEM
CREATE TABLE "CT_ET_TEMPLATES".LM_QRE_ADJUSTMENTS_ITEM
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"FORECAST_NAME" VARCHAR2(50 BYTE) COLLATE "USING_NLS_COMP",
"ADJUSTMENT_AMOUNT" NUMBER(25,8)
);
/
--table old QR_ADJ_ITEM_HEADER - > new LM_QRE_ADJUSTMENTS_ITEM_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_QRE_ADJUSTMENTS_ITEM_HEADER"
( "A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_HEADER_FK" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(3 BYTE) COLLATE "USING_NLS_COMP",
"EFFECTIVE_DATE" DATE,
"LAST_DATE_NOT_FORECAST" DATE
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_QRE_ADJUSTMENTS_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'QuarterlyRevaluationAdjustment'
,pSourceFileDesc => 'Table provides the quarterly revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_QRE_ADJUSTMENTS_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_HEADER'
,pContainerFileKey => NULL
);
/
--Table LM_QRE_ADJUSTMENTS_ITEM
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'QuarterlyRevaluationAdjustment'
,pSourceFileDesc => 'Table provides the quarterly revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_QRE_ADJUSTMENTS_ITEM'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM'
,pContainerFileKey => NULL
);
/
--Table LM_QRE_ADJUSTMENTS_ITEM_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'QuarterlyRevaluationAdjustment'
,pSourceFileDesc => 'Table provides the quarterly revaluations adjustments related to the balance sheet.'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_QRE_ADJUSTMENTS_ITEM_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Change data format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'EFFECTIVE_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--Change data format
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pColumnName => 'LAST_DATE_NOT_FORECAST',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_QRE_ADJUSTMENTS_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_HEADER',
pPrefix => 'INBOX/LM/QuarterlyRevaluationAdjustment/LM_QRE_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_HEADER',
pPrefix => 'ODS/LM/LM_QRE_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_HEADER',
pPrefix => 'ARCHIVE/LM/LM_QRE_ADJUSTMENTS_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_QRE_ADJUSTMENTS_ITEM
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM',
pPrefix => 'INBOX/LM/QuarterlyRevaluationAdjustment/LM_QRE_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM',
pPrefix => 'ODS/LM/LM_QRE_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM',
pPrefix => 'ARCHIVE/LM/LM_QRE_ADJUSTMENTS_ITEM',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--------------------------------------------------------------------
--Table LM_QRE_ADJUSTMENTS_ITEM_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'INBOX/LM/QuarterlyRevaluationAdjustment/LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ODS/LM/LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_QRE_ADJUSTMENTS_ITEM_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pPrefix => 'ARCHIVE/LM/LM_QRE_ADJUSTMENTS_ITEM_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View QRE_ADJUSTMENTS_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.QRE_ADJ_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
YEAR,
QUARTER,
VERSION
FROM ODS.LM_QRE_ADJUSTMENTS_HEADER_ODS;
/
--View QRE_ADJUSTMENTS_ITEM_MARS
CREATE OR REPLACE VIEW OU_LM.QRE_ADJ_ITEM_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
FORECAST_NAME,
ADJUSTMENT_AMOUNT
FROM ODS.LM_QRE_ADJUSTMENTS_ITEM_ODS;
/
--View QRE_ADJUSTMENTS_ITEM_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.QRE_ADJ_ITEM_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_KEY_FK,
A_HEADER_FK,
COUNTRY,
EFFECTIVE_DATE,
LAST_DATE_NOT_FORECAST
FROM ODS.LM_QRE_ADJUSTMENTS_ITEM_HEADER_ODS;
/

View File

@@ -0,0 +1,180 @@
--=============================================================================================================================
--Step 1: Create Template Table
--=============================================================================================================================
--Table LM_STANDING_FACILITIES
CREATE TABLE "CT_ET_TEMPLATES"."LM_STANDING_FACILITIES"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_SFH_FK" NUMBER(38,0) NOT NULL ENABLE,
"COUNTRY" VARCHAR2(12 BYTE),
"MFI_ID" VARCHAR2(1020 BYTE),
"MFI_NAME" VARCHAR2(2000 BYTE),
"MARGINAL_LENDING_FACILITY" NUMBER(28,10),
"DEPOSIT_FACILITY" NUMBER(28,10),
"COMMENT_" VARCHAR2(16000 BYTE)
);
/
--Table STANDING_FACILITIES_HEADER
CREATE TABLE "CT_ET_TEMPLATES"."LM_STANDING_FACILITIES_HEADER"
("A_KEY" NUMBER(38,0) NOT NULL ENABLE,
"A_WORKFLOW_HISTORY_KEY" NUMBER(38,0) NOT NULL ENABLE,
"REV_NUMBER" NUMBER(28,0),
"REF_DATE" DATE,
"FREE_TEXT" VARCHAR2(4000 BYTE),
"MLF_BS_TOTAL" NUMBER(28,10),
"DF_BS_TOTAL" NUMBER(28,10),
"MLF_SF_TOTAL" NUMBER(28,10),
"DF_SF_TOTAL" NUMBER(28,10)
);
/
--=============================================================================================================================
--Step 2: Configure file type for processing
--=============================================================================================================================
--Table LM_STANDING_FACILITIES
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'DistributeStandingFacilities'
,pSourceFileDesc => 'Table provides data on the credit facility available to counterparties at their own initiative. The Eurosystem offers two overnight standing facilities: the marginal lending facility and the deposit facility'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_STANDING_FACILITIES'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES'
,pContainerFileKey => NULL
);
/
--Table LM_STANDING_FACILITIES_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_SOURCE_FILE_CONFIG(
pSourceKey => 'LM'
,pSourceFileType => 'INPUT'
,pSourceFileId => 'DistributeStandingFacilities'
,pSourceFileDesc => 'Table provides data on the credit facility available to counterparties at their own initiative. The Eurosystem offers two overnight standing facilities: the marginal lending facility and the deposit facility'
,pSourceFileNamePattern => '.*.csv'
,pTableId => 'LM_STANDING_FACILITIES_HEADER'
,pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER'
,pContainerFileKey => NULL
);
/
--=============================================================================================================================
--Step 3: Date format adjustment
--=============================================================================================================================
--Table LM_STANDING_FACILITIES_HEADER
CALL CT_MRDS.FILE_MANAGER.ADD_COLUMN_DATE_FORMAT(
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
pColumnName => 'REF_DATE',
pDateFormat => 'YYYY-MM-DD'
);
/
--=============================================================================================================================
--Step 4: Create External Tables
--=============================================================================================================================
--Table LM_STANDING_FACILITIES
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'INBOX/LM/DistributeStandingFacilities/LM_STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'ODS/LM/LM_STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES',
pPrefix => 'ARCHIVE/LM/LM_STANDING_FACILITIES',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
---------------------------------------------------------------
--Table LM_STANDING_FACILITIES_HEADER
--INBOX -For incoming files awaiting processing
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_HEADER_INBOX',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
pPrefix => 'INBOX/LM/DistributeStandingFacilities/LM_STANDING_FACILITIES_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvInboxBucketUri
);
END;
/
--ODS - for processed files in operational data store
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_HEADER_ODS',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
pPrefix => 'ODS/LM/LM_STANDING_FACILITIES_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvDataBucketUri
);
END;
/
--ARCHIVE - For historical/archived files
BEGIN
ODS.FILE_MANAGER_ODS.CREATE_EXTERNAL_TABLE(
pTableName => 'LM_STANDING_FACILITIES_HEADER_ARCHIVE',
pTemplateTableName => 'CT_ET_TEMPLATES.LM_STANDING_FACILITIES_HEADER',
pPrefix => 'ARCHIVE/LM/LM_STANDING_FACILITIES_HEADER',
pBucketUri => CT_MRDS.ENV_MANAGER.gvArchiveBucketUri
);
END;
/
--=============================================================================================================================
--Step 5: Create Compatibility View
--=============================================================================================================================
--View STANDING_FACILITIES_MARS
CREATE OR REPLACE VIEW OU_LM.STANDING_FACILITIES_MARS AS
SELECT
A_KEY,
A_SFH_FK,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_FK,
COUNTRY,
MFI_ID,
MFI_NAME,
MARGINAL_LENDING_FACILITY,
DEPOSIT_FACILITY,
COMMENT_
FROM ODS.LM_STANDING_FACILITIES_ODS;
/
--View STANDING_FACILITIES_HEADER_MARS
CREATE OR REPLACE VIEW OU_LM.STANDING_FACILITIES_HEADER_MARS AS
SELECT
A_KEY,
A_WORKFLOW_HISTORY_KEY AS A_ETL_LOAD_SET_FK,
REV_NUMBER,
REF_DATE,
FREE_TEXT,
MLF_BS_TOTAL,
DF_BS_TOTAL,
MLF_SF_TOTAL,
DF_SF_TOTAL
FROM ODS.LM_STANDING_FACILITIES_HEADER_ODS;
/

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_MARS783 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY;
SET DEFINE ON

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_SOURCE_MARS783 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY_SOURCE;
SET DEFINE ON

View File

@@ -0,0 +1,7 @@
SET DEFINE OFF
CREATE TABLE BKP_MRDS.CT_MRDS_A_TASK_HISTORY_TARGET_MARS783 AS SELECT * FROM CT_MRDS.A_TASK_HISTORY_TARGET;
SET DEFINE ON

Some files were not shown because too many files have changed in this diff Show More