Refactor MARS-956 package for C2D MPEC data export
- Updated README.md to enhance clarity on the one-time export process, including detailed prerequisites, data structure, and validation steps. - Revised install_mars956.sql to streamline installation commands and improve logging mechanisms. - Modified rollback_mars956.sql to include user confirmation before proceeding with deletions and to clarify rollback steps. - Removed unnecessary tracking and validation scripts (track_package_versions.sql, validate_export.sql, verify_packages_version.sql) to simplify the package structure. - Enhanced error handling and validation checks throughout the installation and rollback processes.
This commit is contained in:
Grzegorz Michalski
@@ -1,128 +1,97 @@
|
||||
-- ===================================================================
|
||||
-- MARS-956 MASTER INSTALLATION SCRIPT
|
||||
-- ===================================================================
|
||||
-- Purpose: Export Historical C2D MPEC data from OU_C2D to DATA bucket
|
||||
-- Author: Grzegorz Michalski
|
||||
-- Date: 2026-02-11
|
||||
--
|
||||
-- Requirements:
|
||||
-- - ADMIN user access for MARS installation
|
||||
-- - OU_C2D schema access for source tables
|
||||
-- - DATA_EXPORTER package v2.7.4+ deployed
|
||||
-- - DEF_CRED_ARN credentials configured
|
||||
-- - DATA bucket accessible
|
||||
-- MARS-956 INSTALL SCRIPT: C2D MPEC Data Export to External Tables
|
||||
-- ===================================================================
|
||||
-- Purpose: One-time bulk export of 3 C2D MPEC tables from OU_C2D schema
|
||||
-- to OCI buckets (ODS bucket CSV format)
|
||||
-- Uses DATA_EXPORTER v2.7.5 with pRegisterExport for file registration
|
||||
-- Author: System Generated
|
||||
-- Date: 2026-02-12
|
||||
-- Version: 1.0.0
|
||||
|
||||
-- Dynamic spool file generation
|
||||
-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
|
||||
-- Log files are automatically created in log/ subdirectory
|
||||
-- IMPORTANT: Ensure log/ directory exists before SPOOL (use host mkdir)
|
||||
host mkdir log 2>nul
|
||||
define spoolfile = 'log\install_mars956_'
|
||||
define timestamp = ''
|
||||
|
||||
-- Get current timestamp for unique log filename
|
||||
column current_time new_value timestamp
|
||||
SELECT TO_CHAR(SYSDATE, 'YYYYMMDD_HH24MISS') AS current_time FROM dual;
|
||||
var filename VARCHAR2(100)
|
||||
BEGIN
|
||||
:filename := 'log/INSTALL_MARS_956_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
|
||||
END;
|
||||
/
|
||||
column filename new_value _filename
|
||||
select :filename filename from dual;
|
||||
spool &_filename
|
||||
|
||||
-- Start logging
|
||||
spool &spoolfile.×tamp..log
|
||||
SET ECHO OFF
|
||||
SET TIMING ON
|
||||
SET SERVEROUTPUT ON SIZE UNLIMITED
|
||||
SET PAUSE OFF
|
||||
|
||||
-- Display environment information
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-956 INSTALLATION - Export Historical C2D MPEC Data
|
||||
PROMPT =========================================================================
|
||||
PROMPT Installation Start:
|
||||
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS INSTALL_START FROM DUAL;
|
||||
|
||||
PROMPT Current User:
|
||||
SELECT USER AS CURRENT_USER FROM DUAL;
|
||||
|
||||
PROMPT Database Info:
|
||||
SELECT INSTANCE_NAME, VERSION, STATUS FROM V$INSTANCE;
|
||||
-- Set current schema context (optional - use when modifying packages in specific schema)
|
||||
-- ALTER SESSION SET CURRENT_SCHEMA = CT_MRDS;
|
||||
|
||||
PROMPT =========================================================================
|
||||
PROMPT Installation Details:
|
||||
PROMPT - Purpose: One-time export of historical C2D MPEC delta data
|
||||
PROMPT - Source: OU_C2D schema tables (operational database)
|
||||
PROMPT - Target: DATA bucket as CSV files
|
||||
PROMPT - Tables: MPEC_ADMIN, MPEC_CONTENT, MPEC_CONTENT_CRITERION
|
||||
PROMPT - Method: DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE
|
||||
PROMPT MARS-956: C2D MPEC Data Export to External Tables (One-Time Migration)
|
||||
PROMPT =========================================================================
|
||||
PROMPT
|
||||
PROMPT This script will export 3 C2D MPEC tables to OCI buckets:
|
||||
PROMPT
|
||||
PROMPT TARGET: ODS Bucket (CSV format):
|
||||
PROMPT - MPEC_ADMIN: 5 records with ETL keys 2001-2005
|
||||
PROMPT - MPEC_CONTENT: 5 records with ETL keys 2006-2008
|
||||
PROMPT - MPEC_CONTENT_CRITERION: 9 records with ETL keys 2009-2010
|
||||
PROMPT
|
||||
PROMPT Key Features:
|
||||
PROMPT - Files registered in A_SOURCE_FILE_RECEIVED for tracking
|
||||
PROMPT - Template table column order matching (CT_ET_TEMPLATES.C2D_MPEC_*)
|
||||
PROMPT - ODS/C2D bucket path structure
|
||||
PROMPT
|
||||
PROMPT Expected Duration: 2-5 minutes (small datasets)
|
||||
PROMPT =========================================================================
|
||||
|
||||
SET SERVEROUTPUT ON SIZE 1000000
|
||||
SET LINESIZE 200
|
||||
SET PAGESIZE 1000
|
||||
-- Confirm installation with user
|
||||
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
|
||||
WHENEVER SQLERROR EXIT SQL.SQLCODE
|
||||
BEGIN
|
||||
IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
|
||||
RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
|
||||
END IF;
|
||||
END;
|
||||
/
|
||||
WHENEVER SQLERROR CONTINUE
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 1: Verify Prerequisites
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Pre-Check: Verify prerequisites and table readiness
|
||||
PROMPT =========================================================================
|
||||
@@00_MARS_956_pre_check_prerequisites.sql
|
||||
|
||||
-- Verify DATA_EXPORTER package is available
|
||||
PROMPT Checking DATA_EXPORTER package availability...
|
||||
SELECT 'DATA_EXPORTER v' || CT_MRDS.DATA_EXPORTER.PACKAGE_VERSION ||
|
||||
' (Build: ' || CT_MRDS.DATA_EXPORTER.PACKAGE_BUILD_DATE || ')' AS PACKAGE_INFO
|
||||
FROM DUAL;
|
||||
|
||||
-- Verify source tables exist in OU_C2D
|
||||
PROMPT Checking source tables in OU_C2D schema...
|
||||
SELECT table_name, num_rows
|
||||
FROM all_tables
|
||||
WHERE owner = 'OU_C2D'
|
||||
AND table_name IN ('MPEC_ADMIN', 'MPEC_CONTENT', 'MPEC_CONTENT_CRITERION')
|
||||
ORDER BY table_name;
|
||||
|
||||
-- Verify template tables exist in CT_ET_TEMPLATES
|
||||
PROMPT Checking template tables in CT_ET_TEMPLATES schema...
|
||||
SELECT table_name
|
||||
FROM all_tables
|
||||
WHERE owner = 'CT_ET_TEMPLATES'
|
||||
AND table_name IN ('C2D_MPEC_ADMIN', 'C2D_MPEC_CONTENT', 'C2D_MPEC_CONTENT_CRITERION')
|
||||
ORDER BY table_name;
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 2: Execute Historical Data Export
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT Step 1: Export C2D MPEC Data to ODS Bucket
|
||||
PROMPT =========================================================================
|
||||
@@01_MARS_956_export_c2d_mpec_data.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 3: Track Package Versions
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
@@track_package_versions.sql
|
||||
PROMPT Step 2: Verify Exports (File Registration Check)
|
||||
PROMPT =========================================================================
|
||||
@@02_MARS_956_verify_exports.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT Step 4: Verify Package Versions
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
@@verify_packages_version.sql
|
||||
PROMPT Step 3: Verify Data Integrity (Source vs Exported)
|
||||
PROMPT =========================================================================
|
||||
@@03_MARS_956_verify_data_integrity.sql
|
||||
|
||||
PROMPT
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT MARS-956 INSTALLATION SUMMARY
|
||||
PROMPT MARS-956 Installation - COMPLETED
|
||||
PROMPT =========================================================================
|
||||
|
||||
-- Display final summary
|
||||
PROMPT Installation Completed:
|
||||
SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS') AS INSTALL_END FROM DUAL;
|
||||
|
||||
PROMPT Export Results Summary:
|
||||
SELECT COUNT(*) AS EXPORT_LOG_ENTRIES,
|
||||
MIN(EVENT_TIMESTAMP) AS FIRST_EXPORT,
|
||||
MAX(EVENT_TIMESTAMP) AS LAST_EXPORT
|
||||
FROM CT_MRDS.A_PROCESS_LOG
|
||||
WHERE PACKAGE_NAME = 'MARS-956'
|
||||
AND EVENT_TIMESTAMP >= SYSDATE - 1; -- Last 24 hours
|
||||
|
||||
PROMPT
|
||||
PROMPT =========================================================================
|
||||
PROMPT POST-INSTALLATION TASKS
|
||||
PROMPT =========================================================================
|
||||
PROMPT 1. Verify CSV files created in DATA bucket:
|
||||
PROMPT - mrds_data_dev/ODS/C2D/C2D_MPEC_ADMIN/*.csv
|
||||
PROMPT - mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT/*.csv
|
||||
PROMPT - mrds_data_dev/ODS/C2D/C2D_MPEC_CONTENT_CRITERION/*.csv
|
||||
PROMPT
|
||||
PROMPT 2. Check file structure matches template tables
|
||||
PROMPT 3. Validate row counts match source tables
|
||||
PROMPT 4. Confirm data available for delta queries
|
||||
PROMPT 5. Sync deployment timing with REL_02 deployment
|
||||
PROMPT Check the log file for complete installation details.
|
||||
PROMPT For rollback, use: rollback_mars956.sql
|
||||
PROMPT =========================================================================
|
||||
|
||||
spool off
|
||||
|
||||
quit;
|
||||
Reference in New Issue
Block a user