mars/confluence/Package_Deployment_Guide.md

# Package Deployment Guide - Developer Instructions

## Overview

This guide provides step-by-step instructions for developers on how to properly deploy new package versions using the integrated version tracking and hash-based change detection system.

**Target Audience:** Database developers working with CT_MRDS packages  
**System:** Oracle Database 23ai with ENV_MANAGER v3.1.0+  
**Last Updated:** 2025-10-22

---

## MANDATORY STANDARDS (NEVER SKIP!)

### 🔴 Critical: Author Field

**ALL scripts MUST have:**
```sql
-- Author: Grzegorz Michalski
```

**❌ WRONG:**
```sql
-- Author: System
-- Author: Developer
-- Author: [your name here]
```

**✅ CORRECT:**
```sql
-- Author: Grzegorz Michalski
```

### 📅 Date Format Standard

**ISO 8601 format only:**
```sql
-- Date: 2026-02-03  (YYYY-MM-DD)
```

### 📝 Script Header Template

**Copy-paste this template for ALL scripts:**
```sql
-- =====================================================================
-- Script: XX_MARS_XXX_description.sql
-- MARS Issue: MARS-XXX
-- Purpose: Brief description of what this script does
-- Author: Grzegorz Michalski
-- Date: YYYY-MM-DD
-- =====================================================================
```

### 🔐 Database User Requirements

**Execute installations as ADMIN:**
```powershell
# ✅ CORRECT
Get-Content "install_marsXXX.sql" | sql "ADMIN/password@service"

# ❌ WRONG
Get-Content "install_marsXXX.sql" | sql "CT_MRDS/password@service"
```

### 🔍 Data Dictionary Views

**Use ALL_* views when installing as ADMIN:**
```sql
-- ✅ CORRECT (ADMIN user)
SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PACKAGE_NAME';

-- ❌ WRONG (shows ADMIN schema, not CT_MRDS)
SELECT * FROM USER_ERRORS WHERE NAME = 'PACKAGE_NAME';
```

### 📂 Log Directory Management

**MANDATORY in all master scripts:**
```sql
-- Create log/ directory before SPOOL (prevents failures)
host mkdir log 2>nul

-- Then configure dynamic SPOOL filename
var filename VARCHAR2(100)
BEGIN
    :filename := 'log/INSTALL_MARS_XXX_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
```

### ✋ User Confirmation

**MANDATORY ACCEPT validation in master scripts:**
```sql
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
    IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
        RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
    END IF;
END;
/
WHENEVER SQLERROR CONTINUE
```

### 🚪 Clean Exit

**End all master scripts with:**
```sql
spool off
quit;  -- ← MANDATORY for clean SQLcl/SQL*Plus exit
```

### 📋 Quick Compliance Checklist

Before committing any script, verify:
- [ ] `Author: Grzegorz Michalski` in header
- [ ] Date in ISO 8601 format (YYYY-MM-DD)
- [ ] Master scripts use `host mkdir log 2>nul`
- [ ] Master scripts require ACCEPT validation
- [ ] Master scripts end with `quit;`
- [ ] Verification queries use ALL_* views with OWNER filter
- [ ] Installation instructions specify ADMIN user

**💡 Remember:** These standards prevent common mistakes and ensure consistency across all MARS packages!

---

## Table of Contents

1. [Before You Start](#before-you-start)
2. [Standard Deployment Workflow](#standard-deployment-workflow)
3. [Version Update Guidelines](#version-update-guidelines)
4. [Deployment Scenarios](#deployment-scenarios)
5. [Creating MARS Installation Packages](#creating-mars-installation-packages)
6. [Troubleshooting](#troubleshooting)
7. [Best Practices](#best-practices)
8. [Quick Reference](#quick-reference)

---

## Before You Start

### Step 0: Prepare Dedicated Working Directory and Git Branch

**CRITICAL:** Before starting work on any MARS package, create a dedicated working directory with its own feature branch:

```powershell
# Navigate to your Git repository parent directory
cd c:\_git\_local_rep

# Clone main repository to new working directory for MARS issue
# This creates complete isolation for the feature branch
git clone working_dir_02 working_dir_02_MARS-XXXX

# Navigate to new working directory
cd working_dir_02_MARS-XXXX

# Ensure you're on main branch and it's up to date
git checkout main
git pull origin main

# Create new feature branch for MARS issue
git checkout -b feature/MARS-XXXX

# Verify you're on the new branch
git branch
# You should see: * feature/MARS-XXXX (asterisk indicates current branch)
```

**Alternative: Git Worktree (Recommended for Advanced Users):**
```powershell
# Navigate to your main Git repository
cd c:\_git\_local_rep\working_dir_02

# Create new worktree with dedicated directory for feature branch
git worktree add ..\working_dir_02_MARS-XXXX feature/MARS-XXXX

# Navigate to new working directory
cd ..\working_dir_02_MARS-XXXX

# Verify you're on the new branch
git branch
```

**Branch and Directory Naming Convention:**
- **Branch Format**: `feature/MARS-XXXX` (e.g., `feature/MARS-1057`)
- **Directory Format**: `working_dir_02_MARS-XXXX` (e.g., `working_dir_02_MARS-1057`)
- Always branch from `main` to ensure clean starting point
- One working directory per MARS issue for complete isolation

**Working Directory Structure (isolated per MARS issue):**
```
c:\_git\_local_rep\
├── working_dir_02\              # Main working directory (main branch)
│   ├── MARS_Packages\
│   ├── database\
│   └── .git\
├── working_dir_02_MARS-1057\    # Dedicated directory for MARS-1057
│   ├── MARS_Packages\
│   │   └── REL01\
│   │       └── MARS-1057\       # New package folder
│   │           ├── rollback_version\
│   │           ├── new_version\
│   │           └── *.sql files
│   ├── database\
│   └── .git\                     # Tracks feature/MARS-1057 branch
└── working_dir_02_MARS-1058\    # Dedicated directory for MARS-1058 (parallel work)
    └── ...
```

**Git Status Check:**
```powershell
# Always verify which branch and directory you're working in
pwd
# Output should show: c:\_git\_local_rep\working_dir_02_MARS-XXXX

git status
# Output should show: On branch feature/MARS-XXXX

# View all changes in current branch
git status --short
```

**Why dedicated working directories are mandatory:**
- **Complete Isolation**: Each MARS issue has its own physical directory
- **Parallel Work**: Work on multiple MARS issues simultaneously without conflicts
- **Clean Separation**: No risk of mixing changes from different features
- **Testing Safety**: Test different features independently in separate environments
- **Rollback Simplicity**: Delete entire directory to abandon feature
- **Code Review**: Clear separation for pull request workflow
- **Deployment Control**: Deploy from specific working directory with confidence

**Git Worktree Benefits:**
- Shares `.git` repository (saves disk space)
- Faster than full clone
- Automatic branch tracking
- Recommended for experienced Git users

**Cleanup After Merge:**
```powershell
# After feature branch is merged to main, clean up working directory
cd c:\_git\_local_rep

# If using git worktree
git worktree remove working_dir_02_MARS-XXXX

# If using clone, simply delete directory
Remove-Item -Recurse -Force working_dir_02_MARS-XXXX
```

---

### Prerequisites

- **ADMIN user access** - MARS package installations require ADMIN privileges for schema operations
- Access to CT_MRDS schema (or appropriate schema)
- SQLcl or SQL*Plus installed
- Understanding of Semantic Versioning (MAJOR.MINOR.PATCH)
- Package source files available locally
- **Git installed and configured** - Required for version control workflow

**CRITICAL - Data Dictionary Views:**

When installing as ADMIN user, you MUST use `ALL_*` views instead of `USER_*` views for verification queries:

| ❌ WRONG (USER_*) | ✅ CORRECT (ALL_* with OWNER filter) |
|-------------------|--------------------------------------|
| `SELECT * FROM USER_ERRORS WHERE NAME = 'PKG'` | `SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PKG'` |
| `SELECT * FROM USER_OBJECTS WHERE OBJECT_NAME = 'X'` | `SELECT * FROM ALL_OBJECTS WHERE OWNER = 'CT_MRDS' AND OBJECT_NAME = 'X'` |
| `SELECT * FROM USER_TAB_COLUMNS WHERE TABLE_NAME = 'T'` | `SELECT * FROM ALL_TAB_COLUMNS WHERE OWNER = 'CT_MRDS' AND TABLE_NAME = 'T'` |
| `SELECT * FROM USER_TABLES WHERE TABLE_NAME = 'T'` | `SELECT * FROM ALL_TABLES WHERE OWNER = 'CT_MRDS' AND TABLE_NAME = 'T'` |

**Why this matters:** `USER_*` views show objects in the **current schema** (ADMIN). When creating objects in other schemas (ODS, CT_MRDS), you must query `ALL_*` views with explicit OWNER filter to verify the changes.

### System Components

The deployment system uses these key components:

- **ENV_MANAGER** - Centralized version management and hash tracking
- **A_PACKAGE_VERSION_TRACKING** - History table for all package versions
- **CALCULATE_PACKAGE_HASH** - SHA256 hash calculation for change detection
- **TRACK_PACKAGE_VERSION** - Records package versions with automatic change detection
- **CHECK_PACKAGE_CHANGES** - Validates if package was modified

---

## Universal Tracking and Verification Scripts

### MANDATORY: Use Standard Template Files

**CRITICAL:** Every MARS package MUST include two universal scripts for tracking and verification. These scripts are reusable across all MARS issues and provide consistent version management.

#### Required Files Structure

```
MARS_Packages/
└── REL0X/
    └── MARS-XXXX/
        ├── 01_MARS_XXXX_install_*.sql
        ├── 02_MARS_XXXX_install_*.sql
        ├── track_package_versions.sql      # ✅ MANDATORY - Universal tracking
        ├── verify_packages_version.sql     # ✅ MANDATORY - Universal verification
        ├── install_marsXXXX.sql            # Master script
        └── rollback_marsXXXX.sql
```

### track_package_versions.sql

**Purpose:** Universal script for tracking multiple package versions with editable package list.

**Key Features:**
- ✅ **Reusable** - Same file for all MARS issues
- ✅ **Configurable** - Edit package list array to specify which packages to track
- ✅ **Automatic** - Calls ENV_MANAGER.TRACK_PACKAGE_VERSION for each package
- ✅ **Summary Output** - Shows tracked packages count and versions

**Template:**
```sql
-- ===================================================================
-- PACKAGE LIST - Edit this array to specify packages to track
-- ===================================================================
vPackageList t_string_array := t_string_array(
   'CT_MRDS.DATA_EXPORTER',
   'CT_MRDS.FILE_MANAGER',
   'ODS.FILE_MANAGER_ODS'
);
-- ===================================================================
```

**Usage in Install Script:**
```sql
PROMPT Step 4: Track Package Versions
PROMPT =========================================================================
@@track_package_versions.sql
```

**Benefits over custom tracking scripts:**
- No hardcoded package names in multiple places
- Single point of configuration
- Consistent error handling
- Standardized output format
- Copy-paste ready for new MARS issues

### verify_packages_version.sql

**Purpose:** Universal script for verifying ALL tracked packages for code changes.

**Key Features:**
- ✅ **Automatic Discovery** - Queries A_PACKAGE_VERSION_TRACKING to find all tracked packages
- ✅ **Hash Validation** - Calls CHECK_PACKAGE_CHANGES for each package
- ✅ **Status Report** - Shows OK or WARNING for each package
- ✅ **Zero Configuration** - No editing needed, works automatically

**Usage in Install Script:**
```sql
PROMPT Final Step: Verify All Package Versions
PROMPT =========================================================================
@@verify_packages_version.sql
```

**Output Example:**
```
PACKAGE_OWNER   PACKAGE_NAME        VERSION    STATUS
--------------- ------------------- ---------- -------------------------------
CT_MRDS         DATA_EXPORTER       2.2.0      OK: Package has not changed
CT_MRDS         FILE_MANAGER        3.3.0      OK: Package has not changed
ODS             FILE_MANAGER_ODS    2.1.0      OK: Package has not changed
```

### Master Install Script Structure (MANDATORY)

Every `install_marsXXXX.sql` MUST follow this exact structure:

```sql
-- ===================================================================
-- MARS-XXXX INSTALL SCRIPT
-- ===================================================================

-- Dynamic spool file generation
host mkdir log 2>nul
-- ... spool setup ...

PROMPT Step 1: Deploy Package Specification
@@01_MARS_XXXX_install_PACKAGE_SPEC.sql

PROMPT Step 2: Deploy Package Body
@@02_MARS_XXXX_install_PACKAGE_BODY.sql

PROMPT Step 3: Track Package Versions
@@track_package_versions.sql              -- ✅ MANDATORY universal script

PROMPT Step 4: Verify Package Versions
@@verify_packages_version.sql             -- ✅ MANDATORY universal script

spool off
quit;
```

### DO NOT Create Custom Track/Verify Scripts

**❌ WRONG (Custom tracking per MARS issue):**
```sql
@@04_MARS_XXXX_track_version.sql          -- DON'T DO THIS
@@05_MARS_XXXX_verify_installation.sql    -- DON'T DO THIS
```

**✅ CORRECT (Universal scripts):**
```sql
@@track_package_versions.sql              -- Always use this
@@verify_packages_version.sql             -- Always use this
```

**Why universal scripts are mandatory:**
- **Consistency** - Same behavior across all MARS packages
- **Maintainability** - Bug fixes propagate to all packages
- **Simplicity** - No need to create custom scripts for each issue
- **Compliance** - Ensures all packages follow same standard
- **Code Review** - Easier to review when structure is identical

### Copying Template Files

**For each new MARS package, copy from reference implementation:**

```powershell
# Copy universal scripts from MARS-826-PREHOOK (reference implementation)
Copy-Item "MARS_Packages\REL01_POST_DEACTIVATION\MARS-826-PREHOOK\track_package_versions.sql" `
          "MARS_Packages\RELXX\MARS-YYYY\track_package_versions.sql"

Copy-Item "MARS_Packages\REL01_POST_DEACTIVATION\MARS-826-PREHOOK\verify_packages_version.sql" `
          "MARS_Packages\RELXX\MARS-YYYY\verify_packages_version.sql"

# Edit package list in track_package_versions.sql
# No changes needed to verify_packages_version.sql (works automatically)
```

---

## Standard Deployment Workflow

### Step 1: Check Current Package Status

Before making any changes, verify the current state:

```sql
-- Connect to database
sql CT_MRDS/password@service_name

-- Check current version
SELECT FILE_MANAGER.GET_VERSION() FROM DUAL;

-- Check for any untracked changes
SELECT ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;
```

**Expected Output:** `OK: Package CT_MRDS.FILE_MANAGER has not changed.`

If you see **WARNING**, it means there are untracked modifications. See [Handling Untracked Changes](#handling-untracked-changes).

---

### Step 2: Make Your Code Changes

Edit the package SPEC and/or BODY files in your local workspace:

```
database/CT_MRDS/packages/FILE_MANAGER.pkg
database/CT_MRDS/packages/FILE_MANAGER.pkb
```

**Important:** Do not modify version constants yet.

---

### Step 3: Update Version Information

Based on the type of changes, update version constants in the **SPEC** file:

#### A. Determine Version Increment

Follow Semantic Versioning rules:

- **MAJOR (X.0.0)** - Breaking changes, incompatible API modifications
- **MINOR (x.Y.0)** - New features, backward-compatible additions
- **PATCH (x.y.Z)** - Bug fixes, backward-compatible corrections

#### B. Update Package Constants

Edit the package SPEC file and update three constants:

```sql
-- Package Version Information (Semantic Versioning: MAJOR.MINOR.PATCH)
PACKAGE_VERSION     CONSTANT VARCHAR2(10) := '3.3.0';  -- INCREMENT THIS
PACKAGE_BUILD_DATE  CONSTANT VARCHAR2(20) := '2025-10-22 21:00:00';  -- UPDATE THIS
PACKAGE_AUTHOR      CONSTANT VARCHAR2(100) := 'Your Name';  -- YOUR NAME

-- Version History (Latest changes first)
VERSION_HISTORY     CONSTANT VARCHAR2(4000) := 
   '3.3.0 (2025-10-22): Added new export procedure for historical data' || CHR(13)||CHR(10) ||  -- ADD THIS LINE
   '3.2.0 (2025-10-15): Enhanced validation with column mismatch detection' || CHR(13)||CHR(10) ||
   '3.1.0 (2025-10-01): Initial release with file processing capabilities';
```

**Version History Format:**
```
'VERSION (DATE): Brief description of changes' || CHR(13)||CHR(10) ||
```

---

### Step 4: Deploy to Database

Deploy SPEC first, then BODY:

```powershell
# Using PowerShell and SQLcl
Get-Content "database\CT_MRDS\packages\FILE_MANAGER.pkg" | sql "CT_MRDS/password@service_name"
Get-Content "database\CT_MRDS\packages\FILE_MANAGER.pkb" | sql "CT_MRDS/password@service_name"
```

Or using SQL*Plus:

```sql
@database/CT_MRDS/packages/FILE_MANAGER.pkg
@database/CT_MRDS/packages/FILE_MANAGER.pkb
```

**Check for compilation errors:**

```sql
-- When installing as ADMIN, check specific schema
SELECT * FROM ALL_ERRORS 
WHERE OWNER = 'CT_MRDS'
  AND TYPE IN ('PACKAGE', 'PACKAGE BODY') 
  AND NAME = 'FILE_MANAGER'
ORDER BY SEQUENCE;
```

---

### Step 5: Track the New Version

After successful deployment, register the new version in the tracking system:

```sql
BEGIN
    ENV_MANAGER.TRACK_PACKAGE_VERSION(
        pPackageOwner => 'CT_MRDS',
        pPackageName => 'FILE_MANAGER',
        pPackageVersion => FILE_MANAGER.PACKAGE_VERSION,
        pPackageBuildDate => FILE_MANAGER.PACKAGE_BUILD_DATE,
        pPackageAuthor => FILE_MANAGER.PACKAGE_AUTHOR
    );
END;
/
```

**System Response:**
- If version updated correctly: `First tracking record` or `End TRACK_PACKAGE_VERSION - Record inserted`
- If version NOT updated: `WARNING: Source code changed without version update!`

---

### Step 6: Verify Deployment

Confirm the deployment was successful:

```sql
-- 1. Check deployed version
SELECT FILE_MANAGER.GET_BUILD_INFO() FROM DUAL;

-- 2. Verify no untracked changes
SELECT ENV_MANAGER.CHECK_PACKAGE_CHANGES('CT_MRDS', 'FILE_MANAGER') FROM DUAL;

-- 3. Review tracking history
SELECT 
    PACKAGE_VERSION,
    PACKAGE_BUILD_DATE,
    DETECTED_CHANGE_WITHOUT_VERSION,
    TO_CHAR(TRACKING_DATE, 'YYYY-MM-DD HH24:MI:SS') AS TRACKED_AT
FROM A_PACKAGE_VERSION_TRACKING
WHERE PACKAGE_OWNER = 'CT_MRDS' 
  AND PACKAGE_NAME = 'FILE_MANAGER'
ORDER BY TRACKING_DATE DESC
FETCH FIRST 5 ROWS ONLY;
```

---

### Step 7: Update Source Repository (CODE-FIRST Compliance)

**CRITICAL:** After successful database deployment, updated packages MUST be copied back to the main source repository to maintain version control integrity.

```powershell
# Copy updated packages to source repository
# Replace PACKAGE_NAME with actual package name (e.g., FILE_MANAGER)

Copy-Item "database\CT_MRDS\packages\PACKAGE_NAME.pkg" `
          "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\SCHEMA\packages\PACKAGE_NAME.pkg"

Copy-Item "database\CT_MRDS\packages\PACKAGE_NAME.pkb" `
          "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\SCHEMA\packages\PACKAGE_NAME.pkb"
```

**Why this is mandatory:**
- **CODE-FIRST PRINCIPLE**: Source files are the single source of truth
- **Version Control**: Git tracks changes in source repository
- **Team Collaboration**: Other developers need access to latest code
- **Deployment Reproducibility**: Enables clean deployments to other environments
- **Rollback Capability**: Source repository serves as backup

**Directory Structure:**
```
MARS_Packages/
└── mrds_elt-dev-database/
    └── mrds_elt-dev-database/
        └── database/
            ├── CT_MRDS/
            │   └── SCHEMA/
            │       ├── packages/
            │       │   ├── FILE_MANAGER.pkg       # ← Specification
            │       │   ├── FILE_MANAGER.pkb       # ← Body
            │       │   ├── DATA_EXPORTER.pkg
            │       │   ├── DATA_EXPORTER.pkb
            │       │   ├── ENV_MANAGER.pkg
            │       │   └── ENV_MANAGER.pkb
            │       ├── tables/
            │       ├── views/
            │       └── triggers/
            ├── ODS/
            │   └── SCHEMA/
            │       └── packages/
            │           ├── FILE_MANAGER_ODS.pkg   # ← Specification
            │           └── FILE_MANAGER_ODS.pkb   # ← Body
            └── CT_ET_TEMPLATES/
                └── SCHEMA/
                    └── tables/
```

**Verification:**
```powershell
# Verify files were updated
Get-ChildItem "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\SCHEMA\packages\*.pkg", `
              "MARS_Packages\mrds_elt-dev-database\mrds_elt-dev-database\database\CT_MRDS\SCHEMA\packages\*.pkb" | 
    Select-Object Name, LastWriteTime | 
    Sort-Object LastWriteTime -Descending
```

---

## Version Update Guidelines

### Semantic Versioning Rules

| Change Type | Version Impact | Example | Description |
|-------------|---------------|---------|-------------|
| **Breaking Change** | MAJOR (3.0.0 → 4.0.0) | Changed procedure signature, removed parameter | Requires code changes in calling applications |
| **New Feature** | MINOR (3.2.0 → 3.3.0) | Added new procedure, new optional parameter | Backward compatible, adds functionality |
| **Bug Fix** | PATCH (3.2.1 → 3.2.2) | Fixed calculation error, corrected validation | No API changes, fixes existing functionality |
| **Refactoring** | PATCH (3.2.1 → 3.2.2) | Code cleanup, performance optimization | Internal changes only, no behavior change |

### Build Date Format

Always use ISO 8601 format with time:

```sql
PACKAGE_BUILD_DATE CONSTANT VARCHAR2(20) := 'YYYY-MM-DD HH24:MI:SS';
```

**Example:** `'2025-10-22 21:00:00'`

---

## Deployment Scenarios

### Scenario 1: New Feature Addition (MINOR Version)

**Example:** Adding new export procedure to FILE_MANAGER

1. **Current Version:** 3.2.0
2. **New Version:** 3.3.0
3. **Changes:**
   - Add new procedure `EXPORT_HISTORICAL_DATA` to SPEC
   - Implement procedure in BODY
   - Update `PACKAGE_VERSION` from `'3.2.0'` to `'3.3.0'`
   - Update `PACKAGE_BUILD_DATE` to current timestamp
   - Add entry to `VERSION_HISTORY`

4. **Deploy & Track:**
```sql
-- Deploy files
@database/CT_MRDS/packages/FILE_MANAGER.pkg
@database/CT_MRDS/packages/FILE_MANAGER.pkb

-- Track new version
BEGIN
    ENV_MANAGER.TRACK_PACKAGE_VERSION(
        pPackageOwner => 'CT_MRDS',
        pPackageName => 'FILE_MANAGER',
        pPackageVersion => FILE_MANAGER.PACKAGE_VERSION,
        pPackageBuildDate => FILE_MANAGER.PACKAGE_BUILD_DATE,
        pPackageAuthor => FILE_MANAGER.PACKAGE_AUTHOR
    );
END;
/
```

---

### Scenario 2: Bug Fix (PATCH Version)

**Example:** Fixing date format validation in DATA_EXPORTER

1. **Current Version:** 2.1.0
2. **New Version:** 2.1.1
3. **Changes:**
   - Fix validation logic in BODY (no SPEC changes)
   - Update `PACKAGE_VERSION` from `'2.1.0'` to `'2.1.1'`
   - Update `PACKAGE_BUILD_DATE`
   - Add entry to `VERSION_HISTORY`

4. **Deploy & Track:** Same as Scenario 1

---

### Scenario 3: Breaking Change (MAJOR Version)

**Example:** Removing deprecated parameter from FILE_ARCHIVER

1. **Current Version:** 2.5.3
2. **New Version:** 3.0.0
3. **Changes:**
   - Remove old parameter from procedure signature in SPEC
   - Update BODY implementation
   - Update `PACKAGE_VERSION` from `'2.5.3'` to `'3.0.0'`
   - Update `PACKAGE_BUILD_DATE`
   - Add detailed migration notes to `VERSION_HISTORY`

4. **Additional Steps:**
   - Document breaking changes
   - Notify dependent application teams
   - Update API documentation

---

### Scenario 4: Emergency Hotfix

When urgent fix is needed in production:

1. **Identify Issue:** Document the problem clearly
2. **Create Fix:** Make minimal changes to address the issue
3. **Increment PATCH:** Update to next patch version (e.g., 3.2.5 → 3.2.6)
4. **Fast Track Deployment:**

```powershell
# Quick deployment script
Get-Content "database\CT_MRDS\packages\PACKAGE_NAME.pkg" | sql "CT_MRDS/password@service_name"
Get-Content "database\CT_MRDS\packages\PACKAGE_NAME.pkb" | sql "CT_MRDS/password@service_name"

# Track immediately
echo "BEGIN ENV_MANAGER.TRACK_PACKAGE_VERSION('CT_MRDS', 'PACKAGE_NAME', 'VERSION', 'BUILD_DATE', 'AUTHOR'); END;" | sql "CT_MRDS/password@service_name"
```

5. **Verify & Document:** Record hotfix details in change log

---

## Troubleshooting

### Issue 1: Warning - Untracked Changes Detected

**Symptom:**
```
WARNING: Source code changed without version update!
Last Version: 3.2.0
Current Version: 3.2.0
SPECIFICATION Changed: Hash mismatch detected
```

**Cause:** Code was modified but version constants were not updated.

**Solution:**
1. Update `PACKAGE_VERSION` to next appropriate version
2. Update `PACKAGE_BUILD_DATE` to current timestamp
3. Add entry to `VERSION_HISTORY`
4. Redeploy package
5. Call `TRACK_PACKAGE_VERSION` again

---

### Issue 2: Compilation Errors After Deployment

**Symptom:**
```
Package Body compiled with errors
LINE/COL ERROR
-------- -------------
42/10    PLS-00201: identifier 'PROCEDURE_NAME' must be declared
```

**Solution:**
1. Check `ALL_ERRORS` for detailed error information:
```sql
-- Check specific schema when installing as ADMIN
SELECT LINE, POSITION, TEXT 
FROM ALL_ERRORS 
WHERE OWNER = 'CT_MRDS'  -- Replace with target schema
  AND NAME = 'PACKAGE_NAME' 
  AND TYPE = 'PACKAGE BODY'
ORDER BY SEQUENCE;
```

2. Fix errors in source file
3. Redeploy package
4. Verify compilation: `SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PACKAGE_NAME';`

---

### Issue 3: Package State Discarded Error

**Symptom:**
```
ORA-04068: existing state of packages has been discarded
ORA-04061: existing state of package body has been invalidated
```

**Cause:** Package was recently recompiled, session state needs refresh.

**Solution:** Simply retry the command. This is a transient error that resolves after first execution.

---

### Issue 4: Cannot Track Package - Not Found

**Symptom:**
```
ERROR tracking PACKAGE_NAME: Package not found in database
```

**Solution:**
1. Verify package exists:
```sql
-- Check specific schema when installing as ADMIN
SELECT OBJECT_NAME, OBJECT_TYPE, STATUS 
FROM ALL_OBJECTS 
WHERE OWNER = 'CT_MRDS'  -- Replace with target schema
  AND OBJECT_NAME = 'PACKAGE_NAME';
```

2. If not found, deploy package first
3. If status = INVALID, recompile package
4. Retry tracking

---

## Creating MARS Installation Packages

### Overview

MARS packages (e.g., MARS-1049, MARS-1011) are deployment packages that bundle database changes for specific features or fixes. They follow a standardized structure with logging, version tracking, and rollback capabilities.

**CRITICAL:** All MARS package installations MUST be executed as **ADMIN user** to ensure proper schema creation, privilege management, and cross-schema operations.

**Installation User Requirements:**
- **User:** ADMIN (required for all MARS installations)
- **Privileges:** Full DBA privileges, CREATE ANY TABLE, ALTER ANY TABLE, EXECUTE on DBMS_CLOUD
- **Connection:** `sql ADMIN/password@service_name`

### Package Structure

**IMPORTANT:** Currently, all new MARS packages are being created in the **REL01** directory. This is the active release folder for ongoing development.

**CRITICAL - .gitignore Configuration:**

Before starting MARS package development, create `.gitignore` file in package root directory to exclude temporary folders from version control:

**File Location:** `MARS_Packages/REL01/MARS-XXXX/.gitignore`

**Required Content:**
```gitignore
# Exclude temporary folders from version control
confluence/
log/
test/
mock_data/
```

**Why exclude these folders:**
- **confluence/**: Working documentation files (may contain drafts, not final versions)
- **log/**: SPOOL log files (generated during installation, environment-specific)
- **test/**: Test artifacts, temporary test data, and test execution logs
- **mock_data/**: Test data files for development and validation (environment-specific)

**What SHOULD be committed:**
- Installation scripts (01_*, 02_*, 91_*, 92_*)
- Master scripts (install_*.sql, rollback_*.sql)
- track_package_versions.sql and verify_packages_version.sql
- README.md
- .gitignore (with standardized exclusions)
- rollback_version/ and new_version/ folders (if package modifications)
- Core deployment files only

Standard MARS package directory structure:

```
MARS_Packages/REL01/MARS-XXXX/
├── .gitignore                      # Git exclusions (REQUIRED)
├── install_marsXXXX.sql           # Master installation script (with SPOOL to log/)
├── rollback_marsXXXX.sql          # Master rollback script (with SPOOL to log/)
├── 01_MARS_XXXX_install_*.sql     # Individual installation scripts
├── 02_MARS_XXXX_install_*.sql
├── ...
├── 91_MARS_XXXX_rollback_*.sql    # Individual rollback scripts (91-99 range)
├── 92_MARS_XXXX_rollback_*.sql
├── track_package_versions.sql      # Version tracking script
├── verify_packages_version.sql     # Package verification script
├── ...
├── README.md                       # Package documentation
├── rollback_version/                # Backup of objects BEFORE changes (for rollback)
│   ├── PACKAGE_NAME.pkg           # Previous package specification
│   └── PACKAGE_NAME.pkb           # Previous package body
├── new_version/                    # Updated objects AFTER changes (for installation)
│   ├── PACKAGE_NAME.pkg           # New package specification
│   └── PACKAGE_NAME.pkb           # New package body
├── test/                           # Test files and verification scripts
│   ├── test_marsXXXX.sql          # Unit tests
│   ├── TEST_RESULTS.md            # Test documentation
│   └── test_data_*.csv            # Test data files
├── log/                            # SPOOL log files (REQUIRED - auto-created)
│   ├── INSTALL_MARS_XXXX_*.log    # Installation logs from SPOOL
│   └── ROLLBACK_MARS_XXXX_*.log   # Rollback logs from SPOOL
└── mock_data/                      # Mock data for testing (optional)
    ├── 00_LOAD_ALL_MOCK_DATA.sql  # Master data loader
    ├── 01_load_*.sql              # Individual table loaders
    └── README.md                  # Mock data documentation
```

**Naming Conventions:**
- **Installation scripts**: `01-89` range, executed in numerical order
- **Rollback scripts**: `91-99` range, executed in reverse order
- **Verification scripts**: Any number, typically after main scripts
- **Master scripts**: `install_marsXXXX.sql` and `rollback_marsXXXX.sql`

**Version Management Folders (for Database Object Modifications):**
- **`rollback_version/`** - Contains backup of database objects BEFORE changes (for rollback)
- **`new_version/`** - Contains updated database objects AFTER changes (for installation)

**When to use version folders:**
- Required when MARS package modifies ANY existing database objects (packages, tables, views, indexes, triggers, etc.)
- Copy original object definitions to `rollback_version/` before making changes
- Place modified object definitions in `new_version/` after changes
- Examples: FILE_MANAGER.pkg/pkb, A_SOURCE_FILE_CONFIG.sql (table), V_STATUS.sql (view), IDX_CONFIG.sql (index)

**Test Files Organization:**
- **`test/`** - Contains ALL test-related files (unit tests, test data, verification scripts)
- **`log/`** - Contains ALL SPOOL log files from installation/rollback executions (REQUIRED directory)
- **`mock_data/`** - Contains mock data files for development and testing (optional)
- Keep deployment scripts clean - move test artifacts to test/ subfolder after validation
- Include track_package_versions.sql and verify_packages_version.sql in test/ subfolder
- SPOOL log files automatically created in log/ directory by master scripts
- Benefits: Clean deployment structure, easy test replication, organized documentation

---

### Master Install Script Template

All MARS packages MUST include SPOOL logging for audit trail and debugging:

```sql
-- ===================================================================
-- MARS-XXXX INSTALL SCRIPT: Brief Description
-- ===================================================================
-- Purpose: Detailed description of what this package does
-- Author: Grzegorz Michalski
-- Date: YYYY-MM-DD
-- Version: X.Y.Z

-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
-- Log files are automatically created in log/ subdirectory
-- IMPORTANT: Ensure log/ directory exists before SPOOL (use host mkdir)
host mkdir log 2>nul

var filename VARCHAR2(100)
BEGIN
    :filename := 'log/INSTALL_MARS_XXXX_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename

SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF

-- Set current schema context (optional - use when modifying packages in specific schema)
-- ALTER SESSION SET CURRENT_SCHEMA = CT_MRDS;

PROMPT =========================================================================
PROMPT MARS-XXXX: Package Description
PROMPT =========================================================================
PROMPT 
PROMPT This script will:
PROMPT - Change 1
PROMPT - Change 2
PROMPT - Change 3
PROMPT 
PROMPT Expected Duration: X-Y minutes
PROMPT =========================================================================

-- Confirm installation with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with installation, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
    IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
        RAISE_APPLICATION_ERROR(-20001, 'Installation aborted by user');
    END IF;
END;
/
WHENEVER SQLERROR CONTINUE

PROMPT 
PROMPT =========================================================================
PROMPT Step 1: First Installation Step
PROMPT =========================================================================
@@01_MARS_XXXX_install_step1.sql

PROMPT 
PROMPT =========================================================================
PROMPT Step 2: Second Installation Step
PROMPT =========================================================================
@@02_MARS_XXXX_install_step2.sql

-- ... more deployment steps ...

PROMPT 
PROMPT =========================================================================
PROMPT Step N: Track Package Versions
PROMPT =========================================================================
@@track_package_versions.sql

PROMPT 
PROMPT =========================================================================
PROMPT Final Step: Verify Package Versions
PROMPT =========================================================================
@@verify_packages_version.sql

PROMPT 
PROMPT =========================================================================
PROMPT MARS-XXXX Installation - COMPLETED
PROMPT =========================================================================
PROMPT Check the log file for complete installation details.
PROMPT =========================================================================

spool off

quit;
```

**Key Elements:**
1. **Log Directory Creation**: `host mkdir log 2>nul` - Creates log/ directory before SPOOL (suppresses error if exists)
2. **Dynamic Log Name**: `log/INSTALL_MARS_XXXX_<PDB_NAME>_YYYYMMDD_HH24MISS.log` (stored in log/ subdirectory)
3. **SYS_CONTEXT Usage**: Uses `SYS_CONTEXT('USERENV', 'CON_NAME')` instead of `DBA_PDBS` - no DBA privileges required
4. **SPOOL START**: After filename generation, before first PROMPT
5. **SPOOL OFF**: At the very end of the script
6. **SET ECHO OFF**: Prevents double output of PROMPT commands (shows only results, not the SQL commands themselves)
7. **ACCEPT Validation**: User confirmation required before execution (safety feature)
8. **ALTER SESSION SET CURRENT_SCHEMA** (optional): Sets working schema for package operations
9. **@@ Includes**: All sub-scripts executed via `@@` command
10. **track_package_versions.sql**: MANDATORY universal tracking script (must be included before verify)
11. **verify_packages_version.sql**: MANDATORY universal verification script (must be last step before completion)
12. **quit;**: Exits SQLcl/SQL*Plus after completion (important for automated deployments)

**SET ECHO OFF Benefits:**
- **Clean Output**: PROMPT messages appear only once in console and log files
- **Readability**: Log files are easier to read without SQL command echo
- **Professional**: Produces cleaner, more professional-looking installation logs
- **Consistency**: Works the same in both SQLcl and SQL*Plus
- **Sub-Scripts**: Must NOT include `SET ECHO ON` - they inherit setting from master script

**ACCEPT Validation Benefits:**
- **Safety Check**: Prevents accidental execution - requires explicit "YES" confirmation
- **User Control**: User must type YES (case-insensitive) to proceed with changes
- **Abort Capability**: Any other input or Ctrl+C safely aborts before database modifications
- **SQLcl & SQL*Plus Compatible**: Works identically in both tools, unlike PAUSE
- **Best Practice**: Mandatory for both installation and rollback master scripts

**Log File Storage:**
- **Directory**: All SPOOL logs automatically created in `log/` subdirectory
- **Naming Convention**: `log/INSTALL_MARS_XXXX_<PDB>_<timestamp>.log` or `log/ROLLBACK_MARS_XXXX_<PDB>_<timestamp>.log`
- **Auto-Creation**: Directory created by `host mkdir log 2>nul` command before SPOOL (MANDATORY)
- **Error Suppression**: `2>nul` suppresses "directory already exists" errors in PowerShell/Windows
- **Version Control**: Excluded via .gitignore (environment-specific files)
- **Importance**: Without `host mkdir`, SPOOL may fail if log/ directory doesn't exist

**Why `host mkdir log 2>nul` is mandatory:**
- **Reliability**: Ensures log/ directory exists before SPOOL attempts to write
- **Cross-Environment**: Works on fresh Git clones where log/ doesn't exist yet
- **No Errors**: Silently succeeds whether directory exists or not
- **Best Practice**: Prevents installation failures due to missing log directory
- **PowerShell Compatible**: Works in both PowerShell (pwsh.exe) and Windows CMD

---
9. **quit;**: Exits SQLcl/SQL*Plus after completion (important for automated deployments)

**SET ECHO OFF Benefits:**
- **Clean Output**: PROMPT messages appear only once in console and log files
- **Readability**: Log files are easier to read without SQL command echo
- **Professional**: Produces cleaner, more professional-looking installation logs
- **Consistency**: Works the same in both SQLcl and SQL*Plus
- **Sub-Scripts**: Must NOT include `SET ECHO ON` - they inherit setting from master script

**ACCEPT Validation Benefits:**
- **Safety Check**: Prevents accidental execution - requires explicit "YES" confirmation
- **User Control**: User must type YES (case-insensitive) to proceed with changes
- **Abort Capability**: Any other input or Ctrl+C safely aborts before database modifications
- **SQLcl & SQL*Plus Compatible**: Works identically in both tools, unlike PAUSE
- **Best Practice**: Mandatory for both installation and rollback master scripts

---

### Complete MARS Package Example

**MARS-835-PREHOOK** - Real-world example following all standards:

**File Structure:**
```
MARS_Packages/REL01_POST_DEACTIVATION/MARS-835-PREHOOK/
├── .gitignore
├── install_mars835_prehook.sql          # Master install (uses universal scripts)
├── rollback_mars835_prehook.sql         # Master rollback
├── 01_MARS_835_install_DATA_EXPORTER_SPEC.sql
├── 02_MARS_835_install_DATA_EXPORTER_BODY.sql
├── 91_MARS_835_rollback_DATA_EXPORTER_BODY.sql
├── 92_MARS_835_rollback_DATA_EXPORTER_SPEC.sql
├── track_package_versions.sql           # ✅ Universal tracking
├── verify_packages_version.sql          # ✅ Universal verification
├── README.md
├── rollback_version/
│   ├── DATA_EXPORTER.pkg               # v2.1.1 (before changes)
│   └── DATA_EXPORTER.pkb
├── new_version/
│   ├── DATA_EXPORTER.pkg               # v2.2.0 (after changes)
│   └── DATA_EXPORTER.pkb
└── log/                                 # Auto-created by install script
```

**track_package_versions.sql** (edited for this MARS issue):
```sql
-- ===================================================================
-- PACKAGE LIST - Edit this array to specify packages to track
-- ===================================================================
vPackageList t_string_array := t_string_array(
   'CT_MRDS.DATA_EXPORTER'       -- Only package modified in MARS-835
);
-- ===================================================================
```

**install_mars835_prehook.sql** (master script):
```sql
-- ===================================================================
-- MARS-835-PREHOOK INSTALL SCRIPT
-- ===================================================================

host mkdir log 2>nul

var filename VARCHAR2(100)
BEGIN
    :filename := 'log/INSTALL_MARS_835_PREHOOK_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename

SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED

PROMPT =========================================================================
PROMPT MARS-835-PREHOOK: DRY Refactoring for DATA_EXPORTER
PROMPT =========================================================================

ACCEPT continue CHAR PROMPT 'Type YES to continue: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
    IF UPPER(TRIM('&continue')) != 'YES' THEN
        RAISE_APPLICATION_ERROR(-20001, 'Installation aborted');
    END IF;
END;
/
WHENEVER SQLERROR CONTINUE

PROMPT 
PROMPT Step 1: Deploy DATA_EXPORTER Package Specification (v2.2.0)
@@01_MARS_835_install_DATA_EXPORTER_SPEC.sql

PROMPT 
PROMPT Step 2: Deploy DATA_EXPORTER Package Body (v2.2.0)
@@02_MARS_835_install_DATA_EXPORTER_BODY.sql

PROMPT 
PROMPT Step 3: Track Package Versions
@@track_package_versions.sql

PROMPT 
PROMPT Step 4: Verify Package Versions
@@verify_packages_version.sql

PROMPT 
PROMPT =========================================================================
PROMPT MARS-835-PREHOOK Installation - COMPLETED
PROMPT =========================================================================

spool off
quit;
```

**Why this is the correct pattern:**
- ✅ Uses universal `track_package_versions.sql` (no custom 04_MARS_835_track_version.sql)
- ✅ Uses universal `verify_packages_version.sql` (no custom 03_MARS_835_verify_installation.sql)
- ✅ Tracking happens AFTER deployment (Step 3)
- ✅ Verification happens LAST (Step 4) to confirm everything is correct
- ✅ Single configuration point (edit package list in track_package_versions.sql)
- ✅ Consistent with ALL other MARS packages
- ✅ Easy to copy/paste for new MARS issues

---

### SPOOL Logging Benefits

**Why SPOOL is mandatory:**

1. **Audit Trail** - Complete record of all installation activities
2. **Debugging** - Capture errors, warnings, and execution details
3. **Compliance** - Required for production deployments
4. **Troubleshooting** - Review what actually happened during installation
5. **Documentation** - Proof of successful deployment

**Log File Contains:**
- All PROMPT messages
- SQL execution results
- Error messages (ORA-XXXXX)
- Timing information
- Package compilation status
- Version tracking confirmations

**Example Log File Name:**
```
log/INSTALL_MARS_1049_ggmichalski_20251126_143052.log
```

**Log Location:** `log/` subdirectory in MARS package root (auto-created by SPOOL)

---

### Creating Individual Installation Scripts

Each numbered script should be self-contained and focused:

**Example: `01_MARS_1049_install_CT_MRDS_ADD_ENCODING_COLUMN.sql`**

```sql
--=============================================================================================================================
-- MARS-1049: Add ENCODING Column to A_SOURCE_FILE_CONFIG Table
--=============================================================================================================================
-- Purpose: Add VARCHAR2(50) ENCODING column with UTF8 default
-- Author: Grzegorz Michalski
-- Date: 2025-11-24
-- Related: MARS-1049 CSV Encoding Support
--=============================================================================================================================

SET SERVEROUTPUT ON

PROMPT ========================================================================
PROMPT Adding ENCODING column to CT_MRDS.A_SOURCE_FILE_CONFIG
PROMPT ========================================================================

-- Add column
ALTER TABLE CT_MRDS.A_SOURCE_FILE_CONFIG 
ADD (ENCODING VARCHAR2(50) DEFAULT 'UTF8');

-- Add comment
COMMENT ON COLUMN CT_MRDS.A_SOURCE_FILE_CONFIG.ENCODING IS 
'Character encoding for CSV files (e.g., UTF8, WE8MSWIN1252)';

-- Verify change (check specific schema when installing as ADMIN)
SELECT COUNT(*) AS ENCODING_COLUMN_EXISTS
FROM ALL_TAB_COLUMNS 
WHERE OWNER = 'CT_MRDS'
  AND TABLE_NAME = 'A_SOURCE_FILE_CONFIG' 
  AND COLUMN_NAME = 'ENCODING';

PROMPT SUCCESS: ENCODING column added to A_SOURCE_FILE_CONFIG

--=============================================================================================================================
-- End of Script
--=============================================================================================================================
```

**Script Best Practices:**
- Clear header with purpose and metadata
- `SET SERVEROUTPUT ON` for debugging
- Descriptive PROMPT messages
- Verification queries after changes
- Success confirmation message

---

### Rollback Script Template

Always provide rollback capability:

```sql
-- ===================================================================
-- MARS-XXXX ROLLBACK SCRIPT: Brief Description
-- ===================================================================
-- Purpose: Rollback all changes from MARS-XXXX installation
-- Author: Grzegorz Michalski
-- Date: YYYY-MM-DD

-- Dynamic spool file generation (using SYS_CONTEXT - no DBA privileges required)
-- IMPORTANT: Ensure log/ directory exists before SPOOL (use host mkdir)
host mkdir log 2>nul

var filename VARCHAR2(100)
BEGIN
    :filename := 'log/ROLLBACK_MARS_XXXX_' || SYS_CONTEXT('USERENV', 'CON_NAME') || '_' || TO_CHAR(SYSDATE,'YYYYMMDD_HH24MISS') || '.log';
END;
/
column filename new_value _filename
select :filename filename from dual;
spool &_filename

SET ECHO OFF
SET TIMING ON
SET SERVEROUTPUT ON SIZE UNLIMITED
SET PAUSE OFF

PROMPT =========================================================================
PROMPT MARS-XXXX: Rollback Package
PROMPT =========================================================================
PROMPT This will reverse all changes from MARS-XXXX installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Restore previous package version
PROMPT 2. Remove added columns/tables
PROMPT 3. Revert configuration changes
PROMPT =========================================================================

-- Confirm rollback with user
ACCEPT continue CHAR PROMPT 'Type YES to continue with rollback, or Ctrl+C to abort: '
WHENEVER SQLERROR EXIT SQL.SQLCODE
BEGIN
    IF '&continue' IS NULL OR TRIM('&continue') IS NULL OR UPPER(TRIM('&continue')) != 'YES' THEN
        RAISE_APPLICATION_ERROR(-20001, 'Rollback aborted by user');
    END IF;
END;
/
WHENEVER SQLERROR CONTINUE

-- Execute rollback scripts in REVERSE order
@@92_MARS_XXXX_rollback_step2.sql
@@91_MARS_XXXX_rollback_step1.sql

PROMPT 
PROMPT =========================================================================
PROMPT MARS-XXXX Rollback - COMPLETED
PROMPT =========================================================================

spool off
```

**Rollback Principles:**
- Execute scripts in **REVERSE order** (92, 91, not 91, 92)
- Undo changes from newest to oldest
- Restore previous package versions
- Remove added columns/tables
- Revert configuration changes
- Include verification step after rollback (@@verify_packages_version.sql)

**Rollback Communication Style:**
- ✅ **DO**: Use calm, professional tone describing what will be done
- ✅ **DO**: List specific rollback steps clearly
- ❌ **DON'T**: Use dramatic words like "WARNING", "CRITICAL IMPACT", "DANGER"
- ❌ **DON'T**: Create panic - rollback is a normal part of deployment process
- **Rationale**: Rollbacks happen when installations fail or need to be reversed. This is a standard procedure, not an emergency.

**Example - Professional vs. Dramatic:**

```sql
-- ✅ CORRECT - Professional and Clear
PROMPT This will reverse all changes from MARS-828 installation
PROMPT
PROMPT Rollback steps:
PROMPT 1. Remove validation trigger
PROMPT 2. Drop ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns
PROMPT 3. Restore FILE_ARCHIVER package to v2.0.0

-- ❌ WRONG - Unnecessarily Dramatic
PROMPT WARNING: This will restore FILE_ARCHIVER to v2.0.0
PROMPT 
PROMPT CRITICAL IMPACT:
PROMPT 1. All archival strategies revert to THRESHOLD_BASED
PROMPT 2. ARCHIVAL_STRATEGY and MINIMUM_AGE_MONTHS columns will be dropped
```

---

### Version Tracking Integration

Every MARS package should track package versions using the universal tracking script:

**Recommended Approach: Universal Tracking Script**

```sql
-- ===================================================================
-- Simple Package Version Tracking Script
-- ===================================================================
-- Purpose: Track specified Oracle package versions
-- Author: Grzegorz Michalski  
-- Date: 2025-11-26
-- Version: 3.1.0 - List-Based Edition
--
-- USAGE:
-- 1. Edit package list below (add/remove packages as needed)
-- 2. Include in your install/rollback script: @@track_package_versions.sql
-- ===================================================================

SET SERVEROUTPUT ON;

DECLARE
   TYPE t_package_rec IS RECORD (
      owner VARCHAR2(50),
      package_name VARCHAR2(50),
      version VARCHAR2(50)
   );
   TYPE t_packages IS TABLE OF t_package_rec;
   TYPE t_string_array IS TABLE OF VARCHAR2(100);
   
   -- ===================================================================
   -- PACKAGE LIST - Edit this array to specify packages to track
   -- ===================================================================
   -- Add or remove entries as needed for your MARS issue
   -- Format: 'SCHEMA.PACKAGE_NAME'
   -- ===================================================================
   vPackageList t_string_array := t_string_array(
      'CT_MRDS.FILE_MANAGER',
      'ODS.FILE_MANAGER_ODS'
   );
   -- ===================================================================
   
   vPackages t_packages := t_packages();
   vVersion VARCHAR2(50);
   vCount NUMBER := 0;
   vOwner VARCHAR2(50);
   vPackageName VARCHAR2(50);
   vDotPos NUMBER;
BEGIN
   DBMS_OUTPUT.PUT_LINE('========================================');
   DBMS_OUTPUT.PUT_LINE('Package Version Tracking');
   DBMS_OUTPUT.PUT_LINE('========================================');
   
   -- Process each package in the list
   FOR i IN 1..vPackageList.COUNT LOOP
      vDotPos := INSTR(vPackageList(i), '.');
      IF vDotPos > 0 THEN
         vOwner := SUBSTR(vPackageList(i), 1, vDotPos - 1);
         vPackageName := SUBSTR(vPackageList(i), vDotPos + 1);
         
         -- Get package version
         BEGIN
            EXECUTE IMMEDIATE 'SELECT ' || vOwner || '.' || vPackageName || '.GET_VERSION() FROM DUAL' INTO vVersion;
            vPackages.EXTEND;
            vPackages(vPackages.COUNT).owner := vOwner;
            vPackages(vPackages.COUNT).package_name := vPackageName;
            vPackages(vPackages.COUNT).version := vVersion;
            
            -- Track in ENV_MANAGER
            BEGIN
               CT_MRDS.ENV_MANAGER.TRACK_PACKAGE_VERSION(
                  pPackageOwner => vOwner,
                  pPackageName => vPackageName,
                  pPackageVersion => vVersion,
                  pPackageBuildDate => TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS'),
                  pPackageAuthor => 'Grzegorz Michalski'
               );
               vCount := vCount + 1;
            EXCEPTION
               WHEN OTHERS THEN NULL; -- Continue even if tracking fails
            END;
         EXCEPTION
            WHEN OTHERS THEN NULL; -- Skip packages that fail
         END;
      END IF;
   END LOOP;
   
   -- Display results
   IF vPackages.COUNT > 0 THEN
      DBMS_OUTPUT.PUT_LINE('Packages tracked: ' || vCount || ' of ' || vPackages.COUNT);
      DBMS_OUTPUT.PUT_LINE('');
      
      FOR i IN 1..vPackages.COUNT LOOP
         DBMS_OUTPUT.PUT_LINE(vPackages(i).owner || '.' || vPackages(i).package_name || ' = ' || vPackages(i).version);
      END LOOP;
   ELSE
      DBMS_OUTPUT.PUT_LINE('No packages found in list');
   END IF;
   
   DBMS_OUTPUT.PUT_LINE('========================================');
END;
/
```

**Key Benefits:**
- **List-Based**: Simply edit the `vPackageList` array to add/remove packages
- **Automatic Version Detection**: Calls `GET_VERSION()` for each package
- **Error Tolerant**: Continues even if individual packages fail
- **Summary Output**: Shows which packages were successfully tracked
- **Reusable**: Same script can be used across all MARS packages

---

### Package Verification Script

Every MARS package should include a universal verification script to check all tracked packages for untracked changes:

**Recommended: `verify_packages_version.sql`**

```sql
-- ===================================================================
-- Universal Package Version Verification Script
-- ===================================================================
-- Purpose: Verify all tracked Oracle packages for code changes
-- Author: Grzegorz Michalski  
-- Date: 2025-11-25
-- Version: 1.0.0
--
-- USAGE:
-- Include at the end of install/rollback scripts: @@verify_packages_version.sql
--
-- OUTPUT:
-- - List of all tracked packages with their current status
-- - OK: Package has not changed since last tracking
-- - WARNING: Package code changed without version update
-- ===================================================================

SET LINESIZE 200
SET PAGESIZE 1000
SET FEEDBACK OFF

PROMPT
PROMPT ========================================
PROMPT Package Version Verification
PROMPT ========================================
PROMPT

COLUMN PACKAGE_OWNER FORMAT A15
COLUMN PACKAGE_NAME FORMAT A20
COLUMN VERSION FORMAT A10
COLUMN STATUS FORMAT A80

SELECT 
   PACKAGE_OWNER,
   PACKAGE_NAME,
   PACKAGE_VERSION AS VERSION,
   CT_MRDS.ENV_MANAGER.CHECK_PACKAGE_CHANGES(PACKAGE_OWNER, PACKAGE_NAME) AS STATUS
FROM (
   SELECT 
      PACKAGE_OWNER, 
      PACKAGE_NAME,
      PACKAGE_VERSION,
      ROW_NUMBER() OVER (PARTITION BY PACKAGE_OWNER, PACKAGE_NAME ORDER BY TRACKING_DATE DESC) AS RN
   FROM CT_MRDS.A_PACKAGE_VERSION_TRACKING
)
WHERE RN = 1
ORDER BY PACKAGE_OWNER, PACKAGE_NAME;

PROMPT
PROMPT ========================================
PROMPT Verification Complete
PROMPT ========================================
PROMPT
PROMPT Legend:
PROMPT   OK - Package has not changed since last tracking
PROMPT   WARNING - Package code changed without version update
PROMPT
PROMPT For detailed hash information, use:
PROMPT   SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;
PROMPT ========================================

SET FEEDBACK ON
```

**Key Features:**
- **Universal**: Checks ALL packages in A_PACKAGE_VERSION_TRACKING table
- **Latest Version**: Uses ROW_NUMBER() to get most recent tracking record for each package
- **Status Display**: Shows OK or WARNING for each package
- **Formatted Output**: Clean, readable table format with column alignment
- **Reusable**: Same script works for all MARS packages without modification
- **Legend**: Includes helpful legend explaining OK vs WARNING status

**Usage in Master Scripts:**
```sql
PROMPT
PROMPT === Step 3: Verify all tracked packages ===
@@verify_packages_version.sql
```

**Benefits over Package-Specific Verification:**
- No need to edit script for each MARS package
- Automatically detects all tracked packages
- Consistent verification across all installations
- Easier to maintain (single script for all packages)

---

### Package README Template

Every MARS package MUST include README.md:

```markdown
# MARS-XXXX: Brief Description

## Overview
Detailed description of what this package does and why it's needed.

## Contents
- `install_marsXXXX.sql` - Master installation script with SPOOL logging
- `rollback_marsXXXX.sql` - Master rollback script
- `01_MARS_XXXX_*.sql` - Individual installation scripts
- `91_MARS_XXXX_*.sql` - Individual rollback scripts

## Prerequisites
- Oracle Database 23ai
- Required package versions (e.g., ENV_MANAGER v3.1.0+)
- Schema privileges needed

## Installation

### Option 1: Master Script (Recommended)
```powershell
# IMPORTANT: Execute as ADMIN user for proper privilege management
Get-Content "MARS_Packages/REL01/MARS-XXXX/install_marsXXXX.sql" | sql "ADMIN/password@service"

# Log file created: INSTALL_MARS_XXXX_<PDB>_<timestamp>.log
```

### Option 2: Individual Scripts
```powershell
# IMPORTANT: Execute as ADMIN user
Get-Content "01_MARS_XXXX_*.sql" | sql "ADMIN/password@service"
Get-Content "02_MARS_XXXX_*.sql" | sql "ADMIN/password@service"
# ... etc
```

## Verification
```sql
-- Verify package versions
SELECT PACKAGE_NAME.GET_VERSION() FROM DUAL;

-- Check for errors (ADMIN user checks specific schema)
SELECT * FROM ALL_ERRORS 
WHERE OWNER = 'CT_MRDS'  -- Replace with target schema
  AND NAME = 'PACKAGE_NAME';
```

## Rollback
```powershell
# IMPORTANT: Execute as ADMIN user
Get-Content "MARS_Packages/REL01/MARS-XXXX/rollback_marsXXXX.sql" | sql "ADMIN/password@service"
```

## Expected Changes
- Package X: v1.0.0 → v1.1.0
- Table Y: Added COLUMN_Z
- Configuration: Updated DEFAULT_VALUE

## Testing
Describe how to test the changes after installation.

## Known Issues
List any known limitations or issues.

## Related
- MARS-YYYY: Related package
- Confluence: Link to detailed documentation
```

---

### MARS Package Checklist

Before creating a MARS package, ensure:

- [ ] **Directory Structure**: Created `log/` and `mock_data/` subdirectories (if needed)
- [ ] **.gitignore file**: Created with standardized exclusions (confluence/, log/, test/, mock_data/)
- [ ] **Author field**: All scripts have `Author: Grzegorz Michalski`
- [ ] **ADMIN user**: All installation instructions specify ADMIN user connection
- [ ] All individual scripts numbered correctly (01-89, 91-99)
- [ ] Master install script includes SPOOL logging to `log/` directory
- [ ] Master rollback script includes SPOOL logging to `log/` directory
- [ ] SPOOL filenames use dynamic PDB name and timestamp with `log/` prefix
- [ ] **ACCEPT validation**: Master scripts require explicit "YES" confirmation
- [ ] **quit; command**: Master scripts end with quit; for clean exit
- [ ] All scripts have proper headers with metadata
- [ ] Version tracking scripts included (in test/ subfolder)
- [ ] Verification scripts confirm changes (in test/ subfolder)
- [ ] README.md documents installation process
- [ ] Rollback scripts reverse all changes
- [ ] Rollback includes verification step (@@test/verify_packages_version.sql)
- [ ] **rollback_version/ folder**: Contains backup of objects before changes (if modifying packages)
- [ ] **new_version/ folder**: Contains updated objects after changes (if modifying packages)
- [ ] **test/ folder**: All test files and verification scripts organized in subfolder
- [ ] **log/ folder**: SPOOL log files automatically created by master scripts
- [ ] **mock_data/ folder**: Mock data files organized (if applicable)
- [ ] Tested in DEV environment
- [ ] Log files reviewed for errors (in log/ subdirectory)
- [ ] **Source repository updated**: Changed packages copied to `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/`
- [ ] Git commit with descriptive message (log/, mock_data/ excluded via .gitignore)

---

### Example 1: Complete MARS-1046 Package (Package Modification)

Real-world example from MARS-1046 (ISO 8601 Date Format Fix):

**Structure:**
```
MARS_Packages/REL01/MARS-1046/
├── install_mars1046.sql                          # Master with SPOOL, ACCEPT, quit;
├── rollback_mars1046.sql                         # Master rollback with ACCEPT, quit;
├── 01_MARS_1046_install_CT_MRDS_FILE_MANAGER_SPEC.sql
├── 02_MARS_1046_install_CT_MRDS_FILE_MANAGER_BODY.sql
├── 91_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_BODY.sql
├── 92_MARS_1046_rollback_CT_MRDS_FILE_MANAGER_SPEC.sql
├── track_package_versions.sql               # Version tracking
├── verify_packages_version.sql              # Package verification
├── README.md                                     # Comprehensive documentation
├── rollback_version/                              # v3.3.0 backup for rollback
│   ├── FILE_MANAGER.pkg
│   └── FILE_MANAGER.pkb
├── new_version/                                  # v3.3.1 updated packages
│   ├── FILE_MANAGER.pkg
│   └── FILE_MANAGER.pkb
├── test/                                         # All test artifacts
│   ├── test_mars1046.sql                        # Unit tests (7/7 passed)
│   ├── TEST_RESULTS.md                          # Test documentation
│   ├── iso8601_test_data.csv                    # Test data (full format)
│   ├── iso8601_test_data_simple.csv             # Test data (simple format)
│   ├── create_iso8601_test_template.sql         # Template table creation
│   ├── configure_iso8601_test.sql               # Date format configuration
│   ├── process_iso8601_test.sql                 # E2E test
│   └── INSTALL_MARS_1046_*.log                  # Log files (3 iterations)
└── log/                                          # Optional log storage
```

**Key Features:**
- SPOOL logging: `INSTALL_MARS_1046_<PDB>_<timestamp>.log`
- ACCEPT validation: Requires explicit "YES" to proceed (both install & rollback)
- quit; command: Clean SQLcl exit after completion
- Version folders: rollback_version/ (v3.3.0) and new_version/ (v3.3.1)
- Test subfolder: All test files organized separately
- 4-step workflow: Install packages → Track version → Verify packages → quit
- Full rollback: Restore to previous version with verification
- Comprehensive README: Problem, solution, testing, verification
- Track & verify scripts: Universal scripts work for any package

---

### Example 2: Complete MARS-1049-ADHOC Package (Configuration Change)

Real-world example from MARS-1049-ADHOC (CSDB External Tables):

**Structure:**
```
MARS_Packages/REL01/MARS-1049-ADHOC/
├── install_mars1049_adhoc.sql                    # Master with SPOOL
├── 00_MARS_1049_ADHOC_update_encoding.sql        # Update config
├── 01_MARS_1049_ADHOC_drop_external_tables.sql   # Drop tables
├── 02_MARS_1049_ADHOC_recreate_external_tables.sql  # Recreate with encoding
├── 03_MARS_1049_ADHOC_verify_external_tables.sql # Verify creation
└── README.md                                     # Documentation
```

**Key Features:**
- SPOOL logging: `INSTALL_MARS_1049_ADHOC_<PDB>_<timestamp>.log`
- Numbered sequence: 00 → 01 → 02 → 03
- No rollback (external tables are metadata-only)
- No version folders (no package modifications)
- Verification included
- Clear README with installation options

---

## Post-Installation File Organization

### Organizing Test Files and Logs

After successful installation and verification, organize files for clean Git commit:

**Step 1: Create test/ subfolder**
```powershell
# Create test directory if it doesn't exist
New-Item -ItemType Directory -Force -Path "test" | Out-Null
```

**Step 2: Move test-related files**
```powershell
# Move test scripts and data
Move-Item -Path "test_*.sql" -Destination "test" -Force
Move-Item -Path "*_test_*.csv" -Destination "test" -Force
Move-Item -Path "TEST_RESULTS.md" -Destination "test" -Force
```

**Step 3: Move log files**
```powershell
# Option 1: Move to test/ subfolder
Move-Item -Path "*.log" -Destination "test" -Force

# Option 2: Create dedicated log/ subfolder
New-Item -ItemType Directory -Force -Path "log" | Out-Null
Move-Item -Path "*.log" -Destination "log" -Force
```

**Step 4: Verify clean structure**
```powershell
# Check root directory - should only have deployment scripts and README
Get-ChildItem | Where-Object { $_.Name -notmatch "^(01|02|91|92|install|rollback|README|rollback_version|new_version|test|log)" }
# Expected: No output (all non-deployment files moved)
```

**Final MARS Package Structure:**
```
MARS_Packages/REL01/MARS-XXXX/
├── install_marsXXXX.sql           # Master installation
├── rollback_marsXXXX.sql          # Master rollback
├── 01_MARS_XXXX_*.sql             # Installation scripts
├── 02_MARS_XXXX_*.sql
├── 91_MARS_XXXX_*.sql             # Rollback scripts
├── 92_MARS_XXXX_*.sql
├── track_package_versions.sql
├── verify_packages_version.sql
├── README.md                       # Documentation
├── rollback_version/                # Backup packages
├── new_version/                    # Updated packages
└── test/                           # ALL test artifacts
    ├── test_marsXXXX.sql
    ├── TEST_RESULTS.md
    ├── *.csv (test data)
```

**Benefits:**
- **Clean Git commits**: Only deployment-relevant files in root
- **Easy testing**: All test materials in one place
- **Reproducibility**: Test setup can be recreated from test/ folder
- **Documentation**: Log files preserved for troubleshooting
- **Professional**: Organized structure for code review

---

## Best Practices

### 1. Always Update Version Before Deployment

**DO:**
```sql
-- Update version first, then deploy
PACKAGE_VERSION := '3.3.0';  -- Updated
PACKAGE_BUILD_DATE := '2025-10-22 21:00:00';  -- Updated
VERSION_HISTORY := '3.3.0 (2025-10-22): New feature...' || CHR(13)||CHR(10) || ...
```

**DON'T:**
```sql
-- Deploy changes without updating version
-- This triggers "untracked changes" warning
```

---

### 2. Use Descriptive Version History

**DO:**
```sql
'3.3.0 (2025-10-22): Added EXPORT_HISTORICAL_DATA procedure for Parquet exports with Hive partitioning'
```

**DON'T:**
```sql
'3.3.0 (2025-10-22): Updates'  -- Too vague
```

---

### 3. Deploy SPEC Before BODY

**Correct Order:**
1. Deploy PACKAGE.pkg (specification)
2. Deploy PACKAGE.pkb (body)
3. Track version

**Rationale:** BODY depends on SPEC, deploying SPEC first ensures clean compilation.

---

### 4. Test Before Production Deployment

1. Deploy to DEV environment first
2. Run unit tests
3. Verify functionality
4. Check for performance issues
5. Deploy to PROD only after validation

---

### 5. Document Breaking Changes

For MAJOR version increments, always document:
- What changed
- Why it changed
- Migration path for existing code
- Example of old vs new usage

---

### 6. Use Batch Deployment Scripts

For multiple packages, create deployment script:

```powershell
# deploy_all_packages.ps1
$packages = @('ENV_MANAGER', 'FILE_MANAGER', 'DATA_EXPORTER', 'FILE_ARCHIVER')

foreach ($pkg in $packages) {
    Write-Host "Deploying $pkg..."
    Get-Content "database\CT_MRDS\packages\${pkg}.pkg" | sql "CT_MRDS/password@service"
    Get-Content "database\CT_MRDS\packages\${pkg}.pkb" | sql "CT_MRDS/password@service"
}

# Track all packages
Get-Content "database\scripts\track_all_packages.sql" | sql "CT_MRDS/password@service"
```

---

### 7. Regular Change Detection Audits

Run periodic checks to catch untracked modifications:

```sql
-- Check all packages for changes
@database/scripts/check_package_changes_all.sql
```

Schedule this as part of weekly code review process.

---

## Quick Reference

### Essential Commands

| Task | Command |
|------|---------|
| **Check current version** | `SELECT PACKAGE_NAME.GET_VERSION() FROM DUAL;` |
| **Check build info** | `SELECT PACKAGE_NAME.GET_BUILD_INFO() FROM DUAL;` |
| **Check for changes** | `SELECT ENV_MANAGER.CHECK_PACKAGE_CHANGES('OWNER', 'PACKAGE') FROM DUAL;` |
| **Get hash info** | `SELECT ENV_MANAGER.GET_PACKAGE_HASH_INFO('OWNER', 'PACKAGE') FROM DUAL;` |
| **Track version** | `CALL ENV_MANAGER.TRACK_PACKAGE_VERSION('OWNER', 'PKG', 'VER', 'DATE', 'AUTHOR');` |
| **Check errors** | `SELECT * FROM ALL_ERRORS WHERE OWNER = 'CT_MRDS' AND NAME = 'PACKAGE_NAME';` |
| **View tracking history** | `SELECT * FROM A_PACKAGE_VERSION_TRACKING WHERE PACKAGE_NAME = 'X' ORDER BY TRACKING_DATE DESC;` |

---

### Version Number Examples

| Current | Bug Fix | Feature | Breaking |
|---------|---------|---------|----------|
| 1.0.0 | 1.0.1 | 1.1.0 | 2.0.0 |
| 2.5.3 | 2.5.4 | 2.6.0 | 3.0.0 |
| 3.2.0 | 3.2.1 | 3.3.0 | 4.0.0 |

---

### Deployment Checklist

- [ ] Check current package status (`CHECK_PACKAGE_CHANGES`)
- [ ] Make code changes in local files
- [ ] Update `PACKAGE_VERSION` constant (increment appropriately)
- [ ] Update `PACKAGE_BUILD_DATE` constant (current timestamp)
- [ ] Add entry to `VERSION_HISTORY` constant
- [ ] Deploy SPEC file
- [ ] Deploy BODY file
- [ ] Check for compilation errors (`ALL_ERRORS` with OWNER filter when using ADMIN)
- [ ] Track new version (`TRACK_PACKAGE_VERSION`)
- [ ] Verify deployment (`GET_BUILD_INFO`, `CHECK_PACKAGE_CHANGES`)
- [ ] **Update source repository** - Copy updated packages to `MARS_Packages/mrds_elt-dev-database/mrds_elt-dev-database/database/`
- [ ] Update documentation (if API changed)
- [ ] Commit changes to Git
- [ ] Notify team (if breaking changes)

---

## Related Documentation

- [Package Versioning Guide](PACKAGE_VERSIONING_README.md) - Overview of versioning system
- [Package Hash Tracking Guide](PACKAGE_HASH_TRACKING_GUIDE.md) - Technical details of hash system
- [Package Hash Quick Guide](PACKAGE_HASH_QUICK_GUIDE.md) - Quick reference for hash commands
- [Developer Quick Guide](../database/DEVELOPER_QUICK_GUIDE.md) - General development guidelines

---

## Support

For questions or issues with the deployment process:

1. Check [Troubleshooting](#troubleshooting) section above
2. Review package documentation: `SELECT GET_PACKAGE_DOCUMENTATION('PACKAGE_NAME', 'SCHEMA') FROM DUAL;`
3. Check process logs: `SELECT * FROM A_PROCESS_LOG WHERE LOG_TIMESTAMP > SYSDATE - 1 ORDER BY LOG_TIMESTAMP DESC;`
4. Contact database team lead

---

**Last Updated:** 2025-10-22  
**Document Version:** 1.0.0  
**System Version:** ENV_MANAGER 3.1.0+