admin/mars
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

dokumentacja

Browse Source
This commit is contained in:
Grzegorz Michalski
2026-02-20 11:34:58 +01:00
parent b63be15f5d
commit 11723f6c88
2 changed files with 115 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
confluence/FILE_ARCHIVER_Guide.md
View File

@@ -11,25 +11,55 @@ The FILE_ARCHIVER package provides flexible archival strategies that accommodate
- **Three Archival Strategies**: THRESHOLD_BASED, MINIMUM_AGE_MONTHS (with 0=current month only), HYBRID
- **Flexible Configuration**: Per-table archival strategy configuration via A_SOURCE_FILE_CONFIG
- **Validation**: Automatic validation of strategy-specific configuration requirements
- **OCI Integration**: Works seamlessly with DBMS_CLOUD operations via cloud_wrapper
### Package Information
- **Schema**: CT_MRDS
- **Package**: FILE_ARCHIVER
- **Current Version**: 3.3.0
- **Dependencies**: ENV_MANAGER, FILE_MANAGER, cloud_wrapper, A_SOURCE_FILE_CONFIG, A_SOURCE_FILE_RECEIVED, A_WORKFLOW_HISTORY
- **Dependencies**: ENV_MANAGER, FILE_MANAGER, A_SOURCE_FILE_CONFIG, A_SOURCE_FILE_RECEIVED, A_WORKFLOW_HISTORY
### Critical Prerequisites
⚠️ **IMPORTANT**: FILE_ARCHIVER requires data to be registered in `CT_MRDS.A_SOURCE_FILE_RECEIVED` table. This table is automatically populated when files are processed through the modern Airflow + DBT workflow via `FILE_MANAGER.PROCESS_SOURCE_FILE`.
⚠️ **IMPORTANT**: FILE_ARCHIVER requires data to be registered in `CT_MRDS.A_SOURCE_FILE_RECEIVED` table.
**For new system data (Airflow + DBT):**
- `A_SOURCE_FILE_RECEIVED` records are automatically created by `FILE_MANAGER.PROCESS_SOURCE_FILE` during file validation
- No additional configuration needed - standard workflow handles registration
**For legacy data migrated from Informatica + WLA system:**
- Legacy data exported using `DATA_EXPORTER` does NOT automatically create `A_SOURCE_FILE_RECEIVED` records
- Without these records, FILE_ARCHIVER **CANNOT** archive the data
- See [System Migration Guide](System_Migration_Informatica_to_Airflow_DBT.md) for workaround strategies
- Use `DATA_EXPORTER` with **`pRegisterExport => TRUE`** parameter to automatically register exported files in `A_SOURCE_FILE_RECEIVED`
- This enables FILE_ARCHIVER to process legacy data exports without manual registration
- Available in both `EXPORT_TABLE_DATA` (single CSV) and `EXPORT_TABLE_DATA_TO_CSV_BY_DATE` (partitioned CSV exports)
**Recommendation for legacy data**: Export directly to ARCHIVE bucket using `DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE` with `pBucketArea => 'ARCHIVE'` to bypass this requirement
**Example - Legacy Data Export with Registration**:
```sql
-- Export legacy data to DATA bucket WITH automatic registration
BEGIN
CT_MRDS.DATA_EXPORTER.EXPORT_TABLE_DATA_TO_CSV_BY_DATE(
pSchemaName => 'OU_TOP',
pTableName => 'AGGREGATED_ALLOTMENT',
pKeyColumnName => 'A_ETL_LOAD_SET_KEY_FK',
pBucketArea => 'DATA',
pFolderName => 'legacy_export',
pMinDate => DATE '2024-01-01',
pMaxDate => DATE '2024-12-31',
pRegisterExport => TRUE, -- ✓ Registers files in A_SOURCE_FILE_RECEIVED
pProcessName => 'LEGACY_MIGRATION'
);
END;
/
-- Now FILE_ARCHIVER can process these files
BEGIN
CT_MRDS.FILE_ARCHIVER.ARCHIVE_TABLE_DATA(
pSourceFileConfigKey => vConfigKey
);
END;
/
```
**Alternative approach**: Export directly to ARCHIVE bucket using `DATA_EXPORTER.EXPORT_TABLE_DATA_BY_DATE` with `pBucketArea => 'ARCHIVE'` to bypass archival step entirely
## Archival Strategies