mb-backup-manager/changelogs.md

# Changelog

## Version 1.8

### Added
- **`view_snapshots.sh` Enhancements**:
  - Introduced support for a new `all` tag to display all snapshots regardless of specific tags.
  - Added functionality to dynamically fetch and display all available tags in the repository, improving usability and flexibility.
  - Enhanced error handling for empty repositories or inaccessible snapshots with detailed log messages.

- **Improved Snapshot Display**:
  - For the `all` tag, all snapshots are displayed in a formatted output including `short_id`, `time`, and associated tags.
  - Enhanced the script to log raw snapshot data during errors for easier debugging.

### Fixed
- Resolved issues with fetching available tags when no snapshots existed in the repository, ensuring consistent script behavior.
- Fixed a missing password issue when executing `restic -r /mnt/backups snapshots --json`, ensuring `RESTIC_PASSWORD` is securely passed in all cases.
- Corrected formatting and display of available tags to ensure no duplicate or unnecessary entries are shown.

### Updated
- **Logging and Usability**:
  - Improved log formatting to include more descriptive timestamps and detailed error or success messages for each step.
  - Standardized output for snapshot retrieval by tag, including snapshot ID, timestamp, and tags in a clean and readable format.

- **Environment Validation**:
  - Updated `view_snapshots.sh` to ensure all necessary environment variables (`RESTIC_REPOSITORY`, `RESTIC_PASSWORD`) are set before executing Restic commands.
  - Validated repository access and password file existence at the start of the script to avoid runtime errors.

### Improved
- **Modularity and Robustness**:
  - Refactored `view_snapshots.sh` to handle dynamic tags more effectively, making it easier to extend or customize in the future.
  - Centralized dependency validation (`restic`, `jq`) and repository checks to reduce redundancy across script operations.
  
- **Error Handling**:
  - Added detailed error messages when repository access fails or tags cannot be fetched, ensuring issues are easy to debug.
  - Ensured graceful script exits with appropriate logs when critical validation checks fail.

- **Code Maintenance**:
  - Reduced redundancy by consolidating snapshot and tag-related logic into reusable functions.
  - Enhanced maintainability by improving script readability, modularity, and inline documentation.


## Version 1.7

### Added
- **`restore_backup_direct.sh` Enhancements**:
  - Improved restoration script to handle specific backup types (core files, media files, database backups) based on tags.
  - Integrated functionality to restore databases directly by piping `.sql` files from Restic to MySQL.
  - Added automatic detection of database credentials from `wp-config.php` with fallback to default values for `DB_HOST`.
  - Ensured proper environment setup with `RESTIC_PASSWORD` retrieved securely from `/etc/restic-password`.

- **Automatic Backup Tagging**:
  - Modified `backup_all.sh` to differentiate between manual (`manual-backup-YYYY-MM-DD_HH-MM-SS`) and automated (`auto-backup-YYYY-MM-DD_HH-MM-SS`) backups.
  - Ensured backups triggered by cron jobs are tagged appropriately for easy identification.

- **`manage_backup_schedule.sh` Enhancements**:
  - Ensured `backup_all.sh` runs in `auto` mode when added to cron, guaranteeing automated backups have the correct tags.
  - Validated cron job syntax and dependencies to avoid misconfigurations.
  - Added explicit logging of cron actions, including schedule additions, updates, removals, and listing.

- **Improved Error Messaging**:
  - Enhanced restoration script to provide detailed error messages when database restoration fails, including incorrect credentials or insufficient privileges.

### Fixed
- Resolved issues with incorrect handling of database restoration paths, ensuring `.sql` files are restored directly into MySQL when detected.
- Addressed potential confusion in tagging by clearly segregating manual and automated backups in logs and tags.
- Corrected empty `DB_HOST` values in the restoration script by setting a default to `localhost` when no value is provided in `wp-config.php`.

### Updated
- **Logging and Traceability**:
  - Enhanced all relevant scripts (`restore_backup_direct.sh`, `backup_all.sh`, `manage_backup_schedule.sh`) to log detailed timestamps and descriptive messages.
  - Improved consistency in log formatting across scripts for better traceability.
  - Added detailed logs for all cron operations in `manage_backup_schedule.sh` to track automated backups effectively.

### Improved
- **Backup and Restore Modularity**:
  - Streamlined the orchestration of backups and restores to ensure modularity and easier debugging.
  - Improved robustness of `restore_backup_direct.sh` by handling different backup types dynamically using snapshot tags.
  - Simplified management of cron schedules with better error handling and validation in `manage_backup_schedule.sh`.

- **Error Handling**:
  - Ensured all scripts exit gracefully on errors with detailed logs to pinpoint issues.
  - Enhanced dependency validation to check for all required tools (`restic`, `mysql`, `jq`, `crontab`, etc.) before execution.

- **Code Maintenance**:
  - Centralized key operations such as password retrieval, lock handling, and backup tagging across scripts for easier maintenance and fewer redundancies.

## Version 1.6

### Added
- **`check_repo_stats.sh`**:
  - Introduced a new script to automate the retrieval of repository statistics and perform maintenance tasks.
  - Logs repository stats, retention policy application, and integrity checks to dynamically named log files (`repo_stats_YYYY-MM-DD.log`).
  - Automatically applies a retention policy (`--keep-last 7 --prune`) to manage repository size.
  - Includes an integrity check (`restic check --read-data-subset=5%`) to validate repository consistency and data reliability.
  - Detects and removes stale locks before performing operations.
  - Ensures dependencies (`restic`) and repository access are validated before execution.
  - Handles errors gracefully with detailed logging for each step.

- **Enhanced `backup_repo_check` Functionality**:
  - Created a standalone script `check_backup_repo.sh` to validate repository integrity and initialize repositories if empty.
  - Logs actions and errors to `/home/litespeed/logs/backup_repo_check.log` for better traceability.
  - Improved stale lock handling by detecting and removing locks when necessary.
  - Validates or creates a password file (`/etc/restic-password`) to ensure smooth operation without prompts.
  - Automatically initializes a new repository if no files are present in the backup path.

### Fixed
- Addressed permission issues in `check_backup_repo.sh` by ensuring the log directory and repository paths are writable by the correct user.
- Corrected potential errors in handling stale locks by adding proper validation before removing them.

### Improved
- Enhanced logging across `check_repo_stats.sh` and `check_backup_repo.sh` for better monitoring and traceability.
- Optimized script robustness with dependency validation, error handling, and reduced redundancy.
- Streamlined repository maintenance by combining retention policy application, stale lock removal, and integrity checks into `check_repo_stats.sh`.

## Version 1.5

### Added
- Simplified `backup_all.sh` by delegating backup tasks to individual scripts (`backup_core_files.sh`, `backup_database.sh`, `backup_media.sh`).
- Introduced centralized orchestration of all backups in `backup_all.sh` with a unified `manual-backup` tag for consistency.
- Ensured individual scripts can be reused independently or called as part of a larger workflow.

### Fixed
- Corrected `trap` command in `backup_all.sh` to ensure errors are properly logged to `backup_error.log`.
- Addressed issues with redundant logic in `backup_all.sh` by consolidating common functionality into the individual backup scripts.
- Resolved potential inconsistencies in tag usage by applying a standard `manual-backup` tag with the current server date across all backup scripts.

### Updated
- Refactored `backup_all.sh` to dynamically call individual backup scripts instead of duplicating logic for core, database, and media backups.
- Adjusted YAML configuration for `backupnow` to call `backup_all.sh` as the primary entry point for the backup process.

### Improved
- Centralized the backup process to enhance modularity and maintainability by separating core, database, and media backup logic.
- Ensured consistent tagging and logging across all backup types for easier monitoring and debugging.
- Streamlined the orchestration script (`backup_all.sh`) for improved readability and reduced duplication.

## Version 1.4

### Added
- Introduced efficient lock management in all backup scripts to prevent conflicts during simultaneous Restic operations.
- Automated removal of stale locks before each Restic operation.
- Added global lock mechanism using `flock` to serialize backup operations across multiple processes.
- Implemented dynamic inclusion of Restic password handling by reading it directly from the `/etc/restic-password` file.
- Added automatic validation of Restic repository access before performing backups.

### Fixed
- Fixed issues with manual Restic prompts for the repository password by ensuring passwords are passed via environment variables.
- Resolved potential conflicts caused by simultaneous Restic processes with the introduction of serialized operations.
- Corrected permissions for Restic lock directories to avoid permission-denied errors during backup and restore processes.
- Enhanced password validation logic to ensure backups fail gracefully if the provided password is incorrect.

### Updated
- Updated core, media, and database backup scripts to handle dynamic exclusion paths using a loop-based approach for `--exclude` options.
- Revised logging mechanisms across all scripts to include detailed timestamps and step-specific logs for better traceability.
- Improved script robustness by validating the Restic repository's accessibility upfront using `restic snapshots`.

### Improved
- Optimized all backup scripts to use efficient lock and unlock handling, ensuring smooth operation during concurrent backups and restores.
- Standardized backup script flow across media, database, and core backups, including consistent use of environment variables and error handling.
- Improved log formatting with more descriptive log messages and clear delineation of errors, warnings, and successes.
- Reduced redundancy in password handling by centralizing Restic password retrieval logic.