backup_for_luciano
Simple backup program tailored for me Luciano. Made with rust, git-bash and powershell.
version: 2026.724.1142 date: 2026-07-24 author: bestia.dev repository: codeberg.org
Hashtags: #tutorial #rust #git-bash #powershell
My projects on Codeberg are more like a tutorial than a finished product: bestia-dev tutorials.
Not only backup, but a workflow
There are many steps in sequence to make backups and copies of my coding projects.
1. INSIDE CRUSTDE CONTAINER
Open the VSCode on the rustprojects folder inside CRUSTDE container.
In the integrated bash terminal in VSCode run the next commands.
First we need to prepare the repos that we will use for the backup workflow.
It is convenient to input the passwords of private keys in one place, so the workflow can go smoothly.
# In CRUSTDE container bash terminal
cd ~/rustprojects
sshadd_emb codeberg_org_bestia_dev_git
sshadd_emb hetzner_com_debian
# from the git_multi_repo download the bash scripts for luciano
curl -L https://codeberg.org/bestia-dev-ready-for-use/backup_for_luciano/releases/download/v2026.724.1142/crustde_luciano_rustprojects.tar.gz --output /tmp/crustde_luciano_rustprojects.tar.gz
tar --no-same-owner -xzv --strip-components=1 -C ~/rustprojects -f /tmp/crustde_luciano_rustprojects.tar.gz
rm /tmp/crustde_luciano_rustprojects.tar.gz
# everything is ready for
sh pull_all_for_refresh.sh
# and
sh git_status.sh
mkdir -p ~/rustprojects/bestia-dev-ready-for-use
cd ~/rustprojects/bestia-dev-ready-for-use
git clone ssh://git@codeberg.org/bestia-dev-ready-for-use/bestia-dev-website.git
git clone ssh://git@codeberg.org/bestia-dev-ready-for-use/readme_copy.git
curl -L https://codeberg.org/bestia-dev-ready-for-use/readme_copy/releases/download/v2026.724.1235/readme_copy-v2026.724.1235-x86_64-unknown-linux-gnu.tar.gz --output /tmp/readme_copy.tar.gz
tar --no-same-owner -xzv --strip-components=2 -C ~/.cargo/bin -f /tmp/readme_copy.tar.gz
chmod +x ~/.cargo/bin/readme_copy
alias readme_copy='~/.cargo/bin/readme_copy'
rm /tmp/readme_copy.tar.gz
curl -L https://codeberg.org/bestia-dev-work-in-progress/git_multi_repo/releases/download/v2026.723.1508/git_multi_repo-v2026.723.1508-x86_64-unknown-linux-gnu.tar.gz --output /tmp/git_multi_repo.tar.gz
tar --no-same-owner -xzv --strip-components=2 -C ~/.cargo/bin -f /tmp/git_multi_repo.tar.gz
chmod +x ~/.cargo/bin/git_multi_repo
rm /tmp/git_multi_repo.tar.gz
cd ~/rustprojects
2. Local website development - readme
The Google SEO is terrible for codeberg.org or github.com repositories.
I would rather have a copy of the README.md on my web site.
# In CRUSTDE container bash terminal
cd ~/rustprojects/bestia-dev-ready-for-use/readme_copy
sh readme_copy_all.sh
3. Local website development - docs
The project bestia-dev-ready-for-use/bestia-dev-website must always be present in my CRUSTDE container. It is the development project for my website bestia.dev. After making changes locally, I just upload/sync the content to my web server using the bash script upload_to_website.sh.
I want to have fresh docs in my local website development project. The automation task of any of my Rust projects cargo auto doc writes directly into the bestia-dev-website local folders.
I can then use rsync to upload those files to my web server. And use git to commit and push.
cd ~/rustprojects/bestia-dev-ready-for-use/bestia-dev-website
sh upload_to_web_server.sh
git add .
git commit -m "upload"
git push
4. Many repos in CRUSTDE container
I work on my Rust projects always inside the CRUSTE container. There are no external volumes or shared drives. All the code is inside the ephemeral container. I must frequently use git pull and git push to synchronize with the remote repository on codeberg.org or github.com.
The container is ephemeral and can be deleted at any time, so I have to always push changes to the remote as soon as possible. \
cd ~/rustprojects
sh git_status.sh
# Follow the instructions if there are repos to commit and push
# Then run it one more time to assert it is all ok
# sh git_status.sh
5. INSIDE WINDOWS GIT-BASH
The git_multi.repo.exe must me already inside ~/.cargo/bin.
It is convenient to input the passwords of private keys in one place, so the workflow can go smoothly.
cd ~/git-bash/rustprojects
sshadd_emb codeberg_org_bestia_dev
sshadd_emb github_com_bestia_dev
sshadd_emb github_com_guitaraoke_leader
sshadd_emb hetzner_com_storage_box_02
sshadd_emb hetzner_com_storage_box_04
sshadd_emb hetzner_com_debian
6. Pull remote repos
Pull the remote repos into d:\box_original_1.
# In Windows git-bash
cd /d/box_original_1/BestiaDev/backup_codeberg_org_bestia_dev
sh pull_all_for_backup.sh
cd /d/box_original_1/BestiaDev/backup_github_com_bestia_dev
sh pull_all_for_backup.sh
cd /d/box_original_1/BestiaDev/backup_github_com_guitaraoke_leader
sh pull_all_for_backup.sh
7. Finally backup_for_luciano
Connect the external drives and connect to the internet.
# In Windows git-bash
cd ~/git-bash/rustprojects/backup_for_luciano
sh backup_for_luciano_auto.sh
# The first time it takes a lot of time.
# Run it the second time to check that all the backups are really up-to-date. This is quick.
sh backup_for_luciano_auto.sh
Backup strategy
My working folder is on the internal drive 1TB. Now it is just 116GB, because the internal SSD can be really small.
My "primary external drive" is a 2TB Samsung T7Shield SSD. It is small, fast, resilient and expensive. My original files that cannot be stored on my laptop's internal drive will be stored on this drive.
My "backup external drive" is a 2TB Transcend StoreJet 25H3 ruggedized portable hard drive. It has an old school HDD spinning disk, but it is pretty tough.
I have two 1TB storage boxes online on Hetzner called Hetzner_02 and Hetzner_04. They don't have an offering of 2TB, just 1TB and then 5TB. So I need to split my files accordingly.
I now have 3 box_original folders. One on my laptop and 2 on my "primary external SSD".
My backup strategy is described in this ASCII art:
┌──────────────┐ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐
│internal_drive│ │T7_Shield│ │Hetzner_04│ │Transcend│ │Hetzner_02│
└───────┬──────┘ └────┬────┘ └─────┬────┘ └────┬────┘ └─────┬────┘
│backup_1_of_box_original_1 (116GB) │ │ │ │
│──────────────────────────────────>│ │ │ │
│ │ │ │ │
│ backup_2_of_box_original_1 │ │ │
│──────────────────────────────────────────────────────────────>│ │ │
│ │ │ │ │
│ backup_3_of_box_original_1 │ │ │
│────────────────────────────────────────────────────────────────────────────────────>│ │
│ │ │ │ │
│ │ backup_1_of_box_original_2 (369GB) │ │
│ │────────────────────────────────────────────────>│ │
│ │ │ │ │
│ │backup_2_of_box_original_2 │ │ │
│ │──────────────────────────>│ │ │
│ │ │ │ │
│ │ backup_1_of_box_original_3 (636GB) │ │
│ │────────────────────────────────────────────────>│ │
│ │ │ │ │
│ │ backup_2_of_box_original_3 │ │
│ │──────────────────────────────────────────────────────────────────────>│
┌───────┴──────┐ ┌────┴────┐ ┌─────┴────┐ ┌────┴────┐ ┌─────┴────┐
│internal_drive│ │T7_Shield│ │Hetzner_04│ │Transcend│ │Hetzner_02│
└──────────────┘ └─────────┘ └──────────┘ └─────────┘ └──────────┘
I am using https://editor.plantuml.com to draw the diagram.
@startuml
internal_drive -> T7_Shield : backup_1_of_box_original_1 116GB
internal_drive -> Hetzner_04 : backup_2_of_box_original_1
internal_drive -> Transcend : backup_3_of_box_original_1
T7_Shield -> Transcend : backup_1_of_box_original_2 369GB
T7_Shield -> Hetzner_04 : backup_2_of_box_original_2
T7_Shield -> Transcend : backup_1_of_box_original_3 636GB
T7_Shield -> Hetzner_02 : backup_2_of_box_original_3
@enduml
rsync
I would love to use rsync for my backups. But git-bash on Windows is super slow to work with Windows files. I need to invent a workaround.
Hetzner Storage box
I pay for 2 storage boxes of 1TB each on Hetzner, around 8€/month. It is not too expensive in comparison with the american Dropbox.
sshadd_emb hetzner_com_storage_box_04
ssh -p 23 -i //wsl.localhost/Debian/home/luciano/.ssh/hetzner_com_debian_1_luciano_ssh_1 u543604@u543604.your-storagebox.de ls
# backup_2_of_box_original_1
# backup_2_of_box_original_2
sshadd_emb hetzner_com_storage_box_02
ssh -p 23 -i //wsl.localhost/Debian/home/luciano/.ssh/hetzner_com_debian_1_luciano_ssh_1 u543602@u543602.your-storagebox.de ls
# backup_2_of_box_original_3
File list from Hetzner
Hetzner shell has very limited functionality, but it has the tree command. Very fast! So I can get the file list simply.
Add to ~/.ssh/config file:
# ~/.ssh/config
Host hetzner_com_storage_box_02
HostName u543602.your-storagebox.de
Port 23
User u543602
IdentityFile //wsl.localhost/Debian/home/luciano/.ssh/hetzner_com_debian_1_luciano_ssh_1
IdentitiesOnly yes
Host hetzner_com_storage_box_04
HostName u543604.your-storagebox.de
Port 23
User u543604
IdentityFile //wsl.localhost/Debian/home/luciano/.ssh/hetzner_com_debian_1_luciano_ssh_1
IdentitiesOnly yes
# In Windows git-bash
cd /c/Users/luciano/git-bash/rustprojects/backup_for_luciano
sshadd_emb hetzner_com_storage_box_04
ssh hetzner_com_storage_box_04 'tree -i -p -s --timefmt "%Y-%m-%d %H:%M:%S" -f backup_2_of_box_original_1' > tmp/aux_files/backup_2_of_box_original_1.txt
# 4 seconds
# 4206 directories, 33874 files
# In Windows git-bash
cd /c/Users/luciano/git-bash/rustprojects/backup_for_luciano
sshadd_emb hetzner_com_storage_box_04
ssh hetzner_com_storage_box_04 'tree -i -p -s --timefmt "%Y-%m-%d %H:%M:%S" -f backup_2_of_box_original_2' > tmp/aux_files/backup_2_of_box_original_2.txt
# 4 seconds
# 1741 directories, 52403 files
# In Windows git-bash
cd /c/Users/luciano/git-bash/rustprojects/backup_for_luciano
sshadd_emb hetzner_com_storage_box_02
ssh hetzner_com_storage_box_02 'tree -i -p -s --timefmt "%Y-%m-%d %H:%M:%S" -f backup_2_of_box_original_3' > tmp/aux_files/backup_2_of_box_original_3.txt
# 4 seconds
# 3465 directories, 57843 files
The format of the resulting files is:
[drwxr-xr-x 6 2026-01-17 11:44:03] backup_2_of_box_original_1/0_temp/peacefulview_export
[-rw-r--r-- 517201991 2026-01-17 11:40:23] backup_2_of_box_original_1/0_temp/peacefulview_export/peacefulview_eu_public_html.zip
File list from local originals
I will use a PowerShell command from git-bash because it is fast, really fast:
# In Windows git-bash
pwsh -Command 'dir -r d:\box_original_1 | % { if ($_.PsIsContainer) {"d`t" + "0" + "`t" + "2026-01-01 01:01:01" + "`t" + $_.FullName } else {"f`t" + $_.Length + "`t" + $_.LastWriteTime.ToString("yyyy-MM-dd hh:mm:ss") + "`t" +$_.FullName } } ' > tmp/aux_files/box_original_1.txt
# 5 seconds
# 69591 lines
# In Windows git-bash
pwsh -Command 'dir -r e:\box_original_2 | % { if ($_.PsIsContainer) {"d`t" + "0" + "`t" + "2026-01-01 01:01:01" + "`t" + $_.FullName } else {"f`t" + $_.Length + "`t" + $_.LastWriteTime.ToString("yyyy-MM-dd hh:mm:ss") + "`t" +$_.FullName } } ' > tmp/aux_files/box_original_2.txt
# 5 seconds
# 55015 lines
# In Windows git-bash
pwsh -Command 'dir -r e:\box_original_3 | % { if ($_.PsIsContainer) {"d`t" + "0" + "`t" + "2026-01-01 01:01:01" + "`t" + $_.FullName } else {"f`t" + $_.Length + "`t" + $_.LastWriteTime.ToString("yyyy-MM-dd hh:mm:ss") + "`t" +$_.FullName } } ' > tmp/aux_files/box_original_3.txt
# 5 seconds
# 60865 lines
The format of the resulting files is:
d -> 0 -> 2026-01-01 01:01:01 -> D:\box_original_1\0_temp\peacefulview_export
f -> 517201991 -> 2026-01-17 12:40:23 -> D:\box_original_1\0_temp\peacefulview_export\peacefulview_eu_public_html.zip
Compare file lists
Now I need a Rust program to parse these files, compare and find the differences.
This is strictly a one-way sync, from local to remote.
Basic rules:
- same name, size and timestamp around 3 seconds is the same file. Ignore
- different size, timestamp or more than 3 seconds difference goes into the list for upload
- not-existing on remote goes into the list for upload
- not existing on local goes into the list for deletion
The 3-seconds rule is stupid because Microsoft's exFAT has a resolution of 2 seconds for the file timestamp. How stupid is that in 2026! Back to the future, 1980 - Hooray, Microsoft.
First, I need to sort the files with the same sort order. Sometimes different OS have different sort orders for non alphabetic symbols.
Rust to the rescue
I created a Rust CLI program in Linux, cross-compiled it to Windows.
# In CRUSTDE container bash terminal
clear && cargo fmt && cargo build --release --target x86_64-pc-Windows-gnu
It can be run inside Windows git-bash.
# In Windows git-bash
cd ~/git-bash/rustprojects/backup_for_luciano/
scp crustde:/home/rustdevuser/rustprojects/backup_for_luciano/target/x86_64-pc-windows-gnu/release/backup_for_luciano.exe backup_for_luciano.exe && BACKUP_FOR_LUCIANO_LOG="debug" ./backup_for_luciano -h
The program has a few commands:
- local_and_remote_file_list
- compare
- upload_changed
- upload_added
- delete_removed
- verify
- backup_local
Smaller video
Some cameras make huge videos. I can use ffmpeg to make them smaller.
I don't want to run every video separately. It takes 20 minutes to convert the video.
I will use Rust instead of bash to read a file with a list of large videos. I can get this list using TotalCmd find advanced size larger than.
Verify backup
Incredible! The normal copy and robocopy do not verify that the destination file is equal to the source file. And it happens that some files are corrupted.
Not a lot, but just enough to lose all confidence. No reason at all for errors like that. How does anything work at all?
The program rsync is better because it verifies the integrity of the file after the copy.
I can use TotalCmd to check the files to have the same 'content'. It calculates the checksum on both sides and compare it.
Set the Synchronization Compare to "by content" and watch only "duplicates".
I want to do the same, but integrate it into my list of files, so I can verify if anything changes.
All the files have the same name, date and size, but the content is corrupted.
It can take a lot of time for 1.5TB.
Windows uses the tool chkdsk to check the disks.
chkdsk C: /f /r
chkdsk D: /f /r
chkdsk E: /f /r
chkdsk G: /f /r
For Samsung SSD drives, I run FullScan in Samsung Magician.
Development details
Read the development details in a separate MD file:
DEVELOPMENT.md
Releases changelog
Read the release's changelog in a separate MD file:
RELEASES.md
TODO
find external drive in /media/
TUI, at the top, shows the commands as a menu or buttons. The button shows the terminal.
Open-source and free as a beer
My open-source projects are free as a beer (MIT license).
I just love programming.
But I also need to drink. If you find my projects and tutorials helpful, please buy me a beer by donating to my PayPal.
You know the price of a beer in your local bar ;-)
So I can drink a free beer for your health :-)
Na zdravje! Alla salute! Prost! Nazdravlje! 🍻
//bestia.dev
//codeberg.org/bestia-dev
//github.com/bestia-dev
//bestiadev.substack.com
//youtube.com/@bestia-dev-tutorials