How to recover and update Proxmox 8 firewall configuration in SQLite when you locked yourself out

Diego Carrasco G. — Fri, 27 Sep 2024 07:00:00 GMT

TLDR

The firewall config is not in /etc/pve/firewall/cluster.fw but in a SQLite Database in /var/lib/pve-cluster/config.db. You need to reboot your system into rescue mode, edit the value enable: 1 to enable: 0 and reboot into Proxmox.

Context

I made a noob mistake and locked myself out of my server. Luckily Hetzner allows me to reboot into rescue mode. This is what happened and how I managed to get my access back.

In other words, this tutorial is for situations where you've accidentally locked yourself out of your Proxmox server due to a firewall misconfiguration (like I did). In my case, I enabled the firewall (enable: 1) with an incorrect configuration, preventing access to the server. The solution involves booting into a rescue system, mounting the Proxmox partition, and manually editing the firewall configuration in the SQLite database.

Prerequisites

Access to a rescue system (e.g., Hetzner Rescue System)
Basic knowledge of Linux commands and SQLite, although you can copy and paste these commands and it should work.

Disclaimer: I am not responsible for data loss or anything else for that matter. The following commands worked for me and nothing bad happened. I out them here in case they help someone else, as I had to research a few hour before solving this (specially the issue of not finding the config).

Step 1: Boot into Rescue System

Boot your server into the rescue system provided by your hosting provider (e.g., Hetzner Rescue System).

Step 2: Identify the Proxmox Partition

Use the lsblk command to list all block devices:

lsblk

Identify the partition where Proxmox is installed. It's often part of a RAID array or LVM setup.

In my case the output was like this:

loop0            7:0    0   3.1G  1 loop  
nvme1n1        259:0    0 476.9G  0 disk  
├─nvme1n1p1    259:1    0   256M  0 part  
│ └─md0          9:0    0 255.9M  0 raid1 
├─nvme1n1p2    259:2    0     1G  0 part  
│ └─md1          9:1    0  1022M  0 raid1 
└─nvme1n1p3    259:3    0 475.7G  0 part  
  └─md2          9:2    0 475.6G  0 raid1 
    ├─vg0-root 253:0    0    64G  0 lvm   
    ├─vg0-swap 253:1    0     8G  0 lvm   
    └─vg0-data 253:2    0   402G  0 lvm   
nvme0n1        259:4    0 476.9G  0 disk  
├─nvme0n1p1    259:5    0   256M  0 part  
│ └─md0          9:0    0 255.9M  0 raid1 
├─nvme0n1p2    259:6    0     1G  0 part  
│ └─md1          9:1    0  1022M  0 raid1 
└─nvme0n1p3    259:7    0 475.7G  0 part  
  └─md2          9:2    0 475.6G  0 raid1 
    ├─vg0-root 253:0    0    64G  0 lvm   
    ├─vg0-swap 253:1    0     8G  0 lvm   
    └─vg0-data 253:2    0   402G  0 lvm

There I saw that I should mount vg0, and that is was in a raid md2

Step 3: Assemble RAID Array (if applicable)

If your Proxmox partition is part of a RAID array, assemble it:

mdadm --assemble --scan

Step 4: Activate Volume Group

Activate the volume group (usually named vg0 in Proxmox):

vgchange -ay vg0

Step 5: Mount the Proxmox Partition

Create a mount point and mount the Proxmox root partition:

mkdir /mnt/proxmox
mount /dev/vg0/root /mnt/proxmox

Verify the mount:

ls /mnt/proxmox/

Here you should see some files and directories.

Step 6: Locate the Configuration Database

The Proxmox configuration is stored in an SQLite database. Locate it:

ls -la /mnt/proxmox/var/lib/pve-cluster

You should see a file named config.db.

Step 7: Access the SQLite Database

Open the SQLite database:

sqlite3 /mnt/proxmox/var/lib/pve-cluster/config.db

sqlite3 is already installed in the rescue system of Hetzner. You need to install it if it's not available in your system.

Step 8: Check the Current Firewall Configuration

View the current firewall configuration:

SELECT * FROM tree WHERE name = 'cluster.fw';

Note: Initially I didn't know where this was, so I used the following to find where the entry was and if there was any.

SELECT * FROM tree WHERE name = 'cluster.fw';

Step 9: Update the `enable` Option

Change the enable option from 1 to 0 to disable the firewall:

UPDATE tree 
SET data = replace(data, 'enable: 1', 'enable: 0') 
WHERE name = 'cluster.fw';

Step 10: Verify the Change

Confirm that the change was made successfully:

SELECT * FROM tree WHERE name = 'cluster.fw';

Step 11: Exit SQLite

Exit the SQLite prompt:

.quit

Step 12: Unmount and Reboot

Unmount the Proxmox partition and reboot the server:

umount /mnt/proxmox
reboot

Important Notes

Disabling the Firewall: This process disables the firewall cluster-wide. Re-enable it after properly configuring it once you regain access.
Security Risks: A disabled firewall may expose your system to security risks. You have been warned.
Backup: Always create backups before making significant changes. I have my proxmox configs in a git repository for reference.
Alternative Methods: When possible, use the Proxmox web interface or CLI tools for configuration changes. At least that's what I've read. I like to use config files, but I also locked myself out of my server.

References

Several sites, but I cannot longer remember all of them.

Some of the sites I visited are:

https://forum.proxmox.com/threads/ssh-connection-no-web-interface.110702/
https://www.reddit.com/r/Proxmox/comments/13hyn0y/how_to_secure_proxmox_web_ui/
https://eulenfunk.readthedocs.io/en/stable/supernode01.html
and many more ...

Diego A. Carrasco Gubernatis | Personal Website (Posts about SQLite)