I messed up the web server -- how to restore?

Platform Access Issues
1

Hi!
Well, I messed up my Syncloud instance because I ran certbot on my device, which apparently took down the Syncloud web server. Or something like this – I don’t know how any of this works. I also installed snapd, which probably wasn’t very smart.
Anyway, my Syncloud instance is not reachable any more now. The browser says “unable to connect”, and ping [name].syncloud.it returns “Destination unreachable: Address unreachable.”.
nginx isn’t even running any more. I still have SSH access.

Is there a way to restart or reconfigure the web server?
How can I find out what’s wrong?

If there is no straightforward way to restore the server, then I guess I have to reset the whole installation, right? However, how can I keep all my data? According to the wiki I would do the following:

  1. Remove my external USB drive.
  2. Flash the device image.
  3. Connect the new instance to my Syncloud account.
  4. Attach my USB drive.
  5. Activate external disk support.
  6. Reinstall my apps.

Would this work?
However, I cannot do any backups beforehand, because I cannot access the web interface. Is really everything important stored in /data?

I’m really sorry to bother you with this, because it is nothing but my own fault for being stupid and trying around with stuff I don’t know shit about …

Thanks in advance!

2

The easiest way is to rewrite the disk image assuming you have activated external disk before to store all the data.

/data contains all files generated or uploaded to apps.

It does not contain app settings like calendars or contacts for nextcloud app, for that you need a backup (which will be stored under /data)

What apps do you have?

3

Also why did you need to run certbot and install snapd.

We use dns based certbot, so in free and premium mode it should give you certificate withing minutes after activating (usually).

Snapd is even more critical as we use a modified one repointed to syncloud store instead of canonical.

Whould it be better if we disable apt by default so it is possible to enable but at least you need to be aware of what you are doing?

4

Yeah I know, that was really stupid. I just wanted to try something out, it didn’t have anything to do with the Syncloud certificate. Syncloud was working perfectly fine. I didn’t expect that running certbot would be so intrusive.

Can I run a backup manually for all apps via SSH, without the web interface?
If not, what does the backup procedure do? Could I manually do the steps that the backup procedure does?

Nextcloud, Syncthing, Users, Files, Grocy.

Maybe. :grimacing:

5

we have a command line for that, try this:

/snap/platform/current/bin/cli backup create grocy
6

probably if you reinstalled snapd it may probably forgot now about syncloud apps and may not work, let’s see what the above command shows.

7

Thanks, I’ll try it out!
I’m not at home this week, I will report back next weekend.

8

The command failed. Apparently because the server is down.

root@syncloud:~# /snap/platform/current/bin/cli backup create grocy
info backup/backup.go:125 Running backup create {"app": "grocy", "file": "/data/platform/backup/grocy-2023-0507-131140.tar.gz"}
info cli/executor.go:23 execute {"cmd": "/usr/bin/du -s /var/snap/grocy/current"}
info cli/executor.go:23 execute {"cmd": "/usr/bin/du -s /var/snap/grocy/common"}
info backup/backup.go:149 temp space left: 23230001152
info backup/backup.go:150 temp space needed: 88
info cli/executor.go:23 execute {"cmd": "/usr/bin/snap run grocy.backup-pre-stop"}
error snap/cli.go:47 snap failed {"command": "run", "error": "exit status 1: 2023-05-07 13:11:55,935 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump\npg_dumpall: could not connect to database \"template1\": could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/snap/grocy/current/database/.s.PGSQL.5436\"?\n2023-05-07 13:11:59,064 - database - ERROR - postgres error:\nTraceback (most recent call last):\n  File \"/snap/grocy/34/hooks/backup-pre-stop\", line 4, in <module>\n    Installer().backup_pre_stop()\n  File \"/snap/grocy/34/hooks/installer.py\", line 127, in backup_pre_stop\n    self.pre_refresh()\n  File \"/snap/grocy/34/hooks/installer.py\", line 76, in pre_refresh\n    self.db.backup()\n  File \"/snap/grocy/34/hooks/postgres.py\", line 45, in backup\n    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))\n  File \"/snap/grocy/34/hooks/postgres.py\", line 55, in run\n    raise e\n  File \"/snap/grocy/34/hooks/postgres.py\", line 51, in run\n    output = check_output(cmd, shell=True).decode()\n  File \"/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py\", line 415, in check_output\n    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,\n  File \"/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py\", line 516, in run\n    raise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.\n"}
github.com/syncloud/platform/snap.(*Cli).run
	/drone/src/backend/snap/cli.go:47
github.com/syncloud/platform/snap.(*Cli).Run
	/drone/src/backend/snap/cli.go:30
github.com/syncloud/platform/snap.(*Cli).RunCmdIfExists
	/drone/src/backend/snap/cli.go:36
github.com/syncloud/platform/backup.(*Backup).Create
	/drone/src/backend/backup/backup.go:164
main.backupCmd.func1.1
	/drone/src/backend/cmd/cli/backup.go:27
reflect.Value.call
	/usr/local/go/src/reflect/value.go:556
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:339
github.com/golobby/container/v3.Container.Call
	/go/pkg/mod/github.com/golobby/container/v3@v3.3.1/container.go:206
main.backupCmd.func1
	/drone/src/backend/cmd/cli/backup.go:25
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895
main.main
	/drone/src/backend/cmd/cli/main.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
Error: exit status 1: 2023-05-07 13:11:55,935 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump
pg_dumpall: could not connect to database "template1": could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/snap/grocy/current/database/.s.PGSQL.5436"?
2023-05-07 13:11:59,064 - database - ERROR - postgres error:
Traceback (most recent call last):
  File "/snap/grocy/34/hooks/backup-pre-stop", line 4, in <module>
    Installer().backup_pre_stop()
  File "/snap/grocy/34/hooks/installer.py", line 127, in backup_pre_stop
    self.pre_refresh()
  File "/snap/grocy/34/hooks/installer.py", line 76, in pre_refresh
    self.db.backup()
  File "/snap/grocy/34/hooks/postgres.py", line 45, in backup
    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))
  File "/snap/grocy/34/hooks/postgres.py", line 55, in run
    raise e
  File "/snap/grocy/34/hooks/postgres.py", line 51, in run
    output = check_output(cmd, shell=True).decode()
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.

Usage:
  cli backup create [app] [flags]

Flags:
  -h, --help   help for create

Global Flags:
      --system-config string   system config file (default "/snap/platform/current/config/platform.cfg")
      --user-config string     user config sqlite db (default "/var/snap/platform/current/platform.db")

exit status 1: 2023-05-07 13:11:55,935 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump
pg_dumpall: could not connect to database "template1": could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/snap/grocy/current/database/.s.PGSQL.5436"?
2023-05-07 13:11:59,064 - database - ERROR - postgres error:
Traceback (most recent call last):
  File "/snap/grocy/34/hooks/backup-pre-stop", line 4, in <module>
    Installer().backup_pre_stop()
  File "/snap/grocy/34/hooks/installer.py", line 127, in backup_pre_stop
    self.pre_refresh()
  File "/snap/grocy/34/hooks/installer.py", line 76, in pre_refresh
    self.db.backup()
  File "/snap/grocy/34/hooks/postgres.py", line 45, in backup
    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))
  File "/snap/grocy/34/hooks/postgres.py", line 55, in run
    raise e
  File "/snap/grocy/34/hooks/postgres.py", line 51, in run
    output = check_output(cmd, shell=True).decode()
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.

Maybe it would be best to make a backup of the current state, reset the installation and manually restore the app-specific files. However, which files are the app-specific files?
Would it work to just backup-and-restore the snap-directory of each app? E. g. /snap/grocy/current.

9

Can you try this:

snap list
10 
root@syncloud:~# snap list
Name       Version    Rev    Tracking  Developer  Notes
core       16-2.58.3  14950  latest    canonical  core
files      158        158    latest    syncloud   -
grocy      34         34     latest    syncloud   -
nextcloud  676        676    latest    syncloud   -
platform   1905       1905   latest    syncloud   base
syncthing  103        103    latest    syncloud   -
users      272        272    latest    syncloud   -
11

Ok, is a mix of syncloud snaps and official canonical snaps now, let’s try to restore syncloud snapd:

/snap/platform/current/bin/upgrade-snapd.sh
12

I think this worked:

root@syncloud:/snap/platform/current/bin# bash upgrade-snapd.sh stable  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     3  100     3    0     0      5      0 --:--:-- --:--:-- --:--:--     5
--2023-05-08 13:43:29--  http://apps.syncloud.org/apps/snapd-361-armhf.tar.gz
Resolving apps.syncloud.org (apps.syncloud.org)... 52.92.227.11, 52.92.210.91, 52.218.180.114, ...
Connecting to apps.syncloud.org (apps.syncloud.org)|52.92.227.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30727445 (29M) [application/gzip]
Saving to: 'snapd-361-armhf.tar.gz'

     0K ........ ........ ........ .....   100% 3.77M=7.8s

2023-05-08 13:43:37 (3.77 MB/s) - 'snapd-361-armhf.tar.gz' saved [30727445/30727445]

snapd/
snapd/bin/
snapd/bin/snap-exec
snapd/bin/snapctl
snapd/bin/unsquashfs
snapd/bin/snap-discard-ns
snapd/bin/snap-repair
snapd/bin/mksquashfs
snapd/bin/snap-update-ns
snapd/bin/snap-seccomp
snapd/bin/snapd
snapd/bin/snap-confine
snapd/bin/snap
snapd/conf/
snapd/conf/snapd.service
snapd/conf/snapd.socket
snapd/lib/
snapd/lib/libseccomp.so.2
snapd/lib/libseccomp.so
snapd/lib/libseccomp.so.2.3.3
snapd/scripts/
snapd/scripts/prepare.sh

No effect on snap list:

root@syncloud:/snap/platform/current/bin# snap list
Name       Version    Rev    Tracking  Developer  Notes
core       16-2.58.3  14950  latest    canonical  core
files      158        158    latest    syncloud   -
grocy      34         34     latest    syncloud   -
nextcloud  676        676    latest    syncloud   -
platform   1905       1905   latest    syncloud   base
syncthing  103        103    latest    syncloud   -
users      272        272    latest    syncloud   -

The backup command still fails:

root@syncloud:/snap/platform/current/bin# ./cli backup create grocy
info backup/backup.go:125 Running backup create {"app": "grocy", "file": "/data/platform/backup/grocy-2023-0508-134551.tar.gz"}
info cli/executor.go:23 execute {"cmd": "/usr/bin/du -s /var/snap/grocy/current"}
info cli/executor.go:23 execute {"cmd": "/usr/bin/du -s /var/snap/grocy/common"}
info backup/backup.go:149 temp space left: 23127977984
info backup/backup.go:150 temp space needed: 88
info cli/executor.go:23 execute {"cmd": "/usr/bin/snap run grocy.backup-pre-stop"}
error snap/cli.go:47 snap failed {"command": "run", "error": "exit status 1: 2023-05-08 13:45:58,275 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump\npg_dumpall: could not connect to database \"template1\": could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/snap/grocy/current/database/.s.PGSQL.5436\"?\n2023-05-08 13:45:58,720 - database - ERROR - postgres error:\nTraceback (most recent call last):\n  File \"/snap/grocy/34/hooks/backup-pre-stop\", line 4, in <module>\n    Installer().backup_pre_stop()\n  File \"/snap/grocy/34/hooks/installer.py\", line 127, in backup_pre_stop\n    self.pre_refresh()\n  File \"/snap/grocy/34/hooks/installer.py\", line 76, in pre_refresh\n    self.db.backup()\n  File \"/snap/grocy/34/hooks/postgres.py\", line 45, in backup\n    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))\n  File \"/snap/grocy/34/hooks/postgres.py\", line 55, in run\n    raise e\n  File \"/snap/grocy/34/hooks/postgres.py\", line 51, in run\n    output = check_output(cmd, shell=True).decode()\n  File \"/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py\", line 415, in check_output\n    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,\n  File \"/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py\", line 516, in run\n    raise CalledProcessError(retcode, process.args,\nsubprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.\n"}
github.com/syncloud/platform/snap.(*Cli).run
	/drone/src/backend/snap/cli.go:47
github.com/syncloud/platform/snap.(*Cli).Run
	/drone/src/backend/snap/cli.go:30
github.com/syncloud/platform/snap.(*Cli).RunCmdIfExists
	/drone/src/backend/snap/cli.go:36
github.com/syncloud/platform/backup.(*Backup).Create
	/drone/src/backend/backup/backup.go:164
main.backupCmd.func1.1
	/drone/src/backend/cmd/cli/backup.go:27
reflect.Value.call
	/usr/local/go/src/reflect/value.go:556
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:339
github.com/golobby/container/v3.Container.Call
	/go/pkg/mod/github.com/golobby/container/v3@v3.3.1/container.go:206
main.backupCmd.func1
	/drone/src/backend/cmd/cli/backup.go:25
github.com/spf13/cobra.(*Command).execute
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958
github.com/spf13/cobra.(*Command).Execute
	/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895
main.main
	/drone/src/backend/cmd/cli/main.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
Error: exit status 1: 2023-05-08 13:45:58,275 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump
pg_dumpall: could not connect to database "template1": could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/snap/grocy/current/database/.s.PGSQL.5436"?
2023-05-08 13:45:58,720 - database - ERROR - postgres error:
Traceback (most recent call last):
  File "/snap/grocy/34/hooks/backup-pre-stop", line 4, in <module>
    Installer().backup_pre_stop()
  File "/snap/grocy/34/hooks/installer.py", line 127, in backup_pre_stop
    self.pre_refresh()
  File "/snap/grocy/34/hooks/installer.py", line 76, in pre_refresh
    self.db.backup()
  File "/snap/grocy/34/hooks/postgres.py", line 45, in backup
    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))
  File "/snap/grocy/34/hooks/postgres.py", line 55, in run
    raise e
  File "/snap/grocy/34/hooks/postgres.py", line 51, in run
    output = check_output(cmd, shell=True).decode()
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.

Usage:
  cli backup create [app] [flags]

Flags:
  -h, --help   help for create

Global Flags:
      --system-config string   system config file (default "/snap/platform/current/config/platform.cfg")
      --user-config string     user config sqlite db (default "/var/snap/platform/current/platform.db")

exit status 1: 2023-05-08 13:45:58,275 - database - INFO - postgres executing: snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump
pg_dumpall: could not connect to database "template1": could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/snap/grocy/current/database/.s.PGSQL.5436"?
2023-05-08 13:45:58,720 - database - ERROR - postgres error:
Traceback (most recent call last):
  File "/snap/grocy/34/hooks/backup-pre-stop", line 4, in <module>
    Installer().backup_pre_stop()
  File "/snap/grocy/34/hooks/installer.py", line 127, in backup_pre_stop
    self.pre_refresh()
  File "/snap/grocy/34/hooks/installer.py", line 76, in pre_refresh
    self.db.backup()
  File "/snap/grocy/34/hooks/postgres.py", line 45, in backup
    self.run('snap run grocy.pgdumpall -f {0}'.format(self.backup_file))
  File "/snap/grocy/34/hooks/postgres.py", line 55, in run
    raise e
  File "/snap/grocy/34/hooks/postgres.py", line 51, in run
    output = check_output(cmd, shell=True).decode()
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/snap/grocy/current/python/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'snap run grocy.pgdumpall -f /var/snap/grocy/current/database.dump' returned non-zero exit status 1.
13

Ok, if you open device ip in the browser https://ip what do you see syncloud or something else?

Also what is the output of this:

snap refresh grocy
14

Oh damn it! Browser access is working again! Something of the above steps must have fixed that, probably the upgrade-snapd …
Now I could make backups of syncthing and files, but no backup was created for the other apps when I tried to do it via the web interface.

snap refresh doesn’t work:

root@syncloud:~# snap refresh grocy
error: cannot communicate with server: Post "http://localhost/v2/snaps/grocy": EOF

Now some apps are accessible again, e. g. syncthing, but others are not reachable, like nextcloud: nextcloud.[name].syncloud.it returns 502 Bad Gateway nginx/1.20.1.

This is eating up too much of your time and you have already helped me so much!
I will now try to manually back up the app settings and simulate the cli backup procedure. According to the backup that worked (syncthing), I need to pack /var/snap/<app>/(common|current) into an archive like /data/platform/backup/<app>-<timestamp>.tar.gz. Then, after resetting the device installation, the apps should be able to recognize the archive as a valid backup, right?

15

Well backup contains files under /var/snap but also it will create a latest backup of a database (nextcloud and grocy have one) and put it under:

/var/snap/[app]/current/database.dump

Then on restore it will try to drop the database (in your case potentially having the latest data) and use the file to create new one. So if you think you have up to date database.dump file then it may be ok to skip what backup does and create archive manually.

I would try to understand why backup is not working which means why grocy database is not running (error from the backup) which needs snap to work.

snap does not work either still which if fixed can fix everything else.

I would check if you have correct version of snap

snap version
16

I see … thanks for the insight! Then it might be worthwhile to dig further …

root@syncloud:~# snap version
snap    361
snapd   361
series  16
debian  10
kernel  4.1.19-v7+
17 
snap refresh grocy

Then

systemctl status snapd
18 
root@syncloud:~# snap refresh grocy
error: cannot communicate with server: Post "http://localhost/v2/snaps/grocy": EOF
root@syncloud:~# systemctl status snapd
● snapd.service - Snappy daemon
   Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2023-05-08 20:01:30 CEST; 5s ago
 Main PID: 4847 (snapd)
   CGroup: /system.slice/snapd.service
           └─4847 /usr/lib/snapd/snapd

May 08 20:01:26 syncloud systemd[1]: Starting Snappy daemon...
May 08 20:01:30 syncloud snapd[4847]: 2023/05/08 20:01:30.279758 daemon.go:322: started snapd/361 (series 16; classic; devmode) debian/10 (armhf) linux/4.1.19-v7.
May 08 20:01:30 syncloud systemd[1]: Started Snappy daemon.
May 08 20:01:30 syncloud snapd[4847]: 2023/05/08 20:01:30.326873 store.go:922: not writing catalogs
19 
systemctl status snapd.socket

Did you reboot since snapd upgrade?

20 
root@syncloud:~# systemctl status snapd.socket
● snapd.socket - Socket activation for snappy daemon
   Loaded: loaded (/lib/systemd/system/snapd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-02-14 11:12:04 CET; 4 years 2 months ago
   Listen: /run/snapd.socket (Stream)
           /run/snapd-snap.socket (Stream)
   CGroup: /system.slice/snapd.socket

Feb 14 11:12:04 syncloud systemd[1]: Starting Socket activation for snappy daemon.
Feb 14 11:12:04 syncloud systemd[1]: Listening on Socket activation for snappy daemon.

Running since 4+ years seems strange because I have been using the server for at most 2 years or so.
I remember that I recently changed the device’s time zone via timedatectl set-timezone. I think it was UTC and I changed it to Europe/Berlin. Might this have any relevance? I just changed it back to UTC.

Yes, several times.