'Univention Proxy Error' after upgrade and restart

rijadp · June 12, 2024, 2:20pm

Hello,

I have UCS 5.0-5 errata872 and OwnCloud 10.13.4.1 running on it. Everything works fine for 7-8 months now.
Now it is time to upgrade UCS to a newer version.
After the upgrade, everything worked perfectly until I made server reboot. After reboot, I can access the Univention portal, but I’m not able to access OwnCloud app. I have this error:

Univention Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server

I have tried upgrading to all available versions: 5.0.6, 5.0.7, 5.0.8, but every time situation is the same. And every time I need to restore a snapshot.

Here are some logs from docker that can help:

[Wed Jun 12 12:13:10.150431 2024] [autoindex:error] [pid 199] [client 172.17.42.1:33026] AH01276: Cannot serve directory /var/www/owncloud: No matching DirectoryIndex (none) found, and server-generated directory index forbidden by Options directive, referer: https://192.x.x.x/univention/portal/

Creating volume folders…
Creating hook folders…
[11.shibd.sh] check if ucs shibd service exists…

[ ? ] hwclock.sh*
[11.shibd.sh] no shibd found…
Waiting for MySQL…
services aren’t ready in 3m0s
Database didn’t come up in time!
Creating volume folders…
Creating hook folders…
[11.shibd.sh] check if ucs shibd service exists…
[ ? ] hwclock.sh*
[11.shibd.sh] no shibd found…
Waiting for MySQL…
services aren’t ready in 3m0s
Database didn’t come up in time!
Creating volume folders…
Creating hook folders…
[11.shibd.sh] check if ucs shibd service exists…
[ ? ] hwclock.sh*
[11.shibd.sh] no shibd found…
Waiting for MySQL…

MySQL service is running and I’m able to connect to the database.
All configurations (apache, owncloud, docker-compose, etc) are the same as on the old server.
I tried to disable the firewall but didn’t help.

I checked a lot of more things, but everything looks ok. I’m not able to catch what makes this problem.
Any suggestion on what can cause the problem?

Thank you!

rijadp · June 14, 2024, 2:38pm

Still looking for a solution…
The main reason I need to upgrade to a newer version is Apache version 2.4.38 which is recognized as vulnerable. So, I tested to upgrade only Apache to the newest version 2.4.59, and it works fine.
But… it seems that after that it’s not possible to upgrade the Univention portal at all. Still need to perform some more tests with this option.

Unfortunately, if I don’t find solution for main error of upgrading UCS, I will be forced to upgrade only Apache to be compliant with security requirements.

giuseppe.borrelli · July 1, 2024, 9:36am

Hi,
i have the same problem. after the update, owncloud no longer works.
the error message is:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server

the problem is very serious

rijadp · July 3, 2024, 1:00pm

Hi Giuseppe, did you find solution for your problem?

giuseppe.borrelli · July 3, 2024, 7:31pm

hello,
on Sunday I thought I had fixed it then I did the latest updates and it broke again.
it worked until today now if you update it doesn’t work anymore

giuseppe.borrelli · July 3, 2024, 7:34pm

they are all breaking down

giuseppe.borrelli · July 3, 2024, 7:40pm

I have now solved it.
I did it like this:

1 I recovered the virtual machine from a backup
2 I renamed the file tc.log
3 i upgraded to version 5.08

so it restarted but i have no exact procedure so i would say i got lucky.

i think it is all very unstable

rijadp · July 15, 2024, 12:11pm

Hi Giuseppe,
Yes, you were very lucky. I tried your solution,but didn’t work.
I’m still looking for a solution.
Can you remember, did you do anything else before or after the upgrade?

ruxley · July 15, 2024, 4:25pm

Hi rijadp, Giuseppe,

Now having the same problem here.

I administer a Univention/ownCloud server for a local charity and it has been ok for a few years.

4th Jun updated server to 5.0-7 errata 1060 without a reboot and ownCloud has been running fine

Last week a hardware issure forced a reboot
The VMs all run up ok but ownCloud gives proxy error and the docker container logs that it can’t contact the sql server

I made a snapshot of the VM and upgraded to 5.0-8 errata 1085 - owncloud running again

Reboot the VM and ownCloud broken

I’ve rolled back and patched again and can confirm ownCloud works after patching, until you reboot the server.

I currently have a working server (as I havn’t rebooted) but I’m sitting on a ticking time bomb.

If ANYONE can shed any light on what the hell is going on I would love to here from you.

giuseppe.borrelli · July 15, 2024, 9:03pm

i can confirm that the same thing is happening. i hope UCS will solve this serious problem

giuseppe.borrelli · July 16, 2024, 1:59pm

Hi,
I have good news!!!

I solved it. I edited the file /var/lib/nivention-appcenter/apps/owncloud/conf/config.php.

You need to add in the line ‘dbhost’ => ‘172.17.42.1:3306’, the number of the listening port (3306).

Then restart the UCS server.

Let me know.
Thank you

giuseppe.borrelli · July 16, 2024, 2:02pm

/var/lib/univention-appcenter/apps/owncloud/conf/config.php.

ruxley · July 19, 2024, 12:51pm

Thanks Guiseppe,

Just tried this and sadly it has not worked on my instance. Still “Waiting for MySql…”.

I’m going to keep my finger crosse we don’t have a powercut, before we migrate away to a new solution.

rijadp · July 23, 2024, 12:35pm

Hi guys,

Finally, I solve this problem. YEAAH!!!

Problem was related to the univention firewall where the rule for mysql is missing.

@ruxley Try this:

#iptables -L -v -n

Check if you have an INPUT entry for tcp:3306 dpt? If not, that’s the problem.

You need to go to: /etc/security/packetfilter.d
#nano 50_local.sh

In this script you need to configure this rule, so it will not be overwritten during the updates or restarts. This file will be called automatically by /etc/init.d/univention-firewall during system boot after default rules (defined by UCR) have been set.

At the end of the file enter this rule:
iptables --wait -A INPUT -p “tcp” --dport 3306 -j ACCEPT

Save and then do server reboot, and after server is UP, wait an about minute, and then try to open your owncloud app. I hope, you will see the magic…

My iptables looks like this:
iptables -L -v -n
Chain INPUT (policy DROP pkts bytes target prot opt in 894 220K ACCEPT all – lo 719 276K ACCEPT all – * 2 72 ACCEPT icmp – * * 33 1716 ACCEPT tcp – * 0 0 ACCEPT tcp – * * 2 128 ACCEPT tcp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT udp – * * 0 0 ACCEPT tcp – * * 0 0 ACCEPT tcp – * * 7 420 ACCEPT tcp – * 0 0 ACCEPT tcp – * 0 0 ACCEPT tcp – * 22 4310 REJECT all – * 0 packets, 0 bytes)
out source destination
* 0.0.0.0/0 0.0.0.0/0
* 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0.0.0.0/0 0.0.0.0/0
* 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0.0.0.0/0 0.0.0.0/0 tcp dpt:22
0.0.0.0/0 0.0.0.0/0 udp dpt:123
0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0.0.0.0/0 0.0.0.0/0 tcp dpt:7777
0.0.0.0/0 0.0.0.0/0 udp dpt:53
0.0.0.0/0 0.0.0.0/0 udp dpt:7777
0.0.0.0/0 0.0.0.0/0 tcp dpt:6669
0.0.0.0/0 0.0.0.0/0 tcp dpt:544
0.0.0.0/0 0.0.0.0/0 tcp dpt:464
0.0.0.0/0 0.0.0.0/0 tcp dpt:749
0.0.0.0/0 0.0.0.0/0 tcp dpt:88
0.0.0.0/0 0.0.0.0/0 udp dpt:464
0.0.0.0/0 0.0.0.0/0 udp dpt:88
0.0.0.0/0 0.0.0.0/0 tcp dpt:389
0.0.0.0/0 0.0.0.0/0 tcp dpt:636
0.0.0.0/0 0.0.0.0/0 tcp dpt:7389
0.0.0.0/0 0.0.0.0/0 tcp dpt:7636
0.0.0.0/0 0.0.0.0/0 tcp dpt:25
0.0.0.0/0 0.0.0.0/0 tcp dpt:465
0.0.0.0/0 0.0.0.0/0 tcp dpt:587
0.0.0.0/0 0.0.0.0/0 tcp dpt:5666
0.0.0.0/0 0.0.0.0/0 tcp dpt:111
0.0.0.0/0 0.0.0.0/0 tcp dpt:2049
0.0.0.0/0 0.0.0.0/0 tcp dpts:32765:32769
0.0.0.0/0 0.0.0.0/0 udp dpt:111
0.0.0.0/0 0.0.0.0/0 udp dpt:2049
0.0.0.0/0 0.0.0.0/0 udp dpts:32765:32769
0.0.0.0/0 0.0.0.0/0 tcp dpt:5432
0.0.0.0/0 0.0.0.0/0 tcp dpt:11212
* 172.17.0.0/16 0.0.0.0/0 tcp dpt:3306
* 172.16.0.0/16 0.0.0.0/0 tcp dpt:3306
* 0.0.0.0/0 0.0.0.0/0 tcp dpt:3306
* 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
100 8348 DOCKER-USER all – * * 0.0.0.0/0 0.0.0.0/0
100 8348 DOCKER-ISOLATION-STAGE-1 all – * * 0.0.0.0/0 0.0.0.0/0
100 8348 DOCKER-USER all – * * 0.0.0.0/0 0.0.0.0/0
100 8348 DOCKER-ISOLATION-STAGE-1 all – * * 0.0.0.0/0 0.0.0.0/0
93 7928 ACCEPT all – * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
7 420 DOCKER all – * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – docker0 !docker0 0.0.0.0/0 0.0.0.0/0
7 420 ACCEPT all – docker0 docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all – * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – docker0 docker0 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
894 220K ACCEPT all – * lo 0.0.0.0/0 0.0.0.0/0

Chain DOCKER-USER (2 references)
pkts bytes target prot opt in out source destination
200 16696 RETURN all – * * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0

Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp – !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:8080
0 0 ACCEPT tcp – !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:8080
0 0 ACCEPT tcp – !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:8080

Chain DOCKER-ISOLATION-STAGE-1 (2 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all – docker0 !docker0 0.0.0.0/0 0.0.0.0/0
200 16696 RETURN all – * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-2 all – docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all – * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all – * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all – * * 0.0.0.0/0 0.0.0.0/0

I hope this will solve your problem. Let me know.
Cheers!

ruxley · July 24, 2024, 10:01am

Hi @rijadp

You are a genius!

All working now. I hadn’t thought to check the firewall

Thank you so much for sharing the fix.

rijadp · July 24, 2024, 10:22am

@ruxley glad to hear that.

MichaelTYMcNamara · December 10, 2024, 5:08am

This was very helpful, thank you. Note that in the file 50_local.sh, the added text should be:
iptables --wait -A INPUT -p tcp --dport 3306 -j ACCEPT

Do not include quotes around the tcp word

Note that you can test this by running the command at the prompt in the UCS shell:

ssh root@ucs
Passwd:
root@ucs # 
root@ucs # iptables --wait -A INPUT -p tcp --dport 3306 -j ACCEPT

When I did this, the rule was added at the end of the ACCEPT section, after the final REJECT rule; which is not correct:

    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpts:32765:32769
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5432
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:11212
  548 36356 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:3306

To move the rule, run iptables-save to create a text file; edit that to move the rul ahead of the RJECT rule, and then restore the rules:

root@ucs# iptables-save > ll
root@ucs# vi ll
root@ucs# iptables-restore < ll

Now we see:

    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5432
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:11212
 1280 76800 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:3306
  163 14358 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable