Adding/removing portal entries in Portal settings doesn't have any effect on the portal page entries

I was trying to change the portal page links to list all my systems, however, adding portal entries doesn’t seem to add them to the portal page. Also, removing entries (like the Univention Blog) doesn’t seem to have any effect. I also tried to use the yellow edit button, but there was no effect either. How does one clear the caches or whatever is causing this?