The answer of @bvs pointed in exactly the right direction of the problem and solved it for me. Here, are some details on the precise commands. As for many the default interface is
eth0, I am taking that in the following commands, which has to replaced by
eth2 for the original question.
For a temporary solution (until reboot) use
sudo sysctl -w net.ipv4.conf.eth0.promote_secondaries=1
for a permanent solution which persists after reboots create a file
/etc/sysctl.d/ip-promote.conf (or other file name ending with
.conf) with the following content:
# The following makes that a second IP on the eth0 interface
# is promoted as primary IP instead of being removed.