There's the possibility your system is running in an inconsistent status.
I would start by doing some basic diagnostics.
Check the UDP socket and its process.
sudo netstat -lunp or
sudo ss -lunp to see whether that UDP socket (on port 7123) is busy.
sudo is needed for a normal user to escalate privileges and see the PID and the process name who's "listening" on that port. Without
sudo there will be a
- instead of the PID and process name. Take note of the the PID. In case there's none listed, then you can start thinking your system has been compromised as the process is capable of hiding itslef by manipulating the internal kernel structures for the processes!
If you cannot see the PID with high privileges, than you'd better to insulate that server as it's probably been compromised and hacked.
Check whether the listening process is also "talking"
I suggest to use nmap or the more general netcat tool. With
nmap you can try this command:
sudo nmap -sV -v -Pn -sU -p U:7123 220.127.116.11
It tries to understand the protocol spoken at port UDP:7123.
sudo is required for the specific scan type (UDP service scan). If you see data that don't match what you are expecting, than probably either the process is gone wild or is in an inconsistent status. Or it's been replaced by some other (malicious?) code.
netcat you need some manual intervention to generate traffic to the server:
netcat -u 18.104.22.168 7123
Whatever you'll type will be sent to your application on port UDP:7123.
You can also pipe some random data (but don't expect any meaningful result) with:
cat /dev/urandom | netcat -u 22.214.171.124 7123
Check the system and application logs (if any)
The former are usually stored at
/var/log/ directory. Application log could be anywhere else. The tool
lsof can be of help if you check the PID on its 2nd column. I'd keep an eye on these logs during the subsequent actions.
Check the process table for that PID
ps can provide for a lot of details about processes. My personal favorite is:
PS_FORMAT="ruser,pid,ppid,s,%cpu,rss,cmd" ps ax --sort=pid
You can see on the second column the PID you are looking for, along with the real user ID (ruser, 1st column), the parent PID (ppid, 3rd column), the status (s, 4th), the % CPU usage (%cpu, 5th), the resident set size (rss, 6th) and the command line with arguments (cmd, 7th). In my opinion, for this very case, the process status (it's a single letter) and the percentage of used CPU are the key values, along with the command line.
Check on the man page for all the details for
ps and to fine tune the output.
Check the binaries
If you have another machine with the same architecture and OS and the same process running in an expected way, you can check whether the binaries match byte-by-byte. In case they don't, you'd better reinstall those binaries from a known secure source.
Let's assume the program is
Calculate and take note of its checksum. Something like this:
If the program has been compiled to use dynamic libraries, then you'd also check them. The list of used dynamic libraries is obtained with:
Be warned: the output can be quite lengthy, but for each line you'd calculate and take note of the checksum for comparison.
In case any discrepancy with the reference trusted system is found, I'd suggest for a re-installation of the whole system and applications. It's a radical approach but I think that the target system cannot be trusted any more.
Kill and restart the process
I would try to kill the rogue process (provided that you know its PID) and restart it to check whether the rogue behavior sticks. In order to kill a process given its PID you can run:
sudo kill -s SIGKILL <PID>
I need to warn you that there is a number of cases in which a process won't die or will die later than expected. It depends mostly upon the process status shown by the
ps command run earlier. The referenced man page reports some information about the process status column at the "PROCESS STATE CODES" paragraph.
Use the command
ps to check again the process status. It should normally be kicked out within a few seconds and disappear from the process list.
Once the process has been killed you can try to restart it and see whether the rogue behavior repeats.
As a general rule of thumb, if you don't change anything to the system, the behavior will hardly change as well. That is, just rebooting the application or the system will hardly solve the problem: it will just be pushed forward in time.