A malware on GPU server

My team does not realize a malware in our system, we just assumed someone is working hard on the shared GPU system. Because we are not frequently checking host resource due to container or web interface oriented working process. however, it turns out one of Coinminer- Multios.Coinminer.Miner - live in hiding on the system.

This page describes how to handle this malware to help someone's time for the similar situation. if you are interrest in more about malware in linux, this wiki page may helpful for better understanding on the background.

Symptom

The malware uses 100% of CPU with very lower priority CPU slice, so other task can work even this malware is running behind.

Because our system is being used by multiple people and running so many processes all the time, we did not realize the malware until we found a strange process like

As the screen show named "/4a9e003a", PID 8304 uses 11195 CPU utilization.

Trace the Process

To find the program of that PID, we use systemctl status <pid> then found suspicious service name at /lib/systemd/system/yayayaf48a2a0e.service with vendor preset enabled so that it can start up from the begining of boot.

$systemctl status 8304
● yayayaf48a2a0e.service - Server Service
     Loaded: loaded (/lib/systemd/system/yayayaf48a2a0e.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-05-26 12:10:33 KST; 3min 33s ago
    Process: 8235 ExecStart=/usr/lib/x86_64-linux-gnu/yayaya39908e0f (code=exited, status=0/SUCCESS)
   Main PID: 8304 (4a9e003a)
      Tasks: 134 (limit: 154406)
     Memory: 60.1M
     CGroup: /system.slice/yayayaf48a2a0e.service
             └─8304 /4a9e003a

Remove the malware

To get rid of the malware with following steps.

stopping the process itself does not helpful, cause the systemd based script will resume their job several minute later.

1. Kill the process
#kill 8304 

2. Remove the service
#systemctl disable yayayaf48a2a0e.service
#rm -rf /lib/systemd/system/yayayaf48a2a0e.service

3. Find similar name in the system
#find / -name yayaya*
/etc/sysconfig/yayaya
/sys/fs/cgroup/pids/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/memory/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/devices/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/blkio/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/cpu,cpuacct/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/systemd/system.slice/yayayaf48a2a0e.service
/sys/fs/cgroup/unified/system.slice/yayayaf48a2a0e.service
/usr/lib/x86_64-linux-gnu/yayaya39908e0f

4. remove all related file or directory 
#find . -name yaya*  | xargs sudo rmdir

Finally we removed all infected files and related things of the Multios.Coinminer.Miner which usually skulk at /etc/sysconfig/yayaya. We do not found how the system has been infected yet.