[SOLVED] How to delete files in clientmqueue?

We have a VPS server running Magento flawlessly for nearly a year. The filesystem has been completely filled with mails queued in clientmqueue, and MySQL can't add more data causing the website to halt. Sendmail was running fine, but we've had difficult time deleting files residing in /var/spool/clientmqueue folder. The following commands failed:

# cd /var/spool/clientmqueue
# rm -rf *
-bash: /bin/rm: Argument list too long
# foreach i in {a..z}; do rm -f ${i}*; done
-bash: /bin/rm: Argument list too long
# find . -type f exec rm -f {} \;
find: cannot fork: Cannot allocate memory

There are just too many files in the clientmqueue folder, and Linux couldn't handle the number of files for deletion. Our rescue was xargs command. But, before we do this we wanted to make sure the queue isn't building up as we delete the files.

# service sendmail stop
# cd /var/spool
# mv clientmqueue clientmqueue.x
# cd clientmqueue.x
# find . -type f |xargs rm -f

It took more than a day to complete the execution of above command, but it was deleting files slowly but surely freeing up space on the hard disk.

Removing those files is a half of the problem, and identifying the process creating those files and preventing this from happening in the future is the another problem. So, how do we find the process which is writing those files in clientmqueue. The lsof command will help us identify the process.

# lsof +D /var/spool/clientmqueue
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
sendmail 14378 smmsp  cwd    DIR  202,0     4096 163992 clientmqueue

The clientmqueue is the mail queue for handling locally generated outbound emails. If you have sendmail running, and seeing this problem then it indicates you may have a compromised website spamming the world using your server. Your compromised website (or server) is generating a lot more mail than the sendmail can handle causing clientmqueue to fill up. You will have to view the df* and qf* files residing in the /var/spool/clientmqueue to see where these messages are come from, and resolve root cause of the problem.