Linux / Sendmail one liner having trouble with large directories

Posted on

Linux / Sendmail one liner having trouble with large directories – Managing your servers can streamline the performance of your team by allowing them to complete complex tasks faster. Plus, it can enable them to detect problems early on before they get out of hand and compromise your business. As a result, the risk of experiencing operational setbacks is drastically lower.

But the only way to make the most of your server management is to perform it correctly. And to help you do so, this article will share nine tips on improving your server management and fix some problem about linux, sendmail, bash, scripting, .

For whatever reason, spammers found a way to relay mail through a side-system. The exploit has already been addressed.

The problem is that I had massive amount of emails in my /var/spool/mqueue directory (at least 100,000+) that I still need to filter through. I stopped sendmail and moved the contents of the mqueue directory to a new location…

Since then, I’ve been trying to use the following one liner to help:

for x in `find . -type f -name “qf*” | xargs grep -l "foo" | cut -b3-`; do y=d`echo $x | cut -b2-`; mv $x /root/spammessages; mv $y /root/spammessages/; done

The idea being to:

1) check QF files for unique spam-related header content (foo).

2) Find the DF counterpart file

3) Move both df and qf files to a isolated area.

The problem is that the query is running but doesn’t look to be moving any files. If I run top, I see xargs and grep occasionally using some resources but never more than 1%-2%. Furthermore… when I check the spammessages folder, I don’t see any files there.

If I run the same command on a smaller sub-set of messages, it seems to work fine. Are there some linux file limitations here? Are there ways to optimize the one-liner?

Thanks.

-M

Solution :

Let’s break it out into multiple lines

for x in `find . -type f -name "qf*" | xargs grep -l "foo" | cut -b3-`
do 
    y=d`echo $x | cut -b2-`
    mv $x /root/spammessages
    mv $y /root/spammessages/
done
  • You should always use -print0 with find and -0 with xargs when you are piping filenames from one to the other
  • Use $() instead of backticks for readability and the ability to nest and reduce the need for escapes
  • Always quote variable names that hold filenames
  • Calling an external (cut) many times in a loop is slow (I’ve eliminated it as it was not needed anyway)
  • You have a terminal slash on one directory but not the other (consistency)
  • You’re cutting off the “qf” when you make the variable x, but you don’t put it back when you do the move.
  • You’re cutting off “qf” but not putting back “f” when you make the variable y (which you’re cutting off an additional character from x)
  • Also, you had typographical quotes (smart or Unicode or whatever you want to call them) around your filespec in find which may have interfered

Try this:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

Put back into a one-liner:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-); do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

Edit:

Here’s a version using a while loop which may work better for very large numbers of files:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- |
while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

In a one-liner:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- | while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

A variation that uses Bash’s process substitution:

while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)

And:

while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)

Leave a Reply

Your email address will not be published. Required fields are marked *