FTP doesn’t work on server using AFP

Although you should NOT be using FTP, if you need to for some reason, like we did on one server, and some people can't upload files, this might be the issue. We run APF firewall on some servers. A couple of servers have either ProFTP or Pure-FTP installed. Over the last couple of weeks we discovered that some people could not read or write with their FTP client when ftping into our servers. If you experience this, check that MONOKERN is off. SET_MONOKERN="0" This is set in /etc/apf/conf.apf Good luck!

Debugging MailScanner: cleaning messages

Last week I did a short post on testing MailScanner to be sure it was scanning properly. An alert reader pointed out that I missed the easy solution, which was to simply use ‘MailScanner—lint’ to check if MailScanner was running properly.

Since I wrote that article, I found that my MailScanner was again hogging the CPU. Last time it was an issue with f-prot interaction, this time it is a problem apparently with MailScanner.

Here is what I did to debug this issue, and here is the post that helped me learn more about MailScanner.

First thing was to check top:
7078 root     25   0 28936 23m 1960 R   99 0.6   3:19.60 MailScanner

When I did a strace -p 7078 I saw nothing. Literally. So that was not much help. Next up, see what ps has for me:
ps -ax | grep MailScanner
[07:40:59 root@secure site1]# ps ax | grep MailScanner
7077 ?      Ss   0:00 MailScanner: master waiting for children, sleeping
7078 ?      R   14:43 MailScanner: cleaning messages
7085 ?      S     0:00 MailScanner: waiting for messages
7123 ?      S     0:00 MailScanner: waiting for messages
7127 ?      S     0:00 MailScanner: waiting for messages
7170 ?      S     0:00 MailScanner: waiting for messages

There is my stuck process, and apparently it is stuck cleaning messages. But which messages? The MailScanner working directory has more than one email in it, and to be picky I would prefer not to delete a valid email. Next step to be thorough is to do an strace on MailScanner itself and see what it is up to. You will see a lot of output, and then finally you should see a list of PID’s coupled with the email that is stuck:

strace -f /usr/sbin/MailScanner
[pid 24167] open(”/home/virtual/FILESYSTEMTEMPLATE/services/sendmail/mqueue/qfmBJAmQ7J011131”, O_RDWR|O_LARGEFILE) = 7
[pid 24167] ioctl(7, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfddf558) = -1 ENOTTY (Inappropriate ioctl for device)
[pid 24167] _llseek(7, 0, [0], SEEK_CUR) = 0
[pid 24167] fstat64(7, {st_mode=S_IFREG|0600, st_size=1182, ...}) = 0
[pid 24167] fcntl64(7, F_SETFD, FD_CLOEXEC) = 0
[pid 24167] fcntl64(7, F_SETLK64, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}, 0x99a50a8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 24167] close(7)              = 0
[pid 24167] open(”/home/virtual/FILESYSTEMTEMPLATE/services/sendmail/mqueue/qfmBJE3lHG021993”, O_RDWR|O_LARGEFILE) = 7

Note the different PID on these files that seem to cause MailScanner to loop indefinitely: [pid 24167]. Now we can run a trace on that PID:

strace -p 24167
open(”/home/virtual/FILESYSTEMTEMPLATE/services/sendmail/mqueue/qfmBJAmQ7J011131”, O_RDWR|O_LARGEFILE) = 7
ioctl(7, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfddf558) = -1 ENOTTY (Inappropriate ioctl for device)
_llseek(7, 0, [0], SEEK_CUR)        = 0
fstat64(7, {st_mode=S_IFREG|0600, st_size=1182, ...}) = 0
fcntl64(7, F_SETFD, FD_CLOEXEC)      = 0
fcntl64(7, F_SETLK64, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}, 0x99a50a8) = -1 EAGAIN (Resource temporarily unavailable)
close(7)                      = 0
open(”/home/virtual/FILESYSTEMTEMPLATE/services/sendmail/mqueue/qfmBJE3lHG021993”, O_RDWR|O_LARGEFILE) = 7

Now we will get more looping text, but this time it will show the emails that MailScanner is stuck on:

While MailScanner is still running, you can go to the MailScanner working directory and see a list of folders named after the MailScanner PID’s:
cd /home/virtual/FILESYSTEMTEMPLATE/services/mailscanner/MailScanner.work
[08:53:51 root@secure MailScanner.work]# ls -la
total 28
drwx———7 root root 4096 Dec 19 08:13 .
drwxr-xr-x 4 root root 4096 Mar 20 2008 ..
drwx———2 root root 4096 Dec 19 08:39 24167
drwx———2 root root 4096 Dec 19 08:46 26134
drwx———2 root root 4096 Dec 19 08:29 26135
drwx———2 root root 4096 Dec 19 08:46 26136
drwx———2 root root 4096 Dec 19 08:47 26137

Go into the problem PID’s directory and you will see a list of files that make up the email that is stuck. This might help you determine why the email is stuck. In my case, there was a virus attached to an email, which, like the this post pointed out, was also causing MailScanner to loop.

Yes, you can easily just go to /home/virtual/FILESYSTEMTEMPLATE/services/sendmail/mqueue/ and delete all the emails in the queue and restart MailScanner, but I wanted to be sure that one of the emails was really the problem.

This little how-to does not fix the real issue of MailScanner getting stuck in an endless loop, but merely solves the current issue of MailScanner hogging the CPU. We can fully expect the issue to rear its ugly little head again at some point. Hopefully by that time, we will have solved the root issue.

Test your servers Virus filters - don’t assume

On one of our RHEL5 servers running Ensim, we noticed after the last big update of RHEL using ‘yum update’, that MailScanner was capped out, using tons of our CPU power. In the past when this happens I usually find it is ClamAV acting up. However, we were using on this server a free version of f-prot. I took a chance and purchased the latest greatest mail server version of f-prot and installed it. Bingo, the problem went away.

Why f-prot? It works and it is easier on the server. I have noticed that the server load with f-prot is noticeably lower than when using ClamAV.

Once the new version of f-prot was properly installed, I wanted to test that it worked as it should. A friend suggested to test the virus scanner I should put it to a real-world test using one of Declude.com‘s free tools.

This test would actually send an email to my server with a file that was made to get caught be anti-virus filters.

The test worked well, too bad my anti-virus f-prot filter didn’t. After a quick check I realized I had not modified one path in my MailScanner rules file to point to the new f-prot location. After I fixed that, I ran the test again, and sure enough, my anti-virus filter was working as it should.

Lesson learnt? Never assume your new software works, test it!

Yum Lock table is out of available locker entries

Today while running a yum update on one RHEL5 server, I got the following error and python traceback:

yum update
Loading “security” plugin
Loading “rhnplugin” plugin
rhel-i386-server-5     100% |=========================| 1.2 kB   00:00  
primary.xml.gz         100% |=========================| 1.6 MB   00:01  
rhel-i386-: ################################################## 4462/4462
Skipping security plugin, no data
Setting up Update Process
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 75ba
error: db4 error(22) from dbenv->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File “/usr/bin/yum”, line 29, in ?
  File “/usr/share/yum-cli/yummain.py”, line 105, in main
  result, resultmsgs = base.doCommands()
  File “/usr/share/yum-cli/cli.py”, line 293, in doCommands
  return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File “/usr/share/yum-cli/yumcommands.py”, line 163, in doCommand
  return base.updatePkgs(extcmds)
  File “/usr/share/yum-cli/cli.py”, line 600, in updatePkgs
  installed = self.rpmdb.simplePkgList()
  File “/usr/lib/python2.4/site-packages/yum/rpmsack.py”, line 158, in simplePkgList
  return self.pkglist
  File “/usr/lib/python2.4/site-packages/yum/rpmsack.py”, line 59, in _get_pkglist
  File “/usr/lib/python2.4/site-packages/yum/rpmsack.py”, line 235, in _make_header_dict
  for (hdr, idx) in self._all_packages():
  File “/usr/lib/python2.4/site-packages/yum/rpmsack.py”, line 206, in _all_packages
  mi = ts.dbMatch()
TypeError: rpmdb open failed

As scary as all that looks, it was easy to find the solution. Apparently this is caused by a well known rpm DB lock. First ensure that no instances of yum or rpm are running. You can do this by issuing the command ‘ps ax | grep yum’ and ‘os ax | grep rpm’. If either of these show an instance of yum or running, kill it off or wait for it to finish. Then as super user, following these tho instructions:

rm /var/lib/rpm/__db.*

That is it. You should be able to run ‘yum clean all’ and ‘yum update’. Since we use MailScanner, I had to then contend with the annoying conflict with the MailScanner perl packages. But… that is another story.

Page 2 of 4 pages  < 1 2 3 4 >