Dumb Linux Question.

[swestrup]

Speaking of Linux and tools and stuff, I (vaguely) remember reading, years ago, about a method for setting up a spool directory such that any time a file was dropped into the directory, a daemon would immediately read the file and process it. I think there was also some sort of business involving named pipes, but I don't really recall. This may have been because the handling daemon was in another box than the spool directory -- IIRC this was part of a lan-wide printing setup.

Anyway, fast-forward to today, and I'm thinking of putting in 'Retrain' directories in all of the mail accounts on my mail server. Anytime the mail headers misclassify something as spam or nonspam, you simply drop the misclassified mail into the retrain directory, and it is automagically removed and trained as being in the opposite class as its headers suggest, and then re-delivered through the mail-filter system.

Now, clearly I can do all this with a cron task, but I want snappier responses. Ideally, there should only be a few seconds delay between dropping stuff into the retrain directory and getting it re-delivered correctly. Does anyone have any good ideas about how one goes about setting this up these days? If its any help, I have FAM up and running on the system (although I don't really know how it works.)

Comments

[skjalm.livejournal.com]

man 2 stat is what you want to look at :-P

As far as I've been able to figure out all daemons like LPD, MTAs and so on uses a loop to check the directory's modification time. Unlike files you (afaik) cannot make something like a "pipe file" and automatically start up a program when the file changes. I once made my mailsig like that, btw ;-)

So what you probably want is a daemon that has a loop that a) checks the directory and b) sleeps so your system can get on with its life.

I'm not sure if this one will work out of the box as I'm too tired to want to start compiling anything. But it's my guess at something that might work:

struct stat stat_buf;
int spooldir = open("/my/spool/dir",O_NONBLOCK|O_DIRECT|O_DIRECT);

if (-1 == spooldir) {
  fprintf(stderr,"Unable to open spool directory\n");
  exit(1);
}

while (1) {
  fstat(spooldir,&stat_buf);
  if (stat_buf.st_mtime != old_mtime) {
    old_mtime = stat_buf.st_mtime;
    do_your_spool_stuff();
  }

  sleep(1);
}

[swestrup]

Well, I've found famd and dnotify, both of which use a new kernel signalling method that lets you register a signal handler for individual file and/or directory changes. The problem is, there's a whole bunch of coding needed to take this from being a simple daemon to being what I want, which is a user-controllable daemon to handle calling of arbitrary programs upon file change, and doing so in a safe and sane manner.

Oh, well. I may just end up kludging something together in perl.

[sps.livejournal.com]

Actually you could write a new userland filesystem! It's be a hell of a heavy hammer to solve this problem, but not an unslick idea.

Writing a small at-like thing on one of the new access watcher facilities, as Sti suggests, is the better idea.

[skjalm.livejournal.com]

It'd be like shooting sparrows with cannons (bad translattion of a Danish proverb) but yes, I have to admit that I do like that idea.

Lazyness question: the access watcher things are 2.6 stuff, right?

And have either of you done any research into this? It would be nice to know what kind of overhead we're talking about compared to the overhead in having a daemon process running and sleeping a little ;-)

[swestrup]

I don't know if the kernel file watcher is only 2.6 or what, but there is a file called dnotify.txt in the kernel docs that explains (roughly) how the mechanism works. Since I'm only going to be monitoring a few dozen directories, at most, it didn't seem relevant what the overhead was.

I'm more concerned with ease of use. Ideally, each user would simply have an entry in a .changerc in their home directory that reads:

/home/$USER/.Maildir/retrain : spam-retrain $USER $1

or something vaguely in that style. That way they can remove the feature if they want, or use the facility for their own projects. Plus they could pre-process the spam for retraining if they want.

[sps.livejournal.com]

I'm actually very concerned by the overhead I've seen from software that uses these new facilities; as far as I can tell, they must be doing something idiotic like delivering a signal for each change observed (which will more than double the cost of any file system operations that are watched, of course). As with so many things in Linux, a more classically Unix-y design would be better - say a character device from which you can read an event stream in chunks, so that it can batch up sizeable backlogs. But it really doesn't look like they've done that.

So as you can tell from the above, no, I haven't looked into it yet, but I keep meaning to at almost any minute! ;-}

[skjalm.livejournal.com]

they must be doing something idiotic like delivering a signal for each change observed

That's what I was concerned about :-/

but I keep meaning to at almost any minute! ;-}

Hey, surprise! I feel the exact same way... about a great many things ;-)