Gotcha of the Day: Apache refuses to serve up static files

I've got a web app running on a new hosting provider, and for the most part, everything is running smooth. The app is built around a custom dispatcher, so of course, there's a few lines in the .htaccess that look like so:

 RewriteRule ^index\.php$ - [L]
 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteRule . /index.php [L]

Nothing exotic there. And the dispatcher is working great. However, I've got a few cron jobs that perform some maintenance by kicking off a curl command against a site's URL. Something like this:

 */15 * * * * curl -s http://theapp.com/maintenance/cleanup.php

In this case, I get an error about cleanup.php not being found.

My first thought is that cleanup.php didn't get relocated to the new server, but sure enough, it's there.

The I dig a little deeper and realize that the 404 is being issued by my dispatcher, not by Apache. This means that even though the file maintenance/cleanup.php exists on disk, the RewriteCond -f isn't finding it. If it was, it would have skipped my dispatcher.

But how could that be? I double check the permissions on the files and directory, everything looks good.

It must be some RewriteRule madness or something on my part, so I go as far as moving .htaccess out of the way. There's no dispatcher to be had. Now I get a new error:

  HTTP/1.1 406 Not Acceptable

Huh? Now I'm thinking that the web server I've installed on has some sort of serious misconfiguration. I write up a long winded ticket and send it off. But, just in case it's an easy fix, I jump on the live chat offered by the hosting provider.

I plead my case: the host isn't serving up a static file when I ask for it from the command line.

A moment later, the tech responds: it's because you're not sending a User-Agent.

The snarky version of me wants to type back something about how absurd that response is. Of course I'm sending a User-Agent, I just happen to be sending one mentioning curl in it instead of IE or Firefox. Silly rookie.

And then it hits me, and I forget about being snarky. I ask: is there a module installed in Apache that would be rejecting requests that look like bots?

His answer: yes, mod_sec. So that explains it. mod_sec, some sort of security module, is blocking requests that look sketchy, including curl requests. Seems a bit overboard to me, as I use curl all the time, but what the heck do I know.

I was tempted to have him turn off mod_sec, but seeing as I really don't know what I'm asking him to do, I decide better of it. Instead I just updated my cron to say:

 */15 * * * * curl  -A "Mozilla/5.0 (compatible; MSIE 7.01; Windows NT 5.0)" \
     -s http://theapp.com/maintenance/cleanup.php

Which of course, is exactly what any hacker would do, but that's besides the point.

Regardless, It's fixed, and I'm happy.