seorock, I wasn't dissin' linux. I put "Linux:" in the front of the title as that's what this post is about. I see now that it looks like I'm calling Linux the timebomb. Not at all. Unintentional titlebait.
Jan, yes, thanks. Most of my servers have CPanel which does monthly log rotation (which isn't enough in some cases). And, I find that a daily logrotate will cause some issues with awstats and maybe clean up too early.
In any case, the one that bit me this time is not one of the standard web server logs. It was something I set up myself (of course).
seorock, I wasn't dissin' linux. I put "Linux:" in the front of the title as that's what this post is about. I see now that it looks like I'm calling Linux the timebomb. Not at all. Unintentional titlebait.
Jan, yes, thanks. Most of my servers have CPanel which does monthly log rotation (which isn't enough in some cases). And, I find that a daily logrotate will cause some issues with awstats and maybe clean up too early.
In any case, the one that bit me this time is not one of the standard web server logs. It was something I set up myself (of course).
This isn't a Linux problem, it's a filesystem problem. If you're running a FAT32 filesystem on Linux (eeiiiww!) you'll have this limitation. If you're running over NFS, as many shared hosting providers do, you'll have a similar limitation.
Linux hasn't had a 2GB filesize limit since before the 2.2 kernel (pre 1999). With 2.2 and the ext2 filesystem, which was the standard filesystem back then, the limit is up near 1TB. ext2 now supports files up to 32 TB in size. Most default filesystems in use today support well beyond that.
I currently have a debian system that runs a 8GB database for me. One of the mysql DB files is 6 GB in size. (Debian Etch w/ext3 filesystem - nothing special about that configuration)
I've run into a similar issue a couple of times w/ my shared hosting plan. My web site is very "light" when it comes to storage space (mostly text), but I've blown through my storage limits a couple of times b/c the damn log file got too big...luckily my host warned me instead of just knocking me offline. Now I set a reminder to regularly delete the log archives b/f they get too big.
This isn't a Linux problem, it's a filesystem problem. If you're running a FAT32 filesystem on Linux (eeiiiww!) you'll have this limitation. If you're running over NFS, as many shared hosting providers do, you'll have a similar limitation.
Linux hasn't had a 2GB filesize limit since before the 2.2 kernel (pre 1999). With 2.2 and the ext2 filesystem, which was the standard filesystem back then, the limit is up near 1TB. ext2 now supports files up to 32 TB in size. Most default filesystems in use today support well beyond that.
I currently have a debian system that runs a 8GB database for me. One of the mysql DB files is 6 GB in size. (Debian Etch w/ext3 filesystem - nothing special about that configuration)
I've run into a similar issue a couple of times w/ my shared hosting plan. My web site is very "light" when it comes to storage space (mostly text), but I've blown through my storage limits a couple of times b/c the damn log file got too big...luckily my host warned me instead of just knocking me offline. Now I set a reminder to regularly delete the log archives b/f they get too big.
AltJ, thanks for the clarification. My knowledge on Linux may be as old as that kernel that supports only 2G file size. I'll happily stand corrected to get to the real issues.
It is also a relief to know that my dedicated servers will hopefully not have these iseues which are running more recent software.
The above problem did happen on a small site running on a shared host. I have no idea what versions of kernel and apache they're running.
I do have a hard time believing that Apache cares how big a file is. Aren't they just appending a text file? Why would the size be a factor if it isn't the OS tripping on it?
I've seen cases where Apache can't serve up files larger than 2GB. The more I think about it, the more I think that same limit does not apply to logfiles. (logrotate keeps my logfiles well below 2GB, so I've never tested this limit.)
Since you are on a shared host in this situation, I'll bet that the root cause of this problem was an NFS limitation. I would think that a hosting provider would have alerted you when such an error occurs.
On a side note, one way to detect this and many other problems would be to have a custom error page setup for the common
HTTP error codes. This page could be scripted to e-mail you an alert when something "bad" happens. I haven't done this in the past but will on future sites I create, thanks to your timebomb blog entry.
I've seen cases where Apache can't serve up files larger than 2GB. The more I think about it, the more I think that same limit does not apply to logfiles. (logrotate keeps my logfiles well below 2GB, so I've never tested this limit.)
Since you are on a shared host in this situation, I'll bet that the root cause of this problem was an NFS limitation. I would think that a hosting provider would have alerted you when such an error occurs.
On a side note, one way to detect this and many other problems would be to have a custom error page setup for the common
HTTP error codes. This page could be scripted to e-mail you an alert when something "bad" happens. I haven't done this in the past but will on future sites I create, thanks to your timebomb blog entry.
I've run into a similar issue a couple of times w/ my shared hosting plan. My web site is very "light" when it comes to storage space (mostly text), but I've blown through my storage limits a couple of times b/c the damn log file got too big...luckily my host warned me instead of just knocking me offline. Now I set a reminder to regularly delete the log archives b/f they get too big.
© 2012 http://www.jangro.com/
RSS
This has happened to me at least twice now. It's a sure sign that you've got too many websites when you can't remember them all, never mind visit them daily.
I checked out a site today that I haven't been to in a few weeks, only to be greeted by server errors. That's the worst feeling in the world, especially on a Friday night. Even more so on a site you haven't looked at in many days.
I hit the typical places, error logs, optimized the databases, etc. etc. Nothing, and then...
Dawn Breaks on Marblehead
I'm sure that most people don't know that there's a 2GB file limit in 32-bit versions of Linux. This includes most servers running Linux. And even if you do know this, it's probably the last thing you'll think of when things go wrong.
If you've got a website that creates growing files, typically log files, that do not periodically get cleaned, you've got a timebomb on your hands. It may take a few months, or a few years, but eventually a file that the server is trying to write to may reach 2GB in size.
When it happens, it's probably the least helpful message you can get from a web server -- a "500 Server Error".
The error logs generally aren't much help either and contain errors like:
Premature end of script headersIf you do find yourself with this mysterious problem all of a sudden, especially with no other changes being made, look for a super-large log file being the culprit, specifically one that is this size:
$ ls -l . . . 2147483647 Jun 6 17:16 error_logSniffing out Timebombs
If you have shell access to your server, you can use the 'find' command to locate any potentially large and dangerous files, in this case, files that are at least 100MB.
$ find . -size +expr 100 * 1024 * 2-exec ls -l {} ;That command will search recursively from the current directory. So you may want to try it from different locations. To search log files from the server's root, you may need to be the root user.
And if you're doing the math thinking that doesn't add up to 100MB, that's because the find command's size parameter works in 512 bytes. So that's 100 * 1024 * 2 * 512, which is 100MB.
Even if the files you find aren't created by your webserver, it might be a good idea to find some of these and delete them if possible as they're eating up your disk space.