I ran into a rather interesting situation today with a client. It seems that the mysqld daemon stopped with no errors in the error log. I ran through the obvious problems … not enough disk space, memory utilization etc and came up empty.
The server was running MySQL 4.1 on Fedora Core 5. We can save the discussion about running your database on reasonable up to date hardware and operating system for another post. Core 5 runs the GNU/Linux kernel 2.4 along with the ext3 filesystem and so the thought was in the back of my mind that it might be an issue with file size. Well, as Sun’s own documentation shows this shouldn’t be the case.
During the investigation it was uncovered that the general query log was not only enabled but 16 gigabytes in size. Aside from being so large it was absolutely useless for anything, it was the obvious culprit for too large a file. After zapping the log file, it was possible to start the MySQL server successfully. It was then that I looked up the previously referenced information about file size limits in the kernel and MySQL itself and found out that 16 GB shouldn’t have been a problem.
Checking through bugs.mysql.com didn’t uncover anything. I have already thought of how I can reproduce the problem pretty painlessly if it’s a general problem and not something specific to Fedora Core 5. But before I take several hours to run tests I wanted to see if anyone else had heard of this issue.
The rather obvious lessons that the client should take away from this:
1) There was no reason to have the general query log on in the first place and it should have been turned off.
2) If you need to use either slow or general query logging taking the time to set up log rotation.
Ever seen this before? If so please take 30 seconds and let me know. I would really appreciate it. And who knows, it could be one of your servers down the road that could be saved if it really is a bug. Of course you would never let your log files get that large, right? Right?
Haven’t seen it before… but with kernel, filesystem and MySQL version combination, not at all surprised.
Yeah, I agree. Pretty much it’s a pile of poo. But if no one else has seen it in the flesh I am going to do a bit of testing on later kernels and server versions.
have you had a look at the fs block size? most administrators won’t change it, but ext3 with 1k blocks limits files to 16GiB.
MySQL 4.1 is no longer supported I believe. It’s very old.
Download and rebuild the latest MySQL rpm for Fedora which is:
sunsite.rediris.es/mirror/fedora-redhat/releases/11/Fedora/source/SRPMS/mysql-5.1.32-1.fc11.src.rpm
If this version gives this problem then please file a bug report.
Complaining about old versions of the software isn’t always very helpful.
Note: Fedora and RedHat are often sometimes quite slow at updating to the latest releases of software. That can be quite frustrating. In this case you have a way of getting a newer version so I’d suggest you try that. You also get new features and better performance which may help you.
Thanks for the feedback everyone. I will definitely check the block size. Simon, while I appreciate the feedback, there are often cases where customers have to stay on the version that they are on. Yes, 4.1 is unsupported. Heck 5.0 won’t be supported much longer (maybe another year if I recall), but that doesn’t mean the problem shouldn’t be investigated. It could easily affect multiple versions – not just 4.1.
Would I recommend a customer upgrade from 4.1 to another version? In a heartbeat. Does it always happen? Nope. Not even close. If it makes you feel any better I have clients running still running 3.23.
Oh, and in case you didn’t know…it’s a REALLY bad idea to upgrade from 4.1 to 5.1 without going to 5.0 first. Sun recommends, with good reason, that you upgrade from one major version to the next major version without skipping .. so 4.1 to 5.0 and then to 5.1.
I will do some testing in the next few weeks and see what I come up with on more current setups. While this isn’t a “oh crap” problem in most cases, it could still potentially be a bug that is affecting multiple major version of the server. If it is, it needs to be logged so it can be fixed at some point.