Paying to get my server fixed

[Trikkur]

I have my sites on a dedicated server and recently it keeps crashing. I've contacted the host and they said there are out_of_memory issues and that I should upgrade my RAM. I don't really believe that just more RAM is going to do anything though because I've made almost no changes to the websites in the last month or two and the server has been fine.

After I reboot the server, everything will be working fine for a few hours and then they all crash again. This has happened about 5 times now. I don't know if a script on one of my sites has gone haywire, or if a virus is causing the issue, but I can't fix it. If there is anyone here that has some server experience I would be willing to pay for their time. I assume it should be 30 mins to an hour of work at the most.

If anyone here is interested, hit me up through PM. I could really use the help, all of my sites have been down this week and it sucks hard.

[CasinoMack]

If could be as simple as tuning your mysql. Incorrectly coded php loops can cause this.

People hammering your server can cause this as well.

Have you run "top" in the shell?

[kushmoney]

If you use wordpress on any of your websites. Upgrade them asap. There was one wordpress release that crash your site every time.

[Jeremy]

Dealing with hosting companies is one of the things I dread most as an internet marketer, especially with forums. You're paying for a dedicated, make them dig and dig til they find out what the problem is. I have always held these companies feet to the fire when I pay for a dedicated server and sites just all of the sudden just "break".

Frustrating for sure. Let me know if there is anything I can do for you. If you want to upload a site on my server to a random non indexed I.P., just to see if it performs there versus your current host, just let me know. I can create a quick account.

[Gydian]

Definitely run 'top' in shell and go from there. Could be low swap. Could be faulty memory or low memory. Could be issues with your OS (like I had that was causing kernel failure)

But yeah, if you hop into shell as root and run top, take a screen and post the output here and I'm sure one of us can at least point you in the right direction.

[GoodKarmaKid]

But yeah, if you hop into shell as root and run top, take a screen and post the output here and I'm sure one of us can at least point you in the right direction.

Is there an "Idiots Guide" to server Management?? I have a dedicated server and should know how to do all this stuff but I don't, I just poke around in the WHM control panel, and cross a finger or two when changing a setting.

I know there are tons of resources on the internet, I've googled the subject a few times, but most of the sites are way to technical and lose beginners/intermediates right away.

Thanks,
Jason

[Trikkur]

I have no clue how to run "top" or anything like that. Can I do that through Plesk? If this is going to be a pretty quick fix, I will pay someone $20 to get this working or at least tell me I need to buy more RAM. My entire network has been down for a week and I need to get this running ASAP.

[Trikkur]

BTW, Here is some of the conversations I've had with support. Maybe the reports they are referencing can explain something. These have happened over the last week. They will reboot, say it's running fine, then it crashes in a few hours. Same thing has happened 5-10 times now. I really don't know why I would need more RAM though. It feels like something else is causing the problem imo.

Hi,

When we checked the server through IPMI we could find that the text console and KVM were showing blank screen. So we have rebooted the server. We have been monitoring the server till now. We could not find any issue with the server. The server load is normal and all the services are working fine. We have noticed that there is a lack of free memory in the server at some instance of time. We have checked the server logs. But we could not find any details of the issue anywhere in the server logs. We highly recommend you to upgrade the RAM for better performance. If the issue persists, please update us immediately.

=================================
[root@server01 httpdocs]# uptime
16:10:54 up 1:52, 1 user, load average: 0.00, 0.03, 0.14
[root@server01 httpdocs]# free -m
total used free shared buffers cached
Mem: 3952 1870 2082 0 11 1109
-/+ buffers/cache: 749 3203
Swap: 2047 138 1908
[root@server01 httpdocs]#
=================================


Regards
SoftLayer Support

Hello,

We checked the server via IPMI and found out of memory messages.

-------
Node 0 HighMem: empty
3107 pagecache pages
Swap cache: add 2524342, delete 2524027, find 866211/974711, race 0+7
Free swap = 0kB
Total swap = 2096472kB
-------

Hello,

The server is online now. The server load is also normal.

----------
[root@server01 ~]# w
15:06:02 up 7 min, 1 user, load average: 0.09, 0.32, 0.17
----------

We checked the status of the disk with the smartctl utility. However, we couldn't find any issues.

--------
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
--------

The lack of free memory has caused the issue.

If you encounter any load issue in the server, please update the ticket so that we can check the service causing load and optimize it.

If the issue persists, you would need to upgrade RAM in the server.

When I checked via IPMI, I could see the following message:
===========
Free swap = 0kB
Total swap = 2096472kB
crond invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
[<ffffffff800c0b09>] out_of_memory+0x8e/0x2f5
============

This means that there was lack of free memory in the server which caused this issue. I have initiated a server reboot and it is back on-line.
============
$ ssh root@67.228.86.194
root@67.228.86.194's password:
Last login: Thu Feb 18 15:01:12 2010 from 10.0.80.148
[root@server01 ~]# w
12:39:56 up 1 min, 1 user, load average: 1.05, 0.33, 0.11
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 66.228.118.68 12:39 0.00s 0.01s 0.00s w
============

I could see that there are 8 processors for this server.
============
[root@server01 ~]# cat /proc/cpuinfo | grep processor | wc -l
8
============

Right now there is only 4G of RAM in the server.
============
[root@server01 ~]# free -m
total used free shared buffers cached
Mem: 3952 2298 1653 0 40 474
-/+ buffers/cache: 1783 2169
Swap: 2047 0 2047
============

You can add 2G extra RAM to the server for the smooth running of it.

[Gydian]

Is there an "Idiots Guide" to server Management?? I have a dedicated server and should know how to do all this stuff but I don't, I just poke around in the WHM control panel, and cross a finger or two when changing a setting.

I know there are tons of resources on the internet, I've googled the subject a few times, but most of the sites are way to technical and lose beginners/intermediates right away.

Thanks,
Jason

Honestly, the only reason I know this stuff is trial and error over the past 10 years of having my own servers and managing the 40+ servers I managed for a large corporation. Not sure if there's an idiot's guide to it all. Things I just can't figure out generally fall under the research or ask a few other server admin's that I know for help.

[Gydian]

BTW, Here is some of the conversations I've had with support. Maybe the reports they are referencing can explain something. These have happened over the last week. They will reboot, say it's running fine, then it crashes in a few hours. Same thing has happened 5-10 times now. I really don't know why I would need more RAM though. It feels like something else is causing the problem imo.

It's probably one of the following issues:

1. One of the sticks of ram is faulty and causing problems. (unlikely but possible. I know this is what caused one of my issues in the past)

2. Something is using far too much ram and should be able to track it in shell. Pending on your types of sites, this could be something to do with too many database queries especially if you have a high traffic site and your database stuff isn't optimized. Could even be an issue with a loop or something that's logged in the error logs. (highly likely)

Anyway, to run 'top' in shell, you need a program such as putty (PuTTY Download Page)

Run it and enter your server's IP address. Login as root (username is root, and then enter your root password), then type the command 'top' (without the ' marks) and hit enter and it'll spit out a list of processes running.