On Monday we upgraded one of our PostgreSQL instances from v7.2 to v7.4.
Yesterday the box (sabine) on which this runs became very unhappy:
It runs RHEL ES v3, kernel 2.4.21-20.ELsmp
It's generally a very stable box which runs a number of postgresql
instances. But last night we hit very high low averages - 10+, vs the
normal 0-2.
The culprit appeared to be kswapd, which was using huge amounts of cpu.
I'd like to know why!
We resolved the problem last night via a reboot. And so far all is well,
but I am concerned that the problem may reoccur.
I wondered if you could help. In particular the only recent change is the
postgresql upgrade above, so the timing seems suspect.
We used to run two postgresql instances, each with the following
parameters:
Command line was:
/usr/bin/postmaster '-p' '5679' '-i' '-N' '256' '-B' '128000' '-o' '-S
64000'
As we only upgraded one database we split up the 2nd instance into two,
each with the following parameters:
/usr/bin/postmaster '-p' '5679' '-i' '-N' '256' '-B' '64000' '-o' '-S
32000'
(different ports, obviously)
The ram on the box should be generous. Right now (whilst it's behaving,
and all 3 instances are running) free gives the following output:
sabine% free
~
total used free shared buffers cached
Mem: 3857312 3834676 22636 0 50880 3339568
-/+ buffers/cache: 444228 3413084
Swap: 2096472 319460 1777012
sabine%
We plan to revert back to two instances when we upgrade the remaining
databases next week. Should we change the amount of ram we allocate to
these with the move to v7.4?
Are there other settings we should be altering?
All this is assuming that it's a postgres problem, rather than something
else. Another leading candidate is this, which we are looking into:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132155
It would be good to eliminate all possible causes, though.
--
Peter Haworth pmh@no-spam
"Of course, if you were to print(1..Inf), you'd have plenty of time to go and
get a cup of coffee. And even then (given the comparatively imminent heat
death of the universe) that coffee would be really cold before the output
was complete. So there will probably be a warning when you try to do that."
-- Damian Conway in Exegesis 3
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings
On 12/9/04 9:23 AM, "Peter Haworth" <pmh@no-spam> wrote:
> It runs RHEL ES v3, kernel 2.4.21-20.ELsmp
> It's generally a very stable box which runs a number of postgresql
> instances. But last night we hit very high low averages - 10+, vs the
> normal 0-2.
> The culprit appeared to be kswapd, which was using huge amounts of cpu.
> I'd like to know why!
There were some major changes in swap management somewhere in the linux 2.4
kernel (also RH 3.0 ES). I don't off hand remember exactly which level.
Under a heavy I/O load, we also saw kswapd going nuts with the 2.4.21
kernel, destroying system performance. The only solution we found was to
upgrade to a 2.6 kernel. The problem has not reoccurred since then. Our
database is currently about 67 gigabytes and is growing at about 1 million
records per day. We are using Postgresql 7.4.x.
Wes
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match