We have a Linux Centos 5.7 server with RMA agent 1.28 x64 running. Yesterday the server lost power and when it came back up the RMA will not stay running. We can start it, but upon initiating a refresh from Host Monitor the service stops. Logs show valid connections being made, no other info in logs. We tried reinstalling the agent files. Can't reboot the server quite yet, though.
Any thoughts?
RMA Crashes when connection initiated
Response from interactive
-------------------------------------------------------------
Application: RMA (Remote Monitoring Agent for HostMonitor)
Version: 1.28 for Linux (Red Hat, Mandrake, SuSE)
Copyright: 2004 - 2009 Alexander Kozlov
web: http://www.ks-soft.net
e-mail: support@ks-soft.net
-------------------------------------------------------------
Command line checking .. Ok
Settings checking .. Ok
CPU testing .. Ok
Basic encyption checking .. Ok
RMA encyption checking .. Ok
Sock testing .. Ok
[rma][11:47:39] rma Listening...
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 --> request processed (200)
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] rma Session closed, wait for next ..
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
*** glibc detected *** ./rma: double free or corruption (out): 0x000000001a22ef10 ***
[rma][11:47:51] 198.49.223.101 --> request processed (200)
======= Backtrace: =========
[rma][11:47:51] rma Session closed, wait for next ..
/lib64/libc.so.6[0x390cc7245f]
/lib64/libc.so.6(cfree+0x4b)[0x390cc728bb]
./rma[0x402a3b]
./rma[0x412ff1]
./rma[0x413023]
./rma[0x412f02]
./rma[0x41e8e1]
./rma[0x416440]
./rma[0x423be0]
./rma[0x424aa0]
/lib64/libpthread.so.0[0x390d80673d]
/lib64/libc.so.6(clone+0x6d)[0x390ccd44bd]
======= Memory map: ========
00400000-00431000 r-xp 00000000 fd:00 157778587 /opt/rma/rma
00630000-00631000 rw-p 00030000 fd:00 157778587 /opt/rma/rma
00631000-00633000 rw-p 00631000 00:00 0
1a22b000-1a24c000 rw-p 1a22b000 00:00 0 [heap]
4104d000-4104e000 ---p 4104d000 00:00 0
4104e000-41a4e000 rw-p 4104e000 00:00 0
41d30000-41d31000 ---p 41d30000 00:00 0
41d31000-42731000 rw-p 41d31000 00:00 0
42731000-42732000 ---p 42731000 00:00 0
42732000-43132000 rw-p 42732000 00:00 0
43132000-43133000 ---p 43132000 00:00 0
43133000-43b33000 rw-p 43133000 00:00 0
43b33000-43b34000 ---p 43b33000 00:00 0
43b34000-44534000 rw-p 43b34000 00:00 0
390c800000-390c81c000 r-xp 00000000 fd:00 73498626 /lib64/ld-2.5.so
390ca1c000-390ca1d000 r--p 0001c000 fd:00 73498626 /lib64/ld-2.5.so
390ca1d000-390ca1e000 rw-p 0001d000 fd:00 73498626 /lib64/ld-2.5.so
390cc00000-390cd4e000 r-xp 00000000 fd:00 73498629 /lib64/libc-2.5.so
390cd4e000-390cf4e000 ---p 0014e000 fd:00 73498629 /lib64/libc-2.5.so
390cf4e000-390cf52000 r--p 0014e000 fd:00 73498629 /lib64/libc-2.5.so
390cf52000-390cf53000 rw-p 00152000 fd:00 73498629 /lib64/libc-2.5.so
390cf53000-390cf58000 rw-p 390cf53000 00:00 0
390d000000-390d082000 r-xp 00000000 fd:00 73498936 /lib64/libm-2.5.so
390d082000-390d281000 ---p 00082000 fd:00 73498936 /lib64/libm-2.5.so
390d281000-390d282000 r--p 00081000 fd:00 73498936 /lib64/libm-2.5.so
390d282000-390d283000 rw-p 00082000 fd:00 73498936 /lib64/libm-2.5.so
390d800000-390d816000 r-xp 00000000 fd:00 73498671 /lib64/libpthread-2.5.so
390d816000-390da15000 ---p 00016000 fd:00 73498671 /lib64/libpthread-2.5.so
390da15000-390da16000 r--p 00015000 fd:00 73498671 /lib64/libpthread-2.5.so
390da16000-390da17000 rw-p 00016000 fd:00 73498671 /lib64/libpthread-2.5.so
390da17000-390da1b000 rw-p 390da17000 00:00 0
391ea00000-391ea0d000 r-xp 00000000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391ea0d000-391ec0d000 ---p 0000d000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391ec0d000-391ec0e000 rw-p 0000d000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391f600000-391f6e6000 r-xp 00000000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f6e6000-391f8e5000 ---p 000e6000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8e5000-391f8eb000 r--p 000e5000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8eb000-391f8ee000 rw-p 000eb000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8ee000-391f900000 rw-p 391f8ee000 00:00 0
2aaaac000000-2aaaac021000 rw-p 2aaaac000000 00:00 0
2aaaac021000-2aaab0000000 ---p 2aaaac021000 00:00 0
2aaab0000000-2aaab0001000 rw-p 2aaab0000000 00:00 0
2b8737e93000-2b8737e96000 rw-p 2b8737e93000 00:00 0
2b8737ea8000-2b8737eab000 rw-p 2b8737ea8000 00:00 0
7fffbcd3d000-7fffbcd52000 rw-p 7ffffffe9000 00:00 0 [stack]
7fffbcdfd000-7fffbce00000 r-xp 7fffbcdfd000 00:00 0 [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]
Aborted
Application: RMA (Remote Monitoring Agent for HostMonitor)
Version: 1.28 for Linux (Red Hat, Mandrake, SuSE)
Copyright: 2004 - 2009 Alexander Kozlov
web: http://www.ks-soft.net
e-mail: support@ks-soft.net
-------------------------------------------------------------
Command line checking .. Ok
Settings checking .. Ok
CPU testing .. Ok
Basic encyption checking .. Ok
RMA encyption checking .. Ok
Sock testing .. Ok
[rma][11:47:39] rma Listening...
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 TCP connection accepted
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 --> request processed (200)
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] rma Session closed, wait for next ..
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
[rma][11:47:51] 198.49.223.101 <-- Correct HM request received
*** glibc detected *** ./rma: double free or corruption (out): 0x000000001a22ef10 ***
[rma][11:47:51] 198.49.223.101 --> request processed (200)
======= Backtrace: =========
[rma][11:47:51] rma Session closed, wait for next ..
/lib64/libc.so.6[0x390cc7245f]
/lib64/libc.so.6(cfree+0x4b)[0x390cc728bb]
./rma[0x402a3b]
./rma[0x412ff1]
./rma[0x413023]
./rma[0x412f02]
./rma[0x41e8e1]
./rma[0x416440]
./rma[0x423be0]
./rma[0x424aa0]
/lib64/libpthread.so.0[0x390d80673d]
/lib64/libc.so.6(clone+0x6d)[0x390ccd44bd]
======= Memory map: ========
00400000-00431000 r-xp 00000000 fd:00 157778587 /opt/rma/rma
00630000-00631000 rw-p 00030000 fd:00 157778587 /opt/rma/rma
00631000-00633000 rw-p 00631000 00:00 0
1a22b000-1a24c000 rw-p 1a22b000 00:00 0 [heap]
4104d000-4104e000 ---p 4104d000 00:00 0
4104e000-41a4e000 rw-p 4104e000 00:00 0
41d30000-41d31000 ---p 41d30000 00:00 0
41d31000-42731000 rw-p 41d31000 00:00 0
42731000-42732000 ---p 42731000 00:00 0
42732000-43132000 rw-p 42732000 00:00 0
43132000-43133000 ---p 43132000 00:00 0
43133000-43b33000 rw-p 43133000 00:00 0
43b33000-43b34000 ---p 43b33000 00:00 0
43b34000-44534000 rw-p 43b34000 00:00 0
390c800000-390c81c000 r-xp 00000000 fd:00 73498626 /lib64/ld-2.5.so
390ca1c000-390ca1d000 r--p 0001c000 fd:00 73498626 /lib64/ld-2.5.so
390ca1d000-390ca1e000 rw-p 0001d000 fd:00 73498626 /lib64/ld-2.5.so
390cc00000-390cd4e000 r-xp 00000000 fd:00 73498629 /lib64/libc-2.5.so
390cd4e000-390cf4e000 ---p 0014e000 fd:00 73498629 /lib64/libc-2.5.so
390cf4e000-390cf52000 r--p 0014e000 fd:00 73498629 /lib64/libc-2.5.so
390cf52000-390cf53000 rw-p 00152000 fd:00 73498629 /lib64/libc-2.5.so
390cf53000-390cf58000 rw-p 390cf53000 00:00 0
390d000000-390d082000 r-xp 00000000 fd:00 73498936 /lib64/libm-2.5.so
390d082000-390d281000 ---p 00082000 fd:00 73498936 /lib64/libm-2.5.so
390d281000-390d282000 r--p 00081000 fd:00 73498936 /lib64/libm-2.5.so
390d282000-390d283000 rw-p 00082000 fd:00 73498936 /lib64/libm-2.5.so
390d800000-390d816000 r-xp 00000000 fd:00 73498671 /lib64/libpthread-2.5.so
390d816000-390da15000 ---p 00016000 fd:00 73498671 /lib64/libpthread-2.5.so
390da15000-390da16000 r--p 00015000 fd:00 73498671 /lib64/libpthread-2.5.so
390da16000-390da17000 rw-p 00016000 fd:00 73498671 /lib64/libpthread-2.5.so
390da17000-390da1b000 rw-p 390da17000 00:00 0
391ea00000-391ea0d000 r-xp 00000000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391ea0d000-391ec0d000 ---p 0000d000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391ec0d000-391ec0e000 rw-p 0000d000 fd:00 73498646 /lib64/libgcc_s-4.1.2-20080825.so.1
391f600000-391f6e6000 r-xp 00000000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f6e6000-391f8e5000 ---p 000e6000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8e5000-391f8eb000 r--p 000e5000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8eb000-391f8ee000 rw-p 000eb000 fd:00 93427351 /usr/lib64/libstdc++.so.6.0.8
391f8ee000-391f900000 rw-p 391f8ee000 00:00 0
2aaaac000000-2aaaac021000 rw-p 2aaaac000000 00:00 0
2aaaac021000-2aaab0000000 ---p 2aaaac021000 00:00 0
2aaab0000000-2aaab0001000 rw-p 2aaab0000000 00:00 0
2b8737e93000-2b8737e96000 rw-p 2b8737e93000 00:00 0
2b8737ea8000-2b8737eab000 rw-p 2b8737ea8000 00:00 0
7fffbcd3d000-7fffbcd52000 rw-p 7ffffffe9000 00:00 0 [stack]
7fffbcdfd000-7fffbce00000 r-xp 7fffbcdfd000 00:00 0 [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]
Aborted
We do not see any "double free" calls in RMA code that processes TCP requests from HostMonitor.
And RMA was working fine until server crashed due to power failure, right?
So it would be logical to assume your file system can be damaged...
Have you used fsck, e2fsck utilities to check your file system?
Have you tried to reinstall glibc libraries?
Regards
Alex
And RMA was working fine until server crashed due to power failure, right?
So it would be logical to assume your file system can be damaged...
Have you used fsck, e2fsck utilities to check your file system?
Have you tried to reinstall glibc libraries?
Regards
Alex