OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#5761 — vss-1b-6k
Incident Report for Network & Infrastructure
Resolved
In the recent days we have strange messages:
Sep 8 08:42:54 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B6823C 40B671A4 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4

It's totally bizarre because the router is a failover
failover that does not route the traffic for now.




Update(s):

Date: 2011-09-08 10:32:56 UTC
it's good, no more problem.

Date: 2011-09-08 10:32:24 UTC
and the CPU is okey now. so it's really
the backup on the disk0 which blocked all the router!?

Sep 8 10:13:54 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:14:31 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:03 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:15:39 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:12 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:16:45 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:17:31 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:06 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:18:38 GMT: %IPC-SP-5-WATERMARK: 1687 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:14 GMT: %IPC-SP-5-WATERMARK: 1684 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:19:45 GMT: %IPC-SP-5-WATERMARK: 3391 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:22 GMT: %IPC-SP-5-WATERMARK: 3397 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:20:51 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:21:14 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (2/2),process = RFSS worker process.
-Traceback= 40B8A0E4 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278 40D61EF4 40D628CC 40D615AC
Sep 8 10:21:29 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:00 GMT: %IPC-SP-5-WATERMARK: 3399 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:22:39 GMT: %IPC-SP-5-WATERMARK: 1700 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:23:41 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673B8 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:23:57 GMT: %IPC-SP-5-WATERMARK: 3387 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:24:41 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:17 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:25:48 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:26:24 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:04 GMT: %IPC-SP-5-WATERMARK: 1685 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:27:35 GMT: %IPC-SP-5-WATERMARK: 1701 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:07 GMT: %IPC-SP-5-WATERMARK: 3394 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:28:12 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B68864 40B69890 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 10:28:15 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card removed from disk0
Sep 8 10:28:23 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card inserted into disk0
Sep 8 10:28:23 GMT: %DIAG-SP-6-TEST_RUNNING: Module 5: Running TestCFRW{ID=41} ...
Sep 8 10:28:25 GMT: %DIAG-SP-6-TEST_OK: Module 5: TestCFRW{ID=41} has completed successfully
Sep 8 10:28:41 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:12 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:29:42 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:13 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:30:43 GMT: %IPC-SP-5-WATERMARK: 3353 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:31:01 GMT: %SYS-SP-3-CONFIG_NV_ERR: Nonvolatile storage write error; configuration failed
Sep 8 10:31:16 GMT: %IPC-SP-5-WATERMARK: 2887 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
vss-1b-6k#




Date: 2011-09-08 10:30:49 UTC
vss-1b-6k# sh run
Unable to get configuration. Try again later.


great!

good, let's go

before
http://www.yfrog.com/z/18ki8hvj

action:
http://www.yfrog.com/z/kho5zjxj
Sep 8 10:28:15 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card removed from disk0

action:
http://www.yfrog.com/z/h759nkpuj
Sep 8 10:28:23 GMT: %FILESYS-SP-5-DEV: PCMCIA flash card inserted into disk0

It's good :)

vss-1b-6k#wr me
Building configuration...
Compressed configuration from 2398126 bytes to 471700 bytes
[OK]






Date: 2011-09-08 09:33:46 UTC
Sep 8 09:52:55 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:53:33 GMT: %IPC-SP-5-WATERMARK: 1696 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:10 GMT: %IPC-SP-5-WATERMARK: 1695 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:54:50 GMT: %IPC-SP-5-WATERMARK: 3393 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:33 GMT: %FM-4-FLOWMASK_REDUCED: Features configured on interface Port-channel5 have conflicting flowmask requirements, some features may work in software
Sep 8 09:55:22 GMT: %IPC-SP-5-WATERMARK: 1688 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:55:52 GMT: %IPC-SP-5-WATERMARK: 3396 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:21 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (4/4),process = RFSS worker process.
-Traceback= 40B67178 40B672DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0BC 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4
Sep 8 09:56:23 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:56:53 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:25 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:57:53 GMT: %SYS-SP-3-CPUHOG: Task is running for (2004)msecs, more than (2000)msecs (3/3),process = RFSS worker process.
-Traceback= 40B673DC 40B6984C 40B69758 40B8A0F0 40B89544 40B89FB8 40D6D0A0 40D63424 40D6FB3C 40D700B8 40D63CC4 40D6485C 40D64CB0 40D654A4 40D64DC4 40D66278
Sep 8 09:57:59 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:58:43 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:15 GMT: %IPC-SP-5-WATERMARK: 1682 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:59:50 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:26 GMT: %IPC-SP-5-WATERMARK: 1681 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 10:00:56 GMT: %IPC-SP-5-WATERMARK: 3388 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000


Date: 2011-09-08 09:33:40 UTC
We activate the netflow and we disable it.

vss-1b-6k(config)#mls flow ip interface-full
vss-1b-6k(config)#mls flow ipv6 interface-full
vss-1b-6k(config)#no mls flow ipv6 interface-full
% Reset for ipv6 protocol to the default flow mask type: none

vss-1b-6k(config)#no mls flow ip interface-full
vss-1b-6k(config)#

Date: 2011-09-08 09:32:23 UTC
We haven't find the origin of the problem.

ah, again this morning:

Sep 8 09:28:37 GMT: %IPC-SP-5-WATERMARK: 1690 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:15 GMT: %IPC-SP-5-WATERMARK: 3411 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:29:48 GMT: %IPC-SP-5-WATERMARK: 3402 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:30:34 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:11 GMT: %IPC-SP-5-WATERMARK: 3392 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:31:43 GMT: %IPC-SP-5-WATERMARK: 3408 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:32:17 GMT: %IPC-SP-5-WATERMARK: 1697 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:03 GMT: %IPC-SP-5-WATERMARK: 1699 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:33:32 GMT: %IPC-SP-5-WATERMARK: 1694 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Sep 8 09:34:03 GMT: %IPC-SP-5-WATERMARK: 3410 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000


we have had this problem and it is on the netflow tables.

Sep 8 09:49:29 GMT: %IPC-SP-5-WATERMARK: 1732 messages pending in rcv for the port Card5/0:Request(10000.7) seat 10000
Posted Sep 08, 2011 - 09:29 UTC