Как исследовать случайный сброс на клиентском порту TCP, подключенном через интерфейс обратной связи к серверу

10

У меня есть набор тестов, который инициализирует сервер на локальном хосте, а затем запускает тысячи стресс-тестов, которые включают запуск и остановку клиентов, подключающихся к этому локальному серверу на той же машине. Я пытаюсь найти причину для отправки RST клиентом на сервер.

Это последовательность событий:

  1. клиент инициализирует SYN
  2. Сервер отправляет обратно SYN, ACK
  3. клиент отправляет RST (вместо ACK и установления соединения)

Одним из наблюдений является то, что во многих случаях серверы работают примерно на 99% ЦП, хотя через несколько секунд / минут он возвращается в нормальное состояние.

Сначала я думал, что это может быть связано с тем, что клиент использует порт из ранее закрытого соединения, находящегося в состоянии TIME_WAIT, и поэтому, когда сервер отправляет SYN, ACK для этого соединения клиент выдает RST. Хотя я скептически относился к тому, почему клиент должен использовать порт в состоянии TIME_WAIT, но тогда я использую старое ядро ​​4 Fedora и подумал, что он не имеет надлежащей реализации TCP / IP :-).

Теперь я не верю, что это является причиной, потому что все настройки, сделанные вокруг исправления вышеупомянутой проблемы, не помогли облегчить проблему. К вашему сведению следующие изменения, которые я попробовал /etc/sysctl.conf:

net.ipv4.ip_local_port_range = 1024     65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_tw_reuse = 1

Еще один момент, эта проблема не соответствует. Фактически я могу воспроизвести эту проблему только после 15–35 циклов перезапуска сервера и выполнения всех клиентских тестов.

У меня закончились все причины, по которым я могу подумать, почему это может происходить, поэтому дайте мне знать, если у кого-нибудь есть какие-либо идеи о том, что может происходить.

Детали пакета

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378209 1032719 1858.494749 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=0 WS=128

Frame 1032719: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378209000 EDT
    Epoch Time: 1403722185.378209000 seconds
    [Time delta from previous captured frame: 0.005893000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 1858.494749000 seconds]
    Frame Number: 1032719
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf489 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378222 1032720 1858.494762 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 navisphere > netview-aix-5 [SYN, ACK] Seq=0 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=1587252 WS=128

Frame 1032720: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378222000 EDT
    Epoch Time: 1403722185.378222000 seconds
    [Time delta from previous captured frame: 0.000013000 seconds]
    [Time delta from previous displayed frame: 0.000013000 seconds]
    [Time since reference or first frame: 1858.494762000 seconds]
    Frame Number: 1032720
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 0, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0xf7fa [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 1587252
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032719]
        [The RTT to ACK the segment was: 0.000013000 seconds]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378228 1032721 1858.494768 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032721: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378228000 EDT
    Epoch Time: 1403722185.378228000 seconds
    [Time delta from previous captured frame: 0.000006000 seconds]
    [Time delta from previous displayed frame: 0.000006000 seconds]
    [Time since reference or first frame: 1858.494768000 seconds]
    Frame Number: 1032721
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375927 1032722 1861.492467 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=0 WS=128

Frame 1032722: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375927000 EDT
    Epoch Time: 1403722188.375927000 seconds
    [Time delta from previous captured frame: 2.997699000 seconds]
    [Time delta from previous displayed frame: 2.997699000 seconds]
    [Time since reference or first frame: 1861.492467000 seconds]
    Frame Number: 1032722
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf19b [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375950 1032723 1861.492490 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 [TCP Previous segment lost] navisphere > netview-aix-5 [SYN, ACK] Seq=2997725 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=1588002 WS=128

Frame 1032723: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375950000 EDT
    Epoch Time: 1403722188.375950000 seconds
    [Time delta from previous captured frame: 0.000023000 seconds]
    [Time delta from previous displayed frame: 0.000023000 seconds]
    [Time since reference or first frame: 1861.492490000 seconds]
    Frame Number: 1032723
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: Bad TCP]
    [Coloring Rule String: tcp.analysis.flags]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 2997725, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 2997725    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0x3414 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 1588002
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032722]
        [The RTT to ACK the segment was: 0.000023000 seconds]
        [TCP Analysis Flags]
            [A segment before this frame was lost]
                [Expert Info (Warn/Sequence): Previous segment lost (common at capture start)]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375958 1032724 1861.492498 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032724: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375958000 EDT
    Epoch Time: 1403722188.375958000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000008000 seconds]
    [Time since reference or first frame: 1861.492498000 seconds]
    Frame Number: 1032724
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:54.376072 1032725 1867.492612 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1589502 TSecr=0 WS=128

Frame 1032725: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:54.376072000 EDT
    Epoch Time: 1403722194.376072000 seconds
    [Time delta from previous captured frame: 6.000114000 seconds]
    [Time delta from previous displayed frame: 6.000114000 seconds]
    [Time since reference or first frame: 1867.492612000 seconds]
    Frame Number: 1032725
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xebbf [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1589502, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

iptables-save вывод:

# Generated by iptables-save v1.3.0 on Thu Jun 26 10:15:33 2014
*filter
:FORWARD ACCEPT [0:0]
:INPUT ACCEPT [675:236200]
:OUTPUT ACCEPT [241:21540]
COMMIT
# Completed on Thu Jun 26 10:15:33 2014
fedora tcp tcpdump tcp-ip каролинец
источник
Разъем от клиента находится в неблокирующем режиме? Возможно ли, чтобы сокет клиента был закрыт к тому времени, когда сервер примет? У вас есть время для этих пакетов? А может быть pcap?
Стефан Шазелас
1
Вы проверяете возвращаемое значение connect()s? Возможно ли, что вы делаете два неблокирующих connect()sна одном сокете? Возможно ли, что некоторые клиенты будут убиты или прерваны каким-либо образом ( alarm()?) В середине connect()? Можете ли вы воспроизвести его с клиентом, работающим под strace -ffили autrace?
Стефан Шазелас
1
Что-нибудь необычное (nfqueue, ограничение соединения ...) на выходе iptables-save? Возможность достижения какого-либо лимита ресурсов? Сколько одновременных клиентов? Есть ли это bindдля конкретного порта источника?
Стефан Шазелас
1
Было бы лучше добавить к вопросу. Вопрос в том, достигнуты ли эти пределы (или, вероятно, достигнуты) в вашем тесте, чем они есть на самом деле.
Стефан Шазелас
1
Я предполагаю, что нагрузка на стек приводит к одному падению соединения на стороне клиента тайм-аута при наполовину открытом состоянии. Это соединение сбрасывается (что означает, что клиентское приложение должно видеть его как a ETIMEDOUT), и когда syn / ack окончательно обрабатывается стеком, больше нет подключений, к которым он может быть привязан, и, таким образом, он сбрасывается. Вы пытались увеличить размер syn backlog ( sysctl -w net.ipv4.tcp_max_syn_backlog=???)?
Didierc

Ответы:

1

Просто хочу завершить этот поток с принятым решением в отсутствие исправления основной причины, которая, как я предполагаю, заключается в реализации стека TCP в Fedora 4. В качестве решения я просто представил немедленную попытку повторного подключения в случае, если сбой произошел из-за ETIMEDOUT, а используемый протокол - IPV6. Это исправило проблему для меня и моей команды навсегда с возможным риском дополнительной 3-минутной задержки в сообщении об ошибке в любой другой проблеме соединения, приводящей к ETIMEDOUT. Это не настоящее / идеальное решение проблемы, но заставляет нас двигаться дальше ... так как это влияет только на наш автоматизированный набор тестов и не доставляется заказчику. Я надеюсь, что в конечном итоге кто-нибудь, кто достаточно хорошо знает реализацию стека fedora 4 tcp / ip, решит эту загадку навсегда.

каролинец
источник