more tcpdumpinfo for nfs3 problem: aix-server --- linux 2.4.15pre5 client

From: Birger Lammering (b.lammering@science-computing.de)
Date: Mon Nov 19 2001 - 11:37:40 EST


Hi,

Alan Cox writes:
> Right so someone is truncating frames. I'd start with the hub then work
> outwards.

tcpdump on the aix was truncating the frames. sorry, the snap length was
too small --- with using -s 64000 there were no truncated frames
anymore.

Well, I guess (n.b.: GUESS) the problem is with nfs again...

This is a consistent reproducible bug. We have tested it on several
AIXen, Linuxen on different switched Networks.

After having learned a bit better how to use tcpdump, here we go:

nfs3 Client: Linux 2.4.15pre5 (capc25) 3com 905B / Intel EtherReal 100+
nfs3 Server: Aix 4.3 (caes04), IBM SP2 / 7044-170
100MBit/FullDuplex
switched Network (Cisco Catalyst)

This is the first attempt of copying a 500k file onto the nfs server:

16:24:19.652672 > capc25.muc.654 > caes04.muc.sunrpc: udp 40 (DF)
16:24:19.655073 < caes04.muc.sunrpc > capc25.muc.654: udp 24
16:24:19.655119 > capc25.muc.655 > caes04.muc.sunrpc: udp 56 (DF)
16:24:19.656813 < caes04.muc.sunrpc > capc25.muc.655: udp 28
16:24:19.656853 > capc25.muc.656 > caes04.muc.nfs: S 2952133600:2952133600(0) win 5840 <mss 1460,sackOK,timestamp 2001001 0,nop,wscale 0> (DF)
16:24:19.657695 < caes04.muc.nfs > capc25.muc.656: S 4177984247:4177984247(0) ack 2952133601 win 59860 <mss 1460>
16:24:19.657711 > capc25.muc.656 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF)
16:24:19.657746 > capc25.muc.656 > caes04.muc.nfs: P 1:45(44) ack 1 win 5840 (DF)
16:24:19.658653 < caes04.muc.nfs > capc25.muc.656: P 1:29(28) ack 45 win 60032
16:24:19.658666 > capc25.muc.656 > caes04.muc.nfs: . 45:45(0) ack 29 win 5840 (DF)
16:24:19.658702 > capc25.muc.656 > caes04.muc.nfs: F 45:45(0) ack 29 win 5840 (DF)
16:24:19.658749 > capc25.muc.657 > caes04.muc.sunrpc: udp 40 (DF)
16:24:19.659739 < caes04.muc.nfs > capc25.muc.656: . 29:29(0) ack 46 win 60032
16:24:19.660017 < caes04.muc.nfs > capc25.muc.656: F 29:29(0) ack 46 win 60032
16:24:19.660026 > capc25.muc.656 > caes04.muc.nfs: . 46:46(0) ack 30 win 5840 (DF)
16:24:19.660157 < caes04.muc.sunrpc > capc25.muc.657: udp 24
16:24:19.660199 > capc25.muc.658 > caes04.muc.sunrpc: udp 56 (DF)
16:24:19.662361 < caes04.muc.sunrpc > capc25.muc.658: udp 28
16:24:19.662393 > capc25.muc.50947429 > caes04.muc.nfs: 40 null (DF)
16:24:19.664439 < caes04.muc.nfs > capc25.muc.50947429: reply ok 24 null
16:24:19.665397 > capc25.muc.799 > caes04.muc.nfs: S 2955913379:2955913379(0) win 5840 <mss 1460,sackOK,timestamp 2001002 0,nop,wscale 0> (DF)
16:24:19.667274 < caes04.muc.nfs > capc25.muc.799: S 3186497216:3186497216(0) ack 2955913380 win 59860 <mss 1460>
16:24:19.667293 > capc25.muc.799 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF)
16:24:19.667328 > capc25.muc.799 > caes04.muc.nfs: P 1:109(108) ack 1 win 5840 (DF)
16:24:19.669096 < caes04.muc.nfs > capc25.muc.799: P 1:117(116) ack 109 win 60032
16:24:19.669108 > capc25.muc.799 > caes04.muc.nfs: . 109:109(0) ack 117 win 5840 (DF)
16:24:19.669155 > capc25.muc.799 > caes04.muc.nfs: P 109:217(108) ack 117 win 5840 (DF)
16:24:19.670267 < caes04.muc.nfs > capc25.muc.799: P 117:289(172) ack 217 win 60032
16:24:19.670295 > capc25.muc.799 > caes04.muc.nfs: P 217:325(108) ack 289 win 6432 (DF)
16:24:19.671674 < caes04.muc.nfs > capc25.muc.799: P 289:457(168) ack 325 win 60032
16:24:19.673364 > capc25.muc.799 > caes04.muc.nfs: P 325:461(136) ack 457 win 6432 (DF)
16:24:19.675030 < caes04.muc.nfs > capc25.muc.799: P 457:701(244) ack 461 win 60032
16:24:19.675121 > capc25.muc.799 > caes04.muc.nfs: P 461:625(164) ack 701 win 7504 (DF)
16:24:19.693914 < caes04.muc.nfs > capc25.muc.799: P 701:849(148) ack 625 win 60032
16:24:19.694170 > capc25.muc.799 > caes04.muc.nfs: . 625:2085(1460) ack 849 win 7504 (DF)
16:24:19.694186 > capc25.muc.799 > caes04.muc.nfs: . 2085:3545(1460) ack 849 win 7504 (DF)
16:24:19.774514 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 3545 win 57112
16:24:19.774545 > capc25.muc.799 > caes04.muc.nfs: . 3545:5005(1460) ack 849 win 7504 (DF)
16:24:19.774556 > capc25.muc.799 > caes04.muc.nfs: . 5005:6465(1460) ack 849 win 7504 (DF)
16:24:19.774565 > capc25.muc.799 > caes04.muc.nfs: . 6465:7925(1460) ack 849 win 7504 (DF)
16:24:19.983865 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 7925 win 52732
16:24:19.983887 > capc25.muc.799 > caes04.muc.nfs: . 7925:9385(1460) ack 849 win 7504 (DF)
16:24:19.983897 > capc25.muc.799 > caes04.muc.nfs: . 9385:10845(1460) ack 849 win 7504 (DF)
16:24:19.983906 > capc25.muc.799 > caes04.muc.nfs: . 10845:12305(1460) ack 849 win 7504 (DF)
16:24:19.983916 > capc25.muc.799 > caes04.muc.nfs: P 12305:13765(1460) ack 849 win 7504 (DF)
16:24:20.194551 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 13765 win 46892
16:24:20.194569 > capc25.muc.799 > caes04.muc.nfs: . 13765:15225(1460) ack 849 win 7504 (DF)
16:24:20.194578 > capc25.muc.799 > caes04.muc.nfs: P 15225:16685(1460) ack 849 win 7504 (DF)
16:24:20.194588 > capc25.muc.799 > caes04.muc.nfs: . 16685:18145(1460) ack 849 win 7504 (DF)
16:24:20.194598 > capc25.muc.799 > caes04.muc.nfs: . 18145:19605(1460) ack 849 win 7504 (DF)
16:24:20.194607 > capc25.muc.799 > caes04.muc.nfs: P 19605:21065(1460) ack 849 win 7504 (DF)
16:24:20.396361 < caes04.muc.nfs > capc25.muc.799: . 849:849(0) ack 21065 win 39592

...and so on...

16:24:21.904205 < caes04.muc.nfs > capc25.muc.799: . 2981:2981(0) ack 462513 win 58968
16:24:21.904231 > capc25.muc.799 > caes04.muc.nfs: P 480033:481493(1460) ack 2981 win 7504 (DF)
16:24:21.904239 > capc25.muc.799 > caes04.muc.nfs: . 481493:482953(1460) ack 2981 win 7504 (DF)
16:24:21.904247 > capc25.muc.799 > caes04.muc.nfs: . 482953:484413(1460) ack 2981 win 7504 (DF)
16:24:21.904256 > capc25.muc.799 > caes04.muc.nfs: . 484413:485873(1460) ack 2981 win 7504 (DF)
16:24:21.904265 > capc25.muc.799 > caes04.muc.nfs: . 485873:487333(1460) ack 2981 win 7504 (DF)
16:24:21.904277 > capc25.muc.799 > caes04.muc.nfs: . 487333:488793(1460) ack 2981 win 7504 (DF)
16:24:21.904289 > capc25.muc.799 > caes04.muc.nfs: . 488793:490253(1460) ack 2981 win 7504 (DF)
16:24:21.904298 > capc25.muc.799 > caes04.muc.nfs: . 490253:491713(1460) ack 2981 win 7504 (DF)
16:24:21.904309 > capc25.muc.799 > caes04.muc.nfs: . 491713:493173(1460) ack 2981 win 7504 (DF)
16:24:21.904317 > capc25.muc.799 > caes04.muc.nfs: . 493173:494633(1460) ack 2981 win 7504 (DF)
16:24:21.904327 > capc25.muc.799 > caes04.muc.nfs: . 494633:496093(1460) ack 2981 win 7504 (DF)
16:24:21.904337 > capc25.muc.799 > caes04.muc.nfs: . 496093:497553(1460) ack 2981 win 7504 (DF)
16:24:21.904349 > capc25.muc.799 > caes04.muc.nfs: . 497553:499013(1460) ack 2981 win 7504 (DF)
16:24:21.904357 > capc25.muc.799 > caes04.muc.nfs: . 499013:500473(1460) ack 2981 win 7504 (DF)
16:24:21.904364 > capc25.muc.799 > caes04.muc.nfs: . 500473:501933(1460) ack 2981 win 7504 (DF)
16:24:21.904372 > capc25.muc.799 > caes04.muc.nfs: . 501933:503393(1460) ack 2981 win 7504 (DF)
16:24:21.905180 < caes04.muc.nfs > capc25.muc.799: P 2981:3145(164) ack 465433 win 56048
16:24:21.905979 > capc25.muc.799 > caes04.muc.nfs: . 503393:504853(1460) ack 2981 win 7504 (DF)
16:24:21.905986 > capc25.muc.799 > caes04.muc.nfs: . 504853:506313(1460) ack 2981 win 7504 (DF)
16:24:21.905992 > capc25.muc.799 > caes04.muc.nfs: . 506313:507773(1460) ack 2981 win 7504 (DF)
16:24:21.905998 > capc25.muc.799 > caes04.muc.nfs: . 507773:509233(1460) ack 2981 win 7504 (DF)
16:24:21.906004 > capc25.muc.799 > caes04.muc.nfs: P 509233:510693(1460) ack 2981 win 7504 (DF)
16:24:21.906009 > capc25.muc.799 > caes04.muc.nfs: P 510693:512153(1460) ack 2981 win 7504 (DF)
16:24:21.906016 > capc25.muc.799 > caes04.muc.nfs: . 512153:513613(1460) ack 2981 win 7504 (DF)
16:24:21.906021 > capc25.muc.799 > caes04.muc.nfs: . 513613:515073(1460) ack 3145 win 7504 (DF)
16:24:21.906027 > capc25.muc.799 > caes04.muc.nfs: P 515073:515133(60) ack 3145 win 7504 (DF)
16:24:21.909806 < caes04.muc.nfs > capc25.muc.799: . 3145:3145(0) ack 494633 win 59764
16:24:21.910644 < caes04.muc.nfs > capc25.muc.799: P 3145:3309(164) ack 497553 win 56844
16:24:21.912247 < caes04.muc.nfs > capc25.muc.799: . 3309:3309(0) ack 515133 win 59892
16:24:21.912855 < caes04.muc.nfs > capc25.muc.799: P 3309:3473(164) ack 515133 win 59892
16:24:21.941386 > capc25.muc.799 > caes04.muc.nfs: . 515133:515133(0) ack 3473 win 7504 (DF)
16:24:21.965016 < caes04.muc.nfs > capc25.muc.799: P 3473:3629(156) ack 515133 win 60032
16:24:21.965042 > capc25.muc.799 > caes04.muc.nfs: . 515133:515133(0) ack 3629 win 7504 (DF)
16:24:21.965128 > capc25.muc.799 > caes04.muc.nfs: P 515133:515273(140) ack 3629 win 7504 (DF)
16:24:21.998128 < caes04.muc.nfs > capc25.muc.799: P 3629:3785(156) ack 515273 win 60032
16:24:22.031385 > capc25.muc.799 > caes04.muc.nfs: . 515273:515273(0) ack 3785 win 7504 (DF)

The file was copied all right.

qx09820@capc25 /home/qx09820 > netstat |grep caes04
tcp 0 0 capc25.muc:729 caes04.muc:nfs TIME_WAIT
tcp 0 0 capc25.muc:744 caes04.muc:nfs TIME_WAIT
tcp 0 0 capc25.muc:737 caes04.muc:nfs TIME_WAIT
tcp 0 0 capc25.muc:798 caes04.muc:nfs ESTABLISHED

Now copying the same file for a second time:

16:27:20.716640 > capc25.muc.799 > caes04.muc.nfs: P 2956428652:2956428780(128) ack 3186501001 win 7504 (DF)
16:27:20.717226 < caes04.muc.nfs > capc25.muc.799: P 1:117(116) ack 128 win 60032
16:27:20.717243 > capc25.muc.799 > caes04.muc.nfs: . 128:128(0) ack 117 win 7504 (DF)
16:27:20.717360 > capc25.muc.799 > caes04.muc.nfs: P 128:264(136) ack 117 win 7504 (DF)
16:27:20.717966 < caes04.muc.nfs > capc25.muc.799: P 117:361(244) ack 264 win 60032
16:27:20.718041 > capc25.muc.799 > caes04.muc.nfs: P 264:428(164) ack 361 win 8576 (DF)
16:27:20.735478 < caes04.muc.nfs > capc25.muc.799: P 361:509(148) ack 428 win 60032
16:27:20.735745 > capc25.muc.799 > caes04.muc.nfs: . 428:1888(1460) ack 509 win 8576 (DF)
16:27:20.735760 > capc25.muc.799 > caes04.muc.nfs: . 1888:3348(1460) ack 509 win 8576 (DF)
16:27:20.858656 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 3348 win 57112
16:27:20.858672 > capc25.muc.799 > caes04.muc.nfs: . 3348:4808(1460) ack 509 win 8576 (DF)
16:27:20.858682 > capc25.muc.799 > caes04.muc.nfs: . 4808:6268(1460) ack 509 win 8576 (DF)
16:27:20.858692 > capc25.muc.799 > caes04.muc.nfs: . 6268:7728(1460) ack 509 win 8576 (DF)
16:27:21.068297 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 7728 win 52732
16:27:21.068318 > capc25.muc.799 > caes04.muc.nfs: . 7728:9188(1460) ack 509 win 8576 (DF)
16:27:21.068328 > capc25.muc.799 > caes04.muc.nfs: . 9188:10648(1460) ack 509 win 8576 (DF)
16:27:21.068337 > capc25.muc.799 > caes04.muc.nfs: . 10648:12108(1460) ack 509 win 8576 (DF)
16:27:21.068347 > capc25.muc.799 > caes04.muc.nfs: . 12108:13568(1460) ack 509 win 8576 (DF)
16:27:21.278296 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 13568 win 46892
16:27:21.278321 > capc25.muc.799 > caes04.muc.nfs: . 13568:15028(1460) ack 509 win 8576 (DF)
16:27:21.278331 > capc25.muc.799 > caes04.muc.nfs: . 15028:16488(1460) ack 509 win 8576 (DF)
16:27:21.278341 > capc25.muc.799 > caes04.muc.nfs: . 16488:17948(1460) ack 509 win 8576 (DF)
16:27:21.278350 > capc25.muc.799 > caes04.muc.nfs: . 17948:19408(1460) ack 509 win 8576 (DF)
16:27:21.278359 > capc25.muc.799 > caes04.muc.nfs: . 19408:20868(1460) ack 509 win 8576 (DF)
16:27:21.488343 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 20868 win 39592
16:27:21.488376 > capc25.muc.799 > caes04.muc.nfs: . 20868:22328(1460) ack 509 win 8576 (DF)
16:27:21.488387 > capc25.muc.799 > caes04.muc.nfs: . 22328:23788(1460) ack 509 win 8576 (DF)
16:27:21.488397 > capc25.muc.799 > caes04.muc.nfs: . 23788:25248(1460) ack 509 win 8576 (DF)
16:27:21.488406 > capc25.muc.799 > caes04.muc.nfs: . 25248:26708(1460) ack 509 win 8576 (DF)
16:27:21.488415 > capc25.muc.799 > caes04.muc.nfs: . 26708:28168(1460) ack 509 win 8576 (DF)
16:27:21.488424 > capc25.muc.799 > caes04.muc.nfs: . 28168:29628(1460) ack 509 win 8576 (DF)
16:27:21.690712 < caes04.muc.nfs > capc25.muc.799: . 509:509(0) ack 29628 win 30832
16:27:21.690736 > capc25.muc.799 > caes04.muc.nfs: P 29628:31088(1460) ack 509 win 8576 (DF)
16:27:21.690746 > capc25.muc.799 > caes04.muc.nfs: . 31088:32548(1460) ack 509 win 8576 (DF)
16:27:21.690756 > capc25.muc.799 > caes04.muc.nfs: . 32548:34008(1460) ack 509 win 8576 (DF)
16:27:21.690765 > capc25.muc.799 > caes04.muc.nfs: . 34008:35468(1460) ack 509 win 8576 (DF)
16:27:21.690774 > capc25.muc.799 > caes04.muc.nfs: . 35468:36928(1460) ack 509 win 8576 (DF)
16:27:21.690784 > capc25.muc.799 > caes04.muc.nfs: . 36928:38388(1460) ack 509 win 8576 (DF)
16:27:21.690793 > capc25.muc.799 > caes04.muc.nfs: . 38388:39848(1460) ack 509 win 8576 (DF)

..and so on..

16:27:22.567415 < caes04.muc.nfs > capc25.muc.799: P 2477:2641(164) ack 434048 win 54320
16:27:22.568505 > capc25.muc.799 > caes04.muc.nfs: . 470548:472008(1460) ack 2477 win 8576 (DF)
16:27:22.568512 > capc25.muc.799 > caes04.muc.nfs: . 472008:473468(1460) ack 2477 win 8576 (DF)
16:27:22.568518 > capc25.muc.799 > caes04.muc.nfs: . 473468:474928(1460) ack 2477 win 8576 (DF)
16:27:22.568524 > capc25.muc.799 > caes04.muc.nfs: . 474928:476388(1460) ack 2477 win 8576 (DF)
16:27:22.568530 > capc25.muc.799 > caes04.muc.nfs: . 476388:477848(1460) ack 2477 win 8576 (DF)
16:27:22.568536 > capc25.muc.799 > caes04.muc.nfs: . 477848:479308(1460) ack 2641 win 8576 (DF)
16:27:22.568541 > capc25.muc.799 > caes04.muc.nfs: . 479308:480768(1460) ack 2641 win 8576 (DF)
16:27:22.568550 > capc25.muc.799 > caes04.muc.nfs: . 480768:482228(1460) ack 2641 win 8576 (DF)
16:27:22.568556 > capc25.muc.799 > caes04.muc.nfs: . 482228:483688(1460) ack 2641 win 8576 (DF)
16:27:22.570140 < caes04.muc.nfs > capc25.muc.799: . 2641:2641(0) ack 461788 win 59496
16:27:22.570163 > capc25.muc.799 > caes04.muc.nfs: . 483688:485148(1460) ack 2641 win 8576 (DF)
16:27:22.570172 > capc25.muc.799 > caes04.muc.nfs: . 485148:486608(1460) ack 2641 win 8576 (DF)
16:27:22.570180 > capc25.muc.799 > caes04.muc.nfs: . 486608:488068(1460) ack 2641 win 8576 (DF)
16:27:22.570190 > capc25.muc.799 > caes04.muc.nfs: . 488068:489528(1460) ack 2641 win 8576 (DF)
16:27:22.570200 > capc25.muc.799 > caes04.muc.nfs: . 489528:490988(1460) ack 2641 win 8576 (DF)
16:27:22.570214 > capc25.muc.799 > caes04.muc.nfs: . 490988:492448(1460) ack 2641 win 8576 (DF)
16:27:22.570222 > capc25.muc.799 > caes04.muc.nfs: . 492448:493908(1460) ack 2641 win 8576 (DF)
16:27:22.570230 > capc25.muc.799 > caes04.muc.nfs: . 493908:495368(1460) ack 2641 win 8576 (DF)
16:27:22.570238 > capc25.muc.799 > caes04.muc.nfs: . 495368:496828(1460) ack 2641 win 8576 (DF)
16:27:22.570247 > capc25.muc.799 > caes04.muc.nfs: . 496828:498288(1460) ack 2641 win 8576 (DF)
16:27:22.570257 > capc25.muc.799 > caes04.muc.nfs: P 498288:499748(1460) ack 2641 win 8576 (DF)
16:27:22.570268 > capc25.muc.799 > caes04.muc.nfs: . 499748:501208(1460) ack 2641 win 8576 (DF)
16:27:22.570279 > capc25.muc.799 > caes04.muc.nfs: . 501208:502668(1460) ack 2641 win 8576 (DF)
16:27:22.570287 > capc25.muc.799 > caes04.muc.nfs: P 502668:504128(1460) ack 2641 win 8576 (DF)
16:27:22.570294 > capc25.muc.799 > caes04.muc.nfs: . 504128:505588(1460) ack 2641 win 8576 (DF)
16:27:22.570307 > capc25.muc.799 > caes04.muc.nfs: . 505588:507048(1460) ack 2641 win 8576 (DF)
16:27:22.570433 < caes04.muc.nfs > capc25.muc.799: P 2641:2805(164) ack 464708 win 56576
16:27:22.571906 > capc25.muc.799 > caes04.muc.nfs: . 507048:508508(1460) ack 2641 win 8576 (DF)
16:27:22.571913 > capc25.muc.799 > caes04.muc.nfs: . 508508:509968(1460) ack 2641 win 8576 (DF)
16:27:22.571919 > capc25.muc.799 > caes04.muc.nfs: . 509968:511428(1460) ack 2641 win 8576 (DF)
16:27:22.571924 > capc25.muc.799 > caes04.muc.nfs: . 511428:512888(1460) ack 2641 win 8576 (DF)
16:27:22.571933 > capc25.muc.799 > caes04.muc.nfs: . 512888:514348(1460) ack 2805 win 8576 (DF)
16:27:22.571938 > capc25.muc.799 > caes04.muc.nfs: P 514348:514796(448) ack 2805 win 8576 (DF)
16:27:22.573924 < caes04.muc.nfs > capc25.muc.799: . 2805:2805(0) ack 495368 win 58832
16:27:22.574462 < caes04.muc.nfs > capc25.muc.799: P 2805:2969(164) ack 499748 win 54452
16:27:22.575192 < caes04.muc.nfs > capc25.muc.799: . 2969:2969(0) ack 514796 win 60032
16:27:22.575518 < caes04.muc.nfs > capc25.muc.799: P 2969:3133(164) ack 514796 win 60032
16:27:22.608665 > capc25.muc.799 > caes04.muc.nfs: . 514796:514796(0) ack 3133 win 8576 (DF)
16:27:26.248872 > capc25.muc.799 > caes04.muc.nfs: P 514796:514936(140) ack 3133 win 8576 (DF)
16:27:26.268210 < caes04.muc.nfs > capc25.muc.799: . 3133:3133(0) ack 514936 win 60032
16:27:26.282829 < caes04.muc.nfs > capc25.muc.799: P 3133:3289(156) ack 514936 win 60032
16:27:26.282843 > capc25.muc.799 > caes04.muc.nfs: . 514936:514936(0) ack 3289 win 8576 (DF)

from now on we get lot's of these:

16:27:26.489024 > capc25.muc.576126976 > caes04.muc.nfs: 40 null (DF)
16:27:26.489647 < caes04.muc.nfs > capc25.muc.576126976: reply ok 24 null

The cp command on the Linux nfs3-client side hangs and cannot be
killed. We get:

dmesg: nfs: server caes04 not responding, still trying

then after a while:
dmesg: nfs: server caes04 OK

qx09820@capc25 /home/qx09820 > netstat | grep caes04
tcp 0 0 capc25.muc:798 caes04.muc:nfs ESTABLISHED

The cp seems to be finished and the game can start again. :-(

The interesting bit seems to be at the end, where `cp` on the Linux
nfs3 client hangs. The cause of the problem might just as well be on
the AIX nfs3 server as on the client. The tcpdump on the server is
basically identical -- now that option -s 64000 has been used.

Files of size 500kB, 1MB and 2MB have been used. Sometimes the whole file
is copied before (after/while?) cp locks up, sometimes just about 800k
are copied.

We also got these messages after the cp died and the nfs: server
caes04 OK message appeared. The line with <mss 1460,sackOK,timestamp
2101448 0,nop,wscale 0> looks weird. Selective acknowledgement? Is
this normal?

16:34:56.487168 > capc25.muc.1649999872 > caes04.muc.nfs: 40 null (DF)
16:34:56.493170 < caes04.muc.nfs > capc25.muc.1649999872: reply ok 24 null
16:35:26.488376 > capc25.muc.2186870784 > caes04.muc.nfs: 40 null (DF)
16:35:26.492354 < caes04.muc.nfs > capc25.muc.2186870784: reply ok 24 null
16:35:56.489587 > capc25.muc.2723741696 > caes04.muc.nfs: 40 null (DF)
16:35:56.491055 < caes04.muc.nfs > capc25.muc.2723741696: reply ok 24 null
16:36:26.490783 > capc25.muc.3260612608 > caes04.muc.nfs: 40 null (DF)
16:36:26.505492 < caes04.muc.nfs > capc25.muc.3260612608: reply ok 24 null
16:36:56.491996 > capc25.muc.3797483520 > caes04.muc.nfs: 40 null (DF)
16:36:56.494516 < caes04.muc.nfs > capc25.muc.3797483520: reply ok 24 null
16:37:26.493194 > capc25.muc.39452672 > caes04.muc.nfs: 40 null (DF)
16:37:26.493631 < caes04.muc.nfs > capc25.muc.39452672: reply ok 24 null
16:37:56.484418 > capc25.muc.576323584 > caes04.muc.nfs: 40 null (DF)
16:37:56.485075 < caes04.muc.nfs > capc25.muc.576323584: reply ok 24 null
16:38:26.485620 > capc25.muc.1113194496 > caes04.muc.nfs: 40 null (DF)
16:38:26.486086 < caes04.muc.nfs > capc25.muc.1113194496: reply ok 24 null
16:38:56.486833 > capc25.muc.1650065408 > caes04.muc.nfs: 40 null (DF)
16:38:56.487256 < caes04.muc.nfs > capc25.muc.1650065408: reply ok 24 null
16:39:26.488042 > capc25.muc.2186936320 > caes04.muc.nfs: 40 null (DF)
16:39:26.488555 < caes04.muc.nfs > capc25.muc.2186936320: reply ok 24 null
16:39:56.489253 > capc25.muc.2723807232 > caes04.muc.nfs: 40 null (DF)
16:39:56.489655 < caes04.muc.nfs > capc25.muc.2723807232: reply ok 24 null
16:40:26.490458 > capc25.muc.3260678144 > caes04.muc.nfs: 40 null (DF)
16:40:26.490858 < caes04.muc.nfs > capc25.muc.3260678144: reply ok 24 null
16:40:56.491656 > capc25.muc.3797549056 > caes04.muc.nfs: 40 null (DF)
16:40:56.492277 < caes04.muc.nfs > capc25.muc.3797549056: reply ok 24 null
16:41:04.171081 < caes04.muc.nfs > capc25.muc.798: F 2278783378:2278783378(0) ack 3544800811 win 60032
16:41:04.171126 > capc25.muc.798 > caes04.muc.nfs: F 1:1(0) ack 1 win 9408 (DF)
16:41:04.171307 > capc25.muc.799 > caes04.muc.nfs: S 4020491914:4020491914(0) win 5840 <mss 1460,sackOK,timestamp 2101448 0,nop,wscale 0> (DF)
16:41:04.172213 < caes04.muc.nfs > capc25.muc.798: . 1:1(0) ack 2 win 60032
16:41:04.172379 < caes04.muc.nfs > capc25.muc.799: S 3316289216:3316289216(0) ack 4020491915 win 59860 <mss 1460>
16:41:04.172394 > capc25.muc.799 > caes04.muc.nfs: . 1:1(0) ack 1 win 5840 (DF)
16:41:04.172426 > capc25.muc.799 > caes04.muc.nfs: . 1:1461(1460) ack 1 win 5840 (DF)
16:41:04.172437 > capc25.muc.799 > caes04.muc.nfs: . 1461:2921(1460) ack 1 win 5840 (DF)
16:41:04.193288 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 2921 win 57112
16:41:04.193322 > capc25.muc.799 > caes04.muc.nfs: . 2921:4381(1460) ack 1 win 5840 (DF)
16:41:04.193330 > capc25.muc.799 > caes04.muc.nfs: . 4381:5841(1460) ack 1 win 5840 (DF)
16:41:04.193339 > capc25.muc.799 > caes04.muc.nfs: . 5841:7301(1460) ack 1 win 5840 (DF)
16:41:04.403451 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 7301 win 52732
16:41:04.403467 > capc25.muc.799 > caes04.muc.nfs: . 7301:8761(1460) ack 1 win 5840 (DF)
16:41:04.403477 > capc25.muc.799 > caes04.muc.nfs: . 8761:10221(1460) ack 1 win 5840 (DF)
16:41:04.403486 > capc25.muc.799 > caes04.muc.nfs: . 10221:11681(1460) ack 1 win 5840 (DF)
16:41:04.403496 > capc25.muc.799 > caes04.muc.nfs: P 11681:13141(1460) ack 1 win 5840 (DF)
16:41:04.613256 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 13141 win 46892
16:41:04.613294 > capc25.muc.799 > caes04.muc.nfs: . 13141:14601(1460) ack 1 win 5840 (DF)
16:41:04.613305 > capc25.muc.799 > caes04.muc.nfs: P 14601:16061(1460) ack 1 win 5840 (DF)
16:41:04.613315 > capc25.muc.799 > caes04.muc.nfs: . 16061:17521(1460) ack 1 win 5840 (DF)
16:41:04.613324 > capc25.muc.799 > caes04.muc.nfs: . 17521:18981(1460) ack 1 win 5840 (DF)
16:41:04.613334 > capc25.muc.799 > caes04.muc.nfs: P 18981:20441(1460) ack 1 win 5840 (DF)
16:41:04.815623 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 20441 win 39592
16:41:04.815641 > capc25.muc.799 > caes04.muc.nfs: . 20441:21901(1460) ack 1 win 5840 (DF)
16:41:04.815651 > capc25.muc.799 > caes04.muc.nfs: . 21901:23361(1460) ack 1 win 5840 (DF)
16:41:04.815660 > capc25.muc.799 > caes04.muc.nfs: . 23361:24821(1460) ack 1 win 5840 (DF)
16:41:04.815670 > capc25.muc.799 > caes04.muc.nfs: P 24821:26281(1460) ack 1 win 5840 (DF)
16:41:04.815693 > capc25.muc.799 > caes04.muc.nfs: . 26281:27741(1460) ack 1 win 5840 (DF)
16:41:04.815704 > capc25.muc.799 > caes04.muc.nfs: . 27741:29201(1460) ack 1 win 5840 (DF)
16:41:05.023080 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 29201 win 30832
16:41:05.023117 > capc25.muc.799 > caes04.muc.nfs: . 29201:30661(1460) ack 1 win 5840 (DF)
16:41:05.023127 > capc25.muc.799 > caes04.muc.nfs: . 30661:32121(1460) ack 1 win 5840 (DF)
16:41:05.023137 > capc25.muc.799 > caes04.muc.nfs: P 32121:32917(796) ack 1 win 5840 (DF)
16:41:05.024644 < caes04.muc.nfs > capc25.muc.799: . 1:1(0) ack 32917 win 60032
16:41:05.025192 < caes04.muc.nfs > capc25.muc.799: P 1:165(164) ack 32917 win 60032
16:41:05.025203 > capc25.muc.799 > caes04.muc.nfs: . 32917:32917(0) ack 165 win 5840 (DF)
16:41:05.025272 > capc25.muc.799 > caes04.muc.nfs: P 32917:33057(140) ack 165 win 5840 (DF)
16:41:05.048922 < caes04.muc.nfs > capc25.muc.799: P 165:321(156) ack 33057 win 60032
16:41:05.081819 > capc25.muc.799 > caes04.muc.nfs: . 33057:33057(0) ack 321 win 5840 (DF)
16:41:26.492871 > capc25.muc.39518208 > caes04.muc.nfs: 40 null (DF)
16:41:26.495098 < caes04.muc.nfs > capc25.muc.39518208: reply ok 24 null
16:41:56.494082 > capc25.muc.576389120 > caes04.muc.nfs: 40 null (DF)
16:41:56.500261 < caes04.muc.nfs > capc25.muc.576389120: reply ok 24 null
16:42:26.485359 > capc25.muc.1113260032 > caes04.muc.nfs: 40 null (DF)
16:42:26.487429 < caes04.muc.nfs > capc25.muc.1113260032: reply ok 24 null
16:42:56.486530 > capc25.muc.1650130944 > caes04.muc.nfs: 40 null (DF)
16:42:56.487097 < caes04.muc.nfs > capc25.muc.1650130944: reply ok 24 null
16:43:26.487748 > capc25.muc.2187001856 > caes04.muc.nfs: 40 null (DF)
16:43:26.489881 < caes04.muc.nfs > capc25.muc.2187001856: reply ok 24 null
16:43:34.489177 > capc25.muc.799 > caes04.muc.nfs: F 33057:33057(0) ack 321 win 5840 (DF)
16:43:34.489777 < caes04.muc.nfs > capc25.muc.799: . 321:321(0) ack 33058 win 60032
16:43:34.490370 < caes04.muc.nfs > capc25.muc.799: F 321:321(0) ack 33058 win 60032
16:43:34.490391 > capc25.muc.799 > caes04.muc.nfs: . 33058:33058(0) ack 322 win 5840 (DF)
16:43:34.494041 > capc25.muc.1022 > caes04.muc.32859: udp 80 (DF)
16:43:34.494079 > capc25.muc.1022 > caes04.muc.sunrpc: udp 84 (DF)
16:43:34.494091 > capc25.muc.1022 > caes04.muc.32859: udp 80 (DF)
16:43:34.496654 < caes04.muc.sunrpc > capc25.muc.1022: udp 28
16:43:34.507767 < caes04.muc.32859 > capc25.muc.1022: udp 72
16:43:34.507939 < caes04.muc.32859 > capc25.muc.1022: udp 24
16:43:56.488965 > capc25.muc.2723872768 > caes04.muc.nfs: 40 null (DF)
16:43:56.489405 < caes04.muc.nfs > capc25.muc.2723872768: reply ok 24 null
16:44:26.490297 > capc25.muc.3260743680 > caes04.muc.nfs: 40 null (DF)
16:44:26.490754 < caes04.muc.nfs > capc25.muc.3260743680: reply ok 24 null

Slighlty clueless,
Birger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Fri Nov 23 2001 - 21:00:20 EST