xcp-ng.org

@ olivierlambert said : Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x , and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. What the socket option still does after pipelining is remove the stall, 16.1% of samples down to

Same only remote pools failed. Main XOA pool completed with no errors. { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "compactMjml", "hideSuccessfulItems": true }, "id": "1788224400022", "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d", "jobName": "Salem - Delta Backups", "message": "backup", "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426", "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4", "start": 1788224400022, "status": "failure", "end": 1788224460395, "result": { "message": "fetch failed", "name": "TypeError", "stack": "TypeError: fetch failed\n at node:internal/deps/undici/undici:14976:13\n at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n at _drain

@ nikade yeah that's hot but would need explaining it as if it were for a 5 years old... asking for a friend :') did they overcome a bottleneck in VDI between SR copy ?

@ olivierlambert happy to see the CEO doing this kind of work, makes me proud to be one of your followers

Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x , and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more seg

I'm doing more tests right now, ideally with RAM drives to make sure we measure the right bottleneck

@olivierlambert said: I have no issue using Claude, but I would prefer your own conclusions/recommendations in the end and less text Sory its early morning. I wanted to show the statement what it said about what was missing on Vates side along what was missing from my script. Also to verify if Claude was correct or not. I have been battle with that alot lately.

@ TeddyAstie you were right, and it's the bigger win. I prototyped the pipelining you described and measured it. Same rig, same 107 GiB disk, same direction as the earlier runs. Arm Wall clock vs control Peak rate Stalls A : stock behaviour 2080.8 s 1.00x 63 MiB/s 62.2% B : TCP_NODELAY only 743.8 s 2.80x 210 MiB/s 0.6% C : + pipelined writes, depth 8 377.4 s 5.51x 432 MiB/s 1.6% Pipelining is worth a further 1.97x on top of the Nagle fix. Your diagnosis was correct: the per-request reply gating, not TCP, is the dominant limit. Verified byte for byte, source and destination md5 of the 107 GiB disk both ce647d9436b48401cd4b489c955ef0f7 . That mattered more than the stopwatch here, for reasons below. It's cheaper than you thought: the multiplexer already exists No NBD redesign is needed. nbd/lib/client.ml:78 is already module Rpc = Mux.Make (NbdRpc) , and that multiplexer: assigns every request a unique handle ( get_handle ) registers a waiter in id_to_wakeup keyed by that handle serialises only the send under outgoing_mutex , then returns a promise runs a background dispatcher thread that reads replies and wakes the matching waiter So concurrent Client.write calls already interleave correctly. The whole request/reply machinery is there and unused. The serialisation is one fold_left in stream_nbd that awaits each write before pulling the next element. The rest of the chain was already fine too: Layer Verdict xapi nbdproxy Unixext.proxy , raw bidirectional byte copy, never parses

Also adding @ Team-XAPI-Network

Here is the exact patch used for the benchmarks above, so the XAPI team can pick it up directly. Target: xapi-project/xen-api , file ocaml/vhd-tool/src/impl.ml Note the standalone xapi-project/vhd-tool repo is not the right target. It has been dead since 2021-05-21 (vendored into xen-api on 2021-09-20) and has diverged: socket sits at line 778 there versus 795 in what actually ships. The patch diff --git a/ocaml/vhd-tool/src/impl.ml b/ocaml/vhd-tool/src/impl.ml --- a/ocaml/vhd-tool/src/impl.ml +++ b/ocaml/vhd-tool/src/impl.ml @@ -800,7 +800,20 @@ let socket sockaddr = | Lwt_unix.ADDR_UNIX _ -> Unix.PF_UNIX in - Lwt_unix.socket family Unix.SOCK_STREAM 0 + let sock = Lwt_unix.socket family Unix.SOCK_STREAM 0 in + (* Disable Nagle's algorithm on TCP sockets. The stream protocols used here + (NBD and Chunked) write a small header and its payload with separate + write(2) calls, and NBD then waits for a per-request reply. Combined with + the peer's delayed ACKs this is the classic write-write-read stall: the + header sits in the send queue for up to 40ms waiting for an ACK that the + peer is itself delaying. Not applicable to Unix domain sockets. *) + ( match sockaddr with + | Lwt_unix.ADDR_INET _ -> + Lwt_unix.setsockopt sock Unix.TCP_NODELAY true + | Lwt_unix.ADDR_UNIX _ -> + () + ) ; + sock let split ~limit ~sep str = Xapi_stdext_std.Xstringext.String.split ~limit sep str Why this spot socket is the only Lwt_unix.socket call site in the whole of vhd-tool, and both Lwt_unix.conne

@ TeddyAstie You're right on the important part, and I went and measured the rest. First, the boring argument: TCP_NODELAY is already the default everywhere else in this stack. Where Call QEMU, used for the qcow2 path nbd/client-connection.c:143 , qio_channel_set_delay(..., false) , unconditional on every connection, and deliberately forwarded through TLS in io/channel-tls.c blktap's own NBD client drivers/block-nbd.c:793 , where failing to set it is treated as fatal xapi generally Unixext.set_tcp_nodelay , and stunnel with r:/a:/l:TCP_NODELAY=1 vhd-tool is the only NBD client in the toolstack that leaves Nagle on. If it were a bad default for bulk NBD, QEMU wouldn't do it unconditionally. So this patch is less "new tuning" and more "stop being the exception". On "bulk transfers benefit from Nagle's coalescing": correct, and it costs 1.16%. Two-host 8.3 pool, 10G, live SXM, two RPMs from the same tree differing only by that one setsockopt (control build, so the A/B isn't a build-environment artifact). bytes / data segment vs MSS (1448) segments / GiB Nagle on 1441.7 0.996x 744,763 TCP_NODELAY 1425.2 0.984x 753,391 Nagle really does pack 99.6% of MSS. But NODELAY still packs 98.4%, because vhd-tool's payload writes are 2 MiB and were never sub-MSS in the first place. The only thing being coalesced is the 28-byte NBD header, and it's paid for with a delayed-ACK round trip. No small-packet flood. What Nagle is actually holding (107 GiB migration, ss -tinp every 200 ms): Nagle on