t/one-core.sh: Adding script to run the one-core io benchmark #1272

ErwanAliasr1 · 2021-09-16T21:00:32Z

Associated to fio, the t/io_uring test is used to compute the max IOPS a
single core can get.

Jens published several times the procedure he uses, but trying to
reproduce this setup is error-prone. It's easy to miss a configuration
and get a different result.

This script is about setting up a common setup to reproduce these runs.

From the fio directory, execute like the folliowing :

[user@fio] t/one-core.sh /dev/nvme0n1 [other drives]
system: Checking configuration
Warning: For better performance, you should enable nvme poll queues by setting nvme.poll_queues=32 on the kernel commande line
nvme0n1: set none as io scheduler
nvme0n1: iostats set to 1. Disabling.
nvme0n1: nomerge set to 0. Disabling.
system: End of system checks
io_uring: Running taskset -c 0,12 t/io_uring -b512 -d128 -c32 -s32 -p1 -F1 -B1 -n4 /dev/nvme0n1
    [...]
IOPS=731008, BWPS=356 MB IOS/call=32/31, inflight=(108 127 126 106)

This script will take care of the following items:

nvme poll queues
io scheduler
iostats
nomerge
finding the logical cores running on the first physical core
calling t/io_uring with the proper parameters in 512 bytes fashion

Signed-off-by: Erwan Velu erwanaliasr1@gmail.com

ByteHamster · 2021-09-17T08:14:16Z

Thanks, I think that will help people who also try to reproduce the measurements. Some thoughts:

How about also setting /sys/block/nvmeX/queue/io_poll to 1? I don't remember if that got set to 1 because of the boot parameter or if I did that manually.
How about setting the cpu scheduler to performance? echo "performance" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor?
When iterating over the cpus, how about also making sure that they are available in /sys/devices/system/node/nodeX/cpuY, where X=/sys/block/nvmeZ/device/numa_node? Unfortunately, I can't test this currently because the machine is booted with Numa disabled.
How about naming the file something like t/io_uring_one_core.sh? Makes it more clear that it belongs to t/io_uring but could also be potentially annoying because it will always show up in tab completion.
Maybe also a little note could help? "You are running Kernel uname -r. Make sure to use the latest version for best results." I mean, it won't directly set anything but it could sensitize users of the script.
Not really clean but lspci -vv | grep -P "[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]|LnkSta:|LnkCap:" | grep -A 2 -i -P "(nvme|nvm express)" would try to display the used and maximum PCI lanes of each SSD, for sensitizing users.

sitsofe

@ErwanAliasr1 Could you push this script through Shellcheck and see what you think of its comments?

ErwanAliasr1 · 2021-09-17T10:50:49Z

That was a first run to share the idea. I will improve it Le ven. 17 sept. 2021 à 11:10, Sitsofe Wheeler ***@***.***> a écrit :

…

***@***.**** commented on this pull request. @ErwanAliasr1 <https://github.com/ErwanAliasr1> Could you push this script through Shellcheck and see what you think of its comments? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1272 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCUIHM6NKOB3QNKICKULQDUCMA2FANCNFSM5EFTIOPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

ErwanAliasr1 · 2021-09-17T17:47:55Z

@ErwanAliasr1 Could you push this script through Shellcheck and see what you think of its comments?
Fixed my IDE setup, only minor stuff. I'll push an update.

ErwanAliasr1 · 2021-09-17T20:02:56Z

Just pushed a new version that set the proper cpu scaling & idle governor and set io_poll to 1

ErwanAliasr1 · 2021-09-17T21:02:33Z

@ByteHamster Just push a version that prints the nvme config like :
nvme0n1: MODEL=KCD6XVUL3T20 FW=GPK1 serial=11E0A016TX1R PCI=0000:44:00.0@16.0 GT/s PCIe IRQ=63 NUMA=4 CPUS=16-19,48-51

Hope you'll like it

Associated to fio, the t/io_uring test is used to compute the max IOPS a single core can get. Jens published several times the procedure he uses, but trying to reproduce this setup is error-prone. It's easy to miss a configuration and get a different result. This script is about setting up a common setup to reproduce these runs. From the fio directory, execute like the folliowing : [user@fio] t/one-core.sh /dev/nvme0n1 [other drives] ##################################################: system: CPU: AMD EPYC 7502P 32-Core Processor system: MEMORY: 2933 MT/s system: KERNEL: 5.10.35-1.el8.x86_64 nvme0n1: MODEL=Samsung SSD 970 EVO Plus 2TB FW=2B2QEXM7 serial=S59CNM0R417706B PCI=0000:01:00.0@8.0 GT/s PCIe IRQ=64 NUMA=0 CPUS=0-23 nvme0n1: set none as io scheduler nvme0n1: iostats set to 1. nvme0n1: nomerge set to 0. Warning: For better performance, you should enable nvme poll queues by setting nvme.poll_queues=32 on the kernel commande line ##################################################: io_uring: Running taskset -c 0,12 t/io_uring -b512 -d128 -c32 -s32 -p1 -F1 -B1 -n4 /dev/nvme0n1 [...] IOPS=731008, BWPS=356 MB IOS/call=32/31, inflight=(108 127 126 106) This script will take care of the following items: - nvme poll queues - io scheduler - iostats - io_poll - nomerge - finding the logical cores running on the first physical core - cpu frequency governor on performance - cpu idle governor on menu - calling t/io_uring with the proper parameters in 512 bytes fashion - reporting the nvme & pci configuration Signed-off-by: Erwan Velu <erwanaliasr1@gmail.com>

ErwanAliasr1 · 2021-09-20T18:03:59Z

Just added the processor, memory speed & kernel version

ErwanAliasr1 · 2021-09-20T18:04:23Z

@axboe unless you see other stuff to add here, I think this PR is ready

sitsofe reviewed Sep 17, 2021

View reviewed changes

ErwanAliasr1 force-pushed the one-core branch 2 times, most recently from 879d066 to 5f0f5ec Compare September 17, 2021 20:02

ErwanAliasr1 force-pushed the one-core branch from 5f0f5ec to c5e79bb Compare September 17, 2021 21:01

ErwanAliasr1 force-pushed the one-core branch from c5e79bb to c3e15df Compare September 20, 2021 18:02

ErwanAliasr1 force-pushed the one-core branch from c3e15df to 0c6c0eb Compare September 20, 2021 18:03

axboe merged commit d913730 into axboe:master Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

t/one-core.sh: Adding script to run the one-core io benchmark #1272

t/one-core.sh: Adding script to run the one-core io benchmark #1272

ErwanAliasr1 commented Sep 16, 2021

ByteHamster commented Sep 17, 2021

sitsofe left a comment

ErwanAliasr1 commented Sep 17, 2021 via email

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 20, 2021

ErwanAliasr1 commented Sep 20, 2021

t/one-core.sh: Adding script to run the one-core io benchmark #1272

t/one-core.sh: Adding script to run the one-core io benchmark #1272

Conversation

ErwanAliasr1 commented Sep 16, 2021

ByteHamster commented Sep 17, 2021

sitsofe left a comment

Choose a reason for hiding this comment

ErwanAliasr1 commented Sep 17, 2021 via email

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 17, 2021

ErwanAliasr1 commented Sep 20, 2021

ErwanAliasr1 commented Sep 20, 2021