Download an image not using all bandwidth

[aweiand]

Hi!

We have an issue with download image. The host not using all the bandwidth we have (1Gb), but use average 100 ~ 150MB. But, with write (on capture image), the server use all the bandwidth....

[Sebastian-Roth]

@aweiand Which version of FOG do you use and which kernel (command: file /var/www/{,html/}fog/service/ipxe/bzImage*)?

Do you capture and deploy to the same kind of hardware or different models? Multicast or unicast deploy?

[aweiand]

Hi @Sebastian-Roth ! Thank's by the quickly reply!

This is the output:

/var/www/html/fog/service/ipxe/bzImage:   Linux kernel x86 boot executable bzImage, version 5.15.68 (buildkite-agent@Tollana) #1 SMP Sun Oct 9 06:54:22 CDT 2022, RO-rootFS, swap_dev 0x8, Normal VGA
/var/www/html/fog/service/ipxe/bzImage32: Linux kernel x86 boot executable bzImage, version 5.15.68 (buildkite-agent@Tollana) #1 SMP Sun Oct 9 06:50:09 CDT 2022, RO-rootFS, swap_dev 0x8, Normal VGA

We are capturing and deploying in the same machine/hardware under unicast deploy

[Sebastian-Roth]

@aweiand Again, which version of FOG do you use (bottom right corner when being logged into the FOG web UI)?

Do you have other hardware you can test delpoying to? In most cases this kind of issue is hardware specific.

As well you may start digging into what causing the slowness by testing different components one after the other. Let's start with the hard drive. Please make sure you have a safe backup copy of the data on the hard drive of this client machine you use for testing! Schedule a debug deploy task for this machine, boot it up and when you get to the console run the following command:

dd if=/dev/zero of=/dev/sda bs=4G count=1 oflag=direct status=progress

HINT: Be aware this will wipe the data off the drive! Depending on the drive you might need to use nvme0n1 instead of sda. You can use lsblk before running the above command to print out a lit of available hard drives (and partitions).

Run that command three times in a row and finish up with hdparm -I /dev/sda | grep Number (so we know what kind of hard drive this is - again you might need to use a different device name instead of sda). Take a picture of the output on screen and post here.

[aweiand]

@Sebastian-Roth , we use 1.5.9

Do you have other hardware you can test delpoying to? In most cases this kind of issue is hardware specific.

Yes, we test in two different machines. Capture and deploy his respective data. Let's test it =) Thank's a lot!!

Here is the picture of the commands

[Sebastian-Roth]

@aweiand Looks like hdparm can't handle the NVMe. Don't worry about that, probably not an important information anyway.

While speeds are not up to the full throughput I would expect from a NVMe drive it's still good enough to rule it out as causing the bottle neck I suppose.

I just stumbled upon great posts by @George1421 in the forums which I had forgotten about:
https://forums.fogproject.org/topic/15945/performace-testing-slow-fog-imaging
https://forums.fogproject.org/topic/10459/can-you-make-fog-imaging-go-fast

[aweiand]

Hi @Sebastian-Roth !

Ok, I will complete the test list specified in the links and return here with the results!

Thanks' again!

[aweiand]

Hi @Sebastian-Roth ,

After many tests here (with support of the links), we found that fog has a issue with downloading the image. We test the dd command in a fog debug session and, in a Linux mint distribution on the same machine (mounting the NFS of fog storage) and the results shows that the mint can download the image at 1Gb, but under the fog client this don't pass the 100-200Mb.

[aweiand]

UPDATE:

We are triyng in a different machine, that doesn't have uefy. On this machine, the steps suggested in the first link as you show, demonstrates a dd command download in 1Gb. But, when using fog, the limit is the same as commented above (100-200Mb). There is a image that shows:

This other screenshot shows our firewall, downloading a image to a machine with HDD and Legacy mode (in a UEFY mode shows the same results)

This graphic shows a downloading image to a machine with NVMe and UEFY mode:

[Sebastian-Roth]

@aweiand said:

After many tests here (with support of the links), we found that fog has a issue with downloading the image. We test the dd command in a fog debug session and, in a Linux mint distribution on the same machine (mounting the NFS of fog storage) and the results shows that the mint can download the image at 1Gb, but under the fog client this don't pass the 100-200Mb.

Ok, on first sight this looks like a network (driver) issue then. But we can't be sure because your testing still combines disk IO with network IO and therefore we can't say what is causing the bottle neck.

Please revisit the links mentioned and test using iperf. Best if you can test both on FOS (FOG Linux OS) and Mint.

As well please run the following commands (both on FOS and Mint) to show the network driver used:

lspci -vnn | grep -e "^[0-9a-f]" -e "Kernel module" | grep -i net -A1
ls -al /sys/class/net/*/device/driver

[aweiand]

@Sebastian-Roth , we realized the iperf3 tests but I'm forgot to described here (and I don't remember that). Now I'm on vacation until February... but I will ask my colleagues to test this, and write here the results...

Thank's again!!
Happy Christmas and new Year!

[Sebastian-Roth]

@aweiand Any news on this?

[aweiand]

@Sebastian-Roth I'm still on vacation unit February... My colleagues don't have tested again...

[Sebastian-Roth]

@aweiand Looking forward to hearing from you in February... :-)