8000 Kerner parameter "vfio-pci.ids" is ignored. · Issue #20 · anatol/booster · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Kerner parameter "vfio-pci.ids" is ignored. #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dron1885 opened this issue Jan 29, 2021 · 34 comments
Closed

Kerner parameter "vfio-pci.ids" is ignored. #20

dron1885 opened this issue Jan 29, 2021 · 34 comments

Comments

@dron1885
Copy link

With vfio-pci.ids=1002:67df,1002:aaf0 secondary GPU should not be activated.
@anatol
Copy link
Owner
anatol commented Jan 29, 2021 •
edited
Loading

I am not quite understand the nature of this problem. Could you please provide more information what exactly the issue is, how to reproduce it and why/how this parameter should be handled by booster?

@GabTheBab
Copy link
GabTheBab commented Feb 24, 2021 •
edited
Loading

I believe I have the same problem, so hopefully I can describe it better. The kernel command line option vfio-pci.ids doesnt seem to work with booster images, while the same command line option works with a dracut image. Normally it would assign the vfio-pci driver to the devices you specify, and thats how the same boot entry works when pointed to a dracut image. My best guess would probably be some timing issue, perhaps the nvidia driver is getting loaded before this option can do its thing?

@anatol
Copy link
Owner
anatol commented Mar 25, 2021

Booster does not do anythong with this parameter. It just passed over to linux kernel.

I am looking at dracut project and do not see any mention of it https://github.com/dracutdevs/dracut/search?q=vfio-pci.ids
neither systemd contains any references https://github.com/systemd/systemd/search?q=vfio-pci.ids

I only see it in the linux kernel here https://github.com/torvalds/linux/blob/719bbd4a509f403f537adcaefd8ce17532be2e84/drivers/vfio/pci/vfio_pci.c#L39

So the loadable kernel module vfio_pci is the one that handles this parameter. You need to make sure the module is added to the image. Add following line to the booster config modules: vfio_pci. Additionally you may want to enable the booster debug and make sure the module is loaded at the boot time.

@GabTheBab
Copy link
GabTheBab commented Mar 25, 2021 •
edited
Loading

modules: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd

This is whats inside my booster.yaml, I see the modules inside of the initramfs if I use lsinitrd, running booster debug, I can't seem to find where its logging stuff to, I dont see anything about it inside of journalctl for that boot.

@GabTheBab
Copy link

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Loading_vfio-pci_early

Here is how things would normally be done on mkinitcpio/dracut for reference

@anatol
Copy link
Owner
anatol commented Mar 25, 2021 •
edited
Loading

@GabTheBab to enable log add booster.debug kernel parameter to your bootloader config. Once it is rebooted you'll see a lot of text during the boot process.

thanks for the link. The equivalent of if for booster would be the modules like you specified.

The only question I have here whether the kernel autoload it (using modalias) or booster has to force these modules loading. The force loading does not work yet but I can come up with a quick proof-of-concept if you are willing to test it.

@GabTheBab
Copy link

I do have booster.debug=1 set in my systemd boot entry, and I'm definitely willing to do some testing if needed. The text during the boot process sadly doesn't stay on screen long enough for me to read it, is there any place i could see the debug info after boot?

@anatol
Copy link
Owner
anatol commented Mar 25, 2021 •
edited
Loading

is there any place i could see the debug info after boot?

It is currently not, but I filed #60 to investigate it

@anatol
Copy link
Owner
anatol commented Mar 25, 2021

Please try this branch https://github.com/anatol/booster/tree/force_modules_loading it adds a new booster.yaml config option modules_force_load: that specifies what modules need to be explicitly loaded at the boot time. Add your vfio modules there.

@GabTheBab
Copy link

Built commit c1a3246, added the new setting to booster.yaml and regenerated the initramfs, sadly still seems to load the nvidia driver rather than vfio-pci.

@GabTheBab
Copy link

modules: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd
modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd
compression: zstd

my full booster.yaml

@anatol
Copy link
Owner
anatol commented Mar 25, 2021

It turns out that there was a circular dependencies between vfio and vfio_iommu_type1

modules.softdep:softdep vfio post: vfio_iommu_type1 vfio_iommu_spapr_tce
modules.dep:kernel/drivers/vfio/vfio_iommu_type1.ko.xz: kernel/drivers/vfio/vfio.ko.xz

that booster did not handle correctly. I added a quick fix for this issue and now I see booster loads these modules at the boot time.

I pushed it to force_modules_loading. Please try it again.

WRT your config compression: zstd is the default option, no need to specify it. modules_force_load implicitly added to modules list, no need to specify it twice. Following config is equivalent to one printed above.

modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd

@GabTheBab
Copy link
GabTheBab commented Mar 25, 2021 •
edited
Loading

force_modules_loading or modules_force_load?

@anatol
Copy link
Owner
anatol commented Mar 25, 2021 •
edited
Loading

Sorry for the confusion, it is modules_force_load branch.

@GabTheBab
Copy link

Built d620be6, config is just modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd, rebuilt initramfs, still loading nvidia driver.

@anatol
Copy link
Owner
anatol commented Mar 25, 2021

Once you booted what is the output of your lsmod | grep vfio and cat /proc/cmdline?

< 8000 /svg>

@GabTheBab
Copy link 
vfio_pci               73728  0
vfio_iommu_type1       45056  0
vfio                   45056  2 vfio_iommu_type1,vfio_pci
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm

@GabTheBab
Copy link
GabTheBab commented Mar 25, 2021 •
edited
Loading

initrd=\booster-linux-zen.img root=LABEL=btrfsroot rw add_efi_memmap loglevel=3 quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction vfio-pci.ids=10de:1b80,10de:10f0 booster.debug=1

@anatol
Copy link
Owner
anatol commented Mar 25, 2021 •
edited
Loading

The list of loaded modules is correct. The command line parameter is also in place.

I quickly looked at the mkinitcpio and dracut and I do not see any special about vfio modules. Their code just says load these modules and that's it.

As a last attempt to solve this problem I pushed one more update to modules_force_load branch that loads modules_force_load, waits until they are fully loaded and only then it moves forward with the rest of the modules discovery. Please try it and see if it makes any difference.

@GabTheBab
Copy link

Sadly it still is loading the nvidia driver on commit 52144a5.

@anatol
Copy link
Owner
anatol commented Mar 25, 2021

Unfortunately I am out of ideas what happens here. If anyone has better understanding how vfio binding suppose to work please share your thoughts.

anatol added a commit that referenced this issue Mar 26, 2021
@anatol
Add a config option to force modules loading at boot time
3158e06 
Some configurations require loading additional modules at the boot time.
modules_force_load gives such opportunity.

Any module added to modules_force_load implicitly added to the image as
well.

Issue #20
@anatol
Copy link
Owner
anatol commented Mar 26, 2021

I merged modules_force_load config option to master in case if anyone wants to try to reproduce the issue.

anatol added a commit that referenced this issue Mar 26, 2021
@anatol
Add a config option to force modules loading at boot time
4848bca 
Some configurations require loading additional modules at the boot time.
modules_force_load gives such opportunity.

Any module added to modules_force_load implicitly added to the image as
well.

Issue #20
@anatol
Copy link
Owner
anatol commented Mar 27, 2021

I just added support for saving booster logs into kmsg/journalctl. It is available in master now.

It worth enabling this feature to get more information about what is going on at the boot time.

@GabTheBab
Copy link 
Mar 28 09:23:00 archlinux booster: loading module vfio
Mar 28 09:23:00 archlinux booster: loading module vfio_virqfd
Mar 28 09:23:00 archlinux booster: loading module irqbypass
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux booster: no match found for alias acpi:LNXCPU:
Mar 28 09:23:00 archlinux kernel: VFIO - User Level meta-driver version: 0.3
``` The nvidia driver loads after this, not before it, not sure why its grabbing onto it

@anatol
Copy link
Owner
anatol commented Mar 28, 2021

Could you please make sure to add printk.devkmsg=on to disable kmsg throttling and then share full logs (at least the pats related to booster & vfio)?

The nvidia driver loads after this, not before it, not sure why its grabbing onto it

From what I read above nvidia driver should not be loaded at all. But here you say you expect nvidia is loaded before vfio? What is the expected behavior?

@GabTheBab
Copy link

The expected behavior is the nvidia driver to not load, however the nvidia driver ends up loading after the messages I sent above.

@GabTheBab
Copy link

log.txt

@anatol anatol mentioned this issue Mar 28, 2021
Closed
anatol added a commit that referenced this issue Mar 28, 2021
@anatol
A quick test for #20
8ec6c19
@anatol
Copy link
Owner
anatol commented Mar 28, 2021

@GabTheBab per your log.txt booster boots fine, loads vfio drivers, switches to the host userspace. nvidia driver is loaded in the userspace.

I compared your vfio output with this one https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Verifying_that_the_configuration_worked and do not see following logs

[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000

After reading kmod code I found that kernel does not use cmdline params for loadable modules. It was surprising for me. Instead it is expected that userspace handles cmdline parsing and provides required module params explicitly.

Here is a quick check to verify this hypothesis. @GabTheBab could you please compile and try following branch to see if it makes any difference for you. And if it does not please provide the dmesg logs. https://github.com/anatol/booster/tree/issue-20-quick-check

@GabTheBab
Copy link

That looks to have done it, vfio-pci is loaded and attached to the gpu.

@anatol
Copy link
Owner
anatol commented Mar 28, 2021

Yay! It is great to hear the problem is solved.

I will work on #74 and provide a clean solution for your issue.

@GabTheBab
Copy link

Thanks for working on this issue, looking forward to the final solution in master

@anatol anatol closed this as completed in 355b308 Mar 30, 2021
@anatol
Copy link
Owner
anatol commented Mar 30, 2021

Done. It is part of master now. Please try it.

@GabTheBab
Copy link

Working on master now

@anatol
Copy link
Owner
anatol commented Apr 4, 2021

This feature got released as part of booster-0.4 now in the stable repository.

I updated the wiki page with booster instructions https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#booster
But if you see other ways to improve the documentation please update it as well.

@bileslav bileslav mentioned this issue Jul 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@anatol @dron1885 @GabTheBab
0