Recently I installed three equal IBM server x 3650 machines with VMware and installed and connected two QLogic HBA’s in each server.
After installing ESX I ran into a problem with one of the machines trying to connect to the datastores. Two of the machines perfectly could see all presented luns after a rescan. One machine however, did not even see a card, no WWN’s available. At first I thought the Storage guy made some mistake in presenting the LUNs, so I called him right away, but he assured me that the faulty machine was targeted. So the investigation continued…
Some quick searches on Google did not bring me any further than pointing me at the HBA config-utility that is initialized at boot time to look if there’s something wrong with the HBA.
While rebooting the host I was looking for the QLogic banner which was not appearing, but instead of a banner I got an error message:
PCI device resource allocation error:
Next when booting into the Console screen there also turned up an error:
When I saw these errors I was not happy assuming I had an hardware issue/failure.
When I googled on the specific error message I just got a few hits, one of them (from a IBM developer site I believe) stated that there is a limited amount of memory available to load ROM bios-apps, and if you have a lot of devices like RAID controllers, HBA controllers, network controllers etc. you may reach a point to where a device ROM BIOS will not load. He also suggested to turn off unneeded ROM BIOS apps/PXE boot options.
Because I indeed have up to four nics AND a RAID controller AND two H’BA’s, I went into the BIOS and turned off the PXE boot option of the onboard nics in the BIOS:
After the PXE was disabled on the onboard nics, the system apparently had enough resources to load the Qlogic Firmware and also ESX was able to load the driver and see the presented LUNs and booted normally (as the others).
If you enjoyed this article, please consider sharing it!
Great good-to-know article.Since you were installing 3 servers I assume they all had the same hardware, amount of NIC’s, HBA’s etc. Any idea as to why only one of the servers experienced the problem?
Hi Dennis, Thanks for your response. Your assumption was right, they all have the same hardware amount of nics etc.
I don’t know for sure, but I can imagine that a pci slot order also makes a difference. It may wel be that on the other two the pxe devices are last in row and are not able to load, which is not noticed because they are not used.
Hi Gerben, thanx for the explanation. That could well be the case. I little bit of an unwanted situation then because those ESX servers are not all the same. When I deploy the same servers (be it ESX or just some Windows servers) I make sure everything is the same, even the placement of cards in PCI slots. Anyways a good thing you ran into this, good to know.
Hi,
i add two nics in my 3650x server and i got same error, i followed your steps and it works fine
Thank you so much
My x3650 was having the same issue. Thanks for the helpful fix.
Interesting, I never would’ve thought to look for this.. hit up a similar problem with an x3550. Thanks for publishing this!
I encountered the same issues and error messages you mentioned with PCI NICs and a raid controller already in my ESX host. I would never have thought to look and turn this off in the BIOS. Thanks for sharing, saved me an awful lot of TShooting. Keep up the good work
Hi, i encourntered the same problem after upgrading the firmware of emulex hba cards. I have to upgrade about 7 servers in all. With 2 quad port nics and 2 hba cards in each, i definitly dont’t have enough rom space.
Thank you for sharing !
Hello, I have the same problem with x3650 server, explain what I have done, but the problem perciste, I have 2 plates 6GB SAS HBA, connected to a DS3524 storage, please can you help.