[08:21:21] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4183902 (10Vgutierrez) Right, from ethtool: ``` root@lvs1016:~# ethtool -i enp5s0f0 | grep firmware firmware-version: bc 7.14.10 root@lvs1016:~# ethtool -i enp4s0f0 |grep fir... [08:46:15] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4183923 (10Vgutierrez) Checking kernel logs we've something weird going on with bnx2x fws across LVS servers: ```root@lvs1016:~# dmesg |grep direct-loading [ 15.744692] bnx... [09:25:52] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4183939 (10Vgutierrez) oh, and it looks like it isn't consistent across reboots: ``` root@lvs1016:/var/log# grep direct-loading kern.log May 4 14:34:11 lvs1016 kernel: [... [10:41:03] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4183998 (10BBlack) I don't think the firmware versions you see there in ethtool and/or dmesg are the whole story anyways. The ones visible from bios-level setup have like 6... [10:53:22] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4184003 (10Vgutierrez) I've seen the VPD access failed error in other boxes as well, currently: ``` ===== NODE GROUP ===== (1) lvs2005.codfw.wmnet ----- OUTPUT of 'dmesg | gr... [10:59:44] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4184004 (10BBlack) Yeah, lvs200x are all HPs as well, so they're "different" in many respects for better or worse, and getting replaced this quarter with something more-like... [11:05:40] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4184005 (10Vgutierrez) right, so let's @Cmjohnson update firmware on lvs1016 NICs and I'll check MSI-X status after that :) [18:11:06] 10Wikimedia-Apache-configuration, 10cloud-services-team, 10wikitech.wikimedia.org, 10Patch-For-Review, 10Regression: https://wikitech.wikimedia.org/view/ no longer redirects to /wiki - https://phabricator.wikimedia.org/T193848#4184331 (10Krinkle) >>! In T193848#4182586, @Legoktm wrote: > Did wikitech use...