[08:37:24] hello folks! [08:37:57] something interesting - cloudelastic10[12] seem to have two network cards - one offering two 10G ports (intel based) and one offering two 25G ports (supermicro) [08:38:11] the former only accepts UEFI settings [08:38:38] I checked in the BIOS config and I see only references of UEFI, but I am still not 100% sure [08:38:43] I filed https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1101457 for these cases [08:39:04] but if this holds it may be another push towards UEFI :D [10:22:55] 10netops, 06Data-Platform-SRE, 06Infrastructure-Foundations, 06SRE: Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10389706 (10BTullis) This change looks fine to me, but would it be OK to wait until the New Year to implement it? I'm just a bit cautious... [10:43:04] 10Packaging, 10Thumbor, 10Wikimedia-SVG-rendering, 07User-notice: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674#10389809 (10TheDJ) [10:43:08] 10Packaging, 10Thumbor, 10Wikimedia-SVG-rendering, 07User-notice: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674#10389812 (10TheDJ) 05Open→03Resolved a:03TheDJ Yes, ever since July'ish [14:34:54] elukey that's cool...there is no urgency to deploy the cloudelastic hosts, so if we can do anything to help test UEFI lmk [14:35:57] inflatador: o/ if you want to use UEFI with those it is fine, we need to add the right partman recipes etc.. [14:36:12] otherwise we can keep as is, remembering about those 10g NICs [14:36:19] we'll probably not use it ever, but.. [14:38:08] ACK, I'll give UEFI a shot if that's OK w/y'all. I tried using it on one of our new R450s on Friday, but the Dell chassis seemed to insist on enabling Lifecycle Manager before UEFI? Have y'all run into that? More notes here: https://wikitech.wikimedia.org/w/index.php?title=Talk%3AUEFI_Boot#Results_from_UEFI_boot_of_wdqs1025 [14:42:59] inflatador: very weird. have you tried to reset the idrac as suggested? [14:44:21] oops, completely missed that ;( . Trying now [14:48:49] no, chassis is still not cooperating after reset. Have y'all tried using UEFI w/R450 yet? And if so, did y'all have to enable Lifecycle Manager? I don't have a problem with enabling it, just wanted to make sure that was necessary [14:49:31] I think we only tried with sretest1001, so not a lot of use cases [14:50:28] I am not 100% sure what `racadm jobqueue create BIOS.Setup.1-1` does, and why you enabled in that away, but it seems something different [14:51:08] AFAIK It's the same thing, just going thru DRAC directly instead of redfish API [14:51:43] BTW I have to correct myself-looks like the cookbook is working now [14:51:57] Very strange, I ran it at least 10 times plus the job queue stuff on Friday [14:52:50] maybe the reset worked? [14:53:21] I guess so...I still had to retry 4 times but redfish isn't the most stable interface ;) [14:54:08] it takes a bit of time for the reset to take fully effect, I think it may be it [14:54:14] good that works though! [14:54:50] definitely. Will report back once the job/reboot is done [14:55:11] thanksss [15:52:45] so enabling the life-cycle controller was not needed? [15:53:08] IIUC nope [15:53:31] ok, I wasn't sure if the jobqueue stuff depended on the lifecycle controller [16:16:41] looks like the lifecycle controller is enabled on wdqs1025 after running the cookbook, and it is enabled on sretest1003 (which is also an R450). So maybe that happens implicitly when we flip other bits [16:32:03] inflatador: thanks, the exact features of the lifecycle controller are a bit murky to me [18:05:34] FIRING: DiskSpace: Disk space build2001:9100:/ 0.02103% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:31:38] wdqs1025 is now booting in UEFI mode! left some notes at https://w.wiki/CMnj but TLDR that I just needed to reset the DRAC and reimage the host again [19:32:21] belated thanks to v-olans and moritz-m for cleaning up my mess in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1101020 [19:58:25] good to hear inflatador, thanks for the notes as well [20:04:40] jhathaway NP, thanks to you and everyone else for fixing this up [20:07:44] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [20:12:44] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [20:12:44] FIRING: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [20:17:44] FIRING: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [20:17:44] RESOLVED: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [20:22:44] RESOLVED: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [22:05:34] FIRING: DiskSpace: Disk space build2001:9100:/ 5.905% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace