[11:52:09] cp3007 (cache_misc) upgraded to varnish 5 and repooled, it looks good [11:52:34] * elukey checks webrequest logs [11:52:42] see https://wikitech.wikimedia.org/wiki/Varnish#Upgrading_from_Varnish_4_to_Varnish_5 for the upgrade procedure [11:52:54] elukey: thanks! [11:55:12] I can see logs flowing, nothing weird so far [11:55:18] great [11:58:35] journalctl looks good, process runs fine, seems liking varnish 5.1 :) [13:25:21] 10Traffic, 10Analytics, 10Operations, 10User-Elukey: Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls - https://phabricator.wikimedia.org/T177927#3675554 (10ema) p:05Triage>03Normal [13:36:32] 10Traffic, 10Analytics, 10Operations, 10User-Elukey: Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls - https://phabricator.wikimedia.org/T177927#3675614 (10Ottomata) This would really only require passing `kafka_clusters` as well as `kafka_cluster_name` to the... [13:38:59] 10Traffic, 10netops, 10Operations, 10ops-eqiad: Upgrade BIOS/RBSU/etc on lvs1007 - https://phabricator.wikimedia.org/T167299#3675624 (10BBlack) Still says `101-I/O ROM Error` twice on every boot attempt, new NIC card has older firmware. PXE boot still doesn't work (tried setting `Boot Strap Type` to `int1... [14:17:57] 10Traffic, 10netops, 10Operations, 10ops-eqiad: Upgrade BIOS/RBSU/etc on lvs1007 - https://phabricator.wikimedia.org/T167299#3675717 (10BBlack) I gave in and tried a stretch network install on lvs1009 for comparison. I didn't make any bios/firmware changes there, just used RBSU console to `onetimeboot net... [14:30:54] 10Traffic, 10netops, 10Operations, 10ops-eqiad: Upgrade BIOS/RBSU/etc on lvs1007 - https://phabricator.wikimedia.org/T167299#3675774 (10BBlack) I figured as a next minimal testing step on lvs1009, should just go into the ethernet firmware (Ctrl+S) and try disabling SR-IOV and/or HP Shared Memory Features,... [15:29:13] ema: I've a question for https://gerrit.wikimedia.org/r/#/c/383591/ [15:31:46] * ema patiently waits for the question :) [15:31:57] lol, didn't know if you were around :) [15:32:17] so, if I set a host pooled=inactive in conftool, does it change the value of "total"? [15:33:30] volans: I don't think so, total is len(crd.servers) [15:33:46] crd being an instance of Coordinator [15:34:32] ok then [15:35:17] I was just wondering if mistakenly depooling inactive some hosts in a cluster could make it a 1 host cluster and not alarm anymore [15:35:26] yeah good point [15:35:51] thanks for the fix! [16:08:03] 10Traffic, 10Operations, 10Pybal: Upgrade LVS servers to stretch - https://phabricator.wikimedia.org/T177961#3676143 (10ema) [16:09:56] 10Traffic, 10Operations, 10Pybal: Upgrade LVS servers to stretch - https://phabricator.wikimedia.org/T177961#3676173 (10ema) p:05Triage>03Normal [16:27:27] 10Traffic, 10Operations, 10Pybal: Upgrade LVS servers to stretch - https://phabricator.wikimedia.org/T177961#3676143 (10BBlack) One significant thing to keep in mind is the interface naming changes. We'll be going from e.g. `eth[0-3]` to something like `eno[1-2], ens1f[0-1]`, and we'll have to work that int... [16:27:51] 10Traffic, 10netops, 10Operations, 10ops-eqiad: Upgrade BIOS/RBSU/etc on lvs1007 - https://phabricator.wikimedia.org/T167299#3676209 (10BBlack) Got arrow keys working in Ctrl-S (thanks @Fgiunchedi !) by re-setting the local terminal. There is no "HP Shared Memory Features" prompt in the current NIC firmwa... [16:39:58] ooohh vector tiles :) [16:59:03] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-Incident: Collect Backend-Timing in Graphite (or Prometheus) - https://phabricator.wikimedia.org/T131894#3676319 (10Krinkle) [17:12:00] ? [17:14:55] 10Traffic, 10Operations: Network hardware purchasing for Asia Cache DC - https://phabricator.wikimedia.org/T162683#3676410 (10faidon) [19:00:23] paravoid: happened to see https://gerrit.wikimedia.org/r/#/c/383398/ go by earlier [21:54:37] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo, 10Patch-For-Review: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3677764 (10RobH) xe-2/0/3 up up cp4005 xe-2/0/4 up up cp4006 xe-2/0/5 up up cp4007 xe-2/0/6 up up cp400... [21:54:59] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo, 10Patch-For-Review: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3677765 (10RobH) [22:23:20] 10Traffic, 10Operations, 10ops-ulsfo: cp4026 memory error - https://phabricator.wikimedia.org/T178011#3677852 (10RobH) [22:29:53] 10Traffic, 10Operations, 10RESTBase-API, 10Patch-For-Review: [feature request] Redirect root API path to docs page - https://phabricator.wikimedia.org/T125226#3677898 (10GWicke) a:05GWicke>03None [22:30:57] 10Traffic, 10Operations, 10ops-ulsfo: cp4026 memory error - https://phabricator.wikimedia.org/T178011#3677901 (10RobH) [22:35:44] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3677915 (10RobH) [22:38:32] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo, 10Patch-For-Review: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3677935 (10RobH) [22:39:51] 10Traffic, 10Operations, 10ops-ulsfo: cp4026 memory error - https://phabricator.wikimedia.org/T178011#3677938 (10BBlack) This will self-depool if you do a clean shutdown from software. We just need to verify + repool manually afterwards. [23:23:03] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo, 10Patch-For-Review: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3678113 (10RobH)