[07:47:33] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9705875 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host cp3072.esams.wmnet with OS bullseye [08:26:52] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9705962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host cp3072.esams.wmnet with OS bullseye executed with errors: - cp3072 (... [08:40:42] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9705992 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host cp3072.esams.wmnet with OS bullseye [09:32:21] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9706148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host cp3072.esams.wmnet with OS bullseye completed: - cp3072 (**WARN**)... [09:38:45] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9706201 (10Fabfur) [12:08:05] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: 14Move 70% of mediawiki external requests to mw on k8s - 14https://phabricator.wikimedia.org/T360763#9706729 (10Clement_Goubert) 05In progress→03Resolved [12:13:47] 10netops, 10Ganeti, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: 14Investigate Ganeti in routed mode - 14https://phabricator.wikimedia.org/T300152#9706756 (10ops-monitoring-bot) 14cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: `testvm2008.wikimedia.org` - testv... [12:14:29] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323 (10Clement_Goubert) 03NEW [12:14:53] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9706771 (10Clement_Goubert) p:05Triage→03High [12:32:54] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9706818 (10Clement_Goubert) [12:39:55] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9706843 (10jijiki) [12:40:45] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9706846 (10jijiki) [12:48:14] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9706856 (10Clement_Goubert) [12:51:03] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9706874 (10Clement_Goubert) [13:08:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9706934 (10Papaul) [13:35:09] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9707012 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp3073.esams.wmnet with OS bullseye [13:41:11] 06Traffic, 06DC-Ops, 10ops-codfw, 10ops-eqiad, 10SRE-swift-storage: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9707026 (10ssingh) Traffic reimaged 8 text nodes in esams and all of them PXE-booted the first time, without any issues. I think looking... [13:47:00] 06Traffic, 06DC-Ops, 10ops-codfw, 10ops-eqiad, 10SRE-swift-storage: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9707044 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp2042.codfw.wmnet with OS b... [13:49:52] 06Traffic: Upgrade to HAProxy 2.6.17 - https://phabricator.wikimedia.org/T362063#9707101 (10Vgutierrez) [13:54:41] 10netops, 10Ganeti, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: 14Investigate Ganeti in routed mode - 14https://phabricator.wikimedia.org/T300152#9707155 (10ops-monitoring-bot) 14cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: `testvm2008.wikimedia.org` - testv... [14:04:02] 06Traffic, 06DC-Ops, 10ops-codfw, 10ops-eqiad, 10SRE-swift-storage: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9707240 (10ssingh) @Papaul suggested to try a host in codfw and `cp2042` PXE booted successfully. In one of the above messages, @cmooney... [14:24:51] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9707334 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp3073.esams.wmnet with OS bullseye completed: - cp3073 (**PASS**)... [14:26:33] 06Traffic, 06DC-Ops, 10ops-codfw, 10ops-eqiad, 10SRE-swift-storage: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9707355 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp2042.codfw.wmnet with OS bulls... [14:29:47] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: esams text cp nvme upgrade - https://phabricator.wikimedia.org/T360430#9707409 (10ssingh) [14:46:25] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE, 13Patch-For-Review: 14esams text cp nvme upgrade - 14https://phabricator.wikimedia.org/T360430#9707488 (10Fabfur) 05Open→03Resolved [14:55:36] 06Traffic: Upgrade to HAProxy 2.6.17 - https://phabricator.wikimedia.org/T362063#9707531 (10Vgutierrez) [15:58:33] 06Traffic: Upgrade to HAProxy 2.6.17 - https://phabricator.wikimedia.org/T362063#9707777 (10Vgutierrez) [20:02:40] (VarnishHighThreadCount) firing: (6) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:07:40] (VarnishHighThreadCount) firing: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:12:40] (VarnishHighThreadCount) firing: (9) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:17:40] (VarnishHighThreadCount) firing: (16) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:32:40] (VarnishHighThreadCount) firing: (11) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:37:40] (VarnishHighThreadCount) resolved: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:04:40] (VarnishHighThreadCount) firing: Varnish's thread count on cp3068:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3068 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:09:40] (VarnishHighThreadCount) firing: (8) Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:19:40] (VarnishHighThreadCount) firing: (16) Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:39:40] (VarnishHighThreadCount) resolved: (8) Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount