[00:03:40] 10Tool-delintbot: Fix cases of misnested tags - https://phabricator.wikimedia.org/T415814#11704893 (10Redmin) [00:31:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [01:01:56] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2) [01:01:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [01:41:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [01:59:35] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11704983 (10Abhimarkzz) @Nokib_Sarkar and @Tiven2240 Hey there, I've implemented the **useSession** hook which was empty in the codebase. It handles **session fetching**, **loadin... [03:30:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [05:00:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [05:16:38] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11705155 (10Abhimarkzz) @Nokib_Sarkar annd @Tiven2240 PR #14 update — all Sentry checks now passing with "No issues found" on the latest commit. Fixed 6 bugs across the review cyc... [05:31:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [05:41:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [06:01:26] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11705165 (10Abhimarkzz) Opened a second PR implementing the **useLocalStorage hook ** (listed as missing in Phase 5 of the roadmap). Includes versioning, TTL expiry, migration suppor... [06:20:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [07:46:09] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11705285 (10Abhimarkzz) @Nokib_Sarkar and @Tiven2240 — would appreciate a review when you get a chance. All automated checks are passing. https://github.com/Open-Knowledge-Impact-Fo... [10:50:15] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11705832 (10ShakespeareFan00) Some further errors this morning: https://ws-page-game.toolforge.org/?index=The%20diagnosis%20and%20treatment%20of%20disea... [11:00:04] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11705896 (10TheDJ) This seems to be ratelimiting of the wikisource API, which then returns an errorpage without CORS headers. Doesn't seem related to the... [11:27:41] (03CR) 10Jean-Frédéric: [C:03+2] "Thanks!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1239197 (https://phabricator.wikimedia.org/T410842) (owner: 10Multichill) [11:29:24] (03Merged) 10jenkins-bot: Fix harvesting jobs [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1239197 (https://phabricator.wikimedia.org/T410842) (owner: 10Multichill) [11:35:37] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11706057 (10arya55) Hi @Nokib_Sarkar and @Tiven2240, I had previously completed the CampaignEditForm microtask. Since I did not see any additional specific microtasks listed and noticed that several components still... [12:42:06] 10Tool-Pageviews: "Category" Massviews error: An unknown error occurred when querying - https://phabricator.wikimedia.org/T419975 (10Pschemmo) 03NEW [12:43:35] 10Tool-Pageviews: "Category" Massviews error: An unknown error occurred when querying - https://phabricator.wikimedia.org/T419975#11706293 (10Pschemmo) [13:00:58] 10Tool-delintbot: Edit across all namespaces in one go based on a sensible list - https://phabricator.wikimedia.org/T415890#11706386 (10Redmin) [13:08:36] 10Tool-curator: Integrate Panoramax - https://phabricator.wikimedia.org/T419982 (10DaxServer) 03NEW [13:09:07] 10Tool-curator: Integrate Panoramax - https://phabricator.wikimedia.org/T419982#11706424 (10DaxServer) p:05Triage→03Low [13:18:28] (03CR) 10Majavah: [C:03+2] Don't trigger on reopens [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1079048 (https://phabricator.wikimedia.org/T374157) (owner: 10Pppery) [13:18:56] (03Merged) 10jenkins-bot: Don't trigger on reopens [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1079048 (https://phabricator.wikimedia.org/T374157) (owner: 10Pppery) [13:21:52] (03Abandoned) 10Majavah: deployment: explicitely require nfs mount [cloud/toolforge/jobs-framework-api] - 10https://gerrit.wikimedia.org/r/840266 (owner: 10Majavah) [13:22:33] (03Abandoned) 10Majavah: cli: replace global --verbose with per-command options [cloud/toolforge/toolforge-cli] - 10https://gerrit.wikimedia.org/r/914740 (owner: 10Majavah) [13:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:39:06] 10Tool-inteGraality: integraality column for EntitySchema compliance? - https://phabricator.wikimedia.org/T347897#11706577 (10JeanFred) p:05Triage→03Low [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:17:01] 06cloud-services-team: cloudcumin not able to communicate with openstack.eqiad1.wikimediacloud.org:25000 anymore - https://phabricator.wikimedia.org/T419996 (10fgiunchedi) 03NEW [14:17:58] 06cloud-services-team, 10Cloud-VPS: cloudcumin not able to communicate with openstack.eqiad1.wikimediacloud.org:25000 anymore - https://phabricator.wikimedia.org/T419996#11706817 (10taavi) [14:22:31] 10Tool-inteGraality: Wikidata:WikiProject every politician/Statistics not updating - https://phabricator.wikimedia.org/T387070#11706869 (10JeanFred) 05Open→03Resolved a:03JeanFred Now succeeding when using QLever endpoint. Closing as resolved. [14:32:36] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: integraality user research - https://phabricator.wikimedia.org/T420002 (10JeanFred) 03NEW [14:43:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:48:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:13:21] 06cloud-services-team, 10Beta-Cluster-Infrastructure, 10Bitu, 10CAS-SSO, and 3 others: Update basedn in CAS - https://phabricator.wikimedia.org/T371930#11707220 (10Arendpieter) 05Open→03Resolved [15:15:06] 06cloud-services-team, 10Beta-Cluster-Infrastructure, 10Bitu, 10CAS-SSO, and 3 others: Update basedn in CAS - https://phabricator.wikimedia.org/T371930#11707227 (10taavi) @Arendpieter AFAICS the patch here is still open, how is this Resolved? [15:23:25] (03PS1) 10Jean-Frédéric: Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 [15:26:01] (03PS2) 10Jean-Frédéric: Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) [15:27:38] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11707289 (10Bawolff) Is this a dupe of T418969 ? [15:28:21] (03CR) 10CI reject: [V:04-1] Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [15:30:33] 10Tool-inteGraality: Wikidata:WikiProject_Govdirectory/Albania/Coverage_of_first_level_agencies. - https://phabricator.wikimedia.org/T415439#11707301 (10JeanFred) I see that integraality suceeded on that page − all good now I think ? https://www.wikidata.org/w/index.php?title=Wikidata:WikiProject_Govdirectory/Al... [15:32:09] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudgw2002-dev - https://phabricator.wikimedia.org/T419738#11707313 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:32:19] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudgw2002-dev - https://phabricator.wikimedia.org/T419738#11707318 (10Jhancock.wm) that did it. ty [15:34:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:34:23] 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11707328 (10bd808) >>! In T414674#11704866, @taavi wrote: >>>! In T414674#11704780, @bd808 wrote: >> # Profit!! > Unfortunately this is missing the step of adding a Gatewa... [15:39:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:40:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1076.eqiad.wmnet}' (T419948) [15:54:13] PROBLEM - Host cloudvirt1076 is DOWN: PING CRITICAL - Packet loss = 100% [15:54:41] RECOVERY - Host cloudvirt1076 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [15:54:51] PROBLEM - ensure kvm processes are running on cloudvirt1076 is CRITICAL: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [15:54:51] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1076.eqiad.wmnet}' (T419948) [15:55:51] RECOVERY - ensure kvm processes are running on cloudvirt1076 is OK: PROCS OK: 1 process with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [15:56:54] (03approved) 10wikigit: feat(linter): add rules and tests for response [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/12 (owner: 10aghirelli) [15:57:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1075.eqiad.wmnet}' (T419948) [15:58:20] (03merge) 10wikigit: feat(linter): add rules and tests for response [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/12 (owner: 10aghirelli) [16:00:24] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707434 (10Yug) [16:04:10] (03PS1) 10Jean-Frédéric: Add link to monuments_config from the home page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251455 (https://phabricator.wikimedia.org/T420017) [16:07:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet.eqiad.wmnet}' (T419948) [16:09:58] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet.eqiad.wmnet}' (T419948) [16:10:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet}' (T419948) [16:14:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1075.eqiad.wmnet}' (T419948) [16:14:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1074.eqiad.wmnet}' (T419948) [16:14:15] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11707536 (10ShakespeareFan00) On discord it was mentioned that https://github.com/inductiveload/paginator/blob/master/src/mw_utils.js#L39 was what might... [16:15:01] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11707540 (10ShakespeareFan00) (Aside: to be fair the entire tool needs an overhaul at some point, or conversion to an inwiki gadget.) [16:15:56] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11707541 (10Yug) [16:16:18] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11707545 (10Yug) [16:17:31] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11707550 (10Yug) [16:24:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2006-dev.codfw.wmnet}' (T419948) [16:24:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' (T419948) [16:24:42] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707573 (10Yug) [16:25:46] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707588 (10Yug) @taavi , Michael informed me that he have a different account. Could you implement the change shown in //Developer... [16:26:05] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707589 (10Yug) 05Resolved→03Open p:05Triage→03Medium [16:30:13] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707599 (10Yug) 05Open→03Resolved After verification in https://horizon.wikimedia.org/project/member/ , all is good. Thanks yo... [16:30:50] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707608 (10Yug) [16:31:13] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707611 (10Yug) [16:31:37] (03PS1) 10Multichill: Add missing geo_naam field for nl nl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251467 [16:32:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1074.eqiad.wmnet}' (T419948) [16:32:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1073.eqiad.wmnet}' (T419948) [16:32:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1074 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [16:33:43] (03CR) 10Multichill: [C:03+2] Add link to monuments_config from the home page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251455 (https://phabricator.wikimedia.org/T420017) (owner: 10Jean-Frédéric) [16:33:54] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707648 (10Yug) [16:35:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [16:36:08] (03Merged) 10jenkins-bot: Add link to monuments_config from the home page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251455 (https://phabricator.wikimedia.org/T420017) (owner: 10Jean-Frédéric) [16:37:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1074 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [16:38:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [16:41:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' (T419948) [16:41:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' (T419948) [16:41:19] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11707694 (10fnegri) +1 [16:43:02] (03CR) 10Jean-Frédéric: [C:03+2] Add missing geo_naam field for nl nl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251467 (owner: 10Multichill) [16:44:58] (03Merged) 10jenkins-bot: Add missing geo_naam field for nl nl [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251467 (owner: 10Multichill) [16:48:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1073.eqiad.wmnet}' (T419948) [16:48:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1072.eqiad.wmnet}' (T419948) [17:04:35] andrew@cloudcumin1001 safe_reboot (PID 396561) is awaiting input [17:04:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1072.eqiad.wmnet}' (T419948) [17:04:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1071.eqiad.wmnet}' (T419948) [17:05:19] FIRING: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1072 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:05:34] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1073 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:11:15] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11707873 (10Yug) [17:11:21] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11707874 (10Yug) [17:18:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1071.eqiad.wmnet}' (T419948) [17:18:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1070.eqiad.wmnet}' (T419948) [17:28:33] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11708020 (10Yug) [17:35:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1070.eqiad.wmnet}' (T419948) [17:35:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1069.eqiad.wmnet}' (T419948) [17:36:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1070 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:41:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1070 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:50:22] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11708092 (10BTullis) Thanks @fnegri. I have now run those commands. ` btullis@an-redacteddb1001:~$ sudo /usr... [17:50:38] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11708097 (10BTullis) 05In progress→03Resolved [17:50:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1069.eqiad.wmnet}' (T419948) [17:50:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1068.eqiad.wmnet}' (T419948) [17:55:38] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11708103 (10BTullis) Another problem relating to the wikireplica views maintenance, as discovered by @fnegri in T414240. The x... [18:11:42] andrew@cloudcumin1001 safe_reboot (PID 402721) is awaiting input [19:18:56] andrew@cloudcumin1001 safe_reboot (PID 402721) is awaiting input [19:27:04] andrew@cloudcumin1001 safe_reboot (PID 402721) is awaiting input [19:27:07] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-5 (cluster eqiad1) [19:27:09] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [19:27:46] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm zuul-k8s-v128-qnv6fefra4wd-node-5 (cluster eqiad1) [19:27:46] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [19:28:16] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-master-0 (cluster eqiad1) [19:28:16] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [19:28:35] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' (T419948) [19:29:09] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm zuul-k8s-v128-qnv6fefra4wd-master-0 (cluster eqiad1) [19:29:09] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [19:34:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1068.eqiad.wmnet}' (T419948) [19:34:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1067.eqiad.wmnet}' (T419948) [19:41:17] FIRING: JobUnavailable: Reduced availability for job pdns_rec in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:46:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:47:28] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11708460 (10Abhimarkzz) @Nokib-Sarkar @Tiven2240 PR #17 ready for **review** 🙏 All Sentry checks passing. This PR includes: - 4 utility hooks (usePermission, useMediaQuery, useOnClic... [19:48:32] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:51:47] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:53:51] andrew@cloudcumin1001 safe_reboot (PID 412781) is awaiting input [19:57:47] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:00:01] !log andrew@cloudcumin1001 paws START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1) [20:00:02] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:00:54] !log andrew@cloudcumin1001 paws END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-master-0 (cluster eqiad1) [20:01:22] andrew@cloudcumin1001 safe_reboot (PID 412781) is awaiting input [20:01:45] !log andrew@cloudcumin1001 zuul START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-9 (cluster eqiad1) [20:01:47] !log andrew@cloudcumin1001 zuul END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm zuul-k8s-v128-qnv6fefra4wd-node-9 (cluster eqiad1) [20:02:00] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-9 (cluster eqiad1) [20:02:01] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [20:02:47] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:02:54] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm zuul-k8s-v128-qnv6fefra4wd-node-9 (cluster eqiad1) [20:02:54] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [20:05:02] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:06:59] !log andrew@cloudcumin1001 7209100e0e744a4fbdf447534d4eb825 START - Cookbook wmcs.vps.instance.stop_start vm k3s-envDB (cluster eqiad1) [20:06:59] andrew@cloudcumin1001: Unknown project "7209100e0e744a4fbdf447534d4eb825" [20:07:37] !log andrew@cloudcumin1001 7209100e0e744a4fbdf447534d4eb825 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm k3s-envDB (cluster eqiad1) [20:07:38] andrew@cloudcumin1001: Unknown project "7209100e0e744a4fbdf447534d4eb825" [20:08:03] !log andrew@cloudcumin1001 o11y START - Cookbook wmcs.vps.instance.stop_start vm phi-syslog-01 (cluster eqiad1) [20:08:33] !log andrew@cloudcumin1001 o11y END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm phi-syslog-01 (cluster eqiad1) [20:13:25] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1067.eqiad.wmnet}' (T419948) [20:13:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1066.eqiad.wmnet}' (T419948) [20:13:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1067 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:17:33] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: Ensure integraality tests run offline - https://phabricator.wikimedia.org/T420054 (10JeanFred) 03NEW [20:18:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1067 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:26:26] !log andrew@cloudcumin1001 paws START - Cookbook wmcs.vps.instance.stop_start vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1) [20:27:19] !log andrew@cloudcumin1001 paws END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1) [20:31:52] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: Ensure integraality tests run offline - https://phabricator.wikimedia.org/T420054#11708632 (10JeanFred) [20:31:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1066.eqiad.wmnet}' (T419948) [20:31:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1065.eqiad.wmnet}' (T419948) [20:32:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1066 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:34:30] 10Tool-inteGraality: Wikidata:WikiProject_Govdirectory/Albania/Coverage_of_first_level_agencies. - https://phabricator.wikimedia.org/T415439#11708655 (10JeanFred) 05Open→03Resolved a:03JeanFred I added a case to handle that kind of errors more gracefully [20:34:58] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: MaxlagTimeoutError when using integraality - https://phabricator.wikimedia.org/T415439#11708663 (10JeanFred) [20:35:21] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: Ensure integraality tests run offline - https://phabricator.wikimedia.org/T420054#11708666 (10JeanFred) Confirmed using pytest-socket [20:35:23] 10Tool-inteGraality, 03Hackathon-Northwestern-Europe-2026: Ensure integraality tests run offline - https://phabricator.wikimedia.org/T420054#11708668 (10JeanFred) 05Open→03Resolved [20:36:22] 10Tool-inteGraality: Error (first time using inteGraality) - https://phabricator.wikimedia.org/T415138#11708671 (10JeanFred) 05Open→03Invalid Will close as resolved then :) [20:36:59] 10Tool-inteGraality: Wikidata:WikiProject India/Schools not updated (since 5 jan 2025) - https://phabricator.wikimedia.org/T388226#11708676 (10JeanFred) 05Open→03Resolved a:03JeanFred Switched the dashboard to use QLever, works now :) [20:37:34] !log andrew@cloudcumin1001 zuul START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-2 (cluster eqiad1) [20:37:36] !log andrew@cloudcumin1001 zuul END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm zuul-k8s-v128-qnv6fefra4wd-node-2 (cluster eqiad1) [20:37:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1066 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:37:55] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-2 (cluster eqiad1) [20:37:55] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [20:38:27] 10Tool-inteGraality: Missing a space in the query - https://phabricator.wikimedia.org/T391523#11708679 (10JeanFred) 05Open→03Resolved a:03JeanFred I know I encountered that before too, but I think it is resolved indeed. :) [20:38:33] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm zuul-k8s-v128-qnv6fefra4wd-node-2 (cluster eqiad1) [20:38:34] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [20:47:37] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1065.eqiad.wmnet}' (T419948) [20:47:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1064.eqiad.wmnet}' (T419948) [20:48:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1065 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:53:49] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1065 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [21:03:54] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1064.eqiad.wmnet}' (T419948) [21:03:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}' (T419948) [21:04:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1064 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [21:09:49] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1064 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [21:16:35] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 START - Cookbook wmcs.vps.instance.stop_start vm zuul-k8s-v128-qnv6fefra4wd-node-1 (cluster eqiad1) [21:16:37] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [21:17:14] !log andrew@cloudcumin1001 c26d9d326bdf464fa1025939ded7e5a2 END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm zuul-k8s-v128-qnv6fefra4wd-node-1 (cluster eqiad1) [21:17:17] andrew@cloudcumin1001: Unknown project "c26d9d326bdf464fa1025939ded7e5a2" [21:21:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}' (T419948) [21:21:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}' (T419948) [21:40:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}' (T419948) [21:40:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1061.eqiad.wmnet}' (T419948) [21:40:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1062 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [21:45:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1062 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [21:49:11] 10VPS-project-Codesearch, 10VerySmallGLAM, 10Wikibase Suite Team: Indexing of wikibase-release-pipeline - https://phabricator.wikimedia.org/T420067 (10Olea) 03NEW [21:55:26] 10VPS-project-Codesearch, 10VerySmallGLAM, 10Wikibase Suite Team: Indexing of wikibase related repos - https://phabricator.wikimedia.org/T420067#11708971 (10Olea) [21:58:56] 10VPS-project-Codesearch, 10VerySmallGLAM, 10Wikibase Suite Team: Indexing of wikibase related repos - https://phabricator.wikimedia.org/T420067#11708984 (10Olea) [22:05:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1061.eqiad.wmnet}' (T419948) [22:05:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}' (T419948) [22:05:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1061 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [22:10:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1061 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [22:25:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}' (T419948) [22:25:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1059.eqiad.wmnet}' (T419948) [22:44:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1059.eqiad.wmnet}' (T419948) [22:44:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1058.eqiad.wmnet}' (T419948) [22:44:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1059 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [22:49:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1059 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [23:09:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1058.eqiad.wmnet}' (T419948) [23:09:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1057.eqiad.wmnet}' (T419948) [23:22:14] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1057.eqiad.wmnet}' (T419948) [23:22:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1056.eqiad.wmnet}' (T419948) [23:44:29] 10Cloud-Services, 06LPL Onboarding and Development: Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074 (10srishakatux) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832... [23:45:56] andrew@cloudcumin1001 safe_reboot (PID 433051) is awaiting input [23:46:30] 10Cloud-Services, 06LPL Onboarding and Development: Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11709219 (10srishakatux) p:05Triage→03Medium [23:47:25] 06cloud-services-team, 10Toolforge, 10Tools, 06LPL Onboarding and Development: Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11709221 (10srishakatux) [23:47:55] 10Toolforge, 10Tools, 06LPL Onboarding and Development: Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11709223 (10srishakatux) [23:48:22] 10Tools, 06LPL Onboarding and Development: Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11709226 (10taavi) Untagging, since this does not seem related to Toolforge infrastructure?