[00:12:24] <wikibugs>	 10Toolforge (Toolforge iteration 14): [jobs-api] prepend date and pod name to filelog lines - https://phabricator.wikimedia.org/T372025 (10Raymond_Ndibe) 03NEW
[00:13:59] <wikibugs>	 10Toolforge (Toolforge iteration 14): [jobs-api] prepend date and pod name to filelog lines - https://phabricator.wikimedia.org/T372025#10049986 (10Raymond_Ndibe) a:03Raymond_Ndibe
[00:14:44] <wikibugs>	 10Toolforge (Toolforge iteration 14): [jobs-api] prepend date and pod name to filelog lines - https://phabricator.wikimedia.org/T372025#10049993 (10Raymond_Ndibe)
[00:48:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
[00:48:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
[00:48:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node
[02:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:29:29] <wikibugs>	 (03open) 10samwilson: Add hourly update-focus-areas command [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/1 (https://phabricator.wikimedia.org/T363240)
[03:29:47] <wikibugs>	 (03update) 10samwilson: Add hourly update-focus-areas command [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/1 (https://phabricator.wikimedia.org/T363240)
[05:13:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
[05:13:28] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node
[06:41:20] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[06:41:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[07:43:30] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
[07:43:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[07:44:29] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[07:44:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:16:41] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050224 (10dcaro) I think adding more storage to F4 will not help with the cluster total usage, as the limiting rack now is E4: ` root@cloudcephosd1010:~# ceph osd tree | grep -i...
[08:19:48] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050233 (10dcaro)
[08:29:29] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050282 (10dcaro) It seems we have a bunch of stuff in the 'ceph trash' for the cinder pool (not the compute one), I'll look a bit more, but seems promising: ` root@cloudcephosd10...
[08:30:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050290 (10dcaro) this sounds familiar.... ` root@cloudcephosd1037:~# rbd trash purge eqiad1-cinder 2024-08-08 08:30:22.044 7f5df9771700 -1 librbd::image::PreRemoveRequest: 0x7f5d...
[08:34:12] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050294 (10dcaro) All of them are expired, so that would be nice to clear, looking into the snapshots: ` root@cloudcephosd1037:~# rbd trash ls  eqiad1-cinder -l ID             NAM...
[08:47:34] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050329 (10dcaro) So, in order to delete those and purge the trash, we have to: ` > root@cloudcephosd1037:~# rbd trash ls eqiad1-cinder  # gives trash_id image_id pairs fef8d9d8a7...
[09:34:40] <wikibugs>	 (03approved) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324)
[09:34:44] <wikibugs>	 (03merge) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324)
[09:34:46] <wikibugs>	 (03update) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324)
[09:43:44] <wm-bot2>	 !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
[09:43:53] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
[09:43:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:43:57] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node
[09:56:33] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add +16 cpu / +16 Gb ram to video project quota - https://phabricator.wikimedia.org/T371047#10050498 (10Slst2020) noted!
[10:09:16] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10050512 (10fnegri) The root cause of these expired images that are not getting cleaned up is {T358774}.  An alternative to your approach in the previous comment is to delete the s...
[11:10:53] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), 13Patch-For-Review: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster depr... - https://phabricator.wikimedia.org/T370465#10050635
[11:25:07] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.undrain_node (T371878)
[11:25:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:25:12] <stashbot>	 T371878: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878
[11:26:16] <wikibugs>	 10Cloud-VPS, 10Striker, 10Tool-gitlab-account-approval, 10Tool-phab-ban, and 6 others: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10050668 (10Xover) >>! In T371977#10048477, @LucasWerkmeister wro...
[11:35:16] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T371878)
[11:35:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:35:22] <stashbot>	 T371878: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878
[12:14:04] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[12:14:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:20:39] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
[12:20:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:56:48] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[12:56:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:00:53] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
[13:02:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.wait_for_rebalance
[13:28:48] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10051176 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fnegri@cumin1002 for host clouddb1018.eqiad.wmnet with OS bookworm
[13:32:59] <wikibugs>	 10Data-Services, 10Wikifunctions: WikiLambda tables are not replicated to cloud - https://phabricator.wikimedia.org/T372058 (10Bugreporter) 03NEW
[13:42:57] <wikibugs>	 10Tool-schedule-deployment: Specify the deploy window via URL in the scheduler tool - https://phabricator.wikimedia.org/T372059 (10Tgr) 03NEW
[14:01:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
[14:01:25] <wm-bot2>	 !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
[14:01:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:18:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (T371878)
[14:18:28] <wmcs-alerts>	 FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure
[14:18:38] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[14:18:43] <wm-bot2>	 !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
[14:25:08] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10051453 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fnegri@cumin1002 for host clouddb1018.eqiad.wmnet with OS bookworm completed: - cl...
[14:40:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-55 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:44:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Update designate sink plugins to work with caracal - https://phabricator.wikimedia.org/T371707#10051541 (10Andrew) 05Open→03Resolved
[14:45:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-55 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:45:38] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10051540 (10fnegri) > Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually  The last step of the cookbook failed with https://phabricator.wiki...
[14:51:28] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance
[14:51:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:58:28] <wmcs-alerts>	 RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure
[15:00:14] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
[15:00:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:27:53] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component compontents
[15:27:57] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
[15:28:10] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components
[15:28:14] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
[15:28:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[15:28:38] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
[15:29:01] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[15:29:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[15:31:09] <wikibugs>	 (03CR) 10David Caro: "Need to fix:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro)
[15:49:30] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10051793 (10fnegri)
[15:49:34] <wm-bot2>	 !log andrew@bullseye admin START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)
[15:49:34] <wm-bot2>	 !log andrew@bullseye admin END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) (T369044)
[15:49:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:49:40] <stashbot>	 T369044: Upgrade cloud-vps openstack to version 'Caracal' - https://phabricator.wikimedia.org/T369044
[15:49:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:50:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)
[15:51:07] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T369044)
[15:52:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
[15:52:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
[15:52:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[15:54:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
[15:54:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
[15:58:35] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[15:58:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:59:00] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10051856 (10jijiki) 05Open→03In progress p:05Triage→03High
[15:59:19] <wikibugs>	 (03PS1) 10Andrew Bogott: Replace mention of cloudcontrol2001-dev with 2006-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060882
[16:00:05] <wikibugs>	 (03PS7) 10David Caro: toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890
[16:00:05] <wikibugs>	 (03PS8) 10David Caro: toolforge.component.deploy: use bump_<component> as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905
[16:00:05] <wikibugs>	 (03PS11) 10David Caro: wmcs_libs.common: add run_script [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906
[16:00:05] <wikibugs>	 (03PS13) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907
[16:00:06] <wikibugs>	 (03PS11) 10David Caro: openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919
[16:00:09] <wikibugs>	 (03PS12) 10David Caro: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921
[16:00:27] <wikibugs>	 (03CR) 10David Caro: "Done" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro)
[16:03:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:03:50] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] Replace mention of cloudcontrol2001-dev with 2006-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060882 (owner: 10Andrew Bogott)
[16:04:32] <wikibugs>	 (03CR) 10CI reject: [V:04-1] toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro)
[16:04:36] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[16:04:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:05:05] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component components-api
[16:05:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:07:23] <wikibugs>	 (03Merged) 10jenkins-bot: Replace mention of cloudcontrol2001-dev with 2006-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060882 (owner: 10Andrew Bogott)
[16:08:45] <jinxer-wm>	 FIRING: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[16:09:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
[16:10:22] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10051910 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[16:10:23] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:11:31] <wm-bot2>	 !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[16:11:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:11:45] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10051916 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm execut...
[16:11:59] <wikibugs>	 (03approved) 10dcaro: components-api: bump to 0.0.9-20240805133909-34a6a2c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/472 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:12:01] <wikibugs>	 (03update) 10dcaro: components-api: bump to 0.0.9-20240805133909-34a6a2c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/472 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:12:02] <wikibugs>	 (03merge) 10dcaro: components-api: bump to 0.0.9-20240805133909-34a6a2c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/472 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:12:14] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[16:12:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:12:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:13:24] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[16:13:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:13:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:13:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:14:23] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10051924 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[16:14:53] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10051925 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm execut...
[16:16:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:17:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:18:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:18:54] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T370412#10051934 (10phaultfinder)
[16:23:48] <jinxer-wm>	 FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:23:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
[16:23:59] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T370412#10051953 (10phaultfinder)
[16:24:08] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[16:24:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:24:37] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
[16:25:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
[16:25:51] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
[16:27:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:28:26] <wikibugs>	 (03PS13) 10David Caro: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921
[16:28:48] <jinxer-wm>	 FIRING: [4x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:28:53] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[16:28:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:28:57] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T370412#10051976 (10phaultfinder)
[16:29:26] <wikibugs>	 10Tool-schedule-deployment: Specify the deploy window via URL in the scheduler tool - https://phabricator.wikimedia.org/T372059#10051974 (10bd808) > couldn't figure out how to set up the tool locally for development  I made some notes at https://wikitech.wikimedia.org/wiki/Tool:Schedule-deployment#Local_developm...
[16:30:29] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[16:30:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:31:13] <wikibugs>	 (03update) 10dcaro: builds-api: bump to 0.0.167-20240805134031-250f2aef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/476 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:33:48] <jinxer-wm>	 RESOLVED: [2x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:34:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:34:58] <wikibugs>	 10Tool-schedule-deployment: Specify the deploy window via URL in the scheduler tool - https://phabricator.wikimedia.org/T372059#10051983 (10bd808) p:05Triage→03Medium
[16:35:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
[16:36:36] <wm-bot2>	 !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[16:36:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:36:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
[16:37:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
[16:37:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
[16:38:32] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:39:43] <wikibugs>	 (03approved) 10dcaro: builds-api: bump to 0.0.167-20240805134031-250f2aef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/476 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:39:45] <wikibugs>	 (03update) 10dcaro: builds-api: bump to 0.0.167-20240805134031-250f2aef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/476 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:39:46] <wikibugs>	 (03merge) 10dcaro: builds-api: bump to 0.0.167-20240805134031-250f2aef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/476 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:40:07] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[16:40:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:40:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:42:06] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
[16:42:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:44:31] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
[16:44:40] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
[16:45:26] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[16:45:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:49:34] <wikibugs>	 (03PS1) 10Andrew Bogott: upgrade_openstack_node: add 'glance db load_metadefs' [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060892 (https://phabricator.wikimedia.org/T369044)
[16:49:39] <wikibugs>	 (03update) 10dcaro: envvars-api: bump to 0.0.59-20240805134018-35ffcb05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/475 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:51:09] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
[16:51:21] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[16:51:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:51:28] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[16:51:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:52:47] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
[16:53:00] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
[16:55:15] <jinxer-wm>	 RESOLVED: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[16:57:34] <wm-bot2>	 !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[16:57:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:59:05] <wikibugs>	 (03approved) 10dcaro: envvars-api: bump to 0.0.59-20240805134018-35ffcb05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/475 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[16:59:23] <wikibugs>	 (03merge) 10dcaro: envvars-api: bump to 0.0.59-20240805134018-35ffcb05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/475 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:11:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
[17:31:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
[17:33:31] <wikibugs>	 06Toolforge-standards-committee: Adoption request for Yapperbot - https://phabricator.wikimedia.org/T361426#10052262 (10DavidTornheim) >>! In T361426#10046351, @bd808 wrote: > @DavidTornheim I can turn the tool over to you, but only after breaking it by stopping the jobs and removing the bot password currently s...
[17:47:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
[17:47:40] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] upgrade_openstack_node: add 'glance db load_metadefs' [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060892 (https://phabricator.wikimedia.org/T369044) (owner: 10Andrew Bogott)
[17:52:33] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
[17:57:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
[18:02:32] <wikibugs>	 (03PS1) 10Andrew Bogott: live_upgrade_openstack: update to replace linuxbridge with openvswitch agent [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060903
[18:06:48] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] live_upgrade_openstack: update to replace linuxbridge with openvswitch agent [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060903 (owner: 10Andrew Bogott)
[18:10:56] <wikibugs>	 (03Merged) 10jenkins-bot: live_upgrade_openstack: update to replace linuxbridge with openvswitch agent [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1060903 (owner: 10Andrew Bogott)
[18:14:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
[18:16:54] <wikibugs>	 10Cloud-VPS, 10Striker, 10Tool-gitlab-account-approval, 10Tool-phab-ban, and 6 others: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10052409 (10DavidBrooks) The AWB breakage is specifically: format...
[18:18:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
[18:18:40] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
[18:25:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
[18:25:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
[18:25:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (T371878)
[18:25:24] <stashbot>	 T371878: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878
[18:26:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
[18:26:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (T371878)
[18:26:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
[18:27:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
[18:28:32] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (T371878)
[18:33:02] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878#10052468 (10Andrew)
[18:33:05] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
[18:35:11] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[18:39:22] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
[18:39:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
[18:48:56] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
[18:49:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
[18:58:14] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
[18:59:37] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[19:01:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[19:09:24] <wikibugs>	 10Toolforge: Java application redeploys several times until it starts - https://phabricator.wikimedia.org/T372092 (10Benjavalero) 03NEW
[19:12:10] <wikibugs>	 10Toolforge: Java application redeploys several times until it starts - https://phabricator.wikimedia.org/T372092#10052544 (10Benjavalero) I am attaching a log where we can see there are 3 attempts to deploy the tool until it finally is deployed.  {F57183000}
[19:13:05] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10052548 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[19:32:22] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10052647 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm execut...
[20:57:07] <wikibugs>	 06Toolforge-standards-committee: Adoption request for Yapperbot - https://phabricator.wikimedia.org/T361426#10052943 (10bd808) 05Open→03Stalled a:03DavidTornheim >>! In T361426#10052262, @DavidTornheim wrote: > Above you seem to be willing to publish the content of the key files--minus the protected inform...
[20:58:32] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Caracal' - https://phabricator.wikimedia.org/T369044#10052959 (10Andrew) codfw1dev is now running C
[21:09:14] <wikibugs>	 06Toolforge-standards-committee: Adoption request for geograph2commons - https://phabricator.wikimedia.org/T345707#10052991 (10bd808) I somehow feel nervous about forcing a new co-maintainer on @Magnus even if he is non-responsive to the specific pings that were sent about this adoption request. The source code...
[21:29:32] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10053037 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[21:45:24] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10053074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm comple...
[21:59:24] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10053122 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[22:18:59] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10053141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm execut...
[22:20:09] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10053145 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host testhost2001.codfw.wmnet with OS bookworm
[23:05:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
[23:05:48] <stashbot>	 T371878: [network,D5] reboot cloudsw-d5 - https://phabricator.wikimedia.org/T371878
[23:08:10] <wikibugs>	 06Toolforge-standards-committee: Adoption request for geograph2commons - https://phabricator.wikimedia.org/T345707#10053246 (10bjh21) My intention in adopting this tool was not primarily to resurrect it, but simply to stop people trying to use it. At the moment, the user experience is fairly terrible: you author...