[00:02:49] <wikibugs>	 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T369772#9971816 (10LibUp-bot)
[00:06:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.43% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:11:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 83.68% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:12:25] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.16% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:27:10] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.14% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:38:25] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.35% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:43:25] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.62% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:48:25] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 83.04% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:49:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:58:25] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.2% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[01:36:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:36:28] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:39:59] <wikibugs>	 (03open) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T366209)
[01:43:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:43:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:43:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:43:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:44:32] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:44:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:47:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:47:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:47:22] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[01:48:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:48:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:50:54] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:50:56] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:51:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:51:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:53:11] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[01:53:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[01:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:02:00] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[02:02:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[02:02:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[02:02:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[02:07:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[02:07:04] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[02:12:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
[02:14:54] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate True, for hosts list: ['cloudvirt1060']
[02:15:16] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate True, for hosts list: ['cloudvirt1060']
[02:15:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
[03:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:15:10] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10linkwatcher: Cloud VPS "linkwatcher" project Buster deprecation - https://phabricator.wikimedia.org/T367536#9972116 (10Pppery)
[04:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:45:59] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10linkwatcher: Cloud VPS "linkwatcher" project Buster deprecation - https://phabricator.wikimedia.org/T367536#9972206 (10Beetstra) Sorry for late replies.  I have extremely limited time.  I will try to get back in (I've had issues) and resolve over next 2-3 weeks, start...
[05:54:05] <wikibugs>	 10Toolforge: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#9972217 (10Pppery)
[05:54:18] <wikibugs>	 10Toolforge, 07Kubernetes: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#9972218 (10Pppery)
[05:56:25] <wikibugs>	 (03update) 10sstefanova: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014)
[06:19:29] <wikibugs>	 (03update) 10sstefanova: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014)
[07:27:26] <wikibugs>	 (03PS1) 10David Caro: wmcs: add the new gitlab repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1053538
[07:40:32] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: conftool: now in gitlab [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1053612 (https://phabricator.wikimedia.org/T369594)
[07:47:27] <wikibugs>	 10Data-Services, 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9972354 (10ABran-WMF) @Zabe it seems we were missing the "storage layer" task we usually get. Anyway, [[ https://wikitech.wikimedia.org/wiki/Mar...
[07:47:52] <wikibugs>	 (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569)
[07:49:15] <wikibugs>	 (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369573)
[07:50:25] <wikibugs>	 (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369573)
[07:50:36] <wikibugs>	 (03approved) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369573)
[07:50:42] <wikibugs>	 (03merge) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369573)
[07:53:52] <wikibugs>	 10Toolforge (Toolforge iteration 12): `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369573#9972371 (10dcaro) a:03dcaro
[07:54:04] <wikibugs>	 10Toolforge (Toolforge iteration 12): `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369573#9972375 (10dcaro) 05Open→03In progress
[08:11:29] <wikibugs>	 (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95
[08:24:06] <wikibugs>	 (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95
[08:24:30] <wikibugs>	 (03approved) 10aborrero: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 (owner: 10dcaro)
[08:24:34] <wikibugs>	 (03update) 10aborrero: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 (owner: 10dcaro)
[08:39:39] <wikibugs>	 (03open) 10aborrero: registry-admission: local: ignore foxtrot-ldap namespace [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/407 (https://phabricator.wikimedia.org/T369527)
[08:42:09] <wikibugs>	 (03open) 10sstefanova: remove /api prefix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/50
[08:53:29] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:58:29] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:05:57] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: puppetserver got OOMkilled - https://phabricator.wikimedia.org/T369797 (10aborrero) 03NEW
[09:07:26] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: puppetserver got OOMkilled - https://phabricator.wikimedia.org/T369797#9972692 (10aborrero)
[09:16:17] <wikibugs>	 10Toolforge: [replica_cnf,functional-tests] Run replica_cnf functional tests in lima-kilo with the rest of functional tests - https://phabricator.wikimedia.org/T369800 (10dcaro) 03NEW
[09:16:54] <wikibugs>	 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api,builds-api,envvars-api] consolidate api paths - https://phabricator.wikimedia.org/T365014#9972737 (10Slst2020)
[09:22:31] <wikibugs>	 (03merge) 10aborrero: registry-admission: local: ignore foxtrot-ldap namespace [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/407 (https://phabricator.wikimedia.org/T369527)
[09:24:19] <wikibugs>	 (03open) 10aborrero: Reapply "k8s: deploy registry-admission" [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/165 (https://phabricator.wikimedia.org/T369527)
[09:34:00] <wikibugs>	 (03open) 10aborrero: functiona-tests: add a registry-admission policy smoke test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/408 (https://phabricator.wikimedia.org/T369527)
[09:40:02] <wikibugs>	 (03open) 10aborrero: helpers: add toolforge_redeploy_components.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/166
[09:43:59] <wikibugs>	 (03update) 10aborrero: Draft: basic_system: add lima-kilo-boot.service [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/148
[10:13:15] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: lima-kilo: deploy registry admission - https://phabricator.wikimedia.org/T369527#9972860 (10aborrero) 05Open→03In progress p:05Triage→03Low
[10:18:22] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: kubernetes can't revoke certificates - https://phabricator.wikimedia.org/T365681#9972877 (10aborrero) We have taken the following measures: * reduced the lifetime of the certificates to 10 days for new certificates * certs will get renewed...
[10:24:46] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: kubernetes can't revoke certificates - https://phabricator.wikimedia.org/T365681#9972900 (10aborrero) there are some risks with lower lifetime values, for example 2 or 5 days: * if there is a problem with maintain-kubeusers, or with the k8...
[11:28:20] <wikibugs>	 (03open) 10aborrero: deployment: drop PSP references [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/6 (https://phabricator.wikimedia.org/T369164)
[11:38:35] <wikibugs>	 (03merge) 10aborrero: deployment: drop PSP references [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/6 (https://phabricator.wikimedia.org/T369164)
[11:41:46] <wikibugs>	 (03update) 10aborrero: functiona-tests: add a registry-admission policy smoke test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/408 (https://phabricator.wikimedia.org/T369527)
[11:43:54] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[11:43:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:44:23] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[11:44:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:46:00] <wikibugs>	 (03open) 10aborrero: gitlab-ci: set MEMORY_OPTIMIZED: true [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/7
[11:46:53] <wikibugs>	 (03approved) 10dcaro: gitlab-ci: set MEMORY_OPTIMIZED: true [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/7 (owner: 10aborrero)
[11:48:36] <wikibugs>	 (03merge) 10aborrero: gitlab-ci: set MEMORY_OPTIMIZED: true [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/7
[11:51:09] <jinxer-wm>	 FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[11:51:55] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.13-20240711114848-774571d5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/409 (https://phabricator.wikimedia.org/T369164)
[11:54:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
[11:54:49] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
[11:55:35] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
[11:55:46] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
[11:55:52] <wikibugs>	 (03merge) 10aborrero: envvars-admission: bump to 0.0.13-20240711114848-774571d5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/409 (https://phabricator.wikimedia.org/T369164) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[11:57:56] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[11:57:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:58:07] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[11:58:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:00:09] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[12:00:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:01:46] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
[12:01:47] <wikibugs>	 (03PS1) 10David Caro: ceph: fix off-by-one index when draining/undraining in chunks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1053646
[12:06:16] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
[12:07:39] <jinxer-wm>	 RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[12:12:01] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
[12:12:03] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
[12:13:37] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
[12:26:30] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/15
[12:27:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1053667 (owner: 10L10n-bot)
[12:31:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
[12:31:52] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
[12:34:22] <wikibugs>	 (03open) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569)
[12:34:46] <wm-bot2>	 !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
[12:34:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:39:56] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
[12:40:08] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
[12:48:08] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
[12:50:22] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
[12:51:23] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
[12:51:25] <wikibugs>	 (03open) 10dcaro: toloforge_get_version: handle special toolforge-jobs package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/167
[12:52:25] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
[12:53:19] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
[12:54:03] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
[12:54:57] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
[12:56:44] <wikibugs>	 (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569)
[13:01:24] <wikibugs>	 (03approved) 10aborrero: toloforge_get_version: handle special toolforge-jobs package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/167 (owner: 10dcaro)
[13:13:00] <wikibugs>	 (03open) 10dcaro: d/changelog: bump to 16.0.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T369573)
[13:32:25] <wikibugs>	 (03PS3) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318)
[13:41:53] <wm-bot2>	 !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add
[13:41:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:42:09] <wm-bot2>	 !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
[13:42:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:42:55] <wikibugs>	 (03update) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014)
[13:50:29] <wikibugs>	 (03update) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014)
[13:51:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 05Goal, 13Patch-For-Review: Replace use of openstack environment settings with clouds.yaml - https://phabricator.wikimedia.org/T337577#9973526 (10Andrew) 05Open→03Resolved
[13:53:43] <wikibugs>	 (03approved) 10dcaro: d/changelog: bump to 16.0.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T369573)
[13:53:46] <wikibugs>	 (03update) 10dcaro: d/changelog: bump to 16.0.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T369573)
[13:53:47] <wikibugs>	 (03merge) 10dcaro: d/changelog: bump to 16.0.13 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T369573)
[13:53:47] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: Fix 'openstack database instance rebuild' - https://phabricator.wikimedia.org/T355721#9973531 (10Andrew) I'm seeing puzzling behavior with this. With some trove instances (e.g. test VMs) rebuild acts as I expected: the old VM is fully...
[13:57:02] <wikibugs>	 (03open) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410
[13:57:29] <wikibugs>	 (03open) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181)
[13:57:34] <wikibugs>	 (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181)
[13:58:15] <wikibugs>	 (03update) 10dcaro: toloforge_get_version: handle special toolforge-jobs package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/167
[13:58:23] <wikibugs>	 (03approved) 10dcaro: toloforge_get_version: handle special toolforge-jobs package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/167
[13:58:27] <wikibugs>	 (03merge) 10dcaro: toloforge_get_version: handle special toolforge-jobs package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/167
[14:04:45] <wikibugs>	 (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1
[14:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:19:52] <wikibugs>	 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9973618 (10aborrero) per the fast-api docs https://fastapi.tiangolo.com/tutorial/background-tasks/#caveat:  > Caveat¶ > > If you need to perform heavy background comp...
[14:21:28] <wikibugs>	 10tool-wscontest, 07good first task: Add contestant number (order) for WSContest contest page - https://phabricator.wikimedia.org/T331507#9973623 (10Samwilson) 05Open→03Resolved This is done now, thanks Avarti Rastogi (sorry, I'm not sure what your Phabricator username is, so feel free to assign this t...
[14:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:32:24] <wikibugs>	 (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1
[14:35:20] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "dumps" project Buster deprecation - https://phabricator.wikimedia.org/T367528#9973733 (10Andrew) Hi folks! If you're actually planning to delete these VMs, please do so so that we can close this task. Thanks!
[14:39:49] <wikibugs>	 (03update) 10bd808: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) (owner: 10dcaro)
[14:43:34] <wikibugs>	 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9973792 (10dcaro) We had the decision meeting today and the option chosen was to start with 2, and then evolve to 4 if that ends up not being enough.  There's the que...
[14:58:19] <wikibugs>	 06cloud-services-team, 10Toolforge: Simple logrotate service for users of Tools as stopgap before central logging - https://phabricator.wikimedia.org/T152235#9973848 (10dcaro) 05Open→03Resolved a:03dcaro We have now the documentation for users to setup the logrotate by themselves, I think this can be...
[14:58:29] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolsbeta: upgrade data plane nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369170#9973857 (10aborrero) we updated a bunch of worker nodes today, will finish tomorrow with the rest.
[14:59:23] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: refresh kubernetes cookbooks for the 1.25 upgrade - https://phabricator.wikimedia.org/T369166#9973866 (10aborrero) 05Open→03Resolved we tested them today. They don't require any updates.
[14:59:59] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolsbeta: upgrade data plane nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369170#9973859 (10aborrero) 05Open→03In progress p:05Triage→03Medium
[15:02:35] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade control plane nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369172#9973881 (10aborrero) p:05Triage→03Medium we have scheduled this operation for 2024-07-16 @ 09:00 UTC
[15:06:37] <wikibugs>	 10Cloud-VPS (Quota-requests): Storage quota increase request for project wikidumpparse - https://phabricator.wikimedia.org/T369545#9973908 (10aborrero) LGTM. +1
[15:13:52] <wikibugs>	 (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1
[15:15:43] <wikibugs>	 10Cloud-VPS: grafana.wmcloud.org down - https://phabricator.wikimedia.org/T369832 (10JJMC89) 03NEW
[16:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:27:42] <wikibugs>	 (03open) 10dcaro: Draft: task: remove the analyze step [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/51 (https://phabricator.wikimedia.org/T369840)
[16:34:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:47:23] <wikibugs>	 (03close) 10dcaro: Draft: task: remove the analyze step [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/51 (https://phabricator.wikimedia.org/T369840)
[16:55:30] <wikibugs>	 (03open) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[16:57:19] <wikibugs>	 (03update) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[17:20:22] <wikibugs>	 (03update) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[17:27:33] <wikibugs>	 (03update) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[17:40:15] <wikibugs>	 (03approved) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[17:40:20] <wikibugs>	 (03merge) 10dcaro: task: set the stack.toml to the passed runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/52 (https://phabricator.wikimedia.org/T369840)
[17:41:40] <wikibugs>	 (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.110-20240711174029-fb3c15d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/412 (https://phabricator.wikimedia.org/T369840)
[17:41:44] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.110-20240711174029-fb3c15d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/412 (https://phabricator.wikimedia.org/T369840)
[17:44:53] <wikibugs>	 (03open) 10dcaro: registry-admission: add harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413
[17:44:57] <wikibugs>	 (03update) 10dcaro: registry-admission: add harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413
[17:45:16] <wikibugs>	 (03update) 10dcaro: registry-admission: add local harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413
[17:46:06] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
[17:46:24] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
[17:49:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
[17:49:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
[17:53:55] <wikibugs>	 (03approved) 10dcaro: builds-builder: bump to 0.0.110-20240711174029-fb3c15d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/412 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:54:11] <wikibugs>	 (03merge) 10dcaro: builds-builder: bump to 0.0.110-20240711174029-fb3c15d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/412 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:58:32] <wikibugs>	 10Toolforge, 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9974673 (10dcaro)
[18:03:51] <wikibugs>	 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858 (10Dzahn) 03NEW
[18:04:03] <wikibugs>	 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858#9974731 (10Dzahn)
[18:04:16] <wikibugs>	 10Data-Services, 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9974732 (10Dzahn)
[18:05:54] <wikibugs>	 10Data-Services, 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9974746 (10Dzahn) The other subtasks that normally get auto-created are also not here. Seems like a bug of the ticket-creating bot/script.
[18:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:36:22] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: puppetserver got OOMkilled - https://phabricator.wikimedia.org/T369797#9974816 (10Andrew) I was going to resize this host to have more RAM but it already has 32G which should really be enough! I'd like to wait a bit and see if we can learn more about what's going...
[19:40:28] <wikibugs>	 10Toolforge: Building tool fails with docker TOOMANYREQUESTS - https://phabricator.wikimedia.org/T369844#9974949 (10bd808) >>! In T369844#9974272, @Magnus wrote: > Silly question: Why don't we host the Docker images in the gitlab repo container service at https://gitlab.wikimedia.org?  We actually do host th...
[20:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:08:53] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T366209)
[21:13:57] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wm-bot" project Buster deprecation - https://phabricator.wikimedia.org/T367567#9975219 (10Andrew) @MacFan4000, can you please delete and clean up VMs and resources that are no longer in use?  Thanks!
[21:48:37] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T366209)
[21:53:10] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209)
[21:53:32] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209)
[21:53:43] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209)
[21:53:59] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209)
[22:47:08] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15 - 26)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#9975478 (10MusikAnimal)
[22:47:24] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Community-Tech (Darwin's Fox (July 15 - 26)): Cloud VPS "eventmetrics" project Buster deprecation - https://phabricator.wikimedia.org/T367530#9975479 (10MusikAnimal)
[22:47:45] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Community-Tech (Darwin's Fox (July 15 - 26)): Cloud VPS "eventmetrics" project Buster deprecation - https://phabricator.wikimedia.org/T367530#9975480 (10MusikAnimal) a:03MusikAnimal
[22:48:52] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15 - 26)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#9975476 (10MusikAnimal) Just as with T367530, #community-tech is asking for an extension of a week o...
[23:14:44] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209)
[23:48:44] <wikibugs>	 10Toolforge, 07Kubernetes: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#9975559 (10Hawkeye7) * Job 'autocheck2' (cronjob) (emails: onfailure) had 2 events:   -- Pod 'autocheck2-28678365-b2wv7'. Phase: 'running'. Container state: 'terminated'. Start timestamp 2024-07-11T12:45:...