[02:15:24] 10VPS-Projects, 10incubator.wikimedia.org: wmflabs.org-hosted script no longer works on some Wikimedia Incubator sites - https://phabricator.wikimedia.org/T437814#12353255 (10Bugreporter) [02:52:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:56:24] (03open) 10amire80: Add Hebrew translation [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/21 [02:57:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:14:20] (03merge) 10pharos: Add Hebrew translation [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/21 (owner: 10amire80) [03:21:36] (03open) 10amire80: Automatically load all languages under i18n [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/22 [03:32:24] FIRING: SystemdUnitFailed: keystone.service on cloudcontrol2010-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:33:34] (03close) 10pharos: Automatically load all languages under i18n [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/22 (owner: 10amire80) [03:37:24] RESOLVED: SystemdUnitFailed: keystone.service on cloudcontrol2010-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:37:30] (03reopen) 10pharos: Automatically load all languages under i18n [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/22 (owner: 10amire80) [03:40:26] (03update) 10amire80: Automatically load all languages under i18n [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/22 [03:41:44] (03merge) 10pharos: Automatically load all languages under i18n [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/22 (owner: 10amire80) [04:06:22] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12353325 (10komla) Okay, cool. I will add the brief set up directly in the tutorial. [04:29:08] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Keystone logs no longer appearing in logstash - https://phabricator.wikimedia.org/T421911#12353333 (10Andrew) Attached patch has Keystone in logstash again. {F104417612} This needs a bit of followup to make sure there ar... [06:04:12] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T437407) [06:04:15] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T437407) [06:04:21] T437407: Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407 [06:40:55] (03update) 10tlepage: poetry: add uvicorn to the main dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/44 (https://phabricator.wikimedia.org/T432566) (owner: 10dcaro) [06:41:15] (03update) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) [06:46:09] (03update) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) [07:58:27] (03update) 10tlepage: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) [08:09:03] (03approved) 10dcaro: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [08:09:32] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12353673 (10Redmin) [08:21:08] 10Toolforge, 06tools-platform-team: [lima-kilo,toolforge_deploy,functional-tests] fail if any of the components last deployment failed - https://phabricator.wikimedia.org/T438937 (10dcaro) 03NEW [08:23:25] (03update) 10dcaro: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [08:23:41] (03merge) 10dcaro: explicitly add httpx back too [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/45 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [08:26:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.53-20260923082352-88d5a1ec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [08:26:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.53-20260923082352-88d5a1ec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 [08:26:39] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:33:54] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [08:34:31] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:36:30] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [08:43:21] 10Cloud-VPS (Quota-requests), 06Test Platform: Quota increase request for project integration - https://phabricator.wikimedia.org/T438940 (10Peter) 03NEW [08:44:26] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [08:45:44] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:56:16] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [08:56:38] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:57:54] !log tlepage@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [08:59:41] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:01:17] FIRING: ToolWebserviceDown: Webservice in tool-test is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [09:03:44] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12353842 (10dcaro) Hi @Premeditated, In order to understand better your needs, could you share the dockerfile that you want to build (I don't see a `model_server` directory under https://github... [09:06:17] RESOLVED: ToolWebserviceDown: Webservice in tool-test is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [09:11:16] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:11:53] (03merge) 10tlepage: logs-api: bump to 0.0.53-20260923082352-88d5a1ec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:11:55] (03update) 10tlepage: logs-api: bump to 0.0.53-20260923082352-88d5a1ec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1429 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:12:38] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [09:16:56] (03update) 10dcaro: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [09:22:24] (03update) 10fnegri: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 (owner: 10dcaro) [09:22:26] (03approved) 10fnegri: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 (owner: 10dcaro) [09:23:29] 10Cloud-VPS (Quota-requests), 06Test Platform: Quota increase request for project integration - https://phabricator.wikimedia.org/T438940#12353926 (10fnegri) +1 [09:24:48] (03approved) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [09:24:56] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [09:25:09] (03merge) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [09:25:17] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:26:42] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:27:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.54-20260923092523-1f70c678 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1430 (https://phabricator.wikimedia.org/T432571) [09:30:09] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T437407) [09:30:13] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T437407) [09:30:18] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:30:20] T437407: Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407 [09:31:46] (03update) 10fnegri: dev: add docs on how to setup the dev env [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/1 (owner: 10dcaro) [09:39:55] (03update) 10dcaro: dev: add docs on how to setup the dev env [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/1 [09:40:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:41:12] (03approved) 10fnegri: dev: add docs on how to setup the dev env [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/1 (owner: 10dcaro) [09:43:09] 10Cloud-VPS (Quota-requests), 06tools-platform-team, 06Test Platform: Quota increase request for project integration - https://phabricator.wikimedia.org/T438940#12353971 (10dcaro) 05Open→03In progress a:03dcaro [09:43:18] !log dcaro@cloudcumin1001 integration START - Cookbook wmcs.openstack.quota_increase by 20 cores [09:43:19] !log dcaro@cloudcumin1001 integration END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) by 20 cores [09:43:29] !log dcaro@cloudcumin1001 integration START - Cookbook wmcs.openstack.quota_increase by 20 cores (T438940) [09:43:33] T438940: Quota increase request for project integration - https://phabricator.wikimedia.org/T438940 [09:43:36] !log dcaro@cloudcumin1001 integration END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 20 cores (T438940) [09:46:46] (03update) 10chimaifeanyi29: T438399 update language switcher [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/16 [09:46:56] 10Cloud-VPS (Quota-requests), 06tools-platform-team, 06Test Platform: Quota increase request for project integration - https://phabricator.wikimedia.org/T438940#12353981 (10dcaro) 05In progress→03Resolved This should be done :) Enjoy! [09:53:17] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:53:22] (03update) 10fnegri: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 (owner: 10dcaro) [10:05:55] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [10:06:02] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [10:06:02] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [10:06:15] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [10:09:57] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [10:11:05] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [10:42:30] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 10Wikidata, 06Wikidata-Omega: Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947 (10ArthurTaylor) 03NEW [10:47:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [10:48:17] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 10Wikidata, 06Wikidata-Omega: Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12354188 (10ArthurTaylor) [10:50:42] FIRING: [3x] AlertLintProblem: Linting problems found for MemcachedDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [10:59:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [11:00:41] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12354238 (10JSengupta-WMF) @Doc_James Here is a proposal for the new section on the right side toolbar after discussing a couple of different options with @mike... [11:00:42] FIRING: [5x] AlertLintProblem: Linting problems found for NodeDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:10:42] RESOLVED: [5x] AlertLintProblem: Linting problems found for NodeDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:21:46] (03PS1) 10Majavah: cloudinfra: Add fake etcd secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1344239 [11:22:00] (03CR) 10Majavah: [V:03+2 C:03+2] cloudinfra: Add fake etcd secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1344239 (owner: 10Majavah) [11:30:42] FIRING: [3x] AlertLintProblem: Linting problems found for MemcachedDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:30:46] 10wikitech.wikimedia.org, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 10Wikimedia-Interwiki-links, 13Patch-For-Review: Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#12354392 (10Ifeatu_Nnaobi_WMDE) [11:31:09] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 10Wikidata, 13Patch-For-Review, 06Wikidata-Omega (Triaging): Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12354394 (10mahmoud.abdelsattar.wmde) [11:40:42] FIRING: [6x] AlertLintProblem: Linting problems found for NodeDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:00:42] FIRING: [6x] AlertLintProblem: Linting problems found for NodeDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:10:42] RESOLVED: [6x] AlertLintProblem: Linting problems found for NodeDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:12:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [12:20:00] (03approved) 10dcaro: logs-api: bump to 0.0.54-20260923092523-1f70c678 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1430 (https://phabricator.wikimedia.org/T432571) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:20:05] (03merge) 10dcaro: logs-api: bump to 0.0.54-20260923092523-1f70c678 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1430 (https://phabricator.wikimedia.org/T432571) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:41:17] (03update) 10dcaro: dev: add docs on how to setup the dev env [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/1 [12:42:00] (03merge) 10dcaro: dev: add docs on how to setup the dev env [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/1 [12:42:01] (03update) 10dcaro: add admin console to splash [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [12:43:48] (03update) 10dcaro: add admin console to splash [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [12:44:09] (03approved) 10dcaro: add admin console to splash [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [12:46:10] (03merge) 10dcaro: add admin console to splash [toolforge-repos/admin-web] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [12:46:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:48:08] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12354798 (10dcaro) [13:19:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:26:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:45:33] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12355344 (10Andrew) Overnight things seem promising... the drives on 1046 are quite busy but not constantly maxing out lik... [14:56:30] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12355411 (10fgiunchedi) I got a chance to play with the testing rig @Peter set up on `integration-agent-docke... [15:04:34] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Keystone logs no longer appearing in logstash - https://phabricator.wikimedia.org/T421911#12355446 (10Andrew) The following services are missing logstash logs: cinder-api designate-api placement-api octavia-api magnum-api It seems to be only... [15:06:26] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Set up confd on Toolforge and project-proxy proxy hosts - https://phabricator.wikimedia.org/T438971 (10taavi) 03NEW [15:06:45] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Set up confd on Toolforge and project-proxy proxy hosts - https://phabricator.wikimedia.org/T438971#12355502 (10taavi) [15:06:48] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12355503 (10taavi) [15:10:32] (03open) 10bodhisattwa: proper description for bn [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/1 [15:14:04] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12355522 (10Doc_James) Should we include SVG translate in that when looking at an SVG file? [15:15:56] (03open) 10taavi: eqiad1: cloudinfra: Add security group rule for etcd access [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/362 (https://phabricator.wikimedia.org/T438971) [15:16:14] (03update) 10taavi: eqiad1: cloudinfra: Add security group rule for etcd access [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/362 (https://phabricator.wikimedia.org/T438971) [15:20:20] 10Toolforge, 06tools-platform-team: [logs-cli,logs-api] enable following logs for tool events - https://phabricator.wikimedia.org/T438972 (10dcaro) 03NEW [15:22:31] 10Toolforge, 06tools-platform-team: [jobs-cli,logs-api] Deprecate the osd `get_job_logs` function in favor of `get_tool_logs/stream_tool_logs` endpoints - https://phabricator.wikimedia.org/T438974 (10dcaro) 03NEW [15:22:47] 10Toolforge, 06tools-platform-team: [jobs-cli,logs-api] Deprecate the osd `get_job_logs` function in favor of `get_tool_logs/stream_tool_logs` endpoints - https://phabricator.wikimedia.org/T438974#12355586 (10dcaro) [15:22:48] 10Toolforge, 06tools-platform-team: [logs-cli,logs-api] enable following logs for tool events - https://phabricator.wikimedia.org/T438972#12355587 (10dcaro) [15:27:01] (03approved) 10volans: eqiad1: cloudinfra: Add security group rule for etcd access [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/362 (https://phabricator.wikimedia.org/T438971) (owner: 10taavi) [15:28:06] (03merge) 10taavi: eqiad1: cloudinfra: Add security group rule for etcd access [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/362 (https://phabricator.wikimedia.org/T438971) [15:28:18] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:28:47] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:32:59] 10Cloud-VPS, 06tools-infrastructure-team: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272#12355688 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I'm calling this one done with alerts fixed too [15:34:41] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12355714 (10Premeditated) Hi @dcaro, Thank you for the quick follow-up! The directory is `model-server/`, it is still in development and the branch with the face-service split was still being... [15:53:12] 10Cloud-Services: Request to unlink SUL / Phabricator account from LDAP IDM (user: el3po) - https://phabricator.wikimedia.org/T438990 (10Elilopes) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace... [15:54:55] 10Toolforge, 06tools-platform-team: [jobs-cli,logs-api] Deprecate the old `get_job_logs` function in favor of `get_tool_logs/stream_tool_logs` endpoints - https://phabricator.wikimedia.org/T438974#12355890 (10TLepage-WMF) [15:55:05] 10Cloud-Services: Request to unlink SUL / Phabricator account from LDAP IDM (user: el3po) - https://phabricator.wikimedia.org/T438990#12355895 (10Elilopes) [15:59:06] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12355914 (10dcaro) >>! In T438842#12355714, @Premeditated wrote: > Hi @dcaro, > > Thank you for the quick follow-up! > > The directory is `model-server/`, it is still in development and the br... [16:02:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [16:03:46] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:04:12] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:05:24] FIRING: SystemdUnitFailed: keystone_rotate_keys.service on cloudcontrol2010-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:08:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12355993 (10dcaro) Thanks, why is it collapsed? This is using the old process: {F104450985} The new process is just an include, see https://wikitech.wikimedia.org... [16:09:36] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12355999 (10dcaro) There's also a typo here (Name -> Names) {F104451034} [16:10:54] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12356004 (10TLepage-WMF) 05In progress→03Resolved Endpoint implemented and released. [16:15:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [16:15:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [16:15:24] FIRING: [2x] SystemdUnitFailed: designate_floating_ip_ptr_records_updater.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:15:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [16:15:42] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [16:18:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [16:18:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [16:19:12] 10Tool-Global-user-contributions: Linking toolforge:guc with URL arguments leads to 404 as ? is turned into %3F - https://phabricator.wikimedia.org/T438320#12356037 (10aputhin) Removing the Toolforge project tag as this seems to be tool-specific. [16:20:24] FIRING: [2x] SystemdUnitFailed: designate_floating_ip_ptr_records_updater.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:22:47] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] change the alerts.enabled config value to alerts.notify - https://phabricator.wikimedia.org/T438995 (10dcaro) 03NEW [16:24:39] (03open) 10dcaro: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 [16:25:24] FIRING: [3x] SystemdUnitFailed: designate_floating_ip_ptr_records_updater.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:30:39] 10Toolforge, 06tools-platform-team: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996 (10dcaro) 03NEW [16:31:48] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] change the alerts.enabled config value to alerts.notify - https://phabricator.wikimedia.org/T438995#12356113 (10dcaro) p:05Triage→03High [16:34:09] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12356141 (10komla) @dcaro, I just feel it is tidier when collapsed because the text on the page is getting long. > The new process is just an include, see https://... [16:34:46] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:43:46] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12356269 (10Redmin) [16:45:21] (03open) 10dcaro: models: updated the models to include webservice job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/193 [16:45:32] 10Toolforge, 06tools-platform-team: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996#12356293 (10dcaro) [16:45:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12356292 (10dcaro) [16:45:35] 10Toolforge, 06tools-platform-team: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996#12356294 (10dcaro) p:05Triage→03High [16:48:21] 10Toolforge, 06tools-platform-team: [components-cli] When doing 'deployment show' the config takes most of the screen - https://phabricator.wikimedia.org/T439004 (10dcaro) 03NEW [16:51:54] RESOLVED: SystemdUnitFailed: nova-fullstack.service on cloudcontrol2006-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:52:51] 10Toolforge, 06tools-platform-team: [components-api] fails to generate configuration for existing job that reuses image - https://phabricator.wikimedia.org/T439005 (10dcaro) 03NEW [16:53:44] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12356398 (10komla) Done. I've fixed the typo and used the new `include` directive for `.gitlab-ci.yml` [17:14:00] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12356512 (10Redmin) [17:16:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [17:23:51] FIRING: [4x] ProbeDown: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:29:04] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [17:39:04] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:12:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:14:04] (03update) 10renovatebot: Update pnpm to v12.5.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [18:14:50] (03update) 10renovatebot: Update pnpm to v12.5.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [18:16:15] (03update) 10renovatebot: Update pnpm to v12.5.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [18:19:20] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:19:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12356759 (10komla) I also updated the push-to-deploy set up to use the new `include` directive. [18:28:04] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:31:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:32:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:32:04] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-internal-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:32:26] FIRING: [2x] ProbeDown: Service wan.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:34:03] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [18:34:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:37:04] FIRING: [2x] PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:38:04] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:39:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:41:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:42:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:42:04] RESOLVED: [2x] PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:55:34] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [18:59:03] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:05:50] FIRING: [12x] ProbeDown: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:06:44] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [19:07:48] FIRING: [4x] SystemdUnitFailed: nova-fullstack.service on cloudcontrol2006-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:10:06] 10Toolforge (Push-to-Deploy): Toolforge push-to-deploy bootstrap - https://phabricator.wikimedia.org/T439030 (10LucasWerkmeister) 03NEW [19:10:11] FIRING: [9x] JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:17:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:17:10] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [19:18:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [19:18:47] 10Toolforge (Push-to-Deploy): Toolforge push-to-deploy bootstrap - https://phabricator.wikimedia.org/T439030#12356950 (10LucasWerkmeister) Quoting @dcaro ([comment](https://github.com/lucaswerkmeister/cookiecutter-toolforge/pull/5#issuecomment-5784401020)): > There's an underlying security issue of trusting the... [19:20:14] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [19:21:50] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [19:22:26] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [19:23:26] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12356959 (10Redmin) [19:23:52] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12356963 (10Redmin) [19:29:05] 10Tool-inteGraality: Upgrade to Bootstrap 5 - https://phabricator.wikimedia.org/T439032 (10JeanFred) 03NEW [19:32:30] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [19:37:26] FIRING: [3x] SystemdUnitCrashLoop: designate-central.service crashloop on cloudcontrol2005-dev:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [19:39:10] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [19:41:45] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [19:42:29] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [19:42:48] RESOLVED: [12x] ProbeDown: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_codfw_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:45:42] FIRING: [9x] JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:46:39] FIRING: [9x] JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:49:09] (03CR) 10Jsn.sherman: [C:03+2] "Does what it says on the tin; thanks!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [19:57:25] FIRING: SystemdUnitFailed: designate_floating_ip_ptr_records_updater.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:58:36] FIRING: [9x] JobUnavailable: Reduced availability for job ceph_codfw in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:01:48] FIRING: [2x] SystemdUnitFailed: designate_floating_ip_ptr_records_updater.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:06:02] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [20:06:32] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [20:12:02] RESOLVED: SystemdUnitFailed: security_group_default_to_project_trove.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:21:58] FIRING: [3x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:28:12] RESOLVED: JobUnavailable: Reduced availability for job jmx_zookeeper in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:40:56] (03CR) 10Jsn.sherman: [C:03+1] "looks good with a single inline note" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [20:51:14] (03CR) 10Umherirrender: "Please resubmit, jsn.sherman can do it by adding a comment, others by apply another +2" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [21:19:56] (03CR) 10Jsn.sherman: [C:03+1] "LGTM!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [21:27:08] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12357294 (10Premeditated) >>! In T438842#12355914, @dcaro wrote: >>>! In T438842#12355714, @Premeditated wrote: >> Hi @dcaro, >> >> Thank you for the quick follow-up! >> >> The directory is `m... [22:19:11] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12357442 (10Southparkfan) The PHP 8.3 hosts hav... [22:58:04] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [23:01:18] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [23:05:07] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12357639 (10Redmin) Warning emitted during installation of Playwright on Toolforge: ` Playwright Host validation warning: ╔══════════════════════════════════════════════════════╗ ║ Host system... [23:08:34] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [23:10:25] FIRING: [2x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:50:25] RESOLVED: [2x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed