[00:19:53] (03PS5) 10Alien4444: Add discord channel and invite links to footer [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257431 [00:20:16] (03CR) 10Alien4444: Add discord channel and invite links to footer (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257431 (owner: 10Alien4444) [02:38:57] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:11:34] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11736678 (10Mokshith_25) Hi, I have completed the Root Path microtask by converting the Next.js-based dashboard into a pure React implementation. Key changes: - Replaced async serve... [05:16:22] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11736688 (10Mokshith_25) Hi, I have completed the Root Path microtask by converting the Next.js-based dashboard into a pure React implementation. GitHub Repository: https://github.com/mokshith1222/campwiz-react Loo... [06:24:00] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11736731 (10Monicadola42) Hi @Gerges I’ve reviewed the issue and the discussion so far. I’m interested in working on this as well, especially focusing on the mobile to... [06:38:57] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:26:08] (03open) 10arthurtaylor: Reduce job-time variance using weighted average [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/12 [08:31:37] (03update) 10arthurtaylor: Reduce job-time variance using weighted average [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/12 (https://phabricator.wikimedia.org/T420884) [08:32:18] (03update) 10arthurtaylor: Reduce job-time variance using weighted average [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/12 (https://phabricator.wikimedia.org/T420884) [08:38:00] 10Tool-documentation, 07Documentation, 07good first task: Request for Documentation: Python Flask OAuth2 Integration - https://phabricator.wikimedia.org/T382634#11736999 (10Chinmayee_Mishra) 05Open→03Resolved [08:38:42] 10Tool-documentation, 07Documentation, 07good first task: Request for Documentation: Python Flask OAuth2 Integration - https://phabricator.wikimedia.org/T382634#11737002 (10Chinmayee_Mishra) [09:10:10] (03open) 10taavi: Add default route manifest [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/17 (https://phabricator.wikimedia.org/T418479) [09:10:14] (03update) 10taavi: Add default route manifest [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/17 (https://phabricator.wikimedia.org/T418479) [09:14:33] (03approved) 10dcaro: Add default route manifest [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/17 (https://phabricator.wikimedia.org/T418479) (owner: 10taavi) [09:15:33] (03merge) 10taavi: Add default route manifest [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/17 (https://phabricator.wikimedia.org/T418479) [09:16:43] (03update) 10dcaro: core: update jobs in storage too [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/277 [09:20:45] RESOLVED: IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [09:24:19] RESOLVED: IngressPodMisplaced: ingress-nginx-gen2 pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIngressPodMisplaced [09:33:14] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26): [builds-api,builds-builder] php buildpack is broken on current default buidlpacks and `--use-latest-versions` ones - https://phabricator.wikimedia.org/T420547#11737144 (10dcaro) 05Open→03Resolved [09:48:39] (03approved) 10fnegri: projects: added project etherpads3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/302 (https://phabricator.wikimedia.org/T420532) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [10:09:14] FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:12:08] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11737444 (10dcaro) [10:16:33] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki abstractwiki - https://phabricator.wikimedia.org/T420637#11737476 (10fnegri) p:05Triage→03Medium [10:17:10] (03open) 10dcaro: inject_buildpacks: skip injecting aptfile if builtin exists [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/83 [10:17:15] (03open) 10dcaro: inject_buildpacks: skip dotnet if builtin exits [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/84 [10:18:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11737494 (10DamianZaremba) All of the cluebot tools are running with the... [10:29:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:29:52] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [10:30:16] (03open) 10taavi: istio-gateway: Expose metrics endpoint [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180 (https://phabricator.wikimedia.org/T392356) [10:30:16] (03update) 10taavi: istio-gateway: Expose metrics endpoint [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180 (https://phabricator.wikimedia.org/T392356) [10:31:20] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [10:31:31] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [10:33:08] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment codfw1dev [10:34:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:38:57] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:39:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:40:42] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917) [10:40:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:43:32] (03approved) 10dcaro: istio-gateway: Expose metrics endpoint [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [10:44:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:44:58] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86) [10:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:45:23] (03merge) 10taavi: istio-gateway: Expose metrics endpoint [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180 (https://phabricator.wikimedia.org/T392356) [10:45:34] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [10:45:50] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [10:47:26] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project [10:47:26] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project [10:47:33] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [10:51:36] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [10:54:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDow [11:02:34] (03update) 10fnegri: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) [11:04:08] (03update) 10fnegri: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) [11:04:50] 10Tool-wiktlexbot: Properly retrieve lemmas for sign languages - https://phabricator.wikimedia.org/T418890#11737696 (10Redmin) [11:06:11] (03update) 10fnegri: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) [11:06:18] (03update) 10fnegri: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) [11:28:58] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin, 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11737759 (10dcaro) [11:30:29] (03update) 10dcaro: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) (owner: 10fnegri) [11:30:29] (03approved) 10dcaro: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) (owner: 10fnegri) [11:43:46] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528#11737787 (10dcaro) For "harbor vs s3 orphaned objects cleanup": > Can we indefinitely increase the object count quota? what are the true limitat... [11:55:38] (03merge) 10fnegri: Import unversioned files from Toolforge [toolforge-repos/admin-jobs] - 10https://gitlab.wikimedia.org/toolforge-repos/admin-jobs/-/merge_requests/1 (https://phabricator.wikimedia.org/T415383) [12:02:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin, 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11737832 (10fnegri) 05In progress→03Resolved I have merged the MR, and set https://gitlab.wikimedia.org/t... [12:22:36] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Display flagged edits count in browser tab title (dynamic update) - https://phabricator.wikimedia.org/T420850#11737915 (10Gerges) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things... [12:24:29] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/64 [12:24:29] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/36 [12:25:45] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11737918 (10Wesam_harb) Hi, I'm a frontend developer experienced with React and TypeScript. I'm very interested in contributing to CampWiz NxT Redesign for GSoC 2026. Could you guide... [12:54:58] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Increased openstack latency and rabbitmq rolling restarts on certificate update - https://phabricator.wikimedia.org/T418444#11738012 (10fgiunchedi) Change deployed and rabbit roll-restarted: ` # rabbitmqctl eval 'application:get_env(rabbit, cluster_parti... [13:04:36] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[56] - https://phabricator.wikimedia.org/T419892#11738055 (10Andrew) a:05Andrew→03None [13:06:37] 10Tool-centralnotice-banner-editor: Extend Banner Editor to support more complex banner templates - https://phabricator.wikimedia.org/T420918 (10Oyelola_Victoria) 03NEW [13:07:41] (03CR) 10Majavah: [V:03+2 C:03+2] Add toolsbeta-acme-chief private key [labs/private] - 10https://gerrit.wikimedia.org/r/1240325 (owner: 10Majavah) [13:07:50] (03CR) 10Majavah: [V:03+2 C:03+2] Add fake metricsinfra Grafana admin password [labs/private] - 10https://gerrit.wikimedia.org/r/1240326 (owner: 10Majavah) [13:08:01] (03CR) 10Majavah: [V:03+2 C:03+2] Add fake Docker registry passwrod for cloudinfra [labs/private] - 10https://gerrit.wikimedia.org/r/1245297 (owner: 10Majavah) [13:15:35] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat [13:16:27] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat [13:16:32] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate [13:17:07] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate [13:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:21:15] 10Tool-centralnotice-banner-editor: Banner editor does not preserve line breaks in text elements - https://phabricator.wikimedia.org/T420920 (10Oyelola_Victoria) 03NEW [13:22:17] FIRING: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:25:32] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:27:17] RESOLVED: JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:30:10] 06cloud-services-team, 10Cloud-VPS: Bind all cloudlb services to specific service VIPs only - https://phabricator.wikimedia.org/T420921 (10taavi) 03NEW [13:33:32] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:36:22] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: rabbitmqctl list_queues in eqiad/codfw times out after 60s - https://phabricator.wikimedia.org/T420923 (10fgiunchedi) 03NEW [13:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:46:59] 10Tool-centralnotice-banner-editor: Finish banner sharing feature - https://phabricator.wikimedia.org/T420924 (10Oyelola_Victoria) 03NEW [13:47:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [13:54:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:59:17] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11738373 (10Atharv1345) Hi, I have started working on the CampWiz NxT Redesign microtask. I have successfully migrated the CampaignEditForm component from Next.js to pure React. Changes done: - Removed Next.js depen... [14:00:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [14:01:49] 10Tool-centralnotice-banner-editor: Add contrast checker for accessibility guidelines - https://phabricator.wikimedia.org/T420929 (10Oyelola_Victoria) 03NEW [14:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:24:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:44:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:48:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [14:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:50:38] 06cloud-services-team, 10Cloud-VPS: experiment with moving rabbitmq behind haproxy - https://phabricator.wikimedia.org/T420937 (10Andrew) 03NEW [14:51:22] FIRING: [7x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:56:22] RESOLVED: [7x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:58:26] 10Tool-centralnotice-banner-editor: Alignment buttons are not responsive and are cut off on small screens - https://phabricator.wikimedia.org/T420940 (10Oyelola_Victoria) 03NEW [15:02:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [15:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:14:52] (03approved) 10lucaswerkmeister-wmde: Reduce job-time variance using weighted average [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/12 (https://phabricator.wikimedia.org/T420884) (owner: 10arthurtaylor) [15:24:23] 10Tool-centralnotice-banner-editor: Add support for dark mode in edited banners - https://phabricator.wikimedia.org/T420941 (10Oyelola_Victoria) 03NEW [15:26:17] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11738842 (10xcollazo) > The option of telling the few consumers of the rsync to change the hostname/IP th... [15:33:26] 10Tool-centralnotice-banner-editor: Allow bold to be applied to individual words or phrases within a text element - https://phabricator.wikimedia.org/T420943 (10Oyelola_Victoria) 03NEW [15:41:53] 10Tool-centralnotice-banner-editor: Allow links to be applied to individual words or phrases within a text element - https://phabricator.wikimedia.org/T420945 (10Oyelola_Victoria) 03NEW [15:42:17] 10Tool-wikimonitor: Secure Persistent “Remember Me” Login with Split-Key Encryption for Wikimedia OAuth2 Tokens - https://phabricator.wikimedia.org/T420946 (10Gerges) 03NEW [15:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:52:31] 06cloud-services-team, 06DC-Ops, 10ops-codfw: Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T420948 (10Andrew) 03NEW [15:53:32] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:54:19] 10Tool-centralnotice-banner-editor: Implement fixed CSS feature for default uneditable styles in banners - https://phabricator.wikimedia.org/T420950 (10Oyelola_Victoria) 03NEW [15:55:32] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [16:10:57] 10Tool-curator: Curator should handle 504 Bad Gateway errors from Commons - https://phabricator.wikimedia.org/T420953 (10DaxServer) 03NEW [16:11:20] 10Tool-curator: Curator should handle 504 Bad Gateway errors from Commons - https://phabricator.wikimedia.org/T420953#11739188 (10DaxServer) p:05Triage→03Medium [16:13:45] 10Tool-centralnotice-banner-editor: Add confirmation modal when selecting a new template while editing - https://phabricator.wikimedia.org/T420955 (10Oyelola_Victoria) 03NEW [16:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:14:22] 10Tool-curator: Curator should handle Commons backend storage failure during uploads - https://phabricator.wikimedia.org/T420956 (10DaxServer) 03NEW [16:14:43] 10Tool-curator: Curator should handle Commons backend storage failure during uploads - https://phabricator.wikimedia.org/T420956#11739248 (10DaxServer) p:05Triage→03Medium [16:17:57] 10Tool-curator: Curator should handle 502 server error from Commons - https://phabricator.wikimedia.org/T420957 (10DaxServer) 03NEW [16:18:07] 10Tool-curator: Curator should handle 502 server error from Commons - https://phabricator.wikimedia.org/T420957#11739281 (10DaxServer) p:05Triage→03Medium [16:19:07] 10Tool-curator: Curator should handle 504 Bad Gateway errors from Commons - https://phabricator.wikimedia.org/T420953#11739289 (10Praffq) @DaxServer can I pick this issue? I'll like to work on it [16:19:39] (03merge) 10komla: projects: added project etherpads3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/302 (https://phabricator.wikimedia.org/T420532) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:24:08] 10Tool-centralnotice-banner-editor: Add confirmation modal when deleting a banner - https://phabricator.wikimedia.org/T420958 (10Oyelola_Victoria) 03NEW [16:26:16] !log komla@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [16:26:16] komla@cloudcumin1001: Unknown project "etherpads3" [16:26:17] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [16:29:25] 10Tool-curator: Curator should handle "Nonce already used" errors in Commons API calls during uploads - https://phabricator.wikimedia.org/T420960 (10DaxServer) 03NEW [16:29:40] 10Tool-curator: Curator should handle "Nonce already used" errors in Commons API calls during uploads - https://phabricator.wikimedia.org/T420960#11739382 (10DaxServer) p:05Triage→03Medium [16:30:32] 10Tool-curator: Curator should handle MediaWiki\Upload\Exception\UploadStashFileException Commons failure during uploads - https://phabricator.wikimedia.org/T420959#11739388 (10DaxServer) [16:31:02] komla@cloudcumin1001 create_project (PID 1634849) is awaiting input [16:32:45] 10Tool-curator: Curator should handle invalid CSRF token errors during uploads - https://phabricator.wikimedia.org/T420961 (10DaxServer) 03NEW [16:32:50] 10Tool-curator: Curator should handle invalid CSRF token errors during uploads - https://phabricator.wikimedia.org/T420961#11739410 (10DaxServer) p:05Triage→03Medium [16:54:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:12:48] FIRING: PuppetDisabled: Puppet disabled on cloudvirt1076:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:17:48] FIRING: [3x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:22:48] FIRING: [4x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:23:05] komla@cloudcumin1001 create_project (PID 1634849) is awaiting input [17:28:34] !log komla@cloudcumin1001 etherpads3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project etherpads3 in eqiad1 (T420532) [17:28:34] komla@cloudcumin1001: Unknown project "etherpads3" [17:28:34] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [17:28:46] !log komla@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [17:28:46] komla@cloudcumin1001: Unknown project "etherpads3" [17:33:33] komla@cloudcumin1001 create_project (PID 1640038) is awaiting input [17:39:16] (03update) 10booksmurf: Draft: dev- machine translation [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/6 [17:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:05:38] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11739990 (10HCoplin-WMF) > the new File Export system, which is now in production running jointly with th... [18:13:02] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:13:03] andrew@cloudcumin1001: Unknown project "etherpads3" [18:13:04] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [18:14:04] !log andrew@cloudcumin1001 etherpads3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project etherpads3 in eqiad1 (T420532) [18:14:04] andrew@cloudcumin1001: Unknown project "etherpads3" [18:17:23] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:17:23] andrew@cloudcumin1001: Unknown project "etherpads3" [18:18:26] !log andrew@cloudcumin1001 etherpads3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project etherpads3 in eqiad1 (T420532) [18:18:28] andrew@cloudcumin1001: Unknown project "etherpads3" [18:18:28] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [18:21:31] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:21:31] andrew@cloudcumin1001: Unknown project "etherpads3" [18:22:04] !log andrew@cloudcumin1001 etherpads3 END (ERROR) - Cookbook wmcs.vps.create_project (exit_code=97) for project etherpads3 in eqiad1 (T420532) [18:22:04] andrew@cloudcumin1001: Unknown project "etherpads3" [18:24:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:24:46] PROBLEM - Memcached on cloudcontrol1007 is CRITICAL: connect to address 10.64.148.21 and port 11211: Connection refused https://wikitech.wikimedia.org/wiki/Memcached [18:24:46] PROBLEM - Memcached on cloudcontrol1006 is CRITICAL: connect to address 10.64.150.6 and port 11211: Connection refused https://wikitech.wikimedia.org/wiki/Memcached [18:25:46] RECOVERY - Memcached on cloudcontrol1007 is OK: TCP OK - 0.000 second response time on 10.64.148.21 port 11211 https://wikitech.wikimedia.org/wiki/Memcached [18:26:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:26:46] RECOVERY - Memcached on cloudcontrol1006 is OK: TCP OK - 0.000 second response time on 10.64.150.6 port 11211 https://wikitech.wikimedia.org/wiki/Memcached [18:29:22] RESOLVED: [3x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:29:24] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:29:26] andrew@cloudcumin1001: Unknown project "etherpads3" [18:29:26] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [18:29:53] (03open) 10dcaro: general: update for new buildpacks [toolforge-repos/sample-dotnet-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app/-/merge_requests/1 [18:30:00] !log andrew@cloudcumin1001 etherpads3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project etherpads3 in eqiad1 (T420532) [18:30:00] andrew@cloudcumin1001: Unknown project "etherpads3" [18:30:33] (03merge) 10dcaro: general: update for new buildpacks [toolforge-repos/sample-dotnet-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app/-/merge_requests/1 [18:30:57] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:30:57] andrew@cloudcumin1001: Unknown project "etherpads3" [18:31:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:31:34] !log andrew@cloudcumin1001 etherpads3 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project etherpads3 in eqiad1 (T420532) [18:31:34] andrew@cloudcumin1001: Unknown project "etherpads3" [18:31:53] (03PS1) 10MusikAnimal: 3.24.3 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1259184 [18:32:47] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11740124 (10Odinaldo) Dear Francesco, Thank you very much for these answers. We have been fixing some other related technical problems, due to differences in apache modules and some related packages.... [18:32:53] !log andrew@cloudcumin1001 etherpads3 START - Cookbook wmcs.vps.create_project for project etherpads3 in eqiad1 (T420532) [18:32:53] andrew@cloudcumin1001: Unknown project "etherpads3" [18:33:29] (03CR) 10MusikAnimal: [C:03+2] 3.24.3 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1259184 (owner: 10MusikAnimal) [18:34:13] (03Merged) 10jenkins-bot: 3.24.3 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1259184 (owner: 10MusikAnimal) [18:34:47] !log andrew@cloudcumin1001 etherpads3 END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project etherpads3 in eqiad1 (T420532) [18:34:51] T420532: Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532 [18:45:35] (03open) 10dcaro: deploy: reply to HEAD requests [toolforge-repos/sample-dotnet-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app/-/merge_requests/2 [18:47:02] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11740220 (10Aklapper) > Would it be possible to have a quick chat at some stage at all? See https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [18:59:57] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/64 (owner: 10l10n-bot) [19:00:01] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/64 (owner: 10l10n-bot) [19:00:49] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/36 (owner: 10l10n-bot) [19:00:58] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/36 (owner: 10l10n-bot) [19:18:08] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11740352 (10JeanFred) Is replacing ` lang=sparql FILTER(wikibase:isSomeValue(?grouping)) ` with ` lang=sparql FILTER(STRSTARTS(STR(?grouping), "http://www.wi... [19:20:21] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11740358 (10JeanFred) [19:20:27] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11740359 (10JeanFred) a:03JeanFred [19:20:57] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11740360 (10JeanFred) [19:20:58] 10Tool-inteGraality: Support qlever endpoint for integraality - https://phabricator.wikimedia.org/T385749#11740361 (10JeanFred) [19:47:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:42:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1076' [20:42:46] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirtlocal1076' [20:42:49] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1076.eqiad.wmnet' [20:42:49] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirtlocal1076.eqiad.wmnet' [20:42:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1076.eqiad.wmnet' [20:42:58] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirtlocal1076.eqiad.wmnet' [20:43:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' [20:49:37] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11740952 (10bd808) >>! In T416590#11740124, @Odinaldo wrote: > 3. Thank you for your suggestion about syncing... We are looking at options and seeing what's most efficient for the database in questions (... [20:50:04] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1076.eqiad.wmnet' [20:52:48] FIRING: [4x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [21:02:46] 10Tool-inteGraality: Function "" is currently not supported by QLever. - https://phabricator.wikimedia.org/T420247#11740995 (10JeanFred) I filed https://github.com/ad-freiburg/qlever/issues/2788 upstream [21:17:48] FIRING: [3x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [22:25:46] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11741408 (10Osuji_pius) [22:27:19] (03PS1) 10Jean-Frédéric: Recreate source tables as part of deployment [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1259245 [22:29:01] (03CR) 10CI reject: [V:04-1] Recreate source tables as part of deployment [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1259245 (owner: 10Jean-Frédéric) [23:47:15] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse