[00:17:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-61 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:23:11] 10Toolforge (Quota-requests): Request increased quota for milhistbot toolforge tool - https://phabricator.wikimedia.org/T387950#10608251 (10Hawkeye7) [00:28:55] FIRING: MaxConntrack: Max conntrack at 80.32% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:32:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-61 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:33:55] RESOLVED: MaxConntrack: Max conntrack at 80.1% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:09:01] RESOLVED: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:42:42] 10Toolforge: ipcheck 504 Gateway Time-out - https://phabricator.wikimedia.org/T387947#10608347 (10Jeff_G) It's responding now. [02:12:06] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] add -l|--last to toolforge jobs logs ... - https://phabricator.wikimedia.org/T388088 (10Raymond_Ndibe) 03NEW [02:22:06] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice (in read-only mode) - https://phabricator.wikimedia.org/T388090 (10Raymond_Ndibe) 03NEW [02:33:46] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice - https://phabricator.wikimedia.org/T388090#10608460 (10Raymond_Ndibe) [02:37:55] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] add toolforge jobs shell feature - https://phabricator.wikimedia.org/T388091 (10Raymond_Ndibe) 03NEW [02:48:06] 06cloud-services-team, 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#10608479 (10Raymond_Ndibe) This is a duplicate of https://phabricator.wikimedia.org/T356503 [02:59:22] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] enable ingress for jobs-api continuous jobs - https://phabricator.wikimedia.org/T388092 (10Raymond_Ndibe) 03NEW [03:00:25] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10608506 (10Raymond_Ndibe) [03:03:50] 06cloud-services-team, 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#10608508 (10Raymond_Ndibe) We should probably close this down since we decided to implement this with kubernetes own CRD dir... [03:05:42] 06cloud-services-team, 10Toolforge: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#10608510 (10Raymond_Ndibe) a:03Raymond_Ndibe [03:08:24] 06cloud-services-team, 10Toolforge: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#10608513 (10Raymond_Ndibe) [06:54:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:29:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:01:57] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-legacy-redirector: constant failed probes by prometheus - https://phabricator.wikimedia.org/T385908#10608834 (10dcaro) It failed overnight :/ Just re-tweaked the mpm_event config to start a lot more threads (that are memory cheap) and keep th... [09:03:09] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:03:11] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:41] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:07:42] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.82-20250306090549-677c59f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/699 [09:10:45] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:16:14] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [09:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:19:30] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice - https://phabricator.wikimedia.org/T388090#10608869 (10dcaro) I'm not sure about this, it introduces a lot of complexity and binds toolforge jobs to be backwards compatible wi... [09:19:39] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:21:45] 06cloud-services-team, 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#10608894 (10dcaro) 05Open→03Declined [09:27:45] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [09:27:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:53:25] (03approved) 10dcaro: components-api: bump to 0.0.82-20250306090549-677c59f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/699 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:53:40] (03merge) 10dcaro: components-api: bump to 0.0.82-20250306090549-677c59f7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/699 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:53:56] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:54:00] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:57:31] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: registry-admission: bump to 0.0.57-20250306095410-d4fed2c9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/700 [10:11:42] FIRING: [2x] AlertLintProblem: Linting problems found for NeutronAgentDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [10:26:58] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [10:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:34:59] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [10:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:30:49] 06cloud-services-team: MaxConntrack Max conntrack at 90.6% on cloudvirt1039:9100 - https://phabricator.wikimedia.org/T387179#10609438 (10aborrero) 05Open→03Resolved [11:36:39] 06cloud-services-team, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#10609447 (10fgiunchedi) Change is live, I can confirm I'm not seeing duplicate alerts anymore. i.e. for alerts with more tha... [11:46:32] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): cloud: prometheus: investigate weirdness with metrics and alertmanager - https://phabricator.wikimedia.org/T374599#10609500 (10aborrero) We had this alert today: {F58618848} Thanos does not know about the metric: {F5861... [11:50:11] 10Tool-nfp: Tool-nfp: 500 Internal Server Error - https://phabricator.wikimedia.org/T388113 (10SD_hehua) 03NEW [11:53:22] 10Tool-nfp: Tool-nfp: 500 Internal Server Error - https://phabricator.wikimedia.org/T388113#10609525 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup The service needed a restart, I think many toolforge tools had similar issues. [12:05:52] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [12:05:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:06:54] 06cloud-services-team, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#10609567 (10fnegri) Nice one, I've just seen a duplicate one disappear from the WMCS board! Thanks @fgiunchedi <3 [12:14:48] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): cloud: prometheus: investigate weirdness with metrics and alertmanager - https://phabricator.wikimedia.org/T374599#10609578 (10fnegri) Related: {T374513} [12:15:29] !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [12:15:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:19:34] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): cloud: prometheus: investigate weirdness with metrics and alertmanager - https://phabricator.wikimedia.org/T374599#10609606 (10fnegri) Available Neutron metrics can be seen with ` cloudcontrol1007:~$ curl http://localhost... [12:20:05] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/9 [12:20:05] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/32 [12:27:57] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [12:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:30:17] !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [12:30:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:57:26] (03approved) 10dcaro: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) (owner: 10fnegri) [12:58:59] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [12:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:07:50] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [13:07:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:19:50] (03update) 10raymond-ndibe: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) (owner: 10fnegri) [14:11:42] FIRING: [2x] AlertLintProblem: Linting problems found for NeutronAgentDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [14:13:10] 06cloud-services-team, 10Cloud-VPS: openstack: fix missing prometheus metrics - https://phabricator.wikimedia.org/T373878#10609956 (10fnegri) 05Resolved→03Open Reopening this task because similar alerts are firing again today, the missing metric is `openstack_neutron_agent_state`. I see that metric [does e... [14:21:26] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): cloud: prometheus: investigate weirdness with metrics and alertmanager - https://phabricator.wikimedia.org/T374599#10610007 (10fnegri) 05Open→03Resolved a:03fnegri I have reopened {T373878} which is more specific... [14:25:43] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T387946#10610020 (10fnegri) 05Open→03Resolved This was caused by hardware issues with cloudcontrol1005 that have now been resolved: {T387828}. [14:25:55] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T387946#10610024 (10fnegri) [14:26:01] 10cloud-services-team (Hardware), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: openstack galera no recent writes 2025-03-04, suspected network hardware problem - https://phabricator.wikimedia.org/T387828#10610025 (10fnegri) [14:27:53] 06cloud-services-team, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#10610026 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Thank you @fnegri ! I'm resolving this task, while the fa... [14:29:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:34:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:54:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:13:20] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice - https://phabricator.wikimedia.org/T388090#10610255 (10Raymond_Ndibe) ok maybe it's premature to do this now, but eventually we'll need to find a way to migration legacy webse... [15:17:18] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T387074#10610263 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/483 [15:17:38] vivian-rook opened https://github.com/toolforge/paws/pull/483 [15:21:11] 06cloud-services-team, 10Cloud-VPS: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137 (10aborrero) 03NEW [15:23:01] 06cloud-services-team, 10Cloud-VPS: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610292 (10fnegri) Some additional info from the "cloud@" mailing list: > The final url is https://openstack.eqiad1.wikimediacloud.org:25000/v3/auth/OS-FEDERATION/websso/openid?origin=http... [15:23:29] 06cloud-services-team, 10Cloud-VPS: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610297 (10aborrero) p:05Triage→03High [15:23:38] 06cloud-services-team, 10Cloud-VPS: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610303 (10kamila) This is what I see: {F58620044} I am logged in: {F58620054} [15:23:48] 06cloud-services-team, 10Horizon: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610308 (10fnegri) [15:24:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:28:04] (03open) 10raymond-ndibe: [jobs-cli] set timeout None for jobs logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/86 (https://phabricator.wikimedia.org/T356503) [15:34:38] (03update) 10fnegri: wmcs-k8s-metrics: upgrade charts for K8s v1.29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/681 (https://phabricator.wikimedia.org/T362868) [15:34:54] (03close) 10raymond-ndibe: [jobs-cli] set timeout None for jobs logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/86 (https://phabricator.wikimedia.org/T356503) [15:35:13] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (owner: 10soda) [15:35:19] 06cloud-services-team, 10Horizon: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610413 (10kamila) This happens on different networks (home wifi and mobile hotspot) and also in a private browser window. [15:39:19] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [15:44:38] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [15:45:43] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [15:46:58] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [15:47:16] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [15:52:26] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [16:24:03] vivian-rook closed https://github.com/toolforge/paws/pull/483 [16:24:47] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T387074#10610697 (10rook) 05Open→03Resolved a:03rook [16:26:21] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T386408#10610702 (10rook) 05Open→03Resolved a:03rook [17:05:01] (03approved) 10dcaro: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [17:16:19] 06cloud-services-team, 10Horizon: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137#10610887 (10fnegri) @kamila sent me the http headers for the 401 request, and they contain ` x-openstack-request-id req-f72077da-71c8-455b-8e0f-4cfbbce87e23 ` Looking for that ID in logstash... [17:23:04] (03approved) 10dcaro: registry-admission: bump to 0.0.57-20250306095410-d4fed2c9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/700 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:23:14] (03merge) 10dcaro: registry-admission: bump to 0.0.57-20250306095410-d4fed2c9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/700 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:23:54] (03update) 10dcaro: deploy: support deploying source build components (buildpack) [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T384634) [17:32:00] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] (add_deployment_status_updates) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [17:35:38] 10Cloud-VPS (Project-requests): Request creation of antivandalismai VPS project - https://phabricator.wikimedia.org/T388151 (10ItsNyoty) 03NEW [17:44:39] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/121 (https://phabricator.wikimedia.org/T362868) [17:44:55] (03update) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/121 (https://phabricator.wikimedia.org/T362868) [17:47:02] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/19 (https://phabricator.wikimedia.org/T362868) [17:47:17] (03update) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/19 (https://phabricator.wikimedia.org/T362868) [17:47:31] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T362868) [17:48:04] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T362868) [17:49:13] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] (add_deployment_status_updates) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [17:50:12] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/27 (https://phabricator.wikimedia.org/T362868) [17:51:36] (03update) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/121 (https://phabricator.wikimedia.org/T362868) [17:51:53] (03open) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/21 (https://phabricator.wikimedia.org/T362868) [17:53:04] (03update) 10fnegri: Update deps for K8s 1.29 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/52 (https://phabricator.wikimedia.org/T362868) [17:56:58] (03open) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [17:58:54] (03CR) 10Andrew Bogott: build: Set up Vite for building static assets (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1106032 (https://phabricator.wikimedia.org/T380114) (owner: 10Majavah) [18:17:10] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [18:37:34] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [18:44:32] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [18:47:33] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [18:58:29] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [19:06:29] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [20:11:26] (03update) 10raymond-ndibe: Draft: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:42:21] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:42:28] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:42:37] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:45:02] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:45:49] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:46:07] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:46:34] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:46:52] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:47:15] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:47:39] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:49:30] (03update) 10raymond-ndibe: [jobs-api] fix log streaming timeout bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T356503) [22:52:18] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [22:52:28] (03update) 10raymond-ndibe: Explicitly set timeout to None when following logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/31 (https://phabricator.wikimedia.org/T356503) (owner: 10soda) [23:11:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [toolforge,jobs] toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503#10611987 (10Raymond_Ndibe) [23:11:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [toolforge,jobs] toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503#10611990 (10Raymond_Ndibe) a:03Raymond_Ndibe [23:11:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [toolforge,jobs] toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503#10611992 (10Raymond_Ndibe) 05Open→03In progress