[00:03:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:08:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:13:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:51:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:01:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:20:24] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:17:31] 10Cloud-VPS, 10Striker, 10Tool-gitlab-account-approval, 10Tool-phab-ban, and 6 others: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10055153 (10taavi) >>! In T371977#10049711, @LucasWerkmeister wro... [07:57:43] (03PS2) 10Majavah: Fix a bunch of strict mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055580 [07:58:46] (03CR) 10CI reject: [V:04-1] Fix a bunch of strict mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055580 (owner: 10Majavah) [07:59:46] (03PS3) 10Majavah: Fix a bunch of strict mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055580 [08:58:05] (03CR) 10Majavah: [C:03+2] labsauth: Set OATH input autocomplete to one-time-code [labs/striker] - 10https://gerrit.wikimedia.org/r/1058616 (https://phabricator.wikimedia.org/T371794) (owner: 10XtexChooser) [09:00:24] (03Merged) 10jenkins-bot: labsauth: Set OATH input autocomplete to one-time-code [labs/striker] - 10https://gerrit.wikimedia.org/r/1058616 (https://phabricator.wikimedia.org/T371794) (owner: 10XtexChooser) [09:44:47] 10toolforge_i18n, 10Tools, 07I18n, 03Wikimania-Hackathon-2024: Extract Python library for Wikimedia tool i18n from Wikidata Lexeme Forms tool - https://phabricator.wikimedia.org/T283376#10055337 (10LucasWerkmeister) #wikimania-hackathon-2024 outcome: there are some published docs now \o/ https://toolforge-... [09:51:36] 10Tool-spacemedia: Include Flickr accounts of U.S. Government into Spacemedia tool - https://phabricator.wikimedia.org/T372192 (10Don-vip) 03NEW [09:53:44] 10Tool-spacemedia: Include Flickr accounts of U.S. Government into Spacemedia tool - https://phabricator.wikimedia.org/T372192#10055349 (10Don-vip) 05Open→03In progress p:05Triage→03Medium a:03Don-vip [09:54:26] 10Tool-spacemedia, 03Wikimania-Hackathon-2024: Include Flickr accounts of U.S. Government into Spacemedia tool - https://phabricator.wikimedia.org/T372192#10055357 (10Don-vip) [10:10:01] 06cloud-services-team, 10Toolforge, 07LDAP: python3-ldap3 mixed versions and future traps - https://phabricator.wikimedia.org/T214541#10055366 (10taavi) 05Open→03Resolved Boldly closing this a few years later :-) [11:20:45] 10Toolforge: [infra,builds-api,jobs-api,webservice] Provide metrics about build service and non-NFS adoption - https://phabricator.wikimedia.org/T360190#10055401 (10taavi) [11:20:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 14Toolforge (Toolforge iteration 09): [builds-api] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#10055402 (10taavi) [14:01:25] (03update) 10raymond-ndibe: DO_NOT_MERGE: testing _display_messages move to toolforge-weld [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/90 [14:18:31] (03open) 10raymond-ndibe: DO_NOT_MERGE: testing _display_messages move to toolforge-weld [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/116 [14:48:40] (03update) 10raymond-ndibe: DO_NOT_MERGE: testing _display_messages move to toolforge-weld [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/90 [14:49:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [14:54:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-6 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:13:51] (03open) 10raymond-ndibe: DO_NOT_MERGE: testing _display_messages move to toolforge-weld [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/44 [15:43:58] (03update) 10raymond-ndibe: [envvars-cli] remove display_messages [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/57 [15:44:10] (03update) 10raymond-ndibe: [jobs-cli] remove _display_messages [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/62 [15:44:43] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [17:41:22] (03PS1) 10Tacsipacsi: Add .gitreview [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061166 [17:41:22] (03PS1) 10Tacsipacsi: Support dark mode in mails [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061167 [17:44:06] (03CR) 10Tacsipacsi: Support dark mode in mails (031 comment) [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061167 (owner: 10Tacsipacsi) [20:20:36] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [20:21:28] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29688 bytes in 1.157 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static