[00:13:13] 10wikitech.wikimedia.org: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074 (10GTrang) 03NEW [00:19:53] 10wikitech.wikimedia.org: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074#10223751 (10GTrang) 05Open→03Stalled Per the comments at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1077048. [01:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:25:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:19:07] (03PS3) 10Krinkle: README.md: Add pointer to frontend local setup documentation [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1076198 (owner: 10D3r1ck01) [03:19:10] (03CR) 10Krinkle: [C:03+2] README.md: Add pointer to frontend local setup documentation [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1076198 (owner: 10D3r1ck01) [03:20:08] (03Merged) 10jenkins-bot: README.md: Add pointer to frontend local setup documentation [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1076198 (owner: 10D3r1ck01) [03:43:14] (03open) 10pppery: Update queue_wiki.html [toolforge-repos/nfp] - 10https://gitlab.wikimedia.org/toolforge-repos/nfp/-/merge_requests/2 (https://phabricator.wikimedia.org/T376935) [03:46:33] 10Tool-nfp, 13Patch-For-Review, 07Voice & Tone: Poor grammar on NFP disclaimer header - https://phabricator.wikimedia.org/T376935#10223798 (10Pppery) [03:47:03] 10Tool-nfp: tool-nfp should not manually concatenate SQL - https://phabricator.wikimedia.org/T336978#10223801 (10Pppery) Since the tool is querying public read-only data the impact of a SQL injection bug if one exists seems minimal. (Which doesn't mean it shouldn't be fixed. of course) [04:42:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:47:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:09:44] (03approved) 10ladsgroup: Update queue_wiki.html [toolforge-repos/nfp] - 10https://gitlab.wikimedia.org/toolforge-repos/nfp/-/merge_requests/2 (https://phabricator.wikimedia.org/T376935) (owner: 10pppery) [06:09:48] (03merge) 10ladsgroup: Update queue_wiki.html [toolforge-repos/nfp] - 10https://gitlab.wikimedia.org/toolforge-repos/nfp/-/merge_requests/2 (https://phabricator.wikimedia.org/T376935) (owner: 10pppery) [06:14:24] 10Tool-nfp, 13Patch-For-Review, 07Voice & Tone: Poor grammar on NFP disclaimer header - https://phabricator.wikimedia.org/T376935#10223809 (10Ladsgroup) 05Open→03Resolved a:03Pppery [06:18:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673#10223813 (10taavi) [07:01:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [13:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:57:51] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10224063 (10valerio.bozzolan) [18:01:12] 10Tool-itwiki: Catch unmanaged exceptions to report their time - https://phabricator.wikimedia.org/T377081 (10valerio.bozzolan) 03NEW p:05Triage→03Medium [18:03:21] 10Tool-itwiki: Understand crash in BotCancellazioni - Error reported from cURL [error n. 16] - https://phabricator.wikimedia.org/T375936#10224077 (10valerio.bozzolan) 05Open→03Invalid > man 5 curl > `EXIT CODES` > `16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat g... [18:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [19:33:32] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10224211 (10Audiodude) @fnegri Now we're getting a message that we don't have enough RAM quota. The original server was `g4.cores2.ram4.disk20`, so we'd like to replicate... [19:37:37] (03update) 10raymond-ndibe: Draft: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [19:39:17] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [19:40:08] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [19:46:11] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_registry) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [21:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:36:21] 10Tool-nfp: tool-nfp should not manually concatenate SQL - https://phabricator.wikimedia.org/T336978#10224302 (10Ladsgroup) Also this is not really unsafe: `lang=python conds = [ 'rc_patrolled = 0', 'rc_last_oldid = 0', 'rc_namespace = 6', 'rc_type = 3', "rc_source = '... [22:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:47:34] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 11d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:12:27] 10Tools: zinbot not patrolling - https://phabricator.wikimedia.org/T363552#10224349 (10Tamzin) 05Open→03Resolved a:03Tamzin There were a number of issues here, primarily someone having updated the RfD template without the requested notification, and then some ancillary issues as tend to arise when poki... [23:31:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-22 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [23:36:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-22 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess