[00:05:34] RESOLVED: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 6.968% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:27:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-66 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:32:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-66 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:03:19] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407085 (10phaultfinder) 03NEW [01:18:27] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:29:23] (03open) 10legoktm: Updates [toolforge-repos/i-want-to-vote] - 10https://gitlab.wikimedia.org/toolforge-repos/i-want-to-vote/-/merge_requests/2 [01:31:06] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407071#11267349 (10Bugreporter2) Why is phaultphinder phinding so many phaults like this? It's phound phour already today, and it phound phive yesterday. {F66746264} [01:32:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-66 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:37:19] (03merge) 10legoktm: Updates [toolforge-repos/i-want-to-vote] - 10https://gitlab.wikimedia.org/toolforge-repos/i-want-to-vote/-/merge_requests/2 [01:52:34] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [02:02:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:02:57] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:27:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:42:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:52:13] 10Tool-wsindex, 10Developer-Outreach, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11267370 (10Bodhisattwa) [03:22:03] FIRING: [7x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:45:10] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [03:47:04] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30041 bytes in 3.832 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [04:02:57] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:17:03] FIRING: [8x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:48:45] (03PS1) 10Legoktm: Fix fetching of extension names [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195430 [04:49:35] (03CR) 10Legoktm: [C:03+2] Fix fetching of extension names [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195430 (owner: 10Legoktm) [04:50:11] (03Merged) 10jenkins-bot: Fix fetching of extension names [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195430 (owner: 10Legoktm) [05:06:37] (03PS1) 10Legoktm: Add /healthz route [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195434 [05:07:07] (03CR) 10Legoktm: [C:03+2] Add /healthz route [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195434 (owner: 10Legoktm) [05:07:47] (03Merged) 10jenkins-bot: Add /healthz route [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1195434 (owner: 10Legoktm) [05:08:03] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407090 (10phaultfinder) 03NEW [05:52:34] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:52:03] FIRING: [8x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:06:59] 10Tool-paulina: Hide author data fields when they have no value - https://phabricator.wikimedia.org/T406909#11267548 (10Oluwatumininu.m) >>! In T406909#11266034, @Pepe_piton wrote: > @Oluwatumininu.m, Thanks for your contribution. You catched the essence of the task: adding if statements with Jinja2 in the auth... [07:07:03] FIRING: [9x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:10:11] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:11:01] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30029 bytes in 0.203 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:29:53] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 [07:29:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:37:03] FIRING: [8x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:47:03] FIRING: [8x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:52:03] FIRING: [8x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:57:03] FIRING: [7x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:03:12] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:03:22] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.nfs.migrate_service (T404584) [08:03:26] T404584: [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584 [08:03:31] !log filippo@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T404584) [08:03:57] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.nfs.migrate_service (T404584) [08:04:06] !log filippo@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T404584) [08:04:58] !log godog@r5 tools START - Cookbook wmcs.nfs.migrate_service (T404584) [08:05:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:05:18] !log godog@r5 tools END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T404584) [08:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:07:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-26 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:10:42] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.nfs.migrate_service (T404584) [08:10:47] T404584: [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584 [08:12:01] !log filippo@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T404584) [08:12:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-66 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:14:50] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:15:01] PROBLEM - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 324 bytes in 60.021 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [08:17:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:17:31] FIRING: ToolsNFSDown: No tools nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNFSDown [08:19:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:21:26] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 [08:22:31] RESOLVED: ToolsNFSDown: No tools nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNFSDown [08:23:18] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-23, tools-k8s- [08:23:18] worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36 [08:24:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-10 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:24:43] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [08:27:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:29:00] !log filippo@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-n [08:29:01] fs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36 [08:30:03] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-checker-5 (cluster eqiad1, project tools) [08:30:06] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-checker-5 (cluster eqiad1, project tools) [08:30:15] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-static-15 (cluster eqiad1, project tools) [08:30:18] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-static-15 (cluster eqiad1, project tools) [08:30:40] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-bastion-14 (cluster eqiad1, project tools) [08:30:43] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-bastion-14 (cluster eqiad1, project tools) [08:30:48] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-bastion-15 (cluster eqiad1, project tools) [08:30:51] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-bastion-15 (cluster eqiad1, project tools) [08:31:03] RECOVERY - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.045 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [08:31:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-services-06 (cluster eqiad1, project tools) [08:31:46] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-services-06 (cluster eqiad1, project tools) [08:31:55] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-mail-4 (cluster eqiad1, project tools) [08:31:57] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-mail-4 (cluster eqiad1, project tools) [08:32:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:32:45] PROBLEM - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/nfs/dumps - 258 bytes in 0.009 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [08:33:21] FIRING: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [08:34:26] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-checker-5 (cluster eqiad1, project tools) [08:34:29] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-checker-5 (cluster eqiad1, project tools) [08:34:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:35:58] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs [08:35:59] -26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36 [08:37:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:37:11] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [08:39:28] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-10 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:39:45] RECOVERY - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.015 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [08:42:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:44:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-10 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:44:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:45:28] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [08:45:35] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [08:45:36] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [08:45:43] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [08:46:35] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) [08:46:36] (03PS1) 10David Caro: wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 [08:46:37] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) [08:49:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-26 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:49:29] !log filippo@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, [08:49:29] tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36 [08:49:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:50:01] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-12 (cluster eqiad1, project tools) [08:50:02] (03CR) 10CI reject: [V:04-1] wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 (owner: 10David Caro) [08:50:04] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-12 (cluster eqiad1, project tools) [08:52:31] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-13 (cluster eqiad1, project tools) [08:52:33] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-13 (cluster eqiad1, project tools) [08:52:47] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-14 (cluster eqiad1, project tools) [08:52:49] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-14 (cluster eqiad1, project tools) [08:52:59] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-16 (cluster eqiad1, project tools) [08:53:01] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-16 (cluster eqiad1, project tools) [08:53:11] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-17 (cluster eqiad1, project tools) [08:53:12] (03PS2) 10David Caro: wmsc.k8s.reboot: add --skip-drain [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1195629 [08:53:14] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-17 (cluster eqiad1, project tools) [08:53:19] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-19 (cluster eqiad1, project tools) [08:53:22] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-19 (cluster eqiad1, project tools) [08:53:56] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-2 (cluster eqiad1, project tools) [08:53:59] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-2 (cluster eqiad1, project tools) [08:54:10] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-21 (cluster eqiad1, project tools) [08:54:13] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-21 (cluster eqiad1, project tools) [08:54:18] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-23 (cluster eqiad1, project tools) [08:54:21] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-23 (cluster eqiad1, project tools) [08:54:26] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-24 (cluster eqiad1, project tools) [08:54:29] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-24 (cluster eqiad1, project tools) [08:54:36] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-26 (cluster eqiad1, project tools) [08:54:38] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-26 (cluster eqiad1, project tools) [08:54:44] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-27 (cluster eqiad1, project tools) [08:54:46] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-27 (cluster eqiad1, project tools) [08:54:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:55:16] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-3 (cluster eqiad1, project tools) [08:55:19] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-3 (cluster eqiad1, project tools) [08:55:24] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-32 (cluster eqiad1, project tools) [08:55:27] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-32 (cluster eqiad1, project tools) [08:55:30] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-33 (cluster eqiad1, project tools) [08:55:32] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-33 (cluster eqiad1, project tools) [08:55:35] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-34 (cluster eqiad1, project tools) [08:55:38] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-34 (cluster eqiad1, project tools) [08:55:41] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-35 (cluster eqiad1, project tools) [08:55:43] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-35 (cluster eqiad1, project tools) [08:55:47] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-36 (cluster eqiad1, project tools) [08:55:49] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-36 (cluster eqiad1, project tools) [08:56:26] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-22 (cluster eqiad1, project tools) [08:56:29] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-22 (cluster eqiad1, project tools) [08:57:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:57:24] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) [08:57:27] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) [08:58:38] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) [08:58:41] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) [08:59:43] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) [08:59:45] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) [08:59:50] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:00:02] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) [09:00:05] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) [09:00:21] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) [09:00:24] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) [09:00:40] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) [09:00:43] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) [09:00:59] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) [09:01:02] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) [09:01:18] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) [09:01:21] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) [09:01:37] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) [09:01:40] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) [09:01:56] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) [09:01:59] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) [09:02:03] FIRING: [7x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:02:16] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) [09:02:18] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) [09:02:35] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) [09:02:37] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) [09:02:54] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) [09:02:56] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) [09:03:13] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) [09:03:15] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) [09:03:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-78 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:05:01] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) [09:05:04] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) [09:05:20] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) [09:05:23] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) [09:05:39] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) [09:05:42] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) [09:05:58] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) [09:06:02] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) [09:06:18] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) [09:06:21] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) [09:06:37] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) [09:06:40] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) [09:07:03] FIRING: [7x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:07:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [09:07:37] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) [09:07:40] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) [09:07:56] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) [09:07:59] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) [09:08:15] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) [09:08:18] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) [09:08:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-78 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:08:35] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) [09:08:38] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) [09:08:54] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) [09:08:57] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) [09:09:13] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) [09:09:16] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) [09:09:50] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:12:03] FIRING: [7x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-33 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:13:02] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:13:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:13:43] !log dcaro@acme tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [09:13:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:13:51] RESOLVED: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [09:13:58] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-78 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:14:12] !log dcaro@acme tools START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) [09:14:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:14:15] !log dcaro@acme tools END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) [09:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:14:23] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:14:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:14:44] !log dcaro@acme tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [09:14:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:17:03] RESOLVED: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:17:57] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 [09:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:18:58] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-78 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:19:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:24:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:29:28] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:33:58] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [09:34:28] RESOLVED: [4x] PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:48:21] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11267923 (10fgiunchedi) The flip happened today, with two snags: 1. roles were applied via `tools-nfs-` puppet pre... [09:50:53] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11267926 (10fgiunchedi) We also had to issue `wmcs.vps.instance.force_reboot` for stuck nfs workers (i.e. hard rebo... [09:52:34] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:04:54] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11268002 (10jhsoby) >>! In T335407#11246630, @Nemoralis wrote: >>>! In T335407#8815451, @Ladsgroup wrote: >> I don't think we have a good section that woul... [10:17:08] 10Tool-paulina: Hide author data fields when they have no value - https://phabricator.wikimedia.org/T406909#11268027 (10Rolalove) >>! In T406909#11266034, @Pepe_piton wrote: > @Oluwatumininu.m, Thanks for your contribution. You catched the essence of the task: adding if statements with Jinja2 in the author.html... [10:17:22] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11268028 (10fgiunchedi) I also missed the fact that `tools-nfs-2` is a big instance `g4.cores16.ram64.disk20.10xiop... [10:21:23] (03open) 10miiswom: Add improvements to the text structure in country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/65 [10:43:07] (03open) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [10:44:40] (03update) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [10:45:21] (03update) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [10:49:32] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11268116 (10Dipanshu1223) hi @Pepe_piton @Nat_WDU i made a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_request... [11:29:09] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11268167 (10Ladsgroup) Adding a new section is not hard but not as easy as adding a new repo to existing section. I can try to do it later today but also d... [11:50:24] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407071#11268238 (10Aklapper) @Bugreporter2: Please post general questions unrelated to the topic of this specific task to [more appropriate places](https://www.mediawiki.org/wiki/Communication). Thanks. [12:02:13] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [12:03:12] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:04:06] 10Toolforge, 06tools-infrastructure-team: Pull toolviews data from Kubernetes HAProxy or ingress-nginx instead of the front nginx - https://phabricator.wikimedia.org/T284558#11268291 (10taavi) a:03taavi [12:04:09] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 6.811 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [12:07:13] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [12:08:54] (03update) 10piracalamina: T405385: improve country page layout, reorganize headings [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/29 (owner: 10dipanshu1223) [12:14:05] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30030 bytes in 0.518 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [12:16:31] (03update) 10piracalamina: Improvements to the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/47 (owner: 10joyakinyi) [12:16:38] (03update) 10piracalamina: Minor improvements to the country page (T405385) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/46 (owner: 10aradhya) [12:22:06] (03update) 10piracalamina: Improved layout and organized headings [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/52 (owner: 10oluwatumininu) [12:32:41] (03open) 10dcaro: loki.alloy: don't drop logs and increase rate limit [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/997 [12:33:29] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268346 (10Nat_WDU) Hello, everyone! I've been revising your Pull requests. Here's some feedback @System625 I like that you listed the changes and added a test plan you paid attention to the h1 category hierarch... [12:34:18] (03approved) 10piracalamina: Refactor: Reorganize country page headings for better clarity [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/20 (owner: 10system625) [12:34:51] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/46 [12:34:52] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/17 [12:36:54] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1195669 (owner: 10L10n-bot) [12:38:59] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268349 (10Oluwatumininu.m) > @Oluwatumininu.m > good implementation > If you can include the name if the ticket in the name of the PR that’s helpful for us to keep things tidy Thank you very much for the feedba... [12:42:40] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268354 (10System625) > @System625 > I like that you listed the changes and added a test plan > you paid attention to the h1 category hierarchy Thank you for the feedback! I appreciate [12:44:00] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268358 (10Miiswom) Good morning! I have worked on this issue as a warm-up and would appreciate your feedback and comments @Pepe_piton and @Nat_WDU. Find attached the link to my [[ https://gitlab.wikimedia.org/to... [12:50:11] (03update) 10piracalamina: feat: add wikidata copyright description to term pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/15 (owner: 10rona) [12:51:15] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11268376 (10jhsoby) Like you mention, there's the Commons app, and also the new [[https://github.com/cis-india/Wikisource-Reader|Wikisource Reader]] app; a... [12:54:37] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115 (10jhsoby) 03NEW [12:55:30] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance cvn-nfs-1 in project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [13:00:52] (03update) 10joyakinyi: Improvements to the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/47 [13:04:48] (03update) 10miiswom: Add improvements to the text structure in country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/65 [13:04:48] (03update) 10miiswom: Add improvements to the text structure in country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/65 [13:13:03] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407118 (10phaultfinder) 03NEW [13:13:26] (03update) 10joyakinyi: Improvements to the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/47 [13:19:55] (03update) 10joyakinyi: T405385 - Improvements to the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/47 [13:23:59] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268507 (10Joyakinyi) Hello @Pepe_piton I have rebased the branch to only have one commit related to this specific task . Thank you for the feedback! [13:34:31] (03update) 10joyakinyi: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/49 [13:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:51:41] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268573 (10Nat_WDU) @Joyakinyi thanks for fixing it! @Miiswom just reviewed your, good job! [13:51:54] (03open) 10lebogang: T406950: add Wikimedia Commons link support to work pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/67 [13:52:34] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:54:15] (03update) 10piracalamina: T405598: Add copyright term description paragraph [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/37 (owner: 10lebogang) [13:55:23] (03close) 10piracalamina: T405598: Add copyright term description paragraph [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/37 (owner: 10lebogang) [13:55:30] (03reopen) 10piracalamina: T405598: Add copyright term description paragraph [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/37 (owner: 10lebogang) [13:56:05] (03merge) 10piracalamina: Refactor: Reorganize country page headings for better clarity [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/20 (owner: 10system625) [13:56:56] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11268608 (10Nat_WDU) 05Open→03Resolved [13:57:22] (03close) 10piracalamina: T405385: improve country page layout, reorganize headings [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/29 (owner: 10dipanshu1223) [13:57:37] (03close) 10piracalamina: Minor improvements to the country page (T405385) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/46 (owner: 10aradhya) [13:57:53] (03close) 10piracalamina: T405385 - Improvements to the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/47 (owner: 10joyakinyi) [13:58:24] (03close) 10piracalamina: Add improvements to the text structure in country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/65 (owner: 10miiswom) [13:59:24] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11268620 (10Lebogang) Hi @Pepe_piton I have completed the microtask, tested locally, and it works as expected. Screenshots are included, and a Merge Request has been submitted for review and feedba... [14:07:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:12:50] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:13:05] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11268672 (10Nat_WDU) Hello, dear applicants, I have some feedback for each: @Ronatone I like your idea of having a placeholder paragraph for when there is no description, but I think the chosen... [14:14:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:16:20] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11268702 (10Nat_WDU) [14:19:35] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:20:36] (03approved) 10dcaro: builds-api: bump to 0.0.201-20251007080628-5cba3c29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/996 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:20:38] (03merge) 10dcaro: builds-api: bump to 0.0.201-20251007080628-5cba3c29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/996 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:32:54] (03update) 10anbanguyen: T405604_Prefill nationality of author and country of origin of work at the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/12 [14:35:24] 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11268772 (10AnbaNguyen) @Pepe_piton : hi Jorge! thank you for reviewing! I've pushed a new commit to fix the trailing space issue. [14:36:13] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [14:37:05] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30030 bytes in 0.725 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [14:39:48] (03update) 10oluwatumininu: Improvements to the copyright term page: term description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/64 [14:43:15] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11268794 (10Oluwatumininu.m) > @Oluwatumininu.m > As requested in the ticket description, we’d like to show the wikidata item’s description at the page Hi @Nat_WDU I have corrected this. I ju... [15:01:16] (03open) 10kimondorose: T406950: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/68 [15:05:21] (03update) 10kimondorose: T406950: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/68 [15:06:13] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11268896 (10Kimondorose) @Pepe_piton Here is a link to my Merge Request for this task: **Key Changes** **pdclasses.py:** - Implemented logic in the Work.init method to safely extract the Wikimedia Co... [15:15:44] (03CR) 10Urbanecm: [C:03+2] Update didYouKnow.py [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1193522 (owner: 10Matěj Suchánek) [15:16:15] (03Merged) 10jenkins-bot: Update didYouKnow.py [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1193522 (owner: 10Matěj Suchánek) [15:33:24] (03open) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [15:36:16] (03update) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [15:39:48] (03update) 10oluwatumininu: Improvements to the copyright term page: term description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/64 [15:41:57] 10Tool-paulina: Hide work data fields when they have no value - https://phabricator.wikimedia.org/T406994#11269053 (10Miiswom) Good afternoon! Please find attached the [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 | Merge Request ]] for this microtask. Would really appreciate you... [15:42:51] (03update) 10oluwatumininu: Improvements to the copyright term page: term description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/64 [15:44:35] (03update) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [15:47:32] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11269075 (10Aradhya05) @Nat_WDU thank you for the feedback! [15:48:44] 10Toolforge, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140 (10taavi) 03NEW [15:49:14] 10Toolforge, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11269090 (10taavi) /cc @Andrew @cmooney [15:50:12] (03update) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [15:56:19] (03open) 10shr0x-ya: Improvements to the copyright term page: Term Description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/70 [15:57:51] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269144 (10shreya-bhagat) @Nat_WDU @Pepe_piton Completed the task, kindly review! https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/70 > Other applicants are welcome submi... [16:03:12] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:09:06] (03update) 10piracalamina: Improved copyright term page: changed country list to grid layout with flags (T405603) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/17 (owner: 10rolalove) [16:11:28] (03update) 10pepepiton: Add: Inception field in Work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/26 (owner: 10shr0x-ya) [16:20:17] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11269232 (10Pepe_piton) @Oluwatumininu.m Thanks for your contribution! I think there are some changes from other tasks included in the [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/22/diffs |... [16:26:45] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11269237 (10shreya-bhagat) @Pepe_piton thanks for the feedback! [16:27:40] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11269238 (10Nat_WDU) Hello, everyone! @Shreya-bhagat & @Oluwatumininu.m good implementation, good job with the SPARQL query @Rolalove good job! but I think the blue links don’t go well with... [16:28:05] (03update) 10piracalamina: Change country list to grid with flags icons [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/13 (owner: 10shr0x-ya) [16:28:31] (03approved) 10piracalamina: Change country list to grid with flags icons [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/13 (owner: 10shr0x-ya) [16:28:40] (03merge) 10piracalamina: Change country list to grid with flags icons [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/13 (owner: 10shr0x-ya) [16:28:52] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11269241 (10Oluwatumininu.m) >>! In T406949#11269232, @Pepe_piton wrote: > @Oluwatumininu.m Thanks for your contribution! I think there are some changes from other tasks included in the [[ https://gitlab.wikimedia.org/... [16:29:24] (03update) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [16:31:10] 10Toolforge, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11269245 (10cmooney) Thanks for the task @taavi ! Option 1 is nice and neat, but as you say there is a lot of work to do in terms... [16:31:27] (03update) 10piracalamina: Improvements to the copyright term page: country list styles [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/63 (owner: 10oluwatumininu) [16:31:52] (03close) 10piracalamina: Improvements to the copyright term page: country list styles [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/63 (owner: 10oluwatumininu) [16:32:12] (03close) 10piracalamina: Improved copyright term page: changed country list to grid layout with flags (T405603) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/17 (owner: 10rolalove) [16:32:23] (03close) 10piracalamina: T405603 Display countries in grid with flags [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/56 (owner: 10dipanshu1223) [16:33:05] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11269250 (10Nat_WDU) 05Open→03Resolved [16:36:52] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11269265 (10Oluwatumininu.m) >>! In T405603#11269238, @Nat_WDU wrote: > Hello, everyone! > > @Shreya-bhagat & @Oluwatumininu.m > good implementation, good job with the SPARQL query > T... [16:39:57] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11269268 (10shreya-bhagat) Thanks for the feedback @Nat_WDU [16:47:10] (03update) 10lebogang: T405598: Add copyright term description paragraph [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/37 [16:59:55] (03open) 10oluwatumininu: Added SPARQL query link to author's works list page (T397554) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/71 [17:03:30] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11269303 (10Oluwatumininu.m) Hi @Nat_WDU @Pepe_piton I worked on this issue. Link to MR: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/71 [17:13:18] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407147 (10phaultfinder) 03NEW [17:25:45] (03open) 10miiswom: Draft: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [17:30:39] (03update) 10raymond-ndibe: global: update generated toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 (owner: 10dcaro) [17:30:46] (03approved) 10raymond-ndibe: global: update generated toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 (owner: 10dcaro) [17:31:07] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11269362 (10Dipanshu1223) thanks for feedback @Nat_WDU I try my best not make mistake again. [17:33:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 07Regression: Validation error for CommonJob: file logging is only available with --mount=all - https://phabricator.wikimedia.org/T405828#11269365 (10dcaro) Nothing came up, so probably there's nothing more creating those pods without the mount... [17:33:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 07Regression: Validation error for CommonJob: file logging is only available with --mount=all - https://phabricator.wikimedia.org/T405828#11269367 (10dcaro) 05In progress→03Resolved [17:33:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11269369 (10dcaro) 05Open→03In progress [17:34:35] (03open) 10aradhya: hide work data fields when they have no value (T406994) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/73 [17:36:55] (03update) 10aradhya: hide work data fields when they have no value (T406994) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/73 [17:37:19] (03update) 10aradhya: hide work data fields when they have no value (T406994) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/73 [17:37:42] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269370 (10Lebogang) > @Lebogang > Could be improved by capitalizing and adding a period to the sentence to make it more grammatically correct. @Nat_WDU Thanks for the feedback. I have implem... [17:38:56] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:42:20] 10Tool-paulina: Hide work data fields when they have no value - https://phabricator.wikimedia.org/T406994#11269372 (10Aradhya05) Hello @Pepe_piton @Nat_WDU, I completed the task, please review the merge request - https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/73 thank you [17:48:31] (03update) 10aradhya: hide work data fields when they have no value (T406994) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/73 [17:52:34] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [17:57:12] (03open) 10rolalove: Add copyright term description to term pages above country list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/74 [18:00:03] (03update) 10miiswom: Draft: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [18:00:15] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269402 (10Nat_WDU) Here's some feedback on the latest fixes and MR's: @Oluwatumininu.m you got it right, thanks! @Lebogang and @Oluwatumininu.m As for the capitalization and punctuation, I... [18:05:45] (03update) 10rolalove: Add copyright term description to term pages above country list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/74 [18:06:32] 06cloud-services-team, 10Cloud-VPS, 10Countervandalism-Network: Puppet failure on cvn-nfs-1 - https://phabricator.wikimedia.org/T407151 (10AntiCompositeNumber) 03NEW [18:06:39] (03update) 10piracalamina: Improvements to the copyright term page: term description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/64 (owner: 10oluwatumininu) [18:07:08] (03update) 10piracalamina: Improvements to the copyright term page: Term Description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/70 (owner: 10shr0x-ya) [18:08:54] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/46 (owner: 10l10n-bot) [18:09:00] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/46 (owner: 10l10n-bot) [18:09:32] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269416 (10Rolalove) >>! In T405598#11268672, @Nat_WDU wrote: > Hello, dear applicants, I have some feedback for each: > > @Ronatone > I like your idea of having a placeholder paragraph for whe... [18:09:34] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/17 (owner: 10l10n-bot) [18:09:37] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/17 (owner: 10l10n-bot) [18:11:09] (03update) 10piracalamina: Edit feature for Nationality field [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/18 (owner: 10nurahwakili) [18:11:31] (03update) 10miiswom: Draft: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [18:11:58] (03update) 10miiswom: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [18:12:18] (03update) 10shr0x-ya: Improvements to the copyright term page: Term Description [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/70 [18:12:29] (03update) 10miiswom: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [18:15:09] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269417 (10shreya-bhagat) @Nat_WDU Rebased my PR to the latest main. Please review! [18:17:43] 10Tool-paulina: Hide author data fields when they have no value - https://phabricator.wikimedia.org/T406909#11269419 (10Miiswom) Good afternoon, I’m a little late to the party but since I’ve completed, I would very much appreciate comments and feedback from you @Nat_WDU and @Pepe_piton . Please find attached m... [18:19:08] (03update) 10lebogang: T405598: Add copyright term description paragraph [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/37 [18:25:45] (03update) 10dipanshu1223: T406909: Hide empty biography fields in author page template [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/33 [18:30:35] (03update) 10lebogang: T406950: add Wikimedia Commons link support to work pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/67 [18:33:20] (03update) 10dipanshu1223: T406909: Hide empty biography fields in author page template [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/33 [18:35:48] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [18:37:10] (03update) 10dipanshu1223: T406994:Hide detail fields with "Unknown" values dynamically [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/59 [18:37:44] (03update) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [18:40:36] (03update) 10piracalamina: Implemented a link to the SPARQL query for the author's list of works at the Wikidata Query Service [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/11 (owner: 10sopzy) [18:41:13] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269488 (10Pepe_piton) @Oluwatumininu.m Thanks for your contribution! You managed to place the Wikidata link very well. However, with your current solution, the Wikimedia Commons link isn't displaye... [18:44:16] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269494 (10Kimondorose) >>! In T406950#11269488, @Pepe_piton wrote: > @Oluwatumininu.m Thanks for your contribution! You managed to place the Wikidata link very well. However, with your current solu... [18:52:18] (03open) 10dipanshu1223: Draft: T405598:Add copyright term description paragraph on country lists pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/75 [18:57:23] (03update) 10dipanshu1223: Draft: T405598:Add copyright term description paragraph on country lists pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/75 [18:59:40] (03update) 10dipanshu1223: Draft: T405598:Add copyright term description paragraph on country lists pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/75 [19:02:57] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:05:40] (03update) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] (update_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076) [19:05:51] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11269525 (10Dipanshu1223) Hey @Nat_WDU, I’m also working on the MR related to this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/75/diffs?resolved_conflicts=true So... [19:06:29] (03update) 10dipanshu1223: T405598:Add copyright term description paragraph on country lists pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/75 [19:40:03] (03update) 10pepepiton: Feat(work-page): add Wikidata and Wikimedia Commons linksImproved count [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/23 (owner: 10oluwatumininu) [19:40:18] (03update) 10pepepiton: Added Wikidata & Wikimedia commons link to render if available [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/24 (owner: 10nurahwakili) [19:49:46] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269551 (10Pepe_piton) Excellent @Kimondorose! Your solution works well and you understood the logic. Anyway, you might be interested in reading the comment I left for @Lebogang , since there are sl... [19:51:04] (03update) 10piracalamina: Added SPARQL query link to author's works list page (T397554) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/71 (owner: 10oluwatumininu) [20:02:03] (03update) 10oluwatumininu: Feat(work-page): add Wikidata and Wikimedia Commons linksImproved count [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/23 [20:04:21] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269572 (10Oluwatumininu.m) >>! In T406950#11269488, @Pepe_piton wrote: > @Oluwatumininu.m Thanks for your contribution! You managed to place the Wikidata link very well. However, with your current... [20:09:20] (03update) 10pepepiton: Fix: Hide data fields on work page with unknown value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/34 (owner: 10shr0x-ya) [20:17:29] (03update) 10nurahwakili: Added Wikidata & Wikimedia commons link to render if available [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/24 [20:17:52] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11269589 (10Nat_WDU) Hello @Sopzy, @Oluwatumininu.m and @Dipanshu1223 , the three of you managed to make a good working implementation of this, congrats! @Dipan... [20:22:21] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269592 (10Nurah_Wakili) Hello @Pepe_piton , Thank you so much for your feedback. I have made the changes to fix the issue with getting the Wikicommons link. Kindly review the update MR: https://gi... [20:26:12] (03open) 10chumzine: Add updated template for country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/76 [21:01:25] (03update) 10oluwatumininu: Added SPARQL query link to author's works list page (T397554) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/71 [21:02:57] FIRING: [4x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:03:58] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11269644 (10Oluwatumininu.m) >>! In T397554#11269589, @Nat_WDU wrote: > Hello @Sopzy, @Oluwatumininu.m and @Dipanshu1223 , the three of you managed to make a goo... [21:10:09] (03update) 10chumzine: Add updated template for country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/76 [21:11:21] (03open) 10miiswom: Draft: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/77 [21:11:24] (03update) 10miiswom: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/77 [21:12:04] (03update) 10miiswom: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/77 [21:12:38] (03update) 10miiswom: Hide work data fields when they have no value [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/69 [21:13:07] (03update) 10miiswom: Hide empty fields in the biography details section [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/72 [21:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:27:34] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:35:50] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269711 (10Miiswom) Hellooo everyone! I have completed this task! I very much enjoyed navigating through the different files and learning more about how the data is travelling between each files.... [21:36:46] 06cloud-services-team, 10Quarry: Create simple admin management tool - https://phabricator.wikimedia.org/T224376#11269714 (10Agamyasamuel) Hey @Framawiki Just wanted to check with you, if your task is available to contribute ? Or are you still working on it ? [21:39:50] (03update) 10miiswom: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/77 [21:41:14] (03update) 10miiswom: Add Wikidata and Wikimedia Commons links to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/77 [21:51:55] (03open) 10chumzine: Resolve T405598 [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/78 [21:54:14] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11269767 (10Miiswom) >>! In T406950#11269488, @Pepe_piton wrote: > As a side note, it's possible to simplify the if statements in the work.html template to correctly display the vertical bar symbol (... [21:55:16] (03update) 10chumzine: Resolve T405598 [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/78 [22:03:00] 10Tool-paulina: Hide work data fields when they have no value - https://phabricator.wikimedia.org/T406994#11269777 (10Pepe_piton) @shreya-bhagat Thanks for your contribution! You've solved the problem very well using a strategy similar to the one you used in T406909. I think you're taking full advantage of the J... [22:46:03] 10Tool-paulina: Hide work data fields when they have no value - https://phabricator.wikimedia.org/T406994#11269866 (10Miiswom) >>! In T406994#11269777, @Pepe_piton wrote: > One minor detail: it might be better to put the if statements for each field outside of the `
` tag of the detail-item, so the template...