[00:54:41] (03open) 10chumzine: Add update to hide author data fields when there are no values [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/85 [01:01:58] 10Tool-paulina: Hide author data fields when they have no value - https://phabricator.wikimedia.org/T406909#11274888 (10Chumzine) Hello @Pepe_piton, I would really love your feedback and comments on my solution. I must say that I am delighted to finally be able to submit a merge request before the microtask is... [01:08:31] 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11274893 (10Chumzine) Hi @Nat_WDU, thank you so much for the feedback. Okay. You're welcome! [01:26:01] 10Tool-enderbot-dev: API scenario: Scrape revIDs of unapproved pages to then use approve action repeatedly - https://phabricator.wikimedia.org/T407262#11274901 (10Anderlaxe) p:05Triage→03Low [02:57:02] (03open) 10nurahwakili: T405635 - Created an API call examples md file [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/86 [04:27:09] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [05:42:35] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11275143 (10taavi) [05:42:48] 06cloud-services-team, 10Toolforge: What are the parts of toolforge? - https://phabricator.wikimedia.org/T407299#11275144 (10taavi) [05:45:43] 10Tool-wsindex, 10Developer-Outreach, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11275157 (10Bodhisattwa) [07:19:28] (03update) 10nurahwakili: T405635 - Created an API call examples md file [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/86 [07:25:40] (03update) 10nurahwakili: T405635 - Created an API call examples md file [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/86 [07:28:02] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11275304 (10Nurah_Wakili) Hello @Pepe_piton @Nat_WDU I worked on this task. Please Kindly review : https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/86.... [07:29:55] (03update) 10nurahwakili: T405635 - Created an API call examples md file [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/86 [07:46:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:46:27] 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@eqiad - https://phabricator.wikimedia.org/T407317 (10phaultfinder) 03NEW [07:49:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-prometheus-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:51:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:52:38] (03open) 10shr0x-ya: Pre-fill nationality and country of origin on country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/87 [07:53:39] 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11275358 (10shreya-bhagat) https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/87 Completed the task, kindly review @Pepe_piton @Nat_WDU [07:57:23] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11275364 (10taavi) [07:57:25] 10Toolforge, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11275365 (10taavi) [08:04:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-prometheus-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:04:50] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:09:50] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:32:37] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11275464 (10fgiunchedi) Ok the resize and reboot went well, nfs clients came back by themselves and processes recov... [08:36:52] (03open) 10oluwatumininu: Add API call examples for creating and editing Wikidata works and authors [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/88 [08:50:40] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11275515 (10Oluwatumininu.m) Hii @Pepe_piton @Nat_WDU I worked on this issue Here is the link to my MR: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/88 [08:57:17] 10Tool-paulina: Hide work data fields when they have no value - https://phabricator.wikimedia.org/T406994#11275542 (10Miiswom) Thank you @Afanyulionel :) You also did a nice job on your MR! [09:06:41] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11275557 (10dcaro) In case it's useful, there was a document started on thinking what's needed from toolforge side: https://docs.google.com/document/d/145cfnZYR2QSYlVtxBR2HCgHx46I7681_H3I74c3j_vI/edit?tab=t.0 [09:20:51] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11275606 (10SLyngshede-WMF) [09:36:16] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11275681 (10Kimbrene-kakande) Guys my gitlab is saying am pending approval : Your account is pending approval from your GitLab administrator... [09:41:39] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11275724 (10Oluwatumininu.m) >>! In T392429#11275681, @Kimbrene-kakande wrote: > Guys my gitlab is saying am pending approval : > Your accoun... [10:15:19] (03open) 10dipanshu1223: T405604:Fix/nationality select input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/89 [10:17:17] 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11275917 (10Dipanshu1223) hey @Pepe_piton @Nat_WDU i made a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/89 change... [10:32:49] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115#11275970 (10Ladsgroup) The skin is not pulled in explicitly. It's pulled in because it's in https://raw.githubusercontent.com/MWStake/nonwmf-skins/master/.gitmodules I suggest creating a PR there and ours shou... [10:46:18] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115#11276004 (10A_smart_kitten) >>! In T407115#11275970, @Ladsgroup wrote: > The skin is not pulled in explicitly. It's pulled in because it's in https://raw.githubusercontent.com/MWStake/nonwmf-skins/master/.gitm... [10:49:48] (03PS1) 10Ladsgroup: Add apps service and section [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1196403 (https://phabricator.wikimedia.org/T335407) [10:53:20] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115#11276029 (10A_smart_kitten) (so I guess, maybe it might be better to auto-detect archived GitHub repos within Codesearch itself [T317989], than submit PRs to remove them from upstream `nonwmf-extensions` when... [11:01:32] (03open) 10shr0x-ya: Type to select country of origin filter for work search [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/90 [11:06:42] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11276061 (10shreya-bhagat) @Nat_WDU @Pepe_piton https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/90 Completed the task, Kindly review! R... [11:08:49] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115#11276072 (10jhsoby) The [[https://github.com/MWStake/nonwmf-skins/blob/master/README.mediawiki|README]] in the MWStake repo says "If the skin is no longer supported and does not work with any [[https://www.med... [11:16:35] 10VPS-project-Codesearch: Remove DarkVector from codesearch - https://phabricator.wikimedia.org/T407115#11276114 (10A_smart_kitten) >>! In T407115#11276072, @jhsoby wrote: > The [[https://github.com/MWStake/nonwmf-skins/blob/master/README.mediawiki|README]] in the MWStake repo says "If the skin is no longer supp... [11:31:09] (03update) 10oluwatumininu: Prefilled nationality of author and country of origin of work at the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/62 [11:34:18] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11276204 (10Aklapper) (Also please note that a Phabricator username can be changed on request. Just saying.) [11:40:12] (03update) 10oluwatumininu: Prefilled nationality of author and country of origin of work at the country page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/62 [11:44:01] (03update) 10shr0x-ya: T406989: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/83 [11:45:51] (03update) 10shr0x-ya: T406989: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/83 [11:48:19] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11276223 (10Adwivedii) Hi @Pepe_piton @Nat_WDU , I'm Anjali Dwivedi, interested in contributing to the Paulina project for Outreachy. I'd li... [11:53:33] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [11:56:54] (03update) 10oluwatumininu: Improve country dropdown for the "country of origin" filter in the advanced search for works. [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/14 [11:58:51] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:03:44] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:05:22] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#11276321 (10Bugreporter) 05Declined→03Open Reopen since recent issue at https://www.mediawiki.org/wiki/User_talk:JJMC89 indicates we do need such a field (before Phabricator has a way to store it, we can record... [12:13:13] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:16:45] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:18:14] (03open) 10marostegui: task_template_primary.txt: Change template [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/14 [12:19:09] (03update) 10oluwatumininu: Improve country dropdown for the "country of origin" filter in the advanced search for works. [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/14 [12:19:48] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:21:23] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11276382 (10Nat_WDU) Hello @System625 ! I don't think it's a good idea to import requests in the middle of the definition. Let me know if you have any specific r... [12:21:25] (03update) 10piracalamina: feat: Add SPARQL query link to author's works list page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/81 (owner: 10system625) [12:22:14] (03update) 10piracalamina: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 (owner: 10dipanshu1223) [12:22:22] (03approved) 10piracalamina: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 (owner: 10dipanshu1223) [12:23:55] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:25:21] (03update) 10oluwatumininu: Improve country dropdown for the "country of origin" filter in the advanced search for works. [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/14 [12:35:00] (03update) 10system625: feat: Add SPARQL query link to author's works list page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/81 [12:35:02] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11276435 (10SLyngshede-WMF) @Aklapper that's a very good point. For SUL accounts we link on the ID returned, and just store the usern... [12:37:41] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11276450 (10System625) My bad @Nat_WDU , I have fixed it! I've moved the requests import to the top of the file. There was no specific reason for the inline imp... [12:38:24] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:44:34] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [12:52:15] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [13:00:35] (03approved) 10piracalamina: feat: Add SPARQL query link to author's works list page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/81 (owner: 10system625) [13:10:46] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [13:12:26] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11276583 (10Kimondorose) >>! In T406950#11269551, @Pepe_piton wrote: > Excellent @Kimondorose! Your solution works well and you understood the logic. Anyway, you might be interested in reading the co... [13:12:38] 10VPS-project-Codesearch, 06Wikipedia-iOS-App-Backlog, 13Patch-For-Review: The Wikipedia iOS app should be added to Codesearch - https://phabricator.wikimedia.org/T406500#11276584 (10Nemoralis) a:05Nemoralis→03Ladsgroup [13:12:54] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11276586 (10Nemoralis) a:05Nemoralis→03Ladsgroup [13:17:32] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [13:21:56] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [13:28:59] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [13:35:21] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [14:00:14] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407147#11276826 (10Andrew) 05Open→03Resolved a:03Andrew [14:00:21] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407118#11276828 (10Andrew) 05Open→03Resolved a:03Andrew [14:00:34] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407100#11276830 (10Andrew) 05Open→03Resolved a:03Andrew [14:00:41] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407090#11276832 (10Andrew) 05Open→03Resolved a:03Andrew [14:01:11] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407085#11276834 (10Andrew) 05Open→03Resolved a:03Andrew [14:01:15] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407082#11276839 (10Andrew) 05Open→03Resolved a:03Andrew [14:01:19] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407081#11276841 (10Andrew) 05Open→03Resolved a:03Andrew [14:01:21] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11276838 (10Nat_WDU) Good job, @System625 . I appreciate that you are using the language selected by the user for the query, and that you've updated i18n files.... [14:01:33] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407079#11276844 (10Andrew) 05Open→03Resolved a:03Andrew [14:02:03] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407071#11276849 (10Andrew) 05Open→03Resolved a:03Andrew [14:02:09] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407069#11276851 (10Andrew) 05Open→03Resolved a:03Andrew [14:02:20] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407068#11276854 (10Andrew) 05Open→03Resolved a:03Andrew [14:02:32] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407067#11276857 (10Andrew) 05Open→03Resolved a:03Andrew [14:03:23] 06cloud-services-team: SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://phabricator.wikimedia.org/T406811#11276865 (10Andrew) 05Open→03Resolved a:03Andrew [14:03:31] 06cloud-services-team, 10Toolforge: Job not restarting despite liveness probe failures - https://phabricator.wikimedia.org/T400957#11276868 (10taavi) 05Open→03Resolved David says this is fixed. [14:03:33] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406800#11276870 (10Andrew) 05Open→03Resolved a:03Andrew [14:03:39] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406797#11276872 (10Andrew) 05Open→03Resolved a:03Andrew [14:03:43] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406787#11276874 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:01] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406779#11276878 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:05] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406757#11276880 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:09] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406727#11276882 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:15] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406693#11276884 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:28] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T406527#11276898 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:32] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406654#11276902 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:42] 06cloud-services-team, 10Toolforge: [envvars] - kubernetes error, length limit? - https://phabricator.wikimedia.org/T406236#11276904 (10taavi) p:05Triage→03Low [14:04:46] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406679#11276907 (10Andrew) 05Open→03Resolved a:03Andrew [14:04:58] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407045#11276910 (10Andrew) 05Open→03Resolved a:03Andrew [14:05:06] 06cloud-services-team, 10Cloud-VPS: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516#11276913 (10taavi) p:05Triage→03Medium a:03Andrew [14:05:11] 06cloud-services-team, 10Cloud-VPS: Package mcrouter for Debian Trixie - https://phabricator.wikimedia.org/T406522#11276915 (10taavi) 05Open→03Resolved a:03Andrew [14:05:23] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Do something with cloudcontrol100[8-10]-dev - https://phabricator.wikimedia.org/T406630#11276920 (10taavi) p:05Triage→03Medium [14:05:31] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Do something with cloudcontrol100[8-10]-dev - https://phabricator.wikimedia.org/T406630#11276923 (10taavi) [14:05:34] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11276924 (10taavi) [14:05:59] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406675#11276926 (10Andrew) 05Open→03Resolved a:03Andrew [14:06:03] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406646#11276928 (10Andrew) 05Open→03Resolved a:03Andrew [14:06:41] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406275#11276931 (10Andrew) 05Open→03Resolved a:03Andrew [14:06:48] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11276933 (10taavi) p:05Triage→03Medium a:03Andrew [14:06:52] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2010-dev:9100 - https://phabricator.wikimedia.org/T406290#11276936 (10Andrew) 05Open→03Resolved a:03Andrew [14:07:23] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406538#11276945 (10Andrew) 05Open→03Resolved a:03Andrew [14:07:28] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol1007:9100 - https://phabricator.wikimedia.org/T406274#11276947 (10Andrew) 05Open→03Resolved a:03Andrew [14:07:41] 06cloud-services-team, 10Toolforge, 07TestMe: Can't launch a job on Toolforge: 504 Gateway Time-out - https://phabricator.wikimedia.org/T407054#11276949 (10taavi) p:05Triage→03Medium [14:07:59] 06cloud-services-team, 10Toolforge, 07TestMe: Can't launch a job on Toolforge: 504 Gateway Time-out - https://phabricator.wikimedia.org/T407054#11276951 (10taavi) [14:08:02] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11276954 (10dcaro) 05Open→03In progress p:05Triage→03Medium Setting as medium as there has been an improvement already in this regard, but work is still ongoing {T404726}, I'll make... [14:08:12] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11276959 (10dcaro) [14:08:13] 10Toolforge (Toolforge iteration 24): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11276960 (10dcaro) [14:08:52] 06cloud-services-team, 10PAWS: PAWS is unable to access CloudElastic - https://phabricator.wikimedia.org/T407216#11276962 (10taavi) p:05Triage→03Medium [14:09:33] 06cloud-services-team, 10Toolforge: What are the parts of toolforge? - https://phabricator.wikimedia.org/T407299#11276965 (10taavi) p:05Triage→03High [14:09:39] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11276966 (10taavi) p:05Triage→03High [14:09:52] 10Toolforge, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11276967 (10taavi) p:05Triage→03High [14:10:03] (03open) 10shr0x-ya: Add: Wikidata and Wikimedia Commons link to work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/91 [14:10:48] 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@codfw - https://phabricator.wikimedia.org/T406517#11276968 (10taavi) 05Open→03Invalid [14:11:14] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcumin1001:9100 - https://phabricator.wikimedia.org/T407204#11276989 (10taavi) 05Open→03Resolved Related to the Gerrit maintenance [14:12:48] 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@eqiad - https://phabricator.wikimedia.org/T407317#11276994 (10taavi) 05Open→03Resolved [14:12:49] 10Tool-paulina: Add Wikidata and Wikimedia Commons links to work page - https://phabricator.wikimedia.org/T406950#11276993 (10shreya-bhagat) https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/91 @Pepe_piton @Nat_WDU Kindly review the changes! [14:13:45] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [14:14:33] 10Tool-paulina: Link to Cradle to create new item if no results - https://phabricator.wikimedia.org/T370223#11277000 (10Nat_WDU) Hello, @Sandijigs ! please notice this task is not listed as a microtask in T392429 Thanks for the effort anyway. [14:22:36] !log filippo@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) [14:23:52] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11277079 (10System625) > Good job, @System625 . I appreciate that you are using the language selected by the user for the query, and that you've updated i18n fil... [14:24:58] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [14:35:43] 06cloud-services-team, 10Toolforge: What are the parts of toolforge? - https://phabricator.wikimedia.org/T407299#11277132 (10fnegri) Cross-posting from the parent task: >>! In T407296#11275557, @dcaro wrote: > In case it's useful, there was a document started on thinking what's needed from toolforge side: htt... [14:43:24] (03approved) 10ladsgroup: task_template_primary.txt: Change template [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/14 (owner: 10marostegui) [14:43:27] (03merge) 10ladsgroup: task_template_primary.txt: Change template [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/14 (owner: 10marostegui) [14:55:48] 06cloud-services-team, 10Toolforge: Toolforge on Metal pilot - https://phabricator.wikimedia.org/T407296#11277258 (10dcaro) I agree with @Andrew, I think that the experiment is not yet well defined. And I agree also with the questions he raises. Maybe we can phrase it as hypotheses like with the anual plan, so... [14:59:22] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [15:02:08] (03CR) 10FNegri: [C:04-1] openstack: cloudweb: Add cookbook to batch reboot hosts (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 (owner: 10Majavah) [15:16:17] (03PS6) 10Majavah: openstack: cloudweb: Add cookbook to batch reboot hosts [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 [15:16:28] (03PS2) 10Ladsgroup: Add apps section [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1196403 (https://phabricator.wikimedia.org/T335407) [15:19:57] (03CR) 10Majavah: openstack: cloudweb: Add cookbook to batch reboot hosts (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 (owner: 10Majavah) [15:21:39] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385 (10Nat_WDU) 03NEW [15:22:00] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [15:22:16] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11277447 (10Nat_WDU) [15:22:55] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11277448 (10Nat_WDU) [15:26:59] (03CR) 10FNegri: [C:03+1] openstack: cloudweb: Add cookbook to batch reboot hosts [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 (owner: 10Majavah) [15:30:43] (03update) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [15:33:45] (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367) [15:34:33] (03update) 10dipanshu1223: T397554: Add Wikidata Query Service link to works list [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/66 [15:39:43] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11277514 (10Dipanshu1223) hey @Nat_WDU , thanks for feedback i forgot test other language i made some changes related this issue::-https://gitlab.wikimedia.org/... [15:45:05] (03CR) 10Majavah: [C:03+2] openstack: cloudweb: Add cookbook to batch reboot hosts [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 (owner: 10Majavah) [15:48:45] (03Merged) 10jenkins-bot: openstack: cloudweb: Add cookbook to batch reboot hosts [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1196017 (owner: 10Majavah) [16:20:19] (03open) 10oluwatumininu: Improved count [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/92 [16:28:40] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners): tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11277894 (10fnegri) After more testing, it seems to be a good'ol MTU issue: `lang=shell-session root@runner-10... [16:31:50] (03update) 10oluwatumininu: Create new generic page (closes T407385) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/92 [16:36:40] (03update) 10oluwatumininu: Create new generic page (closes T407385) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/92 [16:53:02] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11278073 (10Oluwatumininu.m) Hi @Pepe_piton @Nat_WDU I fixed this issue. Here is the link to my merge request: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/92 [17:14:36] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11278217 (10dcaro) Awesome find! what's the mtu of the vm itself? Is there a discrepancy? [17:16:26] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11278224 (10dcaro) >>! In T405742#11278217, @dcaro wrote: > Awesome find! > what's the mt... [17:22:47] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11278268 (10dcaro) I suspect that it might be related to the vxlan/overlay network sort o... [17:22:55] 06cloud-services-team: PuppetFailure Puppet has failed on an-redacteddb1001:9100 - https://phabricator.wikimedia.org/T407201#11278270 (10fnegri) 05Open→03Resolved a:03fnegri Puppet is now working fine. [17:24:10] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11278287 (10fnegri) a:03fnegri [17:24:17] (03approved) 10dcaro: restart deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/218 (owner: 10damian) [17:24:23] (03update) 10dcaro: restart deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/218 (owner: 10damian) [17:35:46] (03update) 10dcaro: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) (owner: 10raymond-ndibe) [17:36:34] (03approved) 10dcaro: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) (owner: 10raymond-ndibe) [17:39:36] (03update) 10dcaro: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) (owner: 10raymond-ndibe) [17:39:55] (03update) 10dcaro: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) (owner: 10raymond-ndibe) [17:44:45] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11278442 (10Pepe_piton) @Ritika-Bhambri11 I like very much your approach, and I also like that you added the cancel button and the "Esc" key cancel action. It works well and the code is pretty straightf... [19:07:34] FIRING: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:07:42] 06cloud-services-team: DiskSpace Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://phabricator.wikimedia.org/T407420 (10phaultfinder) 03NEW [19:09:42] FIRING: SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1002-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:14:38] FIRING: [2x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1002-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:24:43] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11278809 (10Andrew) > I wanted to run `cinder attachment-list --volume-id 648504db-18c2-4cee-b731-567dcb4dadf6` and failed miserably to get the right combination of OS_ envir... [19:34:19] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11278834 (10Andrew) >>! In T406688#11278809, @Andrew wrote: > >> I wanted to run `cinder attachment-list --volume-id 648504db-18c2-4cee-b731-567dcb4dadf6` and failed miserabl... [19:44:50] (03open) 10dipanshu1223: T407385:Add page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/93 [19:46:07] (03update) 10nurahwakili: Edit feature for Nationality field [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/18 [19:48:16] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11278878 (10Andrew) Here are those attachments in the database: ` mysql:root@localhost [cinder]> select id, instance_uuid, volume_id, attach_status from volume_attachment w... [19:48:34] (03update) 10dipanshu1223: T407385:Add page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/93 [19:49:20] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11278885 (10Nurah_Wakili) Hello @Pepe_piton, I have now added a wrapper for the 'Publish' & 'Cancel' buttons and also the alert message so all user-facing strings can be properly translated. Thank you... [19:49:28] (03update) 10dipanshu1223: T407385:Add page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/93 [19:51:58] (03update) 10dipanshu1223: T407385:Add new page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/93 [19:52:48] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11278892 (10Dipanshu1223) hi @Nat_WDU i made a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/93 changes made :- - Added a new page: new-page.html - Updated app.py to... [20:22:41] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:24:46] (03update) 10nurahwakili: T406989 - Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/44 [20:25:19] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#11278995 (10bd808) Implementing the data entry and storage as text on https://wikitech.wikimedia.org/wiki/Tool:Phab-ban/Log is relatively straight forward. As an actual audit tool however it feels pretty weak. The... [20:25:31] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11278997 (10Andrew) ` mysql:root@localhost [cinder]> delete from volume_attachment where volume_id='648504db-18c2-4cee-b731-567dcb4dadf6' and deleted=0 and attach_status='rese... [20:26:27] FIRING: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown [20:27:41] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:31:27] RESOLVED: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown [20:33:48] FIRING: PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:33:54] 06cloud-services-team: PuppetFailure Puppet has failed on cloudbackup1002-dev:9100 - https://phabricator.wikimedia.org/T407429 (10phaultfinder) 03NEW [20:43:25] 06cloud-services-team, 10Cloud-VPS: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688#11279106 (10Andrew) @fgiunchedi, I dropped novaadmin.sh in your home directory on cloudcontrol1006; if you source it you can then run cinder commands. Do you want to explore... [20:45:07] (03open) 10nurahwakili: T407385- Created new generic page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/94 [20:47:06] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11279118 (10Nurah_Wakili) Hello @Nat_WDU. I have implemented this task. Kindly review the MR details: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/94 Thanks! [21:03:56] FIRING: SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:04:13] 06cloud-services-team: SystemdUnitDown The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://phabricator.wikimedia.org/T407437 (10phaultfinder) 03NEW [21:05:31] (03update) 10dipanshu1223: T405412 "Edit field feature" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/82 [21:08:56] FIRING: [2x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:09:03] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407438 (10phaultfinder) 03NEW [21:09:49] (03update) 10dipanshu1223: T405412 "Edit field feature" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/82 [21:14:19] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11279272 (10Dipanshu1223) hey @Pepe_piton @Nat_WDU Thanks for feedback @Pepe_piton i made some changes a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_r... [21:56:05] RESOLVED: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:56:05] PROBLEM - Host cloudbackup1002-dev is DOWN: PING CRITICAL - Packet loss = 100% [21:56:18] RESOLVED: PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:56:26] RESOLVED: [2x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1002-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:56:26] RESOLVED: [2x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:57:47] FIRING: NodeDown: Node cloudbackup1002-dev is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [22:05:52] 06cloud-services-team, 10Cloud-VPS, 10Ganeti: cloudbackup1002-dev out of service - https://phabricator.wikimedia.org/T407443 (10Andrew) 03NEW [23:10:21] (03open) 10miiswom: Draft: Create API call examples to create and edit Wikidata works and authors items [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/95 [23:35:50] (03open) 10chumzine: Add new-page template [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/96 [23:37:47] FIRING: NodeDown: Node cloudbackup1002-dev has been down for long. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:37:58] 06cloud-services-team: NodeDown Node cloudbackup1002-dev has been down for long. - https://phabricator.wikimedia.org/T407456 (10phaultfinder) 03NEW [23:43:01] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11279656 (10Chumzine) Hola @Nat_WDU I have implemented this task. Please find my MR at [[https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/96 | this link ]]. The following changes were made; - A new-pa...