[12:05:07] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:30:26] (03CR) 10Addshore: Respect environment overrides in browser tests and custom commands (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1361209 (https://phabricator.wikimedia.org/T252883) (owner: 10Addshore) [15:12:52] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Extended-Inputbox - https://phabricator.wikimedia.org/T442416#12530890 (10Aklapper) 05Open→03Resolved Ah sorry, fixed [15:31:25] 10Gerrit, 07Upstream: Gitiles does not provide raw view of the file for copy-pasting (txt download is base64 encoded) - https://phabricator.wikimedia.org/T270903#12531003 (10Tgr) A similar request has been declined before: https://issues.gerritcodereview.com/issues/40001846 [16:21:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Pywikibot, 10Wikimedia-GitHub: Delete the already deleted Pywikibot branch "tests" on Gitub mirror - https://phabricator.wikimedia.org/T444044#12531038 (10Xqt) a:05Xqt→03None [16:24:02] !log Delete deployment-echostore02 in Horizon - T436479 [16:24:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:24:05] T436479: Replace deployment-echostore02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436479 [16:49:43] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12531095 (10Pppery) 05Open→03Resolved Shazaam! [16:49:52] 10Phabricator Translations Extension: Update phabricator translations for upstream i18n changes - https://phabricator.wikimedia.org/T412650#12531098 (10Pppery) 05Open→03Resolved Calling this close enough to done [16:57:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Pywikibot, 10Wikimedia-GitHub: Delete the already deleted Pywikibot branch "tests" on Gitub mirror - https://phabricator.wikimedia.org/T444044#12531101 (10Meno25) [18:41:28] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T444063 (10Comander123x) 03NEW [18:43:20] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12531179 (10Aklapper) Thank you a lot for your ~~masochi~~persistency! [18:46:17] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Commander_x - https://phabricator.wikimedia.org/T444063#12531180 (10Aklapper) [18:46:25] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T444063#12531182 (10Comander123x) [18:47:11] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Commander_x - https://phabricator.wikimedia.org/T444063#12531186 (10Aklapper) [18:47:24] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Commander_x - https://phabricator.wikimedia.org/T444063#12531188 (10Aklapper) 05Open→03Resolved a:03Aklapper Approved. Happy hacking! Please also [link your LDAP account to your Phabricator acc... [18:53:07] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for commanderx - https://phabricator.wikimedia.org/T444063#12531192 (10Comander123x) 05Resolved→03Open a:05Aklapper→03None [19:02:20] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for commander_x - https://phabricator.wikimedia.org/T444063#12531197 (10Aklapper) 05Open→03Resolved a:03Aklapper [19:23:06] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf31 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:29:24] <_Gerges> I have an issue on arzwiki: a file has been deleted, but for some reason, it still exists in the database: https://arz.wikipedia.org/wiki/%D9%85%D9%84%D9%81:Arguments [19:42:40] _Gerges: you will need to file a task and someone will pick it up in the morning [19:44:33] <_Gerges> RhinosF1: Do you have access to Logstash? [19:44:53] <_Gerges> Check the messages; I think you might find something related to this page. [19:44:58] _Gerges: no [19:45:18] File a task and someone will help during working hours [19:45:28] It is not even a releng issue [19:46:40] !log Switch DNS RRs for etcd to deployment-etcd06, decommission deployment-etcd02 and deployment-etcd05 - T436473 [19:46:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:46:43] T436473: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473 [19:46:49] Evening Southparkfan [19:46:56] Hi :) [19:50:07] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:56:16] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473#12531244 (10Southparkfan) The process was a bit more involved than I had hoped. For future reference, if you wish to replace this host with a ne... [20:05:07] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:09:29] _Gerges: seems to be a known issue: https://phabricator.wikimedia.org/T258321 [20:13:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473#12531265 (10Southparkfan) 05In progress→03Resolved [20:27:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12531287 (10Southparkfan) See T428495, T435405 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1309257 for m... [21:04:09] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222#12531310 (10thcipriani) p:05Triage→03Medium [21:05:03] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T442223#12531314 (10thcipriani) p:05Triage→03Medium [21:05:41] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12531318 (10thcipriani) p:05Triage→03Medium [21:06:19] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T442225#12531322 (10thcipriani) p:05Triage→03Medium [21:06:41] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T442226#12531326 (10thcipriani) p:05Triage→03Medium [21:07:11] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T442227#12531330 (10thcipriani) p:05Triage→03Medium [21:07:57] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T442230#12531334 (10thcipriani) p:05Triage→03Medium [22:51:07] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates