[00:05:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:05:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T408050 (10phaultfinder) 03NEW [00:17:42] PROBLEM - HTTPS on gerrit1003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://phabricator.wikimedia.org/project/view/330/ [00:18:42] RECOVERY - HTTPS on gerrit1003 is OK: SSL OK - Certificate gerrit.wikimedia.org valid until 2025-12-28 16:37:29 +0000 (expires in 66 days) https://phabricator.wikimedia.org/project/view/330/ [00:20:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:22:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:23:12] Project beta-code-update-eqiad build #571105: 04FAILURE in 4 min 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571105/ [00:24:02] Project mediawiki-core-doxygen build #14557: 04FAILURE in 5 min 59 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14557/ [00:24:02] Project beta-code-update-eqiad build #571106: 04STILL FAILING in 49 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571106/ [00:24:02] PROBLEM - HTTPS on gerrit1003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://phabricator.wikimedia.org/project/view/330/ [00:25:04] RECOVERY - HTTPS on gerrit1003 is OK: SSL OK - Certificate gerrit.wikimedia.org valid until 2025-12-28 16:37:29 +0000 (expires in 66 days) https://phabricator.wikimedia.org/project/view/330/ [00:27:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:29:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:31:06] PROBLEM - HTTPS on gerrit1003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://phabricator.wikimedia.org/project/view/330/ [00:31:38] RECOVERY - HTTPS on gerrit1003 is OK: SSL OK - Certificate gerrit.wikimedia.org valid until 2025-12-28 16:37:29 +0000 (expires in 66 days) https://phabricator.wikimedia.org/project/view/330/ [00:33:53] Project beta-code-update-eqiad build #571107: 04STILL FAILING in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571107/ [00:34:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:38:31] FIRING: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:38:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T408050#11301030 (10phaultfinder) [00:43:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:45:09] Yippee, build fixed! [00:45:10] Project beta-code-update-eqiad build #571108: 09FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571108/ [01:32:16] Yippee, build fixed! [01:32:16] Project mediawiki-core-doxygen build #14558: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14558/ [02:09:06] 06Release-Engineering-Team, 10MediaWiki-extensions-Cargo, 06Security-Team: Remove REL-style release branches from the Cargo extension git repository - https://phabricator.wikimedia.org/T407862#11301090 (10Reedy) Can you look at https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/Cargo,branches an... [04:09:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:09:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T408050#11301193 (10phaultfinder) [04:15:11] Project beta-code-update-eqiad build #571129: 04FAILURE in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571129/ [04:24:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:25:10] Yippee, build fixed! [04:25:10] Project beta-code-update-eqiad build #571130: 09FIXED in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/571130/ [05:06:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:16:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:09:17] (03approved) 10jelto: Update custom GitLab logo after upstream background changes [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/79 (https://phabricator.wikimedia.org/T407993) (owner: 10aklapper) [07:16:29] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T408050#11301433 (10Jelto) [07:16:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T408050#11301434 (10Jelto) [08:46:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T405680#11301701 (10tstarling) [09:06:41] (03PS5) 10Jgiannelos: Fix imports so the project is compatible with latest versions [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/1198099 [09:10:16] (03PS6) 10Jgiannelos: Make project buildable with latest debian base [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/1198099 [09:20:20] (03PS7) 10Jgiannelos: Make project buildable with latest debian base [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/1198099 [10:10:03] maintenance-disconnect-full-disks build 748173 integration-agent-docker-1061 (/: 24%, /srv: 96%, /var/lib/docker: 32%): OFFLINE due to disk space [10:15:03] maintenance-disconnect-full-disks build 748174 integration-agent-docker-1061 (/: 24%, /srv: 63%, /var/lib/docker: 31%): RECOVERY disk space OK [10:45:41] (03CR) 10Michael Große: "That's great to hear!" [integration/config] - 10https://gerrit.wikimedia.org/r/1195368 (owner: 10Hashar) [10:59:52] 06Project-Admins: Tag for community collab tasks - https://phabricator.wikimedia.org/T405909#11302192 (10Nikerabbit) What's the process from here? Do I need to gather feedback for this proposal? [11:10:38] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Tag for community collab tasks - https://phabricator.wikimedia.org/T405909#11302247 (10Aklapper) a:03Aklapper A comment that missing info has been added to the task description. :) [11:11:17] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Tag for community collab tasks - https://phabricator.wikimedia.org/T405909#11302264 (10Aklapper) 05Openβ†’03Resolved Requested public project #community-collaboration has been created: https://phabricator.wikimedia.org/project/view/8284/ (In case y... [12:03:25] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11302453 (10Xqt) @bd808 Some of those failing tests are marked to be skipped [[ https://codesearch.wmcl... [12:04:40] 10Beta-Cluster-Infrastructure, 10Pywikibot, 13Patch-For-Review, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError in _json_loads when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#11302455 (10Xqt) 05Openβ†’03Resolved a:03Xqt [12:17:46] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11302560 (10Lucas_Werkmeister_WMDE) Gonna quickly note here that Wikimedia Cloud... [12:29:38] 10GitLab, 06collaboration-services, 06DBA, 06Infrastructure-Foundations: python3-wmfmariadbpy fails to import on apt-staging with "already registered with different checksums" - https://phabricator.wikimedia.org/T408109 (10Jelto) 03NEW [12:43:50] 10GitLab, 06collaboration-services, 06DBA, 06Infrastructure-Foundations: python3-wmfmariadbpy fails to import on apt-staging with "already registered with different checksums" - https://phabricator.wikimedia.org/T408109#11302685 (10elukey) @Jelto what are the files that you see? ` root@apt-staging2001:/sr... [13:10:31] FIRING: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:10:37] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T408113 (10phaultfinder) 03NEW [13:15:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:20:18] 10GitLab, 06collaboration-services, 06DBA, 06Infrastructure-Foundations: python3-wmfmariadbpy fails to import on apt-staging with "already registered with different checksums" - https://phabricator.wikimedia.org/T408109#11302813 (10Jelto) Yes that was one of the files I was looking at. However the `wmfmari... [13:23:16] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T408113#11302816 (10Jelto) [13:56:28] 06Release-Engineering-Team: Mismatch between MediaWiki Uusername and Phabricator username - https://phabricator.wikimedia.org/T408122 (10Meztuinaga-ctr) 03NEW [13:59:11] 06Release-Engineering-Team: Mismatch between MediaWiki Uusername and Phabricator username - https://phabricator.wikimedia.org/T408122#11303023 (10taavi) 05Openβ†’03Resolved a:03taavi [13:59:24] 10Phabricator, 06Release-Engineering-Team: Mismatch between MediaWiki Uusername and Phabricator username - https://phabricator.wikimedia.org/T408122#11303025 (10taavi) [14:43:01] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [14:59:37] (03approved) 10dancy: image: Delegate syntax handling to buildkit-syntax-forwarder [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/157 (owner: 10dduvall) [16:00:03] maintenance-disconnect-full-disks build 748243 integration-agent-docker-1043 (/: 35%, /srv: 98%, /var/lib/docker: 28%): OFFLINE due to disk space [16:03:56] 06Release-Engineering-Team, 10MediaWiki-extensions-Cargo, 06Security-Team: Remove REL-style release branches from the Cargo extension git repository - https://phabricator.wikimedia.org/T407862#11303594 (10sbassett) >>! In T407862#11301090, @Reedy wrote: > Can you look at https://gerrit.wikimedia.org/r/admin/... [16:05:03] maintenance-disconnect-full-disks build 748244 integration-agent-docker-1043 (/: 35%, /srv: 70%, /var/lib/docker: 27%): RECOVERY disk space OK [16:12:34] 06Release-Engineering-Team, 10MediaWiki-extensions-Cargo, 06Security-Team: Remove REL-style release branches from the Cargo extension git repository - https://phabricator.wikimedia.org/T407862#11303644 (10Yaron_Koren) Sorry for the delay. I don't see any way for me to delete these branches, either. [16:43:51] (03merge) 10dduvall: image: Delegate syntax handling to buildkit-syntax-forwarder [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/157 [16:48:19] 10Beta-Cluster-Infrastructure, 07Puppet: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#11303762 (10Krinkle) Is something preventing this fix from applying to labs/private? * https://codesearch.wmcloud.... [16:55:03] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Puppet: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#11303795 (10bd808) >>! In T397877#11303762, @Krinkle wrote: > It seems this script is shared... [16:56:00] bd808: wow, I've never looked at beta's labs/private until now [16:56:04] quite a few hot fixes there... [16:56:31] depthsofbeta [16:57:07] (03open) 10dduvall: version: 2.12.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/158 [17:00:43] (03merge) 10dduvall: version: 2.12.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/158 [17:24:34] Krinkle: I don't think I've looked for a long time, but yeah I'd expect it to be a bit of a tangle. Out Puppet secret management practices are a bit archaic [18:38:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:38:35] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148 (10wmcs-alerts) 03NEW [19:14:58] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [19:45:40] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304367 (10bd808) `lang=shell-session,counterexample bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo run-puppet-agent... [19:46:33] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304371 (10bd808) [19:56:39] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304390 (10ssingh) Yeah I missed this in the previous fix. I am going to take this tomorrow since now essentially we have to guard... [20:26:52] Project beta-scap-sync-world build #229323: 04FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/229323/ [20:37:02] Yippee, build fixed! [20:37:03] Project beta-scap-sync-world build #229324: 09FIXED in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/229324/ [20:54:13] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [21:02:08] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for hacksyn - https://phabricator.wikimedia.org/T408161 (10Hacksyn) 03NEW [21:57:40] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for hacksyn - https://phabricator.wikimedia.org/T408161#11304691 (10Aklapper) 05Openβ†’03Stalled @Hacksyn: Hi and welcome! Could you tell us which project(s) in GitLab you plan to work on, as I don't see much accoun... [22:10:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:15:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown