[00:12:03] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12131048 (10Urbanecm_WMF) Logged few follow-ups: * {T432422} * {T432420} [10:14:59] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#12131800 (10zeljkofilipin) a:05zeljkofilipin→03None [10:15:36] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:18:36] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:26:36] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:29:36] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:21:36] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-deploy04 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:21:41] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-deploy04 in project deployment-prep - https://phabricator.wikimedia.org/T432456 (10wmcs-alerts) 03NEW [12:24:37] That's me as well, fixing [12:25:43] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-deploy04 in project deployment-prep - https://phabricator.wikimedia.org/T432456#12132185 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert That's me, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310589... [12:36:36] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-deploy04 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:12:00] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12132673 (10bd808) p:05Low→03High The logging-logstash-04.logging.eqiad1.wikimedia.cloud instance is non-respon... [15:13:50] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12132683 (10dancy) @colewhite Can you give us some insight into what's going with logging-logstash-04.logging.eqiad... [18:59:10] (03PS3) 10Ahmon Dancy: WMF: Cap dependency-stack depth to throttle oversized patch series [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/1311501 (owner: 10Jforrester) [19:00:41] (03CR) 10Ahmon Dancy: "In patchset3 I made it so that a limit of 0 means no limit (in case there are pipelines where we might want to disable the limit)." [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/1311501 (owner: 10Jforrester) [19:22:31] (03PS4) 10Ahmon Dancy: WMF: Cap dependency-stack depth to throttle oversized patch series [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/1311501 (owner: 10Jforrester) [19:23:22] (03CR) 10Ahmon Dancy: "ps4: Honor the 'quiet' flag passed to addChange (recommended by Claude)." [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/1311501 (owner: 10Jforrester) [20:11:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: Restrict depth of stacked commits processing in Zuul - https://phabricator.wikimedia.org/T432491 (10dancy) 03NEW [20:18:19] (03PS5) 10Ahmon Dancy: WMF: Cap dependency-stack depth to throttle oversized patch series [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/1311501 (https://phabricator.wikimedia.org/T432491) (owner: 10Jforrester) [20:28:18] Girl Scouts are hiring for VP of Agentic AI Innovation. Apply now at https://linkedin.com/jobs/view/4415733532 and build the agentic, data-driven future of American scouting. [20:31:00] (03PS1) 10Ahmon Dancy: deploy: Repoint buster build at archive.debian.org [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1311933 (https://phabricator.wikimedia.org/T416707) [20:31:50] (03CR) 10CI reject: [V:04-1] deploy: Repoint buster build at archive.debian.org [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1311933 (https://phabricator.wikimedia.org/T416707) (owner: 10Ahmon Dancy) [20:41:21] that was some fine spam from the g-glue account :/ [22:06:34] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12133664 (10bd808) Cole has been out of office this week. I expect he can start looking into things when he is back... [23:21:39] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830#12133743 (10jeena) 05Open→03Resolved