[00:02:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:55:53] 10SRE-tools, 10Cumin, 06Infrastructure-Foundations, 10Spicerack: Add shell completion to 'cookbook' - https://phabricator.wikimedia.org/T433228 (10fgiunchedi) 03NEW [10:24:30] 10SRE-tools, 10Cumin, 06Infrastructure-Foundations, 10Spicerack: Add shell completion to 'cookbook' - https://phabricator.wikimedia.org/T433228#12158108 (10fgiunchedi) →14Duplicate dup:03T367230 [10:24:32] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12158110 (10fgiunchedi) [11:50:50] 10netops, 06Infrastructure-Foundations, 06SRE: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752#12158344 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=fe3278b9-2acb-46d1-af9d-66d64acae08f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [12:06:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:37:01] 10netops, 06Infrastructure-Foundations, 06SRE: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752#12158517 (10ayounsi) 05Open→03Resolved a:03ayounsi upgraded! [12:56:08] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12158568 (10ayounsi) BGP rib sharding has been enabled on cr4-ulsfo Some non-scientific numbers. Before enabling RIB sharding, removing the "disabled" statement from t... [13:30:34] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158671 (10ssingh) Thanks @BCornwall. @RobH: Is there a preference for the day of the week from your end? I am assuming we can't do tomorrow Sin... [13:58:56] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158773 (10RobH) Right now it is 13:55 GMT = 21:55 Singapore, so if we schedule now it won't be confirmed until Tuesday, for possibly a Wednesda... [14:07:54] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158818 (10ssingh) >>! In T433097#12158773, @RobH wrote: > > Proposed window: 2026-07-30 @ 01:00 GMT (9AM Singapore). Depool to take place a fe... [14:09:33] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12158831 (10ayounsi) [14:25:40] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158992 (10RobH) I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to complete, so we'd like the site depooled fr... [14:30:01] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159006 (10ssingh) >>! In T433097#12158992, @RobH wrote: > I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to c... [14:37:04] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159020 (10RobH) I've sent an email to Jin to confirm the date window of 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT, which is 2026-07-30 @... [14:45:39] 10Mail, 06collaboration-services, 06translatewiki.net: translatewiki.net: translation support for https://gerrit.wikimedia.org/r/operations/software/mailman-templates may be dropped due to inactivity - https://phabricator.wikimedia.org/T433171#12159064 (10LSobanski) [15:08:24] 10Mail, 06collaboration-services, 06translatewiki.net: translatewiki.net: translation support for https://gerrit.wikimedia.org/r/operations/software/mailman-templates may be dropped due to inactivity - https://phabricator.wikimedia.org/T433171#12159194 (10Ladsgroup) Hi, I'm not sure if this is a good case. T... [15:20:00] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12159255 (10CDanis) It seems to be happening much, much more often since 11:30 UTC t... [15:26:48] 10Mail, 06collaboration-services, 06translatewiki.net: translatewiki.net: translation support for https://gerrit.wikimedia.org/r/operations/software/mailman-templates may be dropped due to inactivity - https://phabricator.wikimedia.org/T433171#12159316 (10LSobanski) p:05Triage→03High [15:29:32] 10Mail, 06collaboration-services, 06translatewiki.net: translatewiki.net: translation support for https://gerrit.wikimedia.org/r/operations/software/mailman-templates may be dropped due to inactivity - https://phabricator.wikimedia.org/T433171#12159333 (10siebrand) >>! In T433171#12159194 schreef @Ladsgroup:... [16:06:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:08:25] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159656 (10BCornwall) Confirming I can do those times (2026-07-29 @ 16:00-20:00 PDT) [16:42:49] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159857 (10RobH) Jin confirmed via email just now for the window: 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT I'll update our directions... [16:43:40] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159861 (10RobH) [17:06:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:08:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:11:23] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12159969 (10Tgr) [[https://logstash.wikimedia.org/goto/c3ee5e7c8b2d83ebf77f5dcb63e4d... [17:11:36] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12159970 (10Tgr) p:05Medium→03Unbreak! [17:19:23] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160010 (10Tgr) I created a test account. I did receive the email, but an error was... [17:21:06] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160017 (10Tgr) Started around 11h on Sunday so I guess it cannot be a code change... [18:35:03] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160361 (10SD0001) SDZeroBot ran into this error 3 times today. The emails were in... [19:41:21] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160594 (10Scott_French) I'm able to consistently received the emails I send via Sp... [20:00:37] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160660 (10Scott_French) Deployment completed around ~ 19:57 UTC (stashbot appears... [20:21:13] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160748 (10Scott_French) p:05Unbreak!→03High Now that the increased timeout is... [20:37:28] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12160824 (10hashar) Thank you! I confirm I have managed to send an email via the wik... [21:08:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:49:06] 10Mail, 06collaboration-services, 06translatewiki.net: translatewiki.net: translation support for https://gerrit.wikimedia.org/r/operations/software/mailman-templates may be dropped due to inactivity - https://phabricator.wikimedia.org/T433171#12161102 (10Ladsgroup) I can't find where these templates are pus... [22:24:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12161199 (10Papaul) [22:33:21] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12161214 (10jhathaway) There is some very strange behavior with regard to the messag...