[01:08:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[01:12:01] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12161475 (10Papaul)
[02:08:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[02:10:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:10:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:06:54] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12161827 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8089257c-9548-4d05-a03b-bdb3c1224c47) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi...
[09:10:59] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5c71b24d-45b0-4c80-bf42-231dd7bab42a) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi...
[09:29:03] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12162102 (10ayounsi)
[09:29:10] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162103 (10ayounsi)
[09:29:24] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162106 (10ayounsi) 05Open→03Resolved a:03ayounsi All done.
[10:10:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:30:47] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162617 (10Papaul)
[12:31:09] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162620 (10Papaul)
[12:33:32] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162627 (10Papaul) First homer run is getting me this error ` ERROR:homer:Device asw1-603-eqsin.mgmt.eqsin.wmnet failed to render the template, skipping....
[13:03:56] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162771 (10Papaul)
[13:51:59] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a2dbde41-3fc7-4120-8a34-b7696ca56502) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi...
[14:10:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:12:11] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163147 (10ayounsi) 05Open→03Resolved a:03ayounsi Upgraded.
[14:13:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12163158 (10ayounsi)
[16:07:53] <wikibugs>	 10SRE-tools, 10Cumin, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409 (10RobH) 03NEW p:05Triage→03Medium
[16:10:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:12:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:31:01] <sukhe>	 heya folks. not sure who is taking care of urldownloader when Moritz is out but I have a patch for review
[17:31:04] <sukhe>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313948
[17:31:09] <sukhe>	 this is for T429175
[17:31:10] <stashbot>	 T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175
[17:31:25] <sukhe>	 full chain at https://gerrit.wikimedia.org/r/q/topic:%22T429175%22
[20:12:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:37:23] <jhathaway>	 sukhe: +1ed
[20:59:11] <sukhe>	 thanks a lot jhathaway! 
[21:10:38] <jhathaway>	 happy to
[21:23:11] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12165186 (10jhathaway) The 4s delay was do to Rspamd’s `fuzzy_check` module timing o...
[23:44:06] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12165505 (10Scott_French) Very nice find, @jhathaway. Reverting the to the original...
[23:57:17] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 10 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12165518 (10Perryprog) >>! In T383047#12165186, @jhathaway wrote: > The 4s delay was...