[08:06:22] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10342865 (10LSobanski)
[09:30:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:18:33] <wikibugs>	 10netops, 06Infrastructure-Foundations: Lumen codfw-ulsfo down (Nov 2024) - https://phabricator.wikimedia.org/T380451 (10ayounsi) 03NEW p:05Triage→03High
[10:19:44] <wikibugs>	 10netops, 06Infrastructure-Foundations: Lumen codfw-ulsfo down (Nov 2024) - https://phabricator.wikimedia.org/T380451#10343097 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox circuit ID 102 --- **Interface cr4-ulsfo:xe-0/1/1** -  admin-status: up - ⚠️  oper-status: down -  interface-flapped: 2024-...
[10:24:58] <wikibugs>	 10netops, 06Infrastructure-Foundations: Lumen codfw-ulsfo down (Nov 2024) - https://phabricator.wikimedia.org/T380451#10343105 (10ayounsi) > Ticket ID 30654682 has been successfully created.
[10:36:20] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10343165 (10LSobanski)
[10:46:09] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10343202 (10MoritzMuehlenhoff) Looking at https://knowledge.validity.com/s/articles/Accessing-Validity-reputation-data-through-DNS?language=en_US we co...
[11:30:59] <_joe_>	 hi! I'm working on importing packages from apt-staging into the main reprepro, specifically packages we do upload currently to "main"
[11:31:30] <_joe_>	 my question is - should I create a new apt component for each of these packages? Or can I just keep adding them to main?
[11:32:21] <_joe_>	 I see arguments in both directions tbh
[11:33:09] <_joe_>	 see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1093875 
[11:41:25] <wikibugs>	 10netops, 10Ceph, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Configure DSCP marking for cloudceph* hosts - https://phabricator.wikimedia.org/T371501#10343409 (10BTullis)
[11:47:01] <jinxer-wm>	 FIRING: NTPNoSynced: NTP not synced on wikikube-worker2157:9100 - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.wikimedia.org/?q=alertname%3DNTPNoSynced
[12:01:30] <moritzm>	 I think we can default to import and use staging/main for most packages, in general there should not be much overlap between the staged packages. and if there's a need for further separattion later (for complex cases like a Hadoop update) we can still add them on a case by case basis
[12:32:22] <_joe_>	 ok, so you think my patch above would work?
[12:43:57] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Traffic: eqiad/esams/drmrs LVS: use Netbox BGP flag - https://phabricator.wikimedia.org/T380469 (10ayounsi) 03NEW
[13:30:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:45:32] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, and 2 others: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10343909 (10akosiaris) >>! In T380396#10343202, @MoritzMuehlenhoff wrote: > Looking at https://knowledge.validity.com/s/articles/Accessing-Validity-...
[13:52:45] <wikibugs>	 10netops, 06Infrastructure-Foundations: Lumen codfw-ulsfo down (Nov 2024) - https://phabricator.wikimedia.org/T380451#10343916 (10cmooney) Seems to be back up: ` Nov 21 13:51:02  cr4-ulsfo rpd[29932]: RPD_OSPF_NBRUP: OSPF neighbor 198.35.26.203 (realm ospf-v2 xe-0/1/1.0 area 0.0.0.0) state changed from Exchang...
[15:09:03] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10344210 (10VRiley-WMF) A 10G connection has been placed into port 25 a...
[15:14:31] <wikibugs>	 10netops, 06Infrastructure-Foundations: Lumen codfw-ulsfo down (Nov 2024) - https://phabricator.wikimedia.org/T380451#10344249 (10cmooney) 05Open→03Resolved a:03cmooney From Lumen: ` 2024-11-21 12:38:45 GMT - Splicing has commenced at the south end. Service affecting alarms are expected to begin clea...
[15:30:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:05:44] <jinxer-wm>	 FIRING: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[16:15:44] <jinxer-wm>	 RESOLVED: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[16:20:59] <jinxer-wm>	 FIRING: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[16:30:59] <jinxer-wm>	 RESOLVED: [2x] NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/18/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[16:32:35] <wikibugs>	 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503 (10cmooney) 03NEW p:05Triage→03High The #Cloud-Services project tag is not intended to have any...
[16:35:55] <wikibugs>	 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10344824 (10JJMC89)
[16:42:05] <wikibugs>	 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10344918 (10dcaro)
[17:04:29] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10345111 (10cmooney) >>! In T379790#10344210, @VRiley-WMF wrote: > A 10...
[17:48:02] <wikibugs>	 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10345576 (10dcaro)
[20:39:29] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10346327 (10eoghan) I've put in a change to disable this specific check, we're also going to look at whether we can sign up for an account with them to...