[00:09:59] 10netops, 10Analytics, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10faidon) To add to the above, I'm also wondering how difficult it would be to also include AS *names*, e.g. coming from the MaxMind GeoIP ASN database. I think we've use... [01:33:00] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10King_of_Hearts) Yet another one, currently still broken as of time of writing: https://upload.wikimedia.org/wikipe... [01:41:25] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10SRE-swift-storage: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10Reedy) [07:03:15] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10SRE-swift-storage: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10Aklapper) [08:08:20] XioNoX: there's a strange outage notification from NTT in ops-maintenance [08:08:26] XioNoX: https://groups.google.com/a/wikimedia.org/forum/#!topic/ops-maintenance/StZFgk9xO1E [08:08:50] they claim the issue started 2020-04-30 00:58 UTC and is still ongoing [08:10:12] for our singapore transit link [08:10:19] and ETR on Sunday.. [08:11:36] ema: I don't see anything wrong for now. Thanks for the head's up, We can disable it if it starts miss-behaving [08:13:30] XioNoX: ok, I'll set the mail as "no action needed" then. I don't know if we have any sort of SLO with them but 2 months of outage seems like a fairly long time regardless :) [08:15:02] yeah it's weird, either a typo or they only recently found the root cause. But as we didn't see any impact I don't think it's worth asking for credits [08:15:50] ack. Thanks! [14:01:14] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) [14:02:47] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) [15:29:17] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10Jclark-ctr) @Cmjohnson host are cabled need to be configured. [15:53:04] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) Offline script in Netbox will do this {F31905493} [16:30:04] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) [16:30:22] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) 05Open→03Resolved This is complete [16:30:49] 10netops, 10Operations, 10ops-codfw: codfw: Decommission old mr1 - https://phabricator.wikimedia.org/T256143 (10Papaul) [16:30:58] 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) [16:31:01] 10netops, 10Operations, 10ops-codfw: codfw: Decommission old mr1 - https://phabricator.wikimedia.org/T256143 (10Papaul) 05Open→03Resolved Compete [17:30:16] bblack: would you like me to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607313/ ? I have a free moment [18:32:50] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10Cmjohnson) This is being worked on, I had to put the OS image back on the usb stick. When I reset the switch to factory default the usb was wiped as welll. [20:46:48] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: 2020-06-12) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10Jclark-ctr) @ayounsi switches are cabled and powered waiting on configuration