[09:02:09] 10netops, 06Infrastructure-Foundations, 06SRE: mr1-codfw is single-homed to lsw1-a2-codfw - https://phabricator.wikimedia.org/T407488#11394993 (10ayounsi) management routers are physically single homed in the old design (eqsin, codfw, eqiad), probably because it was best to not over engineer it, and mgmt net... [09:22:00] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice-archive: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11395040 (10Guycn2) This seems to be blocking legitimate web captures by the Internet Archive (see, for example, [[ https://web.archive.org/web... [09:38:38] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11395092 (10ayounsi) New updated list, we're at 642 hosts if the MySQL query from the initial task is still the proper way. Up from 90 in 2020, probably becaus... [10:27:33] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11395183 (10MatthewVernon) Thanks! I've spent a fair chunk of time searching and have come up with nothing. My next stop is likely #no-stu... [10:30:15] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11395193 (10MatthewVernon) I have solved the easy one, though: ecosia. If you image search on there (e.g. https://www.ecosia.org/images?q=... [11:13:46] 10netops, 06Infrastructure-Foundations, 06SRE: mr1-codfw is single-homed to lsw1-a2-codfw - https://phabricator.wikimedia.org/T407488#11395278 (10cmooney) >>! In T407488#11394993, @ayounsi wrote: > management routers are physically single homed in the old design (eqsin, codfw, eqiad), probably because it was... [11:24:56] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice-archive: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11395297 (10daniel) >>! In T400119#11395040, @Guycn2 wrote: > This seems to be blocking legitimate web captures by the Internet Archive (see, f... [11:26:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add connection from - https://phabricator.wikimedia.org/T410717 (10cmooney) 03NEW p:05Triage→03Low [11:41:07] 06Traffic, 06serviceops, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11395355 (10OWresch-WMF) [11:44:15] 10netops, 06Infrastructure-Foundations, 06SRE: mr1-codfw is single-homed to lsw1-a2-codfw - https://phabricator.wikimedia.org/T407488#11395392 (10cmooney) 05Open→03Resolved a:03cmooney Will open task on getting the second link in place on mr1-codfw. We can look to do the same in eqiad once the row... [12:08:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11395518 (10cmooney) [13:56:15] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11395909 (10MatthewVernon) [14:09:17] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11395970 (10MatthewVernon) [14:20:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11396006 (10ayounsi) The current cable is already to lsw1-a2 (https://netbox.wikimedia.org/dcim/cables/7147/) so probably a3 is the next one. To... [14:25:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: mr1-codfw: add second uplink to lsw1-a2-codfw - https://phabricator.wikimedia.org/T410717#11396019 (10cmooney) >>! In T410717#11396006, @ayounsi wrote: > The current cable is already to lsw1-a2 (https://netbox.wikimedia.org/dcim/cables... [14:36:12] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11396060 (10Ladsgroup) I haven't found anything in gadgets, etc. https://commons.wikimedia.org/w/index.php?title=Special:Search&limit=500&... [14:52:34] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630#11396143 (10cmooney) Just to update this we are not planning to move the IP gateways for vlans to the Nokia switches in t... [14:52:56] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan - https://phabricator.wikimedia.org/T405632#11396146 (10cmooney) Just to update this we are not planning to move the IP gateways for vlans to the No... [15:48:05] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice-archive: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11396452 (10CDanis) >>! In T400119#11395040, @Guycn2 wrote: > This seems to be blocking legitimate web captures by the Internet Archive (see, f... [16:21:12] 06Traffic, 06Commons: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201#11396573 (10ssingh) Yeah, there are certainly other files. I think you can remove your comment that has the SVG file contents otherwise it makes it difficult to read the task. I think I see the error now but I... [17:02:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11396665 (10RobH) Day 8 Update: * 3 hosts moved, 19 remain * John worked with Amir directly today to depool and migrate pc101[678] since the depool and repoo... [20:49:51] 10Wikimedia-Apache-configuration, 13Patch-For-Review: nb.wikiversity.org redirects to 404 page on BetaWikiversity - https://phabricator.wikimedia.org/T407553#11397508 (10Pppery) a:03Pppery