[00:15:45] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review, and 2 others: Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10aaron) >>! In T133821#6092867, @Joe wrote: > At a later time, we could think of changing the logic, and make purges avoid race conditions, removing the need for t... [07:16:58] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install WMCS 10G switches - https://phabricator.wikimedia.org/T251632 (10wiki_willy) a:03Jclark-ctr [08:13:22] 10netops, 10Operations: Upgrade Routinator 3000 to 0.7.0 - https://phabricator.wikimedia.org/T252010 (10ayounsi) 05Open→03Resolved Added `routinator_rtr_current_connections` to the Grafana dashboard. [08:16:28] 10netops, 10Operations: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10ayounsi) ACKed for 6 more hours the time Telia fixes it. [09:55:02] 10netops, 10Operations: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10ayounsi) After ticket 01157098 was resolved, the link didn't come back up. Ticket 01157707 was opened. Telia setup a loop on the Chicago side towards SF, which brought the SF interface up, but the C... [10:22:14] 10netops, 10Operations, 10ops-eqord: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10ayounsi) a:05ayounsi→03None [12:09:44] 10netops, 10Operations: Peer with SFMIX at ulsfo - https://phabricator.wikimedia.org/T251536 (10faidon) [12:09:53] 10netops, 10Operations: Peer with SFMIX at ulsfo (May 2020) - https://phabricator.wikimedia.org/T251536 (10faidon) [12:10:33] 10netops, 10Operations: Peer with SFMIX at ulsfo (May 2020) - https://phabricator.wikimedia.org/T251536 (10faidon) LoA received and cross-connect task created. [12:10:40] 10netops, 10Operations: Peer with SFMIX at ulsfo (May 2020) - https://phabricator.wikimedia.org/T251536 (10faidon) [12:10:57] 10netops, 10Operations: Peer with SFMIX at ulsfo (May 2020) - https://phabricator.wikimedia.org/T251536 (10faidon) [13:51:20] vgutierrez@cp5012:~$ sudo -i netstat -n |grep 3128 |wc -l [13:51:20] 35059 [13:51:23] not bad for a depooled host [14:09:54] lol?:) [14:29:31] varnish-fe bombing ats-be [14:29:40] it's a well known issue [14:31:26] varnish misses ats <3 <3 [14:31:34] (see what I did there?) [14:41:55] lol [14:42:02] godog: 🍻 [14:43:40] vgutierrez: haha thank you sir [15:35:37] 10Traffic, 10DNS, 10Operations, 10Wikimedia-Blog, 10Patch-For-Review: Update DNS records for blog.wikimedia.org - https://phabricator.wikimedia.org/T251931 (10colewhite) 05Open→03Resolved DNS update deployed. Please reach out if something is amiss. [15:40:40] 10Traffic, 10Operations, 10ops-eqsin: cp5012 memory errors - https://phabricator.wikimedia.org/T251219 (10Vgutierrez) do we have an ETA on this one? :) [16:31:10] 10Traffic, 10Core Platform Team, 10Operations, 10serviceops, and 2 others: Reduce rate of purges emitted by MediaWiki - https://phabricator.wikimedia.org/T250205 (10Krinkle) [16:55:07] 10Traffic, 10DNS, 10Operations, 10Wikimedia-Blog: Update DNS records for blog.wikimedia.org - https://phabricator.wikimedia.org/T251931 (10Varnent) It appears we are all set - thank you, @colewhite! [17:48:24] 10Traffic, 10Operations, 10ops-eqsin: cp5012 memory errors - https://phabricator.wikimedia.org/T251219 (10wiki_willy) a:05Cmjohnson→03RobH @Vgutierrez - my apologies, I initially mistook this as a host in eqiad, instead of eqsin, so had assigned it to the wrong person last week. Re-assigning now to @Rob... [18:50:55] 10Traffic, 10MediaWiki-Authentication-and-authorization, 10Operations, 10Security-Team, 10Security: Investigate usefulness of SameSite cookies for logged-in accounts - https://phabricator.wikimedia.org/T158604 (10Krinkle) [19:35:04] 10netops, 10Operations, 10ops-eqiad: Eqiad: C6 mgmt switch down - https://phabricator.wikimedia.org/T249309 (10Jclark-ctr)