[00:50:24] 10netops, 10Operations: AS63541's session down reported by cr1-eqsin - https://phabricator.wikimedia.org/T228617 (10ayounsi) p:05Normal→03Lowest Email sent to Chinacache. If no replies in ~1w then we will remove the session. [02:12:23] 10netops, 10Operations, 10observability: Add VCP stats monitoring - https://phabricator.wikimedia.org/T228824 (10ayounsi) p:05Triage→03Normal [06:25:10] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10elukey) p:05Triage→03High [08:21:27] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10Peachey88) [12:13:32] 10Traffic, 10Operations: Do not cache the beta version of the mobile site - https://phabricator.wikimedia.org/T228861 (10ema) [12:13:43] 10Traffic, 10Operations: Do not cache the beta version of the mobile site - https://phabricator.wikimedia.org/T228861 (10ema) p:05Triage→03Normal [12:32:23] 10Traffic, 10MobileFrontend, 10Operations, 10Patch-For-Review: Do not cache the beta version of the mobile site - https://phabricator.wikimedia.org/T228861 (10ema) [12:48:26] 10Traffic, 10Operations: Separate Traffic layer caches for PHP7/HHVM - https://phabricator.wikimedia.org/T206339 (10ema) [13:53:40] 10netops, 10Operations, 10ops-eqiad: (Need By: Sept 30) update RE-S-X6-64G-S in cr[12]-eqiad - https://phabricator.wikimedia.org/T226424 (10RobH) @ayounsi: This now has a need by date of September 30th (I assume you and @wiki_willy came up with that as he added it?) This is basically blocked on #netops tel... [14:26:54] 10netops, 10Operations, 10ops-eqiad: rack spare switches in c1-eqiad - https://phabricator.wikimedia.org/T185337 (10faidon) These could be racked in any rack, including in row A. It would be useful to have a working lab out of our spares - this came up yesterday/today when we were wondering if we had QSFPs t... [14:36:42] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10ayounsi) Level3/CenturyLink opened a ticket for that circuit and completed an emergency maintenance. I also see some planned maintenance in the last few days. And have at least... [14:45:12] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Allow analytics VLAN to reach eventgate-analytics.discovery.wmnet:31192 - https://phabricator.wikimedia.org/T228882 (10Ottomata) [15:28:21] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10elukey) We usually see impact in 50x and/or nginx availability when the link goes down, so if that could be avoided I'd be +1. [16:09:23] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Allow analytics VLAN to reach eventgate-analytics.discovery.wmnet:31192 - https://phabricator.wikimedia.org/T228882 (10Ottomata) a:05Ottomata→03None [17:07:40] 10netops, 10Operations, 10ops-codfw: Cable mr1-codfw<->cr1/2-codfw through asw-a-codfw - https://phabricator.wikimedia.org/T228112 (10ayounsi) [17:12:20] 10netops, 10Operations, 10ops-codfw: Cable mr1-codfw<->cr1/2-codfw through asw-a-codfw - https://phabricator.wikimedia.org/T228112 (10ayounsi) [20:34:06] 10netops, 10Operations, 10ops-codfw: update RE-S-X6-64G-S in cr[12]-codfw - https://phabricator.wikimedia.org/T226422 (10ayounsi) Scheduled for the 31st at 15:00UTC (1h total). [20:35:12] 10netops, 10Operations, 10ops-codfw: Cable mr1-codfw<->cr1/2-codfw through asw-a-codfw - https://phabricator.wikimedia.org/T228112 (10ayounsi) Scheduled for the 30st at 15:00UTC (1h total). Let me know if it needs to be rescheduled.