[08:32:19] 10Traffic, 10Cloud-Services, 10Operations: All sites served by cloudweb2001-dev return 503 - https://phabricator.wikimedia.org/T230105 (10ema) [08:32:25] 10Traffic, 10Cloud-Services, 10Operations: All sites served by cloudweb2001-dev return 503 - https://phabricator.wikimedia.org/T230105 (10ema) p:05Triage→03Normal [10:07:44] 10Traffic, 10Discovery, 10Operations, 10WMDE-Analytics-Engineering, and 4 others: Allow access to wdqs.svc.eqiad.wmnet on port 8888 - https://phabricator.wikimedia.org/T176875 (10Gehel) A few more comments after discussion with @elukey : * the use of port 8888 to get extended query timeouts is exceptional... [13:48:53] ema: this was mentioned in scrum of scrumbs, review for VCL: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526627/ [13:48:57] scrums [13:49:39] mark: ack, thanks! [16:12:38] 10netops, 10Operations, 10observability: Add VCP stats monitoring - https://phabricator.wikimedia.org/T228824 (10ayounsi) 05Open→03Resolved a:03ayounsi We now have visibility on all VCPs; https://librenms.wikimedia.org/ports/ifType=vcp/format=list_basic/ They also benefit from the same alerting as regu... [16:13:26] 10netops, 10Operations, 10ops-codfw: update RE-S-X6-64G-S in cr[12]-codfw - https://phabricator.wikimedia.org/T226422 (10Papaul) Return information for bad SCB {F30000784} [16:33:02] 10Traffic, 10Operations, 10Performance, 10Performance-Team (Radar), 10User-notice: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Elitre) [16:33:17] 10Traffic, 10Operations, 10Performance, 10Performance-Team (Radar), 10User-notice: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Elitre) Removing our tag then. [17:01:23] 10Traffic, 10Operations, 10Wikidata-Bridge-Sprint-3: Data Bridge can’t load entity data on mobile clients - https://phabricator.wikimedia.org/T229385 (10Lydia_Pintscher) 05Open→03Resolved \o/ [17:26:43] 10netops, 10Operations: BGP session down for AS4739 on cr4-ulsfo - https://phabricator.wikimedia.org/T230005 (10elukey) a:03elukey [17:26:58] 10netops, 10Operations: BGP session down for AS4739 on cr4-ulsfo - https://phabricator.wikimedia.org/T230005 (10elukey) Sent an email just now to their NOC, will wait for the answer before closing. [17:27:06] 10netops, 10Operations: BGP session down for AS 20485 on cr2-esams - https://phabricator.wikimedia.org/T230004 (10elukey) a:03elukey [17:27:18] 10netops, 10Operations: BGP session down for AS 20485 on cr2-esams - https://phabricator.wikimedia.org/T230004 (10elukey) Sent an email to their NOC, going to wait for an answer before closing. [21:28:32] 10netops, 10Operations: csw2-esams's VCP link flapped - https://phabricator.wikimedia.org/T229755 (10ayounsi) Seems like this device is seeing its end coming with the esams refresh. The on disk logs have rolled over but syslog logs are visible on https://logstash.wikimedia.org/goto/08b36fc12fdc83accef419f308f... [21:28:45] 10netops, 10Operations: csw2-esams's VCP link flapped - https://phabricator.wikimedia.org/T229755 (10ayounsi) a:03ayounsi