[09:55:05] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10ops-ulsfo: large number of 504 errors from ulsfo - https://phabricator.wikimedia.org/T236500 (10Bugreporter) [11:26:22] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10ops-ulsfo: large number of 504 errors from ulsfo - https://phabricator.wikimedia.org/T236500 (10jijiki) Looks like it started on Friday ~12:00-13:00 UTC https://grafana.wikimedia.org/d/kHk7W6OZz/ats-cluster-view?panelId=1&fullscreen&orgId=... [11:26:28] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10ops-ulsfo: large number of 504 errors from ulsfo - https://phabricator.wikimedia.org/T236500 (10jijiki) [12:24:24] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10ops-ulsfo: large number of 504 errors from ulsfo - https://phabricator.wikimedia.org/T236500 (10ema) In esams, the WDQS query above fails with a java exception after 60s: ` $ curl -v -w '%{time_connect}:%{time_starttransfer}:%{time_total}'... [18:53:30] 10Traffic, 10DC-Ops, 10Operations, 10decommission: decommission multatuli - https://phabricator.wikimedia.org/T236489 (10Papaul) [18:55:45] 10Traffic, 10DC-Ops, 10Operations, 10decommission: decommission nescio and maerlant - https://phabricator.wikimedia.org/T236452 (10Papaul) [18:58:03] 10Traffic, 10DC-Ops, 10Operations, 10decommission: decommission lvs300[1234] - https://phabricator.wikimedia.org/T236451 (10Papaul) [19:01:38] 10Traffic, 10Operations, 10ops-esams: rack/setup/install lvs300[567] - https://phabricator.wikimedia.org/T236294 (10Papaul) [19:03:40] 10Traffic, 10DNS, 10Operations, 10ops-esams: rack/setup/install dns300[12] - https://phabricator.wikimedia.org/T236217 (10Papaul) [19:05:37] 10Traffic, 10DC-Ops, 10Operations, 10decommission: decommission cp3030-3049 - https://phabricator.wikimedia.org/T236454 (10Papaul)